AI Writing Detectors Falsely Flag Human Work; Study Finds Pangram Most Accurate of Four

2.7 · 19h ago · US · AI

The Facts

  • Ghostwriter Sally McGraw, a professional nonfiction writer for over 20 years, said a prospective client told her multiple AI detection tools flagged her writing sample as machine-generated; McGraw says she wrote it herself. NPR only
  • The client told NPR she screened McGraw's work for AI but said the results were not the determining factor in hiring a different writer, citing experience and fit. NPR only
  • A study published in June in the International Journal for Educational Integrity compared four widely used AI detectors on 160 academic papers across four categories: human-written, AI-written, combined, and AI text rewritten by a human. NPR only
  • Pangram was the most consistently accurate of the four detectors, correctly identifying AI use in 37 of 40 combined-text papers and 37 of 40 papers where a human rewrote AI text. NPR only
  • Clarkesworld received roughly 1,100 submissions a month in 2021; by February 2023 the number had doubled, and the magazine temporarily closed its submission portal. Editor Neil Clarke estimates his workload rose about 30%. NPR only
Dive deeper · 4 more factsShow less
  • Pangram CEO Max Spero said in an emailed statement that the detector can make mistakes, particularly when there is little text to evaluate. NPR only
  • In late 2022, blogger Janelle Shane tested excerpts from her 2019 book and one detector reported 98.7% confidence a passage was machine-generated; she says she wrote it. NPR only
  • University of Pennsylvania professor Chris Callison-Burch said many detectors ask a language model how predictable a passage is, and models find text from their training data unusually predictable, and that word choice and style patterns such as 'delve', em dashes and lists of three can trigger AI-positive results. NPR only
  • The Authors Guild said in May that AI detectors create an environment 'reminiscent of witch hunts' amid worries about false accusations. NPR only

Context

Editors facing rising volumes of AI-generated submissions are increasingly using detection tools, which can falsely flag human-written work. Detector companies acknowledge the tools are not foolproof, and researchers have found different detectors can reach very different conclusions about the same text.

A June study found Pangram the most consistently accurate among four detectors tested, though its CEO says it can err, especially on short samples. Writers flagged by such tools often have little way of knowing what triggered the result or how to disprove it.

How the Agent Panel Scored This

Independent AI personas each scored importance 1–10; the story's score is a trimmed mean (highest and lowest votes dropped). Vote spread: 1.

3 News Editor The story centers on a feature-style report about detector accuracy and a single writer's experience, with a June study as the main concrete event, which is niche and limited in immediate impact.
3 Business Analyst The story reports a single anecdote of a ghostwriter being falsely flagged plus a June academic study comparing detectors, which are concrete but limited in economic impact and affect a niche group of writers and editors rather than markets broadly.
2 Informed Citizen The story reports a study's findings, anecdotes of false flags, and expert commentary rather than a concrete new action or policy change, so its impact on ordinary people is limited and niche.

Source

← All stories