AI-text detectors can identify many human-written essays, but a study of 14 tools found they were much less reliable at recognizing ChatGPT text once it had been lightly changed. The findings suggest that automated detection alone is a weak basis for deciding whether a student used AI improperly.
Light editing sharply reduced detection
Researchers assessed tools including Turnitin, GPT Zero, and Compilatio using 54 documents. They wrote undergraduate-level essays across subjects such as civil engineering, computer science, economics, history, linguistics, and literature, then created texts in other languages and translated them into English with DeepL or Google Translate.
They also asked ChatGPT to generate texts and made small changes to hide their origin. Some were manually edited by reordering sentences and swapping words; others were rewritten using Quillbot, an AI paraphrasing tool.
On average, the detection tools identified human-written text with 96% accuracy. Their accuracy for ChatGPT-generated text was 74%, but fell to 42% for text that had been lightly tweaked. The pattern indicates that even modest rewriting can make it harder for these systems to distinguish AI output from human writing.
What the tools measure—and what they miss
Many AI-writing detectors look for patterns such as repetition and estimate whether a passage is likely to have come from a language model. That approach does not establish who wrote a text or whether a student violated a course rule. It produces a signal based on writing patterns, and the study found that signal could be disrupted by editing or paraphrasing.
The research has not yet been peer reviewed, and it focused on text generated by ChatGPT. Sasha Luccioni, a researcher at Hugging Face who was not involved, said it would have been useful to examine AI tools beyond ChatGPT as well. The results therefore describe the tested systems and material, rather than every possible detector or AI model.
False accusations carry real consequences
Daphne Ippolito, a senior research scientist at Google who did not work on the project, emphasized that schools need to understand both false positives and false negatives. A false positive can wrongly accuse a student of cheating and harm their academic career. A false negative means AI-generated text passes as human writing, limiting the detector's usefulness.
Compilatio said its product flags passages that may be plagiarism or AI-generated, leaving schools and teachers to investigate and assess what the author knows. It described the software as a correction aid, and suggested additional steps such as oral questioning or questions in a controlled classroom environment.
Turnitin's chief product officer, Annie Chechitelli, said the company's feature highlights areas where further discussion may be needed. She said it does not decide whether AI use is appropriate or constitutes misconduct; that judgment depends on the assessment and the teacher's instructions.
Rethinking how student work is assessed
The study adds to existing doubts about automated AI detection. Earlier that year, OpenAI introduced a detector and said it identified only 26% of AI-written text as “likely AI-written.” The company also warned educators that detection tools are “far from foolproof.”
Researchers and educators also question whether the right response is to treat AI authorship as something a detector can settle. Vitomir Kovanović, a senior lecturer at the University of South Australia who was not part of the project, said the results point to outdated ways of assessing student work. His proposed direction was to make AI use less central to the problem.
For schools, the practical lesson is to treat a detector result as a prompt for further review, not a verdict. Clear instructions about permitted AI use and assessments that let teachers evaluate a student's understanding may matter more than relying on a score that can be affected by small edits.