HeadlinesBriefing favicon HeadlinesBriefing.com

AI Writing on arXiv: A Measurement Study

Hacker News •
×

A study analyzed 12,750 arXiv papers to measure the prevalence of AI-generated writing. The detector was calibrated to flag genuine pre-LLM text at a 0.4% rate, ensuring that any reported increase reflects actual AI adoption rather than detector bias. Pre-ChatGPT years (2021-2022) served as a control, showing a consistent 0.4% flag rate.

Following the release of ChatGPT, the share of papers flagged as machine-written began to climb, reaching approximately 32% by mid-2026. Computer science papers showed the highest prevalence at around 65%, while mathematics papers had a significantly lower rate of 0.7%. This low rate in mathematics is attributed to the field's heavy reliance on notation and theorem-proof structures, making its prose less recognizable to the detector.

Limitations include a relatively small control sample size per field and the detector's varying sensitivity to different AI models. The study emphasizes that a flag indicates machine-like writing, which can include heavily AI-assisted text, and is not definitive proof of authorship. The reported figures represent a lower bound on the actual prevalence of AI-generated content.