Traditional machine learning models can effectively distinguish between human-written and LLM-generated text, exhibiting strong statistical patterns. The author developed a detector using scikit-learn's TF-IDF and Linear SVC, achieving approximately 85% accuracy at the sentence level. This approach proved more reliable than methods relying on text perplexity or simply asking another LLM.
The detector was trained on a dataset comprising human-written articles from 2010-2022 and LLM-generated articles created using various APIs like Gemini, Qwen, and GLM. The author used cost-effective methods to generate a large volume of training data.
While attempts at multi-class classification (human + multiple AIs) were unsuccessful, a system of seven binary classifiers combined with majority voting proved effective. The author's findings suggest that classic machine learning still holds significant value for tasks like AI-generated text detection.
Source: Hacker News · Summarized by HeadlinesBriefing