When I trained the previous AI comment classifier, I used partially personal/private data to do it, and built it on a somewhat shaky foundation, so I couldn’t share the code or data. I rebuilt it on public data and a better foundation!

First off, you might want to try it out. Nothing you paste into that web page leaves your browser, so you can safely try it with whatever you like. I have invited some testers to try out an earlier version of it, and they had mainly positive feedback to give. We won’t break down robot-isms the way we broke down Claude-isms in the previous article, because in the UI of the new classifier you can just click any part of the text being classified to see which features activate on that portion of the text, and how they contribute to the overall judgment.

In terms of performance, the headline number is the balanced accuracy of 77%. This is how often the classifier gets the human vs. robot verdict right, assuming human-written and robot-generated comments are equally likely. The classifier also prints a predicted percentage which is calibrated, meaning it can be read as the probability that any specific verdict is correct. We test this through the calibration curve, which shows what probability the classifier assigns to an event with a known probability. Since all dots lie very close to the reference diagonal, we know they are approximately correct.

When the classifier is very confident – e.g. when the confidence is 80% or more – the risk of a false positive drops to 5%. I have also manually tested a smaller non-synthetic set of real-world comments from humans and robots to see how well the classifier generalises slightly out of sample. This translates to an accuracy of 88%. Of course, all of this is tested with code comments only. The classifier is not built to detect robot-generated texts of other kinds.