A chatbot reported an invoice total of $1,000 when the source document stated $10,000. Standard hallucination detectors missed this single-digit substitution.

The author tested five detector families—RAGAS-style LLM judges, embedding similarity, entailment models, Lettuce Detect, and Mini Check—on a custom dataset of 200 paired answers differing only in one number. Results showed AUROC scores of 0.57, 0.51, 0.59, 0.58, and 0.75 respectively, meaning four detectors failed to identify the error.

The core issue is that embedding methods average errors away—a 60-word answer wrong in one digit produces nearly identical vectors. LLM judges assess plausibility, not digit-by-digit accuracy. Trained detectors lack sufficient labeled numeric substitution examples in their training data.

The proposed fix is token-level verification: check each answer token against the source. Words match by meaning; numbers match by exact value after normalization. This approach, implemented in the open-source library groundlens, runs in ~70ms per answer on CPU without training or API calls.

Experiments across five test sets reveal no universal hallucination detector. Each method excels only on its own benchmark. At production operating points requiring 95% recall of bad answers, all methods flag too many correct answers, leaving the "deployable corner" empty.