Most LLM evaluation pipelines rely on vague scoring or human judgment that breaks at scale. The real problem isn't hallucination itself — it's that confident, wrong responses score 0.525, pass your threshold, and ship. A developer built a pure Python scoring layer that separates faithfulness into attribution and specificity, so high specificity paired with low attribution flags fabricated content.

The fix came after a three-word prompt change — "be specific and detailed" — broke the eval system. A response claiming context engineering started at MIT in 1987 scored 0.525 and passed. The scorer couldn't distinguish between real specificity and confident fabrication because a single number collapses both directions. Traditional metrics like BLEU and LLM-as-judge approaches miss this entirely.

The layer runs locally with no per-call API cost and produces deterministic scores, unlike frameworks like RAGAS that depend on non-deterministic LLM judges. Code is available on GitHub for teams building production RAG pipelines where wrong answers slip through easily. A single holistic score cannot catch a hallucination wearing a business suit.