The pattern shows up everywhere: chat‑based outputs, structured outputs, agentic task results. Someone wants a JSON with a response key and a confidence key, usually a continuous score from 0 to 100. That scale feels useful but has no scientific dolls behind it. It’s a psychological safety trick that makes output feel trustworthy without actually improving it.

Anthropic’s research on tracing the thoughts of a language model found that Claude plans several words ahead, and their emergent introspective work shows models can notice injected concepts. Yet the authors warn the capability is highly unreliable and context‑dependent. The internal state that might hint at correctness is far from a reliable confidence estimator. Asking the model to report how confident it is about its own answer leads to a confidence‑of‑confidence cascade—each level questioning the previous.

In classical ML we calibrate scores with Platt scaling, isotonic regression, or temperature scaling against held‑out data. LLMs offer only token‑likelihoods, which vary wildly across a response. Collapsing those into a single number is a vibe, not a measurement. Semantic entropy methods sample the model, cluster by meaning, and measure entropy externally.

To get a useful confidence signal would require external sampling, calibration, and validation—work that no one has yet undertaken. The current self‑reported scores lack the rigor needed for trust.