LLM-based agents can appear to do many tasks, including writing code, summarizing papers, planning trips, and filling in forms. When asked for a probability or confidence score, they will readily provide one. Many agent designers use LLMs to generate confidence scores this way, but there is no good reason to trust that number.
To see why, it helps to understand what confidence means. Consider a weather forecaster who says "I am 70% confident that it's going to rain tomorrow." Their confidence comes from the information they use, such as current air pressure, humidity, and radar, along with years of weather records. In theory, turning this information into a confidence is a calculation. Two forecasters with identical information applying the same rule correctly should reach the same confidence. When they disagree, it is because they have different information or one applied the rule badly.
Accuracy can only be judged over many predictions. If a forecaster says 70% on many days and it rains on about 70% of them, the forecast is calibrated. Calibration alone is not enough, however. A forecaster who always says 30%, the seasonal average, is calibrated but uses no information about tomorrow. A useful forecaster is calibrated while also giving confident scores near 10% or 90% when the evidence warrants it.
A trustworthy confidence score therefore needs two things: it must reflect how much information is available and point strongly in one direction, and it must be checked against a written record of predictions and outcomes. Without that record, there is no way to know whether a score is accurate. The author argues that an LLM's self-generated confidence meets neither condition, which is why relying on it is a bad idea.
Source: Hacker News · Summarized by HeadlinesBriefing