A product analyst presents an A/B test dashboard. The treatment variant’s conversion rate improved by 0.4 percentage points, and the 95% confidence interval for the lift excludes zero. When a stakeholder asks: “So there’s a 95% chance the new version is better?” No one says no. That silence is the central subject of this article. The 95% in a confidence interval is not a probability attached to the model, the parameter, or the hypothesis. It is a probability attached to a procedure, and it has frequently misled people in medicine, product analytics, and finance because of confusion between the two.

The confidence interval as most people use it today comes from Jerzy Neyman's 1934–37 work. A 95% confidence interval is defined in such a way that, when the same construction rule is applied repeatedly to new samples, 95% of the resulting intervals contain the true parameter. This is a statement about the long-run behavior of the rule, not about any single interval it generates.

The Bayesian alternative answers a different question. A 95% credible interval is simply the range over which the posterior integrates to 0.95.