When data scientists hear “What does a p‑value mean?” they often reply with shaky confidence, mistaking it for a probability that a hypothesis is true. In reality, a p‑value tells how surprising observed data would be if nothing were happening.

A common illustration uses a bakery: two cookie batches receive 100 taste tests each, yielding an 8% difference in approval. By simulating thousands of trials under the assumption that both batches are identical, analysts find a p‑value of 0.20—meaning such a gap could arise by chance 20% of the time.

Misreading p‑values fuels false confidence. Saying “p < 0.05 means the effect is real” ignores that the figure merely quantifies rarity under a null hypothesis. It also encourages over‑reacting to noise and overlooking effect size, turning statistically significant yet practically trivial results into misleading headlines.

The 0.05 benchmark, popularized by Ronald Fisher, balances false positives and negatives. Tightening the cut to 0.01 reduces false alarms but risks missing true effects; loosening it to 0.10 captures more findings but invites noise. The key takeaway: a p‑value reflects data oddity, not hypothesis truth.