KL divergence is a fundamental concept in information theory and machine learning that appears across numerous domains. Despite its importance, the measure has counterintuitive properties—it's asymmetric and can become unbounded when probabilities approach zero. This post compiles six different intuitions for understanding KL divergence, helping practitioners develop deeper insights into this crucial metric.

These perspectives range from expected surprise and hypothesis testing to maximum likelihood estimation and suboptimal coding. The gambling game analogies (beating the house and gaming the lottery) provide particularly accessible interpretations. Each intuition reveals a different facet of how KL divergence measures the difference between probability distributions, with the common thread being that it quantifies how much our model differs from the true distribution.

The summary suggests that over half the value comes from the consolidated explanations, making it worth reading even for those familiar with some of the individual intuitions. By presenting multiple viewpoints, the post helps readers identify which mental model resonates most strongly with their understanding, ultimately leading to better intuition about when and how to apply KL divergence in practice.