I was reading about compression recently when I stumbled upon something crazy: that compressors and LLMs are, at their core, trying to solve the exact same problem. In this post, I’m going to walk us through the basics of compression to understand its deep relationship with language modeling.

Compression relies on redundancy. For example, the string “AAAAAAAAABBBBCCDAAADDDDDDDDD” (28 characters, 224 bits) can be encoded via run-length encoding as “A9B4C2D1A3D9” (12 characters, 96 bits). Modern compressors like gzip and Brotli use transforms, models, and entropy coders.

An entropy coder uses probabilities to produce a compressed bitstream. Arithmetic coding illustrates this: it represents an entire dataset as a single number. For “ABABAAC” (7 characters, 56 bits in ASCII), after encoding we get a final number requiring only 10 bits. Better probabilities directly yield better compression.