Joseph Fourier's 1822 work on heat spreading introduced a mathematical tool: representing complicated patterns as combinations of sine and cosine functions. Nearly two centuries later, those same functions appear inside the transformer architecture behind modern language models.
The two fields seem far apart. Fourier studied temperature across a physical object, while a language model processes tokens arranged in a sequence. Yet sine and cosine functions give both a useful structure. They turn positions into structured numerical patterns, which a transformer can use to learn how word order affects meaning. The waves supply position information, and training teaches the model how to use it.
Word order carries information the words alone cannot provide. Two sentences with identical words can mean different things depending on who acts on whom. Software works the same way: income - expenses and expenses - income contain the same variables and operator but produce different results. A language model therefore needs access to positional information even when words arrive as an ordered list.
The article explains how word order is encoded, how the attention mechanism makes sense of that order, and how sine and cosine functions supply positional signals. It also covers how training uses this position information and how RoPE, a rotary position embedding method, works.
Source: ByteByteGo · Summarized by HeadlinesBriefing