While digging into foundation models for time series, I realized that I could not really understand them without first understanding transformers. I did not want to use these models as black boxes, so I started tracing the ideas backward, from foundation models to transformers, and from transformers to self-attention. What made the transition interesting is that although transformers were originally built for language, the core idea carries naturally to time series. The two modalities are very different, but they share something fundamental: both are sequences, and in both cases, order changes meaning. In language, dog bites man is very different from man bites dog. Time series are no different. A temperature of 30° yesterday and 20° today tells a different story from 20° yesterday and 30° today. The values may be the same, but their order changes the meaning of the sequence. The question is that if self-attention looks at all observations at once, how does a transformer know which observation came first, which came later, or how far apart two observations are? That question led me to positional encoding. What surprised me most was how such a simple mathematical idea could give a Transformer a sense of order. The exact techniques have evolved considerably since then, but the underlying problem remains the same. This article is my attempt to build that intuition from the ground up, starting with a simple time series and following the path from raw observations to self-attention and finally to positional encoding.

From scalar observations to vector representations Consider a simple time series containing the temperature recorded over five weekdays: Example of time series: 5-day temperature history Each observation x_t is only a scalar. A transformer, however, operates on vectors of dimensionality d_model. The scalar observations therefore need to be mapped into that representation space first. A simple way to do this is through a learned linear projection i.e. embedding: e_t = W_e x_t + b_e giving us a sequence of vector representations: e_1, e_2, e_3, e_4, e_5 An embedding is a deep, abstract representation of the series in the form of a multidimensional numerical vector that encodes its features and that the model understands. Each time series token is represented by a learned embedding. Each e_t captures information about the observed value at that timestep, but at this point, it is just a representation of the observation. The important word here is learned. The model is not given a predefined vector representation for a temperature such as 18°. The parameters W_e and b_e are learned during training so that the resulting representations become useful for the task. At this point, e_t represents what was observed. It does not yet tell the model where that observation occurred in the sequence.

How self-attention builds context? Self-attention allows each observation to use information from the rest of the sequence. Suppose we want to update Friday’s representation. The model first creates three learned projections from every e_t: q_t = W_Q e_t, k_t = W_K e_t, v_t = W_V e_t Query, key, and value vectors are learned in the self-attention block. The matrices W_Q, W_K and W_V are also learned during training. The model is not told beforehand what a useful query, key, or value should look like. For Friday, its query q_5 is compared with the keys of all observations: k_1, k_2, k_3, k_4, k_5. Each comparison produces a...