This article demystifies the Delta Net family of linear attention variants, including those used in the Qwen and Kimi models. It traces the evolution from quadratic softmax attention to simpler linear forms and finally to Kimi Delta Attention (KDA). The core challenge in understanding modern linear attention lies in its increasing complexity. The derivation begins by removing the softmax from traditional attention, leading to a recurrent update of a fixed-size state matrix $S_t = S_{t-1} + lvert v_t

angle langle k_t

vert$. However, this additive update can cause interference. Delta Net addresses this by replacing the unconditional write with a delta-rule correction. It calculates an error term $lvert e_t

angle = eta_t (lvert v_t

angle - S_{t-1} lvert k_t

angle)$ and updates the state as $S_t = S_{t-1} + lvert e_t

angle langle k_t

vert$. This ensures that when the new key $k_t$ is queried, the state correctly returns the value $lvert v_t

angle$ when $eta_t = mathbf{1}$. The article also presents an alternative derivation based on minimizing a reconstruction loss.

The progression of attention mechanisms discussed is: softmax attention $

ightarrow$ linear attention $

ightarrow$ Delta Net $

ightarrow$ Gated Delta Net $

ightarrow$ KDA. The article uses bra-ket notation for clarity, where $lvert q

angle$ is a column vector and $langle k

vert$ is a row vector. The final KDA state update equations are presented, followed by a brief mention of the recurrent and chunkwise Triton programs used for execution.