The Kimi Team introduces Kimi Linear, a novel hybrid linear attention architecture that surpasses full attention in various scenarios, including short-context, long-context, and reinforcement learning.

At the heart of Kimi Linear is Kimi Delta Attention (KDA), an advanced linear attention module that refines Gated Delta Net with a more precise gating mechanism. This allows for more efficient utilization of limited finite-state RNN memory.

A specialized chunkwise algorithm enhances hardware efficiency by employing a tailored version of Diagonal-Plus-Low-Rank (DPLR) transition matrices. This approach significantly cuts down computation compared to standard DPLR formulations, aligning better with the classical delta rule.

Experiments with a Kimi Linear model, featuring 3B activated parameters and 48B total parameters, demonstrate its superiority over full Multi-Head Latent Attention (MLA) with an identical training setup. Kimi Linear achieves this while reducing KV cache usage by up to 75% and boosting decoding throughput by up to 6 times for a 1M context. The Kimi Team is open-sourcing the KDA kernel and v LLM implementations, along with pre-trained and instruction-tuned model checkpoints, to facilitate further research.