The Kimi K3 report offers an unprecedented look into building a frontier AI model, detailing its architecture and training. Unlike typical frontier labs that keep their methods private, Moonshot has published a 47-page technical report on K3, a 2.8-trillion-parameter mixture-of-experts model that rivals closed-source competitors.

K3's advancements stem from several engineering changes: a fixed-size running state for attention, eliminating the need for positional encodings, and attention residuals allowing layers to access earlier outputs. These, combined with refined data and training, led to a 2.5x gain in scaling efficiency over Kimi K2. The report emphasizes that building such models involves more than just the core architecture, with data and training recipes playing crucial roles.

The model utilizes a mixture-of-experts approach, routing each token to a subset of specialized networks, which allows for a vast parameter count without a proportional increase in per-token computation. However, managing the load distribution among these experts is a significant training challenge. The report highlights that the core innovations are well-tested ideas rather than single breakthroughs, underscoring the iterative nature of frontier AI development.