In December 2024, Deep Seek released a 671-billion-parameter language model with full technical documentation. Rather than keeping their work proprietary, they enabled other teams to build upon their research. Moonshot AI scaled this design to one trillion parameters, while Zhipu AI incorporated Deep Seek innovations into their own architecture. This indirect collaboration through public model releases has accelerated progress across the industry.

The common thread among frontier open-weight models is the Mixture-of-Experts transformer architecture. Unlike dense models where all parameters activate for every input, MoE uses multiple expert sub-networks that activate selectively. This allows massive total parameters—Deep Seek V3 reaches 671 billion—while keeping active parameters manageable at around 37 billion per word. The distinction matters because active parameters determine inference costs, not total counts.

Three attention strategies have emerged for handling the KV-cache bottleneck in long conversations. Grouped-Query Attention shares cached information across attention head groups, adopted by Qwen3 and Llama 4. Multi-Head Latent Attention, introduced by Deep Seek, compresses cached data at the cost of extra computation. Sparse Attention selects only relevant previous words, implemented by Deep Seek and Zhipu AI's GLM-5.

This open-weight ecosystem has fundamentally shifted how AI research progresses. Teams iterate faster by studying published weights and technical reports rather than reverse-engineering black boxes. The result is rapid architectural evolution where innovations spread across organizations within months instead of years.