Baidu has open‑sourced ERNIE 5.5 Flash Base, a sparse Mixture-of-Experts language model featuring 280B total parameters, with 14B activated per token. Trained from scratch on 33.5T tokens and a 128K context, it supports a 1M‑token context for fine‑tuning and inference. The release provides the pre‑training checkpoint only, giving developers a clean baseline for domain adaptation, instruction tuning, and reinforcement‑learning research without the cost of large‑scale pre‑training. The architecture balances efficiency and capability: 43 Transformer layers, 4096 hidden size, 64 attention heads, and a hybrid latent‑gated sparse attention scheme that mixes Multi‑head Latent Attention (MLA) and Heavily Compressed Attention (HCA). This design reduces KV cache and compute while preserving long‑range understanding. By co‑designing the model and training system, Baidu aims to lower barriers for further training, allowing the community to focus resources on specialized tasks and custom training methods.
Key highlights include an affordable, open starting point; efficient sparse MoE and hybrid attention; diverse, high‑quality pre‑training data; and a single research baseline for reproducible comparisons.
The model is intended for researchers and engineers who need a robust, cost‑effective foundation for building domain‑specific AI applications.
Source: Hacker News · Summarized by HeadlinesBriefing