Hugo Vergnes trained a 3.8B-parameter LLM to 0.384 CORE score using 65B tokens over 43 hours on rented B200 GPUs for $998. The project, built as a config-driven framework called little-lm, emphasizes clean infrastructure and software engineering practices. Model details include Llama-style architecture with RMSNorm, RoPE, GQA (24 query, 8 KV heads), relu² MLPs, QK-norm, logit softcap, and ResFormer-style value embeddings totaling 3.848B parameters, where value embeddings constitute 19%.

Compared to nanochat d32 (~1B parameters, ~$1000, 0.310 CORE), this model achieves significantly better performance at similar cost and time. Vergnes highlights that good infrastructure reduces manual code changes, enabling experiments via YAML diffs. Early failures with an 858M model on Fine Web-Edu showed the importance of learning rate scheduling and optimizer choice, with Muon showing promise over AdamW for matrix parameters.

The work demonstrates that meaningful LLM training is accessible outside mega-corporations with modest budgets.