HeadlinesBriefing favicon HeadlinesBriefing.com

Neutrino-1 8B: Efficient 8B‑Parameter Transformer

Hacker News •
×

The Neutrino-1 8B model is an 8.19B‑parameter decoder‑only transformer released as a single 3.88 GB artifact. Its 252 transformer linear layers are stored in a proprietary ternary‑family weight format that is eight times smaller than fp16, keeping the weights bit‑packed at rest and decoding them inside the matrix kernels so no fp16 or fp32 weight material appears in the decode path.

This compact representation lets a 3.88 GB working set decode faster than a 16 GB fp16 model on the same memory subsystem, fitting alongside its KV cache on an 8 GB GPU or a 16 GB laptop. The model scores 72.1 on MMLU, achieves 763 tok/s on an H100, 30.7 tok/s on an L4 via GGUF+CUDA, and 33.7 tok/s on an Apple M5 with MLX, while grouped‑query attention keeps the KV cache at 144 KiB per token.

A companion Neutrino-1 0.6B draft model (328 MB) runs in the same process, sharing binaries and adding only an 8.46 % weight overhead. Together they require about 4.20 GB of resident weight and provide a shared cache of 1 GiB for a 4k‑token context. The container works unchanged on datacenter GPUs, Mac Books, and desktop CPUs.