llama-modchip breaks the 64 GB RAM wall, running 284B DeepSeek-V4-Flash and 30B MoE models on pure CPU with zero GPUs. The 284B model is compressed from 76.87 GiB to 56.56 GiB on disk using closed-form Analytical SwiGLU-H2 Second-Order Hessian Compression and Fractal-64 MLA transcoding, fitting entirely in RAM at 57.61 GiB Vm RSS during 256K context inference. It achieves 44.36 tok/s prefill and 9.1–14.7 tok/s decode on a 64 GB desktop. A 30B MoE like Nemotron-3.5-30B-A3B runs in just 14.35 GiB at 25–60.4 tok/s on a 16-core Zen 5 laptop, scaling to 111.6 tok/s on an 8-channel Threadripper workstation. The toolkit also introduces a 13.1 ms Claude Code restore via /dev/shm Slot-0 KV Warm Restore, eliminating 45-second CPU agent prompt bottlenecks.
Supported models include Nemotron-3.5-30B-A3B, Qwen3-30B, and DeepSeek-V4-Flash, with hybrid GPU/CPU configurations available for mixed systems. Quickstart is a single command: clone the repo and run ./modchip chat. Pre-compiled Linux x86_64 binaries and Hugging Face GGUF downloads are provided. Custom GGML_TYPE_Q2_0 (type ID 40) tensors require the modchip binary; unpatched llama.cpp, Ollama, or LM Studio will fail. Full documentation, leaderboards, and preprint PDFs are available on the GitHub repository.
Source: Hacker News · Summarized by HeadlinesBriefing