Air LLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card without quantization, distillation, or pruning. You can even run 405B Llama 3.1 on 8GB, Deep Seek-V3 (671B) on ~12GB, and Kimi K3 (2.8T) on under 4GB, because sparse Mo E models stream one expert at a time rather than a whole layer.

[2026/07] Kimi K3 (2.8T) support: the largest open-source model runs on a single card in 3.72GB of VRAM, measured end to end on one RTX 6000 Ada. Per-expert streaming loads only the experts a token actually routes to. K3 brings three requirements of its own: pip install compressed-tensors flash-attn (its model code mandates flash attention regardless of what you request), a CUDA 12 build of torch, since no prebuilt flash-attn wheel exists for CUDA 13 yet, and transformers 4.56.x, as its remote code does not load on 5.x.

[2026/06] v3.0: FP8 model support + the latest models. Run Deep Seek-V3 (671B) on ~12GB and Qwen3-235B on ~3GB, plus Qwen3, Llama 3.x/4, Deep Seek V2/V3, Phi-4, Gemma and more — all through a single Auto Model.

Air LLM supports model compression with 3x inference speed up via block-wise quantization. Install with pip install -U airllm and pip install -U bitsandbytes, then pass compression ('4bit' or '8bit') when initializing the model. Configurations include profiling_mode, layer_shards_saving_path, hf_token, prefetching, and delete_original. Mac OS is supported with Apple silicon only.