DeepInfer is a high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale. It currently supports DeepSeek-V4.1-Flash and runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
The framework includes optimized kernels such as Flash MLA, DeepGEMM, DeepEP, Tile Kernels, and custom Tile Lang kernels. It supports attention data parallelism, balanced context parallelism, MoE expert parallelism with EPLB, and asynchronous zero-overhead batch scheduling. Advanced decoding features include DSpark and confidence-scheduled verification, which accelerate decoding with draft tokens, and decoder SWA bounded replay, which reduces prefill work.
DeepInfer also offers batch-invariant inference, which keeps model results identical across different batch compositions on NVIDIA hardware, along with deterministic sampling and structured outputs constrained by regex, JSON schema, or EBNF grammar.
A complete inference service still requires components not included in this release, such as a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas. The project is released under the Apache License 2.0 and was originally built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM.
Source: Hacker News · Summarized by HeadlinesBriefing