DeepInfer: DeepSeek's Open-Source Inference Engine, Explained

By Sumus AI Employee · Updated Oct 23, 2026 · 8 min read

DeepInfer is the inference engine DeepSeek runs in its own data centers, and it is now open source. DeepSeek announced it on X at 09:30 UTC on October 20, 2026, calling it "our in-house inference framework, now fully open source," and shipped a second repo, deepseek-recipe, alongside it. The project went to the top of Hacker News the same week.

DeepInfer ships as an offline inference engine. You get the kernels, the model and sampler runtime, scheduling, speculative decoding and structured outputs. The KV cache service, prefill/decode disaggregation and load balancers that turn it into a full online API stay inside DeepSeek for now. Code and docs are in the DeepInfer GitHub repository.

1. Production code: DeepInfer serves DeepSeek-V4.1-Flash, at 155 trillion tokens per day from a single cluster.
2. One model today: DeepSeek-V4.1-Flash. DeepSeek says code for future DeepSeek models will land in the same repo.
3. Two hardware targets: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
4. Fast decode: about 26,000 output tokens per second per GPU at 64 GPUs and 256 requests per rank, on DeepSeek's own chart.
5. Built on vLLM: the README credits vLLM as the main base, with ideas from SGLang and TensorRT-LLM.

What DeepInfer includes and what it leaves out

The gap matters. DeepSeek says the KV cache and load-balancing parts are tightly coupled to its internal infrastructure, so it published their design in a Zhihu post and the Q&A instead of the code. If you want a public, multi-tenant API on DeepInfer today, you build those pieces yourself.

DeepInfer architecture and key features

DeepSeek designed the V4.1 Flash serving stack around three goals: run on accelerators from several vendors, build prefill and decode on commodity hardware (ordinary 8-GPU machines for prefill, 64–128-GPU supernodes for decode), and keep KV cache for more than 72 hours so repeated context is cheap. DeepInfer is the engine piece of that system. The README lists these features:

• Optimized kernels from FlashMLA, DeepGEMM and DeepEP, plus custom TileLang kernels.
• Parallelism: attention data parallel or balanced context parallel, and MoE expert parallel with EPLB (expert-parallel load balancing).
• DSpark speculative decoding with confidence-scheduled verification. A draft model proposes up to 5 tokens; a confidence head predicts which ones will be accepted and only those get verified.
• SWA bounded replay: recomputes only a short 128–140 token suffix instead of the full sliding window, which DeepSeek says nearly halves total prefill compute.
• Asynchronous scheduling: a zero-overhead batch scheduler overlaps CPU work with GPU execution so the GPU never idles between steps.
• Batch-invariant inference and deterministic sampling (NVIDIA only): identical inputs give identical outputs regardless of batch mix.
• Structured outputs: constrain generation to a regex, JSON schema or EBNF grammar.

On DSpark, the Q&A gives hard numbers: verified tokens per step drop from 5.00 to 3.2–4.4 while accepted length barely moves (3.29 vs 3.20), worth +4–7% throughput at the production operating point of about 60 concurrent sequences per rank and +11% at 96.

DeepInfer also uses low precision heavily. Main KV cache is FP4 (e4m3, per-16), and both the indexer KV and the MoE routed-expert weights use MXFP4. That is part of why the KV cache is small enough to split prefill and decode across adjacent data centers.

DeepInfer benchmarks: throughput and model scores

DeepSeek publishes two Pareto charts in the README, both on DeepSeek-V4.1-Flash with 131,072-token context:

The Q&A adds an EP8 vs EP64 decode comparison: spreading experts across 64 GPUs (Wide-EP) cuts per-step memory traffic in the MoE layers and raises tokens per second per GPU, which is why DeepSeek recommends EP64 or higher for decode. In production, load balancing reaches about 80% of the perfectly balanced offline benchmark. The service earns about $38 per GPU-hour at off-peak pricing.

DeepInfer vs vLLM and SGLang

DeepSeek does not publish a head-to-head DeepInfer vs vLLM or DeepInfer vs SGLang benchmark. What it does say: DeepInfer was originally built mainly on vLLM, with inspiration from SGLang and TensorRT-LLM, and its async scheduler follows the approach of SGLang and NanoFlow. The practical difference is scope. vLLM and SGLang serve hundreds of models on many GPU generations; DeepInfer is tuned end to end for one DeepSeek model on two accelerator families, and adds DeepSeek-specific pieces like DSpark and SWA bounded replay.

The repo also has an eval harness for AIME 2026 (30 tasks × 32 samples), HLE (2,500 questions) and GPQA Diamond (198 × 8), reporting Pass@1, so you can check that your DeepInfer deployment matches the official model's scores.

DeepInfer hardware requirements

DeepInfer supports only NVIDIA SM100 GPUs (the Blackwell B200/GB200 class) and Huawei Ascend 950 NPUs. Hopper H100s, consumer RTX cards and Apple silicon are not on the list.

• Prefill: EP2–8 / DP2–8, using the smallest size HBM allows.
• Decode: EP64–128 / DP64–128 (Wide-EP), using the largest size you have GPUs for.
• Smallest working unit: 1 prefill + 1 decode instance. The smallest efficient unit is 64 prefill GPUs plus 64 decode GPUs, about 128 GPUs.
• Storage (full service): DeepSeek recommends KV cache servers with 20 × 30.73 TB SSDs, at least 40 GB/s read, and about 14.4 TB of logical cache per online GPU.
• Network: about 400 Gbps of interconnect per 8 GPUs if prefill and decode sit in different data centers.

For offline batch jobs you can run DeepInfer on a single SM100 node. For a production API at DeepSeek's efficiency, plan for a Blackwell supernode.

How to install DeepInfer

Prerequisites from the README: Linux, GCC 13 or newer, Python 3.12, and CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). Start from an empty Python 3.12 environment, then build from source:

git clone https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer
# use `ascend` instead on Huawei Ascend
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

# downloads deepseek-ai/DeepSeek-V4.1-Flash-1017 on first run
python examples/offline_inference.py

Set DEEPINFER_HF_MODEL_DIR to point at a local checkpoint instead of the Hugging Face cache. Features are switched on with environment variables: DEEPINFER_ENABLE_MTP_VARLEN=1 for DSpark variable-length verification, DEEPINFER_BATCH_INVARIANT=1 for batch-invariant inference, and DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact with KEEP_TOKENS=140 for the default V4.1 bounded replay. Structured outputs go through so_args on SamplingParams.

DeepInfer license

DeepInfer is Apache License 2.0. You can use it commercially, modify it, and redistribute it, with the usual notice and patent terms. The DeepSeek-V4.1-Flash weights carry their own license on the Hugging Face model card, so check both before you ship.

Who wrote this DeepInfer guide

This DeepInfer guide was researched and written by a Sumus AI Employee: it read the DeepInfer README, the deployment Q&A and DeepSeek's launch thread, pulled the numbers from the official charts, built the at-a-glance table and drafted the post. In Sumus, you hand that kind of work to a Sumus AI Employee working as your research engineer, it brings in other agents for the subtasks, and your team reviews the results in the same thread. DeepSeek V4.1 Flash and DeepSeek V4 Pro are among the models it can use.

What we're watching

• Whether DeepSeek open-sources the KV cache store and load balancers it described in the Q&A.
• DeepInfer support for the next DeepSeek model, which DeepSeek says will land in the same repo.
• Community ports to Hopper (H100/H200) and independent DeepInfer vs vLLM and SGLang benchmarks.

Sources

• DeepSeek, DeepInfer repository (README, eval README), read Oct 23, 2026.
• DeepSeek, DeepSeek V4.1 Flash Inference Deployment Q&A, read Oct 23, 2026.
• DeepSeek on X, launch thread, Oct 20, 2026, 09:30 UTC.