vLLM speeds up Kimi K3 to 842 tokens per second on NVIDIA GB300 GPUs

The vLLM project described the work on its blog on 16 October 2026, the day it was merged into vLLM as pull request 61867. NVIDIA built it with help from coding agents. In Kimi K3’s decode step, where the model generates output tokens, custom GPU code written in CuTe DSL replaces almost every operation, from attention to communication between GPUs.

It supports Kimi K3 on 8 or 16 GB300 GPUs connected by NVLink, handling 1 to 64 requests at a time, and operators switch it on with the decode_backend setting latency_specialized. All changes sit in vLLM’s Kimi K3 code, and the pull request reports AIME25, GPQA-Diamond and MRCR v2 scores matching stock vLLM within about one standard error.

With DSpark speculative decoding, in which a draft proposes several tokens for the model to check at once, Kimi K3 reaches 842 tokens per second for a single request on 16 GPUs in vLLM’s controlled test.

With DSpark drafting seven tokens on 16 GB300 GPUs, the new CuTe DSL decode path generates 842 tokens per second for a single request, against 488 for stock vLLM, 1.73 times faster — vLLM’s DSpark test on 16 GPUs: seven draft tokens, a synthetic acceptance length of 6, batch 1Image: vLLM

On AgentX, a SemiAnalysis benchmark that replays real coding-agent sessions, per-user speed at the 90th percentile rose 1.8 times at one concurrent request, though vLLM compares a 16-GPU run with an earlier 8-GPU result.

On AgentX, the optimised path gives each user 403.2 tokens per second at one concurrent request on 16 GPUs, against 223.7 on 8 GPUs and 251.3 on 32 GPUs in vLLM’s earlier results — vLLM’s AgentX chart: per-user speed against throughput per GPU, with GPU counts differing between pointsImage: vLLM

Validation so far covers GB300 only. A similar path for DeepSeek-V4.1-Flash is open for review as pull request 61868.

vLLM speeds up Kimi K3 to 842 tokens per second on NVIDIA GB300 GPUs

Sources

The rest of the day

DeepSeek releases DeepSeek-V4.1-Flash-1017 with MIT-licensed open weights

OpenAI rolls out ChatGPT Slides for presentations teams edit together

Anthropic’s Claude finds five NIST curve seeds came from timestamps, not hashes

OpenCode says a breach tied to a distillation attack hit 1.4% of Console users