NVIDIA Optimizes Kimi K3 Model Inference Speed by 1.68x
Through upstream vLLM optimizations and Megakernel fusion, NVIDIA achieved an inference speed of 423 tok/s/user for the 2.8-trillion-parameter Kimi K3 model on GB300 NVL72, a 1.68x performance improvement.
Key Takeaways
- Key Highlight：Through upstream vLLM optimizations and Megakernel fusion, NVIDIA achieved an inference speed of 423 tok/s/user for the 2.8-trillion-parameter Kimi K3 model on GB300 NVL72, a 1.68x performance improvement.
- Innovation & Tech：Highlights advancements in NVIDIA, Optimizes, Kimi, demonstrating rapid progress in model capabilities.
- Industry Impact：Reported via IT之家 (CN), offering actionable signals for developers and technology leaders.
NVIDIA CentML engineer Benjamin completed an inference optimization for the Kimi K3 model. The core approach involved large-scale Megakernel fusion, using Cute DSL to write a decoding-specific kernel comprising approximately 12,000 lines of code. This optimization enabled the 2.8-trillion-parameter Kimi K3 model to achieve an inference speed of 423 tok/s/user on GB300 NVL72 hardware, representing a 1.68x performance improvement over previous results.
The significance of this achievement lies in the fact that inference efficiency for ultra-large parameter models has long been a key bottleneck constraining their practical deployment. As a 2.8-trillion-parameter model, Kimi K3 incurs extremely high inference costs. By performing deep kernel-level optimizations, NVIDIA achieved a nearly 1.7x speed improvement without replacing hardware, providing a new technical pathway for deploying ultra-large-scale models.
The combination of Megakernel fusion and Cute DSL is noteworthy. Megakernel fuses multiple computational operations into a single kernel for execution, reducing kernel launch overhead and memory access latency. Cute DSL is a domain-specific language introduced by NVIDIA for new hardware architectures, designed specifically for efficiently writing decoding kernels. This hardware-software co-optimization approach reflects NVIDIA's full-stack layout in the AI infrastructure space, spanning from hardware to the software stack.
From an industry impact perspective, the improvement in inference speed directly reduces the unit cost of large model services, making the deployment of ultra-large-scale models feasible in more scenarios. Meanwhile, the 12,000-line Megakernel also hints at the potential of AI-assisted generation of advanced computational kernels. In the future, compilers and automated optimization tools may further lower the barrier to kernel development.
This page provides an editorial summary based on publicly available information. It is not a republished article. Use the source link below for the original report.
Industry Insights & Analysis
As artificial intelligence rapidly evolves, breakthroughs surrounding NVIDIA, Optimizes, Kimi, K3 are shifting toward scalable, robust real-world implementations.
Driven by both open-source ecosystems and proprietary model architectures, the integration between compute optimization, data engineering, and agentic workflows is accelerating. This development provides a strategic benchmark for upcoming AI tooling and developer workflows.