Su Jianlin Reviews Kimi K3: What Key Technical Trade-offs Lie Behind the 896 Experts?
From abnormal activation to expert congestion, dissecting the difficulties of training ultra-large models. Author: Zheng Jiamai Editor: Cen Feng In recent days, Kimi researcher and RoPE proposer Su Jianlin published an article titled "A Simple Discussion on K3's MoE and Attention," focusing on several key trade-offs in Kimi K3's expert architecture, training stability, load balancing, and attention design. Su Jianlin mentioned in the article that K3 has 2.8 trillion total parameters and is configured with 896 routing experts. However, the model does not let each Token call all experts, but only activates 16 of them. In terms of attention structure, K3 did not follow a single scheme, but instead alternates KDA and Gated MLA. These configurations collectively point to K3's base
From abnormal activation to expert congestion, dissecting the difficulties of training ultra-large models. Author: Zheng Jiamai Editor: Cen Feng In recent days, Kimi researcher and RoPE proposer Su Jianlin published an article titled "A Simple Discussion on K3's MoE and Attention," focusing on several key trade-offs in Kimi K3's expert architecture, training stability, load balancing, and attention design. Su Jianlin mentioned in the article that K3 has 2.8 trillion total parameters and is configured with 896 routing experts. However, the model does not let each Token call all experts, but only activates 16 of them. In terms of attention structure, K3 did not follow a single scheme, but instead alternates KDA and Gated MLA. These configurations collectively point to K3's base.
This page provides an editorial summary based on publicly available information. It is not a republished article. Use the source link below for the original report.