Zhipu Launches Next-Generation Large Model Inference Network Architecture ZCube

In the history of AI development, we are still experiencing a stage of wild growth: scale is everything.

This path works. From thousands to tens of thousands of cards, the leap in computing scale supported the surge of large models from GPT-4 to DeepSeek V4 and GLM-5.

However, we overlooked a reality: when GPU hardware scale reaches a certain level, the links connecting computing power become a performance variable that cannot be ignored.

On May 5, 2026, OpenAI, jointly with NVIDIA, AMD, Intel, Microsoft, and Broadcom, released the MRC (Multipath Reliable Connection) protocol to the industry via the Open Compute Project (OCP). Designed for ultra-large-scale AI clusters, it is deployed in OpenAI's largest NVIDIA GB200 supercomputing cluster for training frontier models like ChatGPT.

- Technical blog link: https://openai.com/index/mrc-supercomputer-networking/

MRC's core idea is dispersing data transmission across hundreds of paths concurrently, bypassing link failures at the microsecond level, fundamentally solving network communication bottlenecks in large-scale GPU clusters.

Just now, Zhipu has pioneered the large-scale deployment of the new-generation networking architecture ZCube in the GLM-5.1 online production cluster.

ZCube was published at the top networking conference ACM SIGCOMM 2025, evaluated as "significantly change the way we think about and understand networking." This is its first large-scale application in a production inference cluster. Zhipu collaborated with Yuxun Network for network architecture upgrade and optimization, based on Ruijie Network switches.

- Paper title: From ATOP to ZCube: Automated Topology Optimization Pipeline and A Highly Cost-Effective Network Topology for Large Model Training

- Paper link: https://dl.acm.org/doi/epdf/10.1145/3718958.3750503

- Technical blog: https://z.ai/blog/zcube

Under the new architecture, without adding any GPUs or modifying any application code, cluster inference throughput increased by 15%, tail latency for first-token response (TTFT P99) decreased by 40.6%, and switch and optical module hardware costs reduced by one-third.

Without adding a single GPU, computing power instantly increases by 15%. Perhaps, beyond expanding GPU scale: network has become the next main battlefield for ultra-large-scale AI infrastructure.

ZCube Technical Interpretation: Eliminating Congestion at the Architecture Level

Inference era traffic is completely different from the training era

Traditional internet data center traffic is "statistically uniform." Fat-Tree / Clos architectures are designed based on this, using multi-layer switch stacking and ECMP (Equal-Cost Multi-Path routing) to distribute traffic across multiple paths, with good results. AI training clusters largely follow this architecture, as communication patterns during training are relatively fixed and regular.

But large model inference brings a new traffic paradigm. With Prefill (processing user input context) and Decode (generating output token by token) separated deployment becoming mainstream, internal data transmission shows high dynamic asymmetry. Prefill nodes process input context, Decode nodes generate tokens, requiring frequent KV Cache transfer. Context length differences vary by tens of times, making KV Cache flow between GPUs almost unpredictable.

In ROFT architecture, traffic load between Leaf switches is prone to imbalance

In clusters using traditional ROFT architecture, different NICs on the same machine bear vastly different KV Cache transmission loads. This source-destination asymmetry makes ROFT rail mapping not naturally equivalent to load balancing, concentrating traffic on few Leaf switches and links.

Diagram of KV Cache transmission load differences across different NICs on the same machine

The root of the problem lies in the special congestion caused by Clos / ROFT architectures for inference traffic, which Zhipu explicitly distinguishes into two types in its technical report:

- Inevitable Congestion: For example, multiple GPUs sending data to the same destination simultaneously, inevitably causing competition on the last hop link. This congestion is determined by physical laws, usually mitigated by congestion control, traffic shaping, and other mechanisms.

- Avoidable Congestion: Caused by topology structure and traffic mapping methods. Traffic is naturally pushed to the same switches and links by topology relationships. Even with sufficient total bandwidth, local hotspots form. Some Leaf switch egress queues remain high, frequently triggering PFC (Priority Flow Control) backpressure, spreading congestion from local to the entire link, amplifying tail latency, and lowering overall throughput. The essence of this congestion is an architecture design issue, which cannot be cured by tuning parameters and protocol optimization.

Diagram of two types of network congestion

In the past, the mainstream industry approach to network congestion focused on the protocol layer: adaptive routing, packet spraying, and even OpenAI's recently released MRC. Essentially, these are remedies or bypasses after congestion occurs.

ZCube chooses to eliminate the root cause of the second type of congestion at the architecture level, making it "not happen if it shouldn't."

ZCube's Triple Design Logic

From the ZCube networking architecture diagram, this architecture breaks the traditional networking approach of hierarchical switch stacking in Clos architecture, designing a completely flat way to interconnect GPU servers.

ZCube architecture can effectively avoid structural network congestion compared to ROFT architecture

First Layer: Cancel Hierarchy, Full Network Flattening. Traditional Clos architecture consists of Spine and Leaf layers. Data transmission between GPUs requires "going up to Spine, then down to Leaf," and cross-layer forwarding itself introduces extra latency and congestion risk. ZCube cancels the Spine layer switches, dividing all Leaf switches into two groups by odd/even serial number, using fully bipartite interconnection between the two groups (i.e., each odd switch connects to all even switches). This means any GPU in the network can reach any other via only two switches, with a network diameter of 2 hops, between single-layer switch networking (1 hop, limited scale) and dual-layer Clos (3 hops, high latency).

Second Layer: Single-Rail + Multi-Rail Hybrid Access, Achieving Ideal Load Balancing. This is ZCube's most ingenious design. Each GPU NIC has two ports, accessing the two switch groups in two completely different ways: the first port uses "multi-rail" method, where GPUs with the same number connect to the same odd switch; the second port uses "single-rail" method, where consecutively numbered GPUs connect to the same even switch.

This "one NIC, two connection methods" topology design brings a key feature: between any two GPUs in the network, there is exactly one optimal path. Multi-path selection is a major source of congestion in traditional architectures, because multiple paths mean load balancing strategies need to make choices, choices can go wrong, and can lead to traffic concentration. ZCube eliminates this uncertainty with a unique path.

In simpler terms: traffic that would conflict between switches in ROFT architecture can enjoy the entire network path exclusively in ZCube architecture.

Third Layer: Lower Cost, Stronger Scalability, Higher Fault Tolerance. ZCube reduces hardware cost while achieving performance improvements. Due to canceling the Spine layer, at the same cluster scale, ZCube reduces switches and optical modules by about one-third compared to Clos / ROFT architectures.

Scalability also performs excellently. Calculating with the current mainstream 400Gb/s network configuration, ZCube can build a flat network connecting 16,384 GPUs with just one switch layer. To achieve the same scale, traditional ROFT architecture requires three switch layers. If using next-generation 102.4Tbps switches and four-port ConnectX-8 NICs, ZCube can support a GPU scale of up to 65,536.

Fault tolerance also has advantages. Since there are no hard isolation planes between GPUs in the ZCube network, under a given link failure rate, the probability of GPU unreachable is more than 50% lower than traditional dual-plane Clos networks.

Diagram of traffic load balancing under ZCube architecture

Traditional Clos architecture is like a multi-layer interchange system, where traffic needs to frequently enter and exit ramps. Specific ramps inevitably congest during peak hours, and a failure in one layer of the interchange affects a large amount of traffic. ZCube is like a mathematically optimized flat road network, where every car has a unique shortest path directly to the destination, traffic on any road segment is evenly distributed, eliminating the conditions for congestion from the planning level.

Experiment: Same Hardware, 15% More Computing Power

What truly makes ZCube noteworthy is that it has completed full production validation in Zhipu's thousand-card level GLM-5.1 coding inference cluster.

The experiment setup was very clean: GPU model unchanged, software stack unchanged, business code unchanged. The only variable was replacing the traditional ROFT architecture with ZCube. The results are as follows:

- Inference throughput increased by over 15%: 15% more API requests per second with the same hardware investment

- TTFT P99 decreased by 40.6%: Tail latency for first-token response significantly reduced, smoother user experience

- Switch and optical module costs reduced by one-third: In a ten-thousand card scale cluster, network hardware alone saves investment of approximately 210 million to 640 million RMB

More importantly, the marginal cost of this upgrade is low, purely a networking architecture replacement. When GPU supply remains tight and prices stay high, a solution that significantly improves system efficiency without relying on hardware stacking has obvious industrial value.

AI Infrastructure Value Center is Shifting

ZCube's deployment and OpenAI MRC protocol release, viewed against the backdrop of 2026 AI infrastructure, point to a deeper industry inflection point: the value center of AI infrastructure is shifting from "compute" to "system efficiency."

Make GPUs Run Better

In the past three years, large model companies' infrastructure strategy can be summarized by one word: Buy. Grab GPUs, grab compute cards, grab cabinets, even grab energy. This logic in.

This page provides an editorial summary based on publicly available information. It is not a republished article. Use the source link below for the original report.