Nvidia also believes that storage is a bigger bottleneck than GPUs Interpretation of nvda's latest article NVIDIA's latest release, "AI Model Co Design," is a technical article introducing TensorRT-LLM and Blackwell, But it is also a roadmap for the development of large-scale model design and AI infrastructure in the coming years. The article really discusses how to make the model adapt to GPU from the beginning of design. This means that AI has entered a new stage: models are starting to revolve around hardware design, rather than hardware optimization around the model. In the past, the focus of big model competition has always been accuracy, but at the beginning of this article, the goal of AI is redefined as a balance between accuracy, throughput, and interactivity. Accuracy is no longer the sole determining factor, and greater commercial value comes from generating more tokens per unit time and lower response latency. The entire article revolves almost entirely around inference, with little discussion of training, which in itself is NVIDIA's assessment of the future market focus. The key to understanding the entire text is the Roofline Model: any program will ultimately be constrained by one of two bottlenecks, either Compute or Memory. The determination of which type of program it belongs to is based on how much computation the GPU can perform for each byte of data transferred: high arithmetic strength, program limited by Compute; Low arithmetic strength, program limited by memory. "Latency-sensitive decoding runs at low concurrency and is memory-bound." Why is natural memory bound easy to reason about? The reason is that reasoning is divided into two completely different stages. Prefill processes the entire prompt and can calculate thousands of tokens at once. A large matrix brings high weight reuse, so it usually belongs to Compute bound. And the decode must be generated on a token by token basis, and the next token must wait for the previous token. Naturally, it cannot be parallelized along the time dimension and can only rely on batches between different users. As the batch size decreases, almost complete model weights still need to be read each time a token is generated. Weight Reuse drops sharply, and GPUs spend more time waiting for data rather than computing. The article provides a typical example using FFN. The GEMM of FFN can be written as M × K multiplied by K × N, where M corresponds to the number of tokens, and N and K correspond to the model dimensions. During the inference process, M rapidly shrinks as concurrency decreases, while N and K remain relatively unchanged. This means that the actual computational load has decreased, but the N × K weight matrix that needs to be read has hardly decreased. The article concludes that 'FFN latency becomes memory bound because the GEMM-N and GEMM-K dimensions remain large as GEMM-M shrinks.' In other words, the GPU is no longer waiting for computation to complete, but for Weight Read. That's also why the article constantly emphasizes Memory and hardly discusses FLOPS. The three most representative sentences in the Roofline section are: "Workload with low arithmetic intensity is capped by memory banding." "Latency sensitive decoration is memory bound." "Memory time exceeds math time at every token count The three sentences actually express the same meaning: for modern inference, the real limitations on system performance are no longer GPU computing power, but Memory bandwidth and Weight Read. MoE can alleviate this problem, but it cannot completely solve it. MoE transforms the Dense model from fully calculating each layer to activating only a few Experts, significantly reducing Compute. However, as Compute decreases, new bottlenecks begin to emerge. Firstly, the All to All communication brought by Expert Parallel, secondly, the load balancing issue brought by Router, and finally, the Weight Read itself still exists. Each Expert still needs to read a large number of parameters, so MoE has shifted the bottleneck from Compute to Memory and Communication rather than eliminating it. That's also why NVIDIA focuses on introducing Wide Expert Parallel. Because nvda hopes to distribute Experts to more GPUs, it can improve the Aggregate Memory Bandwidth of the entire system, while reducing the number of Experts saved on each GPU, thereby reducing the latency caused by Weight Read. Note that the revenue here does not come from more FLOPS, but from more Memory Bandwidth. This is also the first time that the article explicitly explains GPU extension as Memory extension, rather than Compute extension. The Helix Parallel released by nvda in 2015 solved another type of memory problem - KV Cache. Traditional Tensor Parallel has to replicate the entire KV Cache when the number of GPUs exceeds the KV Head count, resulting in a rapid increase in HBM usage. Helix divides KV Cache into sequences instead of heads. Each GPU only stores a portion of KV Cache, and then exchanges sequences during the Attention phase before switching back to TP × EP to complete FFN. This means that Attention and FFN no longer adopt the same parallel strategy, but each choose the optimal solution. As Long Context grows from tens of thousands of tokens to millions of tokens, the value of Helix will continue to increase. The entire article proposes seven design principles, but essentially all revolve around the same goal. Near square Matrix enhances Arithmetic Intensity; 128, 256, 512 alignment improves Tensor Core utilization; Prefer Width over Depth increases Weight Reuse; NVFP4 reduces Memory Traffic; Wide EP improves aggregate memory bandwidth; Chunked Pipeline reduces Pipeline Bubbles; Helix decouples Attention from FFN and adopts different Parallel Strategies. Seemingly seven independent suggestions, they all revolve around the Roofline Model with only one goal: reducing Data Movements and improving GPU utilization. What is truly worth paying attention to is the content that is not directly stated in the article. Firstly, the growth of GPU Compute is already faster than the Compute that the model truly needs, and the marginal revenue from continuing to increase FLOPS in the future is decreasing. Secondly, the bottleneck of the inference era has shifted from Compute to Memory, and HBM, KV Cache, Weight Read, and Memory Traffic will become increasingly important. Thirdly, the competition among GPUs has evolved into a competition among the entire system, including HBM, NVLink, TensorRT, parallel strategies, and software stacks, rather than just the performance of individual GPUs. Fourthly, the real object that needs to be optimized for AI infrastructure in the future is no longer computation, but data movement. In summary, in an increasing number of AI inference scenarios, storage (specifically Memory, Memory Bandwidth, Weight Read, and Data Movement) has become a system bottleneck more easily than GPU computing power. Author background: Almost all seven authors of this article come from GPU Architecture、TensorRT-LLM、Distributed Inference、Model-Hardware Co-design Together with the Compute Architecture team.