In the new stage of the AI inference explosion, GPUs are still scarce and tight, but the memory that can hold and run data quickly will be the new protagonist. Today, following the previous article, we will delve into HBM in detail. Currently, the popularity of HBM continues to explode, and at the same time, numerous layered mitigation solutions and architecture level alternatives such as TPU are also on the way to alleviate the tension of HBM. The core concern of the market for storage is whether it is a cyclical industry or an AI infrastructure? If it's the former, everyone is waiting for the drums to stop, but if it's the latter, it will be a completely different picture of the future. This may not be a true or false question, but a complex mathematical problem that this article will attempt to analyze from a logical and factual perspective. 1、 Why does reasoning make memory the protagonist? When reasoning with a large model, three things need to be done for each token generated: Read all parameters of the entire model from memory and send them to the computing core; Read the intermediate state (called KV Cache) of all tokens before this token; Then perform matrix multiplication to calculate the next token. The third step is calculation, and the first two steps are transportation. The total time required for transportation usually exceeds the calculated total time This fact applies to almost all models with over 10 billion parameters. A 70 billion parameter open-source model (Llama 3 70B) with FP16 precision has a model weight of approximately 140 GB. To generate each token, this 140 GB must be read from HBM to GPU computing core. To ensure smooth token generation, such as 30 tokens per second, the bandwidth between HBM and the computing core must be able to support a throughput of approximately 4.2 TB per second. That's why the HBM bandwidth of H100 SXM5 is set at 3.35 TB/s - below this number, the 70B model inference starts to lag. Bandwidth is one thing, capacity is another. If the total number of parameters in a model exceeds the HBM capacity of a single GPU, the model must be divided into multiple parts and distributed across multiple GPUs, which is called tensor parallelism. But everything in the model has changed from being able to be calculated at once to multiple times, and GPUs need to communicate with each other to transmit intermediate results - communication overhead is the new bottleneck. So both capacity and bandwidth are important, but the emphasis is different. Capacity determination: Can the model be installed on a single card? Do you want to split it? What is the communication cost after cutting it off? Bandwidth determination: How fast can the token be ejected after installation? How low is the delay? NVIDIA and AMD have taken different paths in addressing the need for reasoning: NVIDIA's latest flagship Rubin R200, single GPU with 288 GB HBM4, memory bandwidth of 22 TB/s; AMD's next-generation MI455X, equipped with a single GPU and 432 GB HBM4, has a memory bandwidth of 19.6 TB/s. AMD has 50% more capacity, but 11% less bandwidth. NVIDIA bet on bandwidth - move data faster. AMD staking capacity - allows the model body to be opened without dismantling. The target customer groups of the two companies are different: AMD targets the open-source faction that wants to run super large models such as 405B and 671B; NVIDIA is targeting the SaaS faction that wants to do high concurrency, low latency commercial inference. And recently, new player Cerebras WSE-3, which is going through an IPO, has only 44 GB of on-chip SRAM per chip, but a memory bandwidth of up to 21 PB/s -950 times that of NVIDIA Rubin. Seven times smaller capacity results in three orders of magnitude more bandwidth. Cerebras' judgment is different from NVIDIA and AMD's. 2. Reasoning makes both bottlenecks more tense at the same time This is the 12th article of my "AI Investment Map". The entire text has gone through several days, N drafts, various deletions, and still has 18000 words. I suggest forwarding and bookmarking it first. Recommend setting this account as a star symbol. Go directly to the official account, and the layout here is a disaster HBM Panorama Research Report: From Training to Reasoning, the Protagonist is No Longer a GPU https://mp.weixin. (qq.com)/s/ch6D62c-4OsOllHfzf4jMA