Written by: Techub News Organized

Introduction

At the global top chip conference Hot Chips 2025 held in September 2025, one of the co-authors of the Transformer paper "Attention Is All You Need," Noam Shazeer, Vice President of Research at Google DeepMind, delivered a keynote speech titled "Predictions for the Next Phase of AI." As a pioneer deeply involved in the entire process from early language model research to the construction of modern large-scale AI systems, Shazeer's perspective combines historical depth with cutting-edge insights. He not only reviewed his journey from Google Brain to founding Character.ai, and then returning to Google to lead the Gemini project, but also systematically analyzed the core elements driving the intelligent progress of large language models (LLMs), offering heartfelt recommendations for the future direction of AI hardware development. This speech is a valuable window for understanding the technological roots of the current AI wave and the trends of future evolution.

Summary

• The intelligence enhancement of large language models relies on the synergistic growth of several factors such as computational power, parameter scale, model depth, training data volume, and information flow efficiency, with each marginal improvement potentially increasing the model's "IQ" exponentially.
• Hardware innovation is a key driving force of the AI revolution; future hardware should provide more computational power (FLOPS), higher memory capacity and bandwidth, while maintaining determinism.
• Low precision computation (like FP8) is increasingly popular and effective in deep learning, but the hardware and software ecosystem needs to synergistically break through to fully realize its potential.
• The computational demands for model inference (especially long-chain reasoning) and post-training optimization (like RLHF) will significantly increase and could become the focal point of future AI computational load.
• Despite the tremendous success of the Transformer architecture, exploring newer architectures that are more organic and possess "neural state" memory capabilities may be the direction for further enhancing model capabilities and efficiency.

What Do Large Language Models Want?

Noam Shazeer began by posing a central question: what makes large language models smarter? He summarized it as a combination of several key factors, with improvements in each one potentially leading to an exponential increase in the model's "IQ." The foremost factor is computational power, which refers to the ability to perform more floating-point operations. He pointed out that the explosion of deep learning, especially LLMs, is largely attributed to the emergence of highly parallel processors capable of executing computations several orders of magnitude higher than traditional single-threaded processors.

Secondly, the number of model parameters is also crucial, as it determines how much knowledge the model can absorb. While the growth in parameter count typically accompanies increased computational power, it is not a necessity; methods like sparsification can increase parameters without significantly increasing computational burdens. Additionally, model depth (i.e., the number of layers) and the non-linear transformations between layers contribute to the model's capabilities, which is the origin of the term "deep learning."

Shazeer emphasized the importance of information flow. The core advantage of the Transformer architecture lies in its ability to efficiently move information between different parts of a sequence. Whether within the sequence (such as attention mechanisms) or between network layers, smoother and richer information flow enhances the model's intelligence. Finally, the scale and uniqueness of training data are likewise critical. Utilizing more and more unique training samples, while avoiding simple repetition, is fundamental for continuously improving model performance.

"At any time, if we can significantly enhance one aspect without severely compromising others, it's usually a huge victory," Shazeer concluded.

From Language Modeling Obsession to Co-Founder of Transformer

Shazeer recounted how he became "hooked" on the LLM field. Back in 2015, while working at Google Brain, he encountered a group of researchers obsessed with the "Billion Words Language Modeling Benchmark." At that time, LSTM-based models ran for weeks on dozens of GPUs just to lower the perplexity from 30.5 to 30.0. This pursuit of extreme performance deeply captivated him.

He believed that language modeling is "the best problem." Text is the most refined form of abstract thinking, and the task of predicting the next word is extraordinarily simple, yet it contains the potential for solving general artificial intelligence (AGI). "If you get better and better at this task, you've essentially solved artificial intelligence. You can pass the Turing test and accomplish any task that can be done through conversation," explained Shazeer. At the time, the available training data seemed infinite, which opened up a realm of possibilities for him.

His first major contribution was exploring sparsity. Since increasing parameters and computation can make models smarter, and computation is expensive, could parameters be increased without a proportional increase in computation? He proposed the "Sparsely-Gated Mixture of Experts" model, whose core idea is to design a block-level sparsity compatible with dense matrix multiplication to avoid reducing the utilization of hardware computing units. This idea wasn't immediately practical at the time, but has now been widely adopted.

Then in 2017, he collaborated with a team of researchers to create the Transformer architecture. They aimed to replace the dominant and often "frustrating" RNN with attention mechanisms. The attention mechanism significantly enhanced the information flow across the dimensions of sequences by constructing a key-value storage system, and the entire architecture could be efficiently implemented using matrix multiplication, enabling fast execution on GPUs or TPUs. "This ultimately achieved considerable success," Shazeer humbly noted.

In 2018, as Google began building TPU Pods (large-scale TPU clusters), Shazeer started considering how to scale models beyond the memory limitations of individual chips. He collaborated with Cliff and other colleagues to develop the Mesh TensorFlow library, which enabled Transformer models to run distributed across multiple chips, laying the foundation for the training of today's super-large-scale models.

Prophecies From Seven Years Ago and Today's Reality

Shazeer showcased a segment from a talk he gave in 2018 at a major conference titled "Natural Language Models: The Bigger, The Better," as a recap of "I told you so." He presented the quality changes in text generation from models with parameter scales ranging from 30 million to 5 billion and predicted the potential of models with hundreds of billions of parameters. At that time, he refuted the common notion that "smaller models benefit generalization, speed, and memory," arguing that insufficient training data, slow computers, or memory limits were not fundamental issues and strongly advocated for the then-existing TPU v3 supercomputer and Mesh TensorFlow.

"Fast forward to 2025, I want to say, I told you to scale up the Transformer," Shazeer said with a smile. Today, this has become a consensus in the industry.

Since 2018, he has also been addressing other critical issues, such as inference latency. While Transformers are efficient at handling parallel sequences, during token-by-token generation (decoding), the attention layers degrade to vector-matrix multiplication, which is not efficient on modern hardware. He contributed to technologies like Multi-Query Attention and Speculative Decoding (previously referred to as block-level parallel decoding) to accelerate the inference process. Additionally, he participated in the T5 project and the development of Google's dialogue robot LaMDA's predecessor "Mina," which inspired him to co-found Character.ai, aiming to create chatbots with personality that deeply interact with users.

"At that time, we were one of the first companies to launch LLMs that consumers could talk to directly online," Shazeer mentioned. "It turned out that many people used it for entertainment purposes, which makes sense because these models are essentially trained to 'make things up' rather than tell you the truth." This was also one of the reasons why he returned to Google in 2024 to co-lead the Gemini project—he saw the immense progress and potential of LLMs in practical applications.

Hardware, Please Give Us More!

As a software and algorithm researcher, Shazeer is acutely aware of the foundational role of hardware. He first thanked the hardware community present, stating that the AI revolution "could not happen without you." So, what do large language models need hardware to provide?

His list was concise and direct:

• More FLOPS (computational power): This allows us to train more computation-intensive models and use more training samples.
• Higher memory capacity and bandwidth: From on-chip storage to HBM, bottlenecks in the entire memory hierarchy may limit model design. If memory bandwidth is relatively low compared to computational power, we have to use very large matrix multiplication dimensions to compensate, which restricts flexibility in model design. Memory capacity directly determines the size of the models we can build.
• Faster network bandwidth: Today's training and inference are almost entirely distributed. Network bandwidth is crucial at all levels. Particularly for inference latency, when users expect the model to perform long "thought chain" reasoning, we want the single-step inference time to be short enough. This is often limited by the access speed to all model parameters, i.e., memory bandwidth. By distributing parameters across more chips through model parallelism, we can aggregate memory bandwidth, but this requires sufficient interconnect bandwidth between chips.

"In summary, if you can give me more of these standard things—more FLOPS, larger memory capacity, higher memory and network bandwidth—we can really leverage it well," Shazeer emphasized, stating that model architects always ask during design, "What can the chips handle? Which parts are underutilized?" The characteristics of hardware directly shape the forms of software and algorithms.

Additionally, Shazeer proposed two more specific preferences:

Low precision computation: For many applications in deep learning, parameters and activation values are constantly changing, and low precision is often sufficient, providing higher computational throughput with lower power consumption. "If you can give me more low precision FLOPS, that is generally very beneficial."

Determinism: This is crucial for research and debugging. In AI research, many experiments fail due to the ineffectiveness of the algorithm idea itself, but also possibly due to software errors or data issues. If experiments are deterministic, researchers can fine-tune and accurately reproduce the problem, thus efficiently troubleshoot. "Unless you can tell me that sacrificing determinism yields a 10x performance boost, determinism is good."

Audience Q&A: Architecture, Alignment, and Future

During the Q&A session, attendees posed numerous thoughtful questions, and Shazeer provided equally insightful answers.

Regarding whether first token latency is important, he believes it depends on the application scenario. For conversational applications, as long as the generation speed is faster than the user's reading speed, it is acceptable; but if a long text needs to be generated before providing an answer, overall latency is more important than first token latency.

When it comes to exploring next-generation model architectures, an audience member inquired whether more "organic," biologically inspired designs with cyclical connections and other complex structures could enhance "per-parameter intelligence." Shazeer acknowledged that many ideas have been tried, but pointed out that the ratio of parameters to training data (experience) in the human brain far exceeds that of current LLMs, likely due to the limited human lifespan (training time). He prefers not to overly restrict parameters, but rather to pursue improvements in "per-computation intelligence."

Concerning the emphasis on computational growth, Shazeer predicted that future computational investment in post-training optimizations (such as human feedback reinforcement learning RLHF) and inference might be comparable to pre-training, as these are effective means of enhancing a model's ultimate capabilities.

Regarding concerns of data exhaustion, Shazeer's attitude is relatively optimistic. He believes that even with sufficient total data, continuous improvement in data quality is always beneficial. This subtly contrasts with his optimistic statements in 2018 about "infinite data," reflecting the industry's deepening understanding.

The challenge of hardware-software co-design was mentioned multiple times. Shazeer noted that this often presents a "chicken or egg" dilemma: if current hardware does not support a certain function (such as low precision), software will not utilize it, leading to the next generation of hardware not including that feature. Breaking the cycle requires forward-thinking risks from hardware teams and simulations of future hardware by software teams.

Regarding AGI and alignment, when asked if AGI could be achieved solely with existing architectures if hardware development halts in 2025, Shazeer provided a thought-provoking answer: "It's possible. I think what will truly accelerate this is self-acceleration—we can use AI to help us build better software and hardware." As for the alignment issue, he candidly admitted he is uncertain whether hardware architectures can contribute to alignment research in "emerging" ways, reflecting his honest attitude toward unknown fields.

Finally, addressing concerns about models potentially mechanically regurgitating training data and the quality degradation that may result from training using model-generated data, Shazeer stated that this requires subsequent data filtering and quality control to resolve. Additionally, an audience member revisited the memory and forgetting capabilities of RNNs with "neural hidden states," asking whether future models would return to similar mechanisms instead of solely relying on continuously growing external context. Shazeer expressed keen interest, considering it a good idea and humorously extended a work invitation.

The entire speech and Q&A ended in a light-hearted, candid atmosphere full of exploration, leaving a profound impression of Shazeer's humility and curiosity, and pointing to a promising direction for future collaboration between the AI hardware and software communities.

免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。