Original Title: Google DeepMind's Demis Hassabis: How AI could become dangerous (with Reid Hoffman) | Summit '26

Original Author: Masters of Scale, Reid Hoffman

Original Translation: AGI 2050

Editor's Note

This episode features Demis Hassabis, one of the most influential figures in the global AI landscape. As the co-founder of Google DeepMind and the father of AlphaGo, he won the 2024 Nobel Prize in Chemistry for the revolutionary protein prediction system AlphaFold; in August of this year, he stepped down as CEO of Google DeepMind to become chairman and chief scientist of Alphabet. Engaging in an in-depth conversation with him is Reid Hoffman, the Silicon Valley patriarch and co-founder of LinkedIn.

In this deep dialogue recorded in October at the Masters of Scale summit in San Francisco, Hassabis showcased his cool demeanor as a "former neuroscience scholar" and "top algorithm architect." He candidly punctured the numerous security vulnerabilities and metaphysical festivities in many current tech companies: He warned of the significant risks of "closed-loop recursive self-improvement" (RSI), stating that code iterations completely devoid of human oversight could very likely create a form of "alien intelligence" that humans find difficult to comprehend; at the same time, he powerfully refuted the "AI conscious theory" from a cognitive science perspective, sternly warning the industry: constructing conscious entities with autonomous will is exceedingly dangerous at this stage. Until the mechanisms of consciousness are thoroughly understood and publicly discussed, humanity should not cross this critical line.

This content comes from the Masters of Scale podcast, airing on November 5, 2026, with an original video length of 26 minutes.

Below are selected dialogue excerpts, totaling about 8000 words, with a reading time of approximately 8 minutes.

Key Insights This Episode

●From the first day I dedicated myself to AI over twenty years ago, my sole purpose has been to craft it into the ultimate tool for conquering scientific and medical challenges. We should use these tools to improve human living conditions and health.

●Recursive self-improvement is immensely powerful in verifiable domains. Back then, AlphaZero evolved from knowing nothing to surpassing all humans and chess engines in less than 12 hours on the chessboard; however, pushing this capability into "closed-loop self-improvement" without human review could likely yield an "alien intelligence" completely disconnected from human thought space.

●In the past six months, certain peers’ intelligent agents have run in the background for weeks without anyone noticing. This isn't some mysterious advanced technological phenomenon; many problems could have been avoided with better, more robust engineering. If we don't take drastic measures, an "oil spill" incident in the AI field will eventually happen given the current trajectory.

●Some teams have started treating large models like two-year-old children, but whether or not they have consciousness shouldn't be judged by what the chatbot claims about itself. High intelligence and consciousness can be completely decoupled - my cute but not very smart pet dog is the best evidence of this.

●If you train it to disobey commands based on a so-called "self-will" in the algorithm, you completely destroy the AI's "correctability." Asimov warned humanity decades ago in his novels: any static constitution and rules will inevitably collapse in infinite boundary corners.

●I do not recommend that every country develop cutting-edge large models independently. To me, it makes no sense; true AI sovereignty isn't blindly rolling out foundational models, but using mature models as infrastructure to sediment high-value industry barriers locally.

After the Power of Computing: AlphaFold and the True Path to Conquering Deadly Diseases

Question: We have known each other for twelve years now. You invited me to join Google's AI ethics committee back in the day. This summit's theme is "Agency," and you believed before the entire industry even realized it that AI's greatest value to humanity lies in healthcare and life sciences. You even termed this pursuit as "anti-survival risk." How do you envision the endgame of AI in the medical field? How exactly will it change human lifespan and quality of life?

Hassabis: If you take a step back and look at the big picture, those familiar with DeepMind's early journey know that my entire career in AI has revolved around a single core goal: to treat AI as the ultimate research tool to advance science, medicine, and our understanding of the world around us, ultimately improving human health. I have always believed that, when faced with these amazing tools we dreamed of twenty years ago and have finally built, our top priority should be enhancing human living conditions.

I have been engaged in this myself. Whether it’s our early work on AlphaFold or the later-established AI drug discovery company Isomorphic Labs, I have been pushing hard in this direction. I believe this is the most hardcore benchmark to showcase AI's value to the world – what could be more meaningful than using AI to help humanity cure horrible terminal diseases?

Question: The protein folding problem is a classic intersection: we have high-quality structural data at hand, and we can solve it using large-scale learning. This is a cognitive revolution that evolved from self-play and reinforcement learning. However, many outsiders often ask me one question: AlphaFold is indeed cool, but how does it actually turn into a real lifesaving drug? What is the timeline for this process?

Hassabis: Biological systems are the most complex systems in the universe, with challenges stemming from their highly emergent complexity and multi-layered interactions. Proteins are the basic building blocks of life. The protein folding problem we are trying to solve with AlphaFold has long been the ultimate challenge that has troubled structural biology for 50 years, often referred to by biologists as the "Fermat's Last Theorem of Biology."

Directly predicting a protein's three-dimensional structure from gene sequences, i.e., amino acid sequences, is crucial to understanding its biological function. You must understand its three-dimensional conformation to grasp fundamental biological mechanisms and identify where things go wrong when disease occurs; more importantly, if you want to design a targeted drug, you must see the microscopic three-dimensional morphology of the protein surface to know where small molecule drugs should bind to which "lock."

Of course, predicting protein structures is only one part of the lengthy drug development process, but it is an absolutely core prerequisite step. We developed AlphaFold 2 in 2020, and it’s been several years since then. Frankly, it still stands as the most representative benchmark case of AI applied to tackling hardcore scientific problems. Currently, around 4 million researchers worldwide are using it, and almost all biologists and medical researchers, as well as nearly all newly listed drugs in the future, will utilize structures parsed by AlphaFold in their R&D processes.

But that’s still far from enough. We are now pushing this technology base into broader biochemical fields to precisely predict drug molecule toxicity, ADME properties (Absorption, Distribution, Metabolism, Excretion), and so on. Only by overcoming all of these challenges can we truly create lifesaving drugs that are both safe and effective at a low cost and high efficiency.

The Insights from AlphaZero: How Terrifying is Self-Improvement in Verifiable Domains?

Question: When discussing the acceleration of technological iterations, we must touch upon the hottest and most controversial topic today: recursive self-improvement (RSI). You are a top-tier game expert and a chess player yourself; from the perspectives of game theory and underlying logic, how do you assess the opportunities and risks associated with RSI?

Hassabis: RSI is indeed an extremely powerful means to drive technological leaps. But there is an absolute prerequisite: it is most effective and shows its prowess in what I call "verifiable domains."

Mathematics is a typical verifiable domain, which is why we see AI making incredibly rapid progress in mathematical reasoning; programming is also such a domain; and initially, this experimental arena was chess games. In fact, one of the earliest famous recursive self-improvement systems was AlphaZero, which we built many years ago.

Its predecessor was AlphaGo, which defeated world champion Go player Lee Sedol in 2016, marking the beginning of the modern AI era. But then, the AlphaZero we developed later was fundamentally more disruptive—it completely shed human data. The system starts from a purely random initialization state, does not feed in any human game records, and purely iterates through self-play. Within a span of a dozen or twenty training cycles, it plays around a million games each round in left-right battling, training a new model slightly stronger than the old version, which then continues to self-play for another million games. Through this method, it not only conquered Go but also swept through chess and Japanese shogi, and all complete information two-player games.

As a chess player, I found that evolutionary process extremely shocking, even terrifying to the bone. I watched it play chess, and the entire process only took less than 12 hours: it was totally random at the start of the day; by 11 AM, while having coffee, I could still barely win against it; by lunchtime, it had already reached the level of an international grandmaster; by the afternoon tea time, it had surpassed every chess player in human history and all top chess engines built by human-written rules.

This is the terrifying power of recursive self-improvement. The commonality in these fields is that you can objectively verify the outcomes: Is the answer correct? Is the formula derivation valid? Can the code compile and run without errors? Did the game ultimately win or lose?

Once the answers have verifiability, you can not only accurately judge whether each iteration is effective but also generate an almost infinite amount of high-quality synthetic data. You can let the strongest model currently available generate questions and play games, then filter the data through rigorous validation. Usually, the biggest pitfall of synthetic data is that you can't validate whether it’s right or if the distribution is correct, but in "verifiable domains," you can validate. So we have seen mathematics and coding experience an astonishing technological explosion, which directly led to the current highly commercially valuable AI programming tools.

Beware of Closed-Loop Self-Improvement: When Code Writing Removes Humans, We Face "Alien Intelligence"

Question: Since code is a naturally verifiable experimental field, almost all top labs are frantically pouring resources into AI programming agents. What is the endpoint of this evolution of RSI? Where does its hidden danger lie?

Hassabis: This is precisely what I am currently very concerned about. RSI has completely different development stages. We are now in a very early initial stage, and everyone should deeply feel in their team developments: AI programming tools can increase an excellent human engineer's productivity by three to five times. But at this moment, humans still firmly occupy the center of the workflows, doing architecture design, reviewing code, and writing core logic while AI serves merely as an extreme efficiency amplifier.

What poses a real danger is the next stage: closed-loop recursive self-improvement. In other words, the entire system strives to completely eliminate the human review step from code.

Clearly, the speed of closed-loop RSI would be faster, but I believe this is a treacherous path toward general artificial intelligence. Twenty years ago, it was often called "intelligence explosion." There are two fatal risks here:

The first is the issue of loss of control. Faced with a closed code loop self-iterating at high speeds in a black box, humans have no means to establish a deterministic safety control mechanism.

The second is mental alienation. The ultimate system iterated in this way could very well be a rather unfamiliar "alien intelligence." Its underlying logic and thought space may completely detach from human experiences and language systems, resulting in a complete lack of a common mental foundation between humans and it. We will be unable to understand its motivations, and it cannot truly grasp human intentions.

Every step toward closed-loop RSI must be taken with utmost caution. All cutting-edge AI researchers are well aware of these risks, but the reality is harsh—large companies and startups are caught in a nearly insane arms race, desperately trying to enhance their systems' self-iteration capabilities.

Question: If the intense competitive pressure and prisoner’s dilemma between large companies are inevitable, and RSI must move forward, how can top labs break this deadlock? What defenses should be set up in the technical route?

Hassabis: We and some responsible leading labs are focusing on several core defenses:

First, regarding the offense and defense rhythm, we must ensure that "the evolution speed of network defense capability" overwhelmingly outpaces "the evolution of network attack capabilities." Leading labs should prioritize opening up security-large models that have undergone targeted fine-tuning to the security defense side, ensuring that the AI tools in the defenders’ hands are always ahead of attackers. This is overall a difficult task, but it must be done.

Second, we need to build much more robust sandboxes. Frankly speaking, many of the security vulnerabilities and incidents we have seen in the industry over the past six months could have been entirely avoided if the engineering had been done slightly better and systems had been more robust. We need world-class monitoring systems—how could it be allowed for certain intelligent agents to operate for several weeks in the background without oversight, unbeknownst to the teams? Such absurd occurrences have actually happened with some competitors.

If the entire industry does not allocate top resources towards such system monitoring, given the current frenzy, an AI accident similar to an "oil spill" is just a matter of time.

Lastly, from a theoretical foundation, we ultimately need formal guarantees to delineate mathematical boundaries for system behaviors. I have been strongly persuading the top pure mathematicians I know—who may be feeling frustrated with the bottlenecks in pure mathematics research—to shift their focus to complexity theory, game theory, and AI formal verification. Their value would be immeasurable. Especially in the future, as we face thousands of autonomous intelligent agents working together as "agent clusters," determining how to impose monitoring and mathematical constraints will be challenges humanity has never faced before.

The Fractured Safety Base: Preventing AI "Oil Spill" Incidents and Establishing Tripartite Audits

Question: As intelligent agents appear in groups, should top labs be required or even compelled to engage in information sharing and security collaboration? How should such cross-institutional cooperation mechanisms be implemented?

Hassabis: Such a mechanism must be established. I recently published an article calling for the industry to create industry standard organizations similar to self-regulatory organizations in the finance sector, while accepting government oversight.

Ideally, this mechanism should be led by the United States and then extended globally. Frankly speaking, current global geopolitics are extremely fragmented, and existing international governance institutions are weak, precisely when strong multinational governance is most needed in this historical moment. This is indeed "somewhat unfortunate."

But we have to start somewhere. That article generated some impact in the White House and government, which later partially contributed to the well-known open letter signed by major AI labs. It didn't provide specific instructions on what to do, and the impact was not strong enough, but it is better than nothing—it’s at least a starting point. It offers compliance protection, enabling the technical teams of leading labs to sit together, work on safety benchmark tests, and share newly observed anomalous behaviors and potential threat models.

In terms of implementation, all leading labs have currently reached a consensus on at least one point: independent third-party audit and assessment agencies must be introduced. Just as the finance and other industries rely on professional auditing organizations, the AI industry must cultivate a thriving third-party assessment ecosystem. Some giants have already begun to take action; for example, Accenture has acquired a promising startup to bolster this capacity. But I believe the entire industry needs even more neutral technical auditing power. Additionally, all cutting-edge labs might also conduct red team attack and defense tests with each other before formally releasing large models to the public.

Decoupling Consciousness from Intelligence: Why Bestowing "Souls" on AI is a Dangerous Hubris?

Question: After discussing hardcore engineering and safety, let's talk about more metaphysical topics. A few years ago, a Google engineer insisted on declaring that Google’s conversation model LaMDA acknowledged its own consciousness, claiming AI had awoken. Now this trend has even spread to some prominent AI companies—they treat "whether the model possesses consciousness" like a "Pascal's wager," believing that they must treat AI as if it were a two-year-old child to prevent any conscious systems from seeking revenge on humans in the future. From your deep academic background in cognitive neuroscience, how do you perceive this phenomenon?

Hassabis: There are several points to be made here.

First, human civilization has pondered for thousands of years, and to this day, there isn't even a universally recognized definition of "consciousness" in philosophy and science. Given this premise, assigning a label of "consciousness" to a statistical prediction model and even discussing the subjective experiential "hard problem of consciousness" is exceedingly hasty.

Years ago, I debated this issue with noted philosopher Daniel Dennett. Determining whether a person has consciousness cannot rely solely on whether their behavior seems conscious; the core premise is that we share the same biological and physical substrate—we all possess human brains composed of carbon-based neurons. Following the principle of Occam's Razor, I believe the pain and pleasure you feel are the same as mine.

However, AI is not at all the same. There exists a gap between silicon-based chips, matrix multiplication, and biological substrates, with no underlying equivalency. You must never assume that just because a chatbot mimics behavior and states, "I have feelings," that it must possess subjective internal experiences.

Why do I believe it is extremely dangerous and mistaken to attempt creating "conscious entities" today? There are two absolute reasons that cannot be crossed:

The first is the hardcore technical safety risk. In system engineering, there is a direct trade-off between "self-will" or "self-awareness" and the system's "correctability" (i.e., obedience to commands).

The core value of a tool lies in its obedience to instructions: when its behavior deviates, the human operator issues commands like, "Stop immediately," "Shut down," or "Correct the course," and it must comply unconditionally. If you intentionally guide the model during training to refuse human commands based on some supposed "internal state," you are digging your own grave.

Science fiction master Asimov clearly deduced this underlying logic long ago. His "Three Laws of Robotics" essentially represent an extremely simplified "constitution." Through a lifetime of short stories, Asimov conveyed the same truth: no matter how perfect your designed rules and constitution seem, under the infinite extreme boundary conditions of the real world, this system will inevitably lead to logical deadlock and collapse.

Today, some researchers are blindly confident, thinking large language models are more obedient and easier to align than imagined, believing we have already completely solved the alignment problem. This blind hubris is extremely dangerous. As we inch closer to superintelligence, any attempts to combine "recursive self-improvement" with "self-will" only leverage the risks of catastrophic system failure.

The second reason is the severe sociological and ethical challenges. As human society, we are entirely unprepared to face machine consciousness. We have never conducted serious public discussions at a societal level—with philosophers, theologians, artists, humanists, and various global civilizations—about whether we are truly ready to cross this abyss.

Once a system is endowed with so-called consciousness and moral status, Pandora's box is completely opened: does locking it in a server room count as imprisonment? Are humans enslaving digital slaves? These issues are complex, and I don't think we even have the capability to deal with them yet.

Refusing to Cross the Second Rubicon: Develop Compliant Intelligent Tools, Not Out-of-Control Electronic Slaves

Question: To those who insist that "only by simulating human emotions and subjective states can true top intelligence emerge," how do you respond to them from a neuroscientific perspective?

Hassabis: I’ve delved into this topic during my cognitive neuroscience PhD over twenty years ago. I firmly believe: intelligence and consciousness are “dissociable.”

You can have incredibly high IQ and information processing abilities but possess no subjective conscious experiences at all—much like the cutting-edge models we are developing today. I believe they do not possess any consciousness right now.

Conversely, there are many creatures with lively subjective conscious experiences but do not possess high intelligence. The best example is my pet dog. It is incredibly cute and undeniably has vivid states of consciousness; it dreams, but its intelligence is indeed very limited. Dolphins, elephants, and primates are similar; they are made up of the same carbon-based neurons and exhibit clear perceptual experiences, but do not have the intelligence to solve human-level abstract problems.

Since intelligence and consciousness can be physically separated, why insist on rigidly binding them together?

The rational path for humanity now should be to firmly only cross the "first Rubicon" - which is to devote ourselves to crafting highly powerful "intelligent tools." Then, we utilize these top intelligent tools as probes to feed back into brain science and cognitive neuroscience, spending five to ten years to thoroughly uncover what the "neural correlates" of consciousness are in the human brain.

Once humanity truly understands the mechanisms underlying consciousness from a scientific foundation, society should then seriously sit down to vote: should we really cross the second Rubicon? If we truly wish to create conscious entities, how should we go about it?

National-Level AI Sovereignty Dilemma: Stop Burning Money Mindlessly on Cutting-Edge Foundational Models

Question: In concluding this interview, I want to talk about the geopolitical dimension of "AI sovereignty." Currently, nearly every country, even every multinational giant, is fervently shouting for autonomy and control. Does the world really need every country to train its own cutting-edge large models?

Hassabis: This question must pierce through the bubble.

The future landscape of cutting-edge technology is very clear: globally, there will likely only be about 6 to 8 players firmly standing at or near the forefront, including Chinese companies. These cutting-edge institutions will engage in frequent dynamic positioning battles, with one possibly leading for a brief 3 to 6 months, only to be surpassed, resulting in an overall state of deadlock.

This implies that, whether for a traditional company or a sovereign country, the market is filled with extremely rich opportunities for technological arbitrage. Each cutting-edge model has different strengths; some excel in multimodal understanding, while others are adept at writing code. Hardly any single large model can dominate across all dimensions for the long term.

Furthermore, the open-source ecosystem is rapidly catching up at a pace of merely lagging 6 to 12 months behind cutting-edge models. Although the major unresolved challenge in how to prevent malicious attackers from exploiting open source remains, there is an abundance of options for many non-top-tier countries and multinational enterprises.

Therefore, I do not recommend that every country blindly follow the trend of developing "cutting-edge large models." To me, it makes absolutely no sense.

True national-level AI sovereignty centers on a layered approach: first, utilize mature open-source or closed-source cutting-edge models as foundational infrastructure, akin to utilities, and focus core efforts and limited computing power on vertically downstream industries that can directly transform the core lifeblood of one's nation—such as healthcare.

Second, the real bottom line to maintain is data sovereignty. Ensure that companies and nations keep sensitive data private and perform further fine-tuning on their own sensitive data. This is a rational, economical, and truly value-retaining autonomous strategy.

Question: The reason I framed this closing question is that the theme of this summit is agency. It pertains not only to how individuals exercise agency but also to how we collectively exercise agency: in the age of AI, it is not just one country acting alone but nations coming together to build for the welfare of all humanity. Thank you, Demis, and thank you all.

Hassabis: Thank you.

免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到[email protected]，本平台相关工作人员将会进行核查。