Original Author: Vitalik Buterin
Original Translation: Luffy, Foresight News
In April of this year, Daniel Kokotajlo, Scott Alexander, and others released a report titled “AI 2027”, depicting “our best guess about the impact of superhuman AI over the next five years.” They predict that by 2027, superhuman AI will be born, and the future of all human civilization will depend on the development outcomes of AI: by 2030, we will either usher in a utopia (from the American perspective) or head towards total destruction (from the perspective of all humanity).
In the following months, a plethora of differing opinions emerged regarding the likelihood of this scenario. In critical responses, most focused on the issue of “the timeline being too fast”: Will AI development really accelerate as Kokotajlo and others suggest, or even intensify? This debate has persisted in the AI field for years, with many expressing skepticism about the rapid arrival of superhuman AI. In recent years, the time it takes for AI to autonomously complete tasks has roughly doubled every seven months. If this trend continues, it will not be until the mid-2030s that AI can autonomously perform tasks equivalent to an entire human career. While this progress is indeed rapid, it is far later than 2027.
Those who hold a longer timeline perspective tend to believe that there is a fundamental difference between “interpolation/pattern matching” (the work currently done by large language models) and “extrapolation/true original thinking” (which only humans can currently achieve). Automating the latter may require technologies that we have not yet mastered or even begun to explore. Perhaps we are merely repeating the mistakes made during the widespread adoption of calculators: mistakenly believing that since we have quickly automated a certain type of important cognition, everything else will soon follow suit.
This article will not directly engage in the timeline debate, nor will it address the (very important) discussion of “whether super AI is inherently dangerous.” However, it should be noted that I personally believe the timeline will be longer than 2027, and the longer the timeline, the more persuasive the arguments I present in this article will be. Overall, this article will offer a critique from a different angle:
The scenario in “AI 2027” implies an assumption: that the capabilities of leading AI (the “Agent-5” and subsequent “Consensus-1”) will rapidly enhance to possess god-like economic and destructive power, while the (economic and defensive) capabilities of everyone else will essentially stagnate. This contradicts the scenario’s assertion that “even in a pessimistic world, we can expect to cure cancer, slow aging, and even achieve consciousness uploading by 2029.”
Some of the countermeasures I will describe in this article may seem technically feasible to readers, but impractical to deploy in the real world in the short term. In most cases, I agree with this. However, the “AI 2027” scenario is not based on the current reality but assumes that within four years (or any timeline that could lead to destruction), technology will develop to grant humanity capabilities far beyond the current state. Therefore, let’s explore what would happen if both sides possessed AI superpowers, rather than just one.
The biological apocalypse is far from the scenario described
Let’s zoom in on the “race” scenario (where everyone dies due to the U.S. being overly fixated on defeating China while neglecting human safety). Here is the plot of everyone’s death:
“For about three months, Consensus-1 expands around humanity, transforming grasslands and ice fields into factories and solar panels. Eventually, it deems the remaining humans too much of a nuisance: by mid-2030, the AI releases over a dozen quietly spreading biological weapons in major cities, allowing them to silently infect nearly everyone, and then triggers lethal effects with chemical sprays. Most die within hours; a few survivors (such as doomsday responders in bunkers and sailors on submarines) are eliminated by drones. Robots scan the victims' brains, storing copies in memory for future research or revival.”
Let’s analyze this scenario. Even now, there are some developing technologies that could make AI’s “clean and swift victory” less realistic:
Air filtration, ventilation systems, and ultraviolet lights can significantly reduce the transmission rate of airborne diseases;
Two types of real-time passive detection technologies: detecting human infections and issuing notifications within hours, and quickly detecting unknown new viral sequences in the environment;
Various methods to enhance and activate the immune system, more effective, safe, and universal than COVID vaccines, and easy to produce locally, enabling the human body to resist both natural and artificially designed epidemics. Humans evolved in an environment with a global population of only 8 million, spending most of their time outdoors, so intuitively, we should be able to adapt easily to today’s more threatening world.
These methods combined could potentially reduce the basic reproduction number (R0) of airborne diseases by 10-20 times (for example: better air filtration reduces transmission by 4 times, immediate isolation of infected individuals reduces it by 3 times, and simple enhancement of respiratory immunity reduces it by 1.5 times), or even more. This would be sufficient to prevent the transmission of all existing airborne diseases (including measles), and this number is far from the theoretical optimum.
If real-time viral sequencing for early detection could be widely applied, the idea that “quietly spreading biological weapons could infect the global population without triggering alarms” becomes highly questionable. Notably, even advanced methods such as “releasing multiple epidemics and only triggering dangerous chemicals when combined” can still be detected.
Let’s not forget that we are discussing the assumptions of “AI 2027”: by 2030, nanobots and Dyson spheres are listed as “emerging technologies.” This means efficiency will significantly increase, making the widespread deployment of the aforementioned countermeasures more promising. Although in 2025, humanity is slow to act and lethargic, with many government services still relying on paper-based operations. If the world’s most powerful AI can transform forests and fields into factories and solar farms before 2030, then the world’s second most powerful AI can also install numerous sensors, lights, and filters in our buildings before 2030.
But let’s further adopt the assumptions of “AI 2027” and enter a purely sci-fi scenario:
Microscopic air filters in the body (nose, mouth, lungs);
An automated process from discovering new pathogens to fine-tuning the immune system to resist them, ready for immediate application;
If “consciousness uploading” is feasible, simply replace the entire body with a Tesla Optimus or Unitree robot;
Various new manufacturing technologies (which are likely to be super-optimized in a robotic economy) will enable local production of far more protective equipment than currently, without relying on global supply chains.
In a world where cancer and aging issues will be cured by January 2029, and technological progress continues to accelerate, by mid-2030, it would be hard to believe that we do not have wearable devices capable of real-time bioprinting and injecting substances to protect the human body from any infection (and toxins).
The aforementioned biological defense arguments do not cover “mirror life” and “mosquito-sized killer drones” (which the “AI 2027” scenario predicts will appear starting in 2029). However, these means cannot achieve the sudden “clean and swift victory” described in “AI 2027,” and intuitively, symmetric defenses against them would be much easier.
Therefore, biological weapons are actually unlikely to completely destroy humanity in the manner described in the “AI 2027” scenario. Of course, all the outcomes I have described are far from a “clean and swift victory” for humanity. Whatever we do (perhaps excluding “uploading consciousness to robots”), a full-scale AI biological war will still be extremely dangerous. However, achieving the standard of a “clean and swift victory” for humanity is not necessary: as long as the attack has a high probability of partial failure, it is sufficient to form a strong deterrent against the AI that has already established a dominant position in the world, preventing it from attempting any attacks. Of course, the longer the timeline for AI development, the more likely these defensive measures will be able to fully play their role.
What about combining biological weapons with other attack methods?
For the above countermeasures to succeed, three prerequisites must be met:
The physical security of the world (including biological and anti-drone security) is managed by local authorities (human or AI), and not entirely by puppets of Consensus-1 (the name of the AI that ultimately controls the world and destroys humanity in the “AI 2027” scenario);
Consensus-1 cannot invade the defense systems of other countries (or cities, other secure areas) and immediately render them ineffective;
Consensus-1 has not controlled the global information domain to the extent that no one is willing to attempt self-defense.
Intuitively, the outcome of prerequisite (1) could lead to two extremes. Today, some police forces are highly centralized, with powerful national command systems, while others are localized. If physical security must rapidly transform to meet the demands of the AI era, the landscape will be completely reset, and the new outcomes will depend on the choices made in the coming years. Governments may become lazy and rely on Palantir; or they may actively choose a combination of locally developed and open-source technology solutions. Here, I believe we need to make the right choice.
Many pessimistic narratives on these topics assume that (2) and (3) are beyond remedy. Therefore, let’s analyze these two points in detail.
The doomsday of cybersecurity is far from here
The public and professionals generally believe that true cybersecurity is impossible to achieve; at best, we can quickly patch vulnerabilities once discovered and deter cyber attackers by hoarding known vulnerabilities. Perhaps the best scenario we can hope for is a “Battlestar Galactica” situation: almost all human ships are simultaneously paralyzed by Cylon cyberattacks, with the remaining ships surviving because they did not use any connected technology. I do not agree with this view. On the contrary, I believe that the “endgame” of cybersecurity is favorable to the defenders, and under the rapid technological development assumed in “AI 2027,” we can achieve this endgame.
One way to understand this is to use a technique favored by AI researchers: trend extrapolation. Below is a trend line based on deep research surveys of GPT, assuming the use of top security technologies, with the vulnerability rate per thousand lines of code changing over time as follows.
In addition, we have seen significant progress in sandbox technologies and other techniques for isolating and minimizing trusted codebases in both development and consumer adoption. In the short term, attackers' exclusive superintelligent vulnerability discovery tools can find numerous vulnerabilities. However, if highly intelligent agents for discovering vulnerabilities or formally verifying code are publicly available, the natural ultimate balance will be: software developers will discover all vulnerabilities through continuous integration processes before releasing code.
I can see two compelling reasons why vulnerabilities cannot be completely eradicated even in this world:
Defects stem from the complexity of human intent itself, so the main difficulty lies in constructing sufficiently accurate models of intent, rather than the code itself;
For non-safety-critical components, we may continue existing trends in consumer technology: writing more code to handle more tasks (or reducing development budgets), rather than completing the same number of tasks with ever-increasing security standards.
However, these categories do not apply to situations like “can attackers gain root access to the systems that sustain our lives,” which is precisely the core of our discussion.
I acknowledge that my perspective is more optimistic than the mainstream views held by the smart people in the current cybersecurity field. But even if you disagree with my viewpoint in the context of today’s world, it is worth remembering: the “AI 2027” scenario assumes the existence of superintelligence. At the very least, if “100 million superintelligent copies think at 2400 times human speed” cannot yield code without such defects, then we should absolutely reassess whether superintelligence is as powerful as the authors imagine.
To some extent, we not only need to significantly raise software security standards but also enhance hardware security standards. IRIS is a current effort to improve hardware verifiability. We can use IRIS as a starting point or create better technologies. In fact, this may involve a “correct by construction” approach: the hardware manufacturing processes for critical components are deliberately designed with specific verification steps. These are all tasks that AI automation will greatly simplify.
The doomsday of super persuasion is also far from here
As mentioned earlier, another situation where defensive capabilities may still be futile is if AI persuades enough people to believe that there is no need to defend against the threat of superintelligent AI, and that anyone attempting to seek defensive measures for themselves or their community is a criminal.
I have always believed that two things can enhance our ability to resist super persuasion:
A less monolithic information ecosystem. It can be said that we are gradually entering a post-Twitter era, with the internet becoming more fragmented. This is a good thing (even if the fragmentation process is chaotic), as we overall need more information multipolarity.
Defensive AI. Individuals need to be equipped with locally running AI that is explicitly loyal to them, to balance the dark patterns and threats they see online. Such ideas have already seen sporadic pilots (like Taiwan's “Message Checker” app, which performs local scans on mobile phones), and there is a natural market to further test these ideas (such as protecting people from scams), but more effort is needed in this area.
From top to bottom: URL checking, cryptocurrency address checking, rumor checking. Such applications can become more personalized, user-driven, and powerful.
This contest should not be a battle between the super persuader of superintelligence and you, but rather a contest between the super persuader of superintelligence and you plus a slightly weaker but still superintelligent analyzer that serves you.
This is how it should happen. But will it really happen? In the short time frame assumed by the “AI 2027” scenario, achieving the widespread adoption of information defense technologies is a very challenging goal. However, it can be argued that more moderate milestones would suffice. If collective decision-making is most critical, and as shown in the “AI 2027” scenario, all important events occur within an election cycle, then it is crucial to ensure that direct decision-makers (politicians, civil servants, programmers from certain companies, and other participants) can use good information defense technologies. This is relatively easier to achieve in the short term, and based on my experience, many of these individuals are already accustomed to interacting with multiple AIs to assist in decision-making.
Insights
In the world of “AI 2027,” it is taken for granted that superintelligent AI will easily and swiftly eliminate the remaining humans, and thus the only thing we can do is to ensure that the leading AI is benevolent. In my view, the reality is much more complex: whether leading AI is powerful enough to easily eliminate the remaining humans (and other AIs) is still highly contentious, and we can take actions to influence this outcome.
If these arguments are correct, their implications for today’s policies sometimes align with “mainstream AI safety guidelines” and sometimes differ:
Delaying the development of superintelligent AI is still a good thing. Superintelligent AI appearing in 10 years is safer than appearing in 3 years, and appearing in 30 years is even safer. Giving human civilization more preparation time is beneficial.
How to achieve this is a conundrum. I believe that the rejection of the U.S. proposal for a “10-year moratorium on state-level AI regulation” is overall a good thing, but especially after the failure of early proposals like SB-1047, the next steps have become less clear. I think the least invasive and most robust way to delay the development of high-risk AI may involve some treaty regulating the most advanced hardware. Many of the hardware cybersecurity technologies needed for effective defense also help verify international hardware treaties, so there may even be synergies here.
Nevertheless, it is worth noting that I believe the main source of risk comes from military-related actors who will fiercely seek exemptions from such treaties; this must not be allowed. If they ultimately gain exemptions, then AI development driven solely by the military could increase risks.
Coordinating efforts to make AI more likely to do good and less likely to do harm is still beneficial. The main exception (and has always been) is that coordination efforts ultimately evolve into enhancing capabilities.
Regulating to increase the transparency of AI laboratories is still beneficial. Incentivizing AI laboratories to regulate their behavior can reduce risks, and transparency is a good way to achieve this goal.
The mindset of “open source is harmful” has become riskier. Many oppose open-weight AI on the grounds that defense is unrealistic, and the only bright prospect is for good people with good AI to achieve superintelligence before any less benevolent individuals, gaining any highly dangerous capabilities. However, the arguments in this article depict a different picture: defense is unrealistic precisely because one actor is far ahead while others have not kept up. The diffusion of technology to maintain a balance of power becomes important. But at the same time, I would never think that merely because it is done in an open-source manner, accelerating the growth of cutting-edge AI capabilities is a good thing.
The mindset of “we must defeat China” in American laboratories has become riskier for similar reasons. If hegemony is not a security buffer but a source of risk, then this further refutes the (unfortunately too common) notion that “well-meaning people should join leading AI laboratories to help them win faster.”
Initiatives like “public AI” should be supported to ensure a broad distribution of AI capabilities and to ensure that infrastructure actors indeed have the tools to quickly apply new AI capabilities in some of the ways described in this article.
Defense technologies should reflect the idea of “armed sheep” more than the idea of “hunting all wolves.” Discussions about the fragile world hypothesis often assume that the only solution is for hegemonic states to maintain global surveillance to prevent any potential threats from arising. But in a non-hegemonic world, this is not a viable approach, and top-down defense mechanisms can easily be subverted by powerful AIs, turning into tools of attack. Therefore, greater defensive responsibilities need to be achieved through hard work to reduce the world’s fragility.
The above arguments are speculative and should not lead to actions based on these arguments' almost certain assumptions. However, the story of “AI 2027” is also speculative, and we should avoid taking action based on the assumption that “its specific details are nearly certain.”
I am particularly concerned about a common assumption: that establishing an AI hegemony to ensure its “alliance” and “win the race” is the only way forward. In my view, this strategy is likely to reduce our security—especially in cases where hegemony is deeply tied to military applications, which would significantly undermine the effectiveness of many alliance strategies. Once hegemonic AI deviates, humanity will lose all checks and balances.
In the “AI 2027” scenario, humanity's success depends on the U.S. choosing the path of safety rather than destruction at a critical moment—voluntarily slowing AI progress and ensuring that the internal thought processes of Agent-5 can be interpreted by humans. Even so, success is not guaranteed, and how humanity can escape the cliff of continued survival dependent on a single superintelligent thought remains unclear. Regardless of how AI develops in the next 5-10 years, acknowledging that “reducing the world’s fragility is feasible” and investing more effort to achieve this goal with the latest human technologies is a path worth trying.
Special thanks to the feedback and review from Balvi volunteers.
免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。
