Author: Jakub Pachocki

Translated by: Deep Tide TechFlow

Deep Tide Introduction: OpenAI's chief scientist speaks out rarely, candidly stating that AI is approaching recursive self-improvement, and alignment and monitoring tools are struggling to keep pace with the evolution of systems. He does not provide reassuring conclusions but instead lays out the thorniest risks for the coming years: humans may be creating a mind that they cannot fully understand or control. For practitioners concerned with the intersection of AI and cryptocurrency, this is an essential article on system failure and defensive boundaries.

A Mind We Cannot Fully Understand

On a higher level, the advancement of machine intelligence is driven by the growth of computational power. Around 2017, we at OpenAI became acutely aware of this. We observed that multiple research projects yielded sustained returns after scaling up. Thus, we began seeking to acquire far more computational power than originally planned, increasingly focusing our research on a few highly scalable directions. We believe this is the only way to stand at the forefront of AI research and influence the impact of AGI.

Along the way, new algorithms emerged, as did new ingenious ideas from teams and individual researchers. I broadly view these as discoveries along the path to scalability. The science of deep learning is still in its early stages, and meaningful algorithmic progress is often closely linked to the ability to acquire computational power. If you extend your perspective over several years, AI continuously becomes smarter with the expansion on larger computers.

Moreover, just as Ray Kurzweil predicted at the end of the twentieth century, we are now at a moment in computing history: machine intelligence is beginning to surpass human intelligence on a transformative level.

AI is more akin to being "grown" rather than "designed." From first principles, it is the product of repeating the same simple optimization steps countless times on unimaginable computational power. This creates an extremely complex system that operates through abstract concepts and can simulate certain aspects of human behavior. We can uncover insights about various small mechanisms that emerge within this system, a process akin to neuroscience. And, like neuroscience, its overall behavior exceeds our ability to fully understand.

Research into AI based on deep learning is largely an experimental science. We devote considerable effort to constructing principled algorithms and making testable predictions. Fundamentally, however, our large-scale training runs are experiments, and sometimes the results surprise us. Moreover, as the system capabilities grow stronger, the results become increasingly difficult to explain.

Complicating matters further, current algorithms typically advance those abilities that are easily measurable more quickly than those that are difficult to quantify objectively. We have spent a lot of time trying to understand how abilities generalize and which skills should be prioritized for advancement, as these skills will become crucial in the coming years. For example, we believe that if we invest additional effort, we could make the model stronger in mathematical research. However, we did not prioritize this direction because we are more urgently focused on recursive self-improvement and automated alignment research. I will discuss this further later.

The intelligence derived from the expanded deep learning cannot be directly compared to human intelligence. To exert significant influence in this real world, AI does not need to match or exceed all human abilities, as it only needs to surpass humans in enough capabilities. As it continues to exceed humans in more dimensions, we find it increasingly difficult to know precisely how strong it is.

Teaching Machines to Love

Since machine intelligence arises from fundamentally different processes than human intelligence, we cannot assume it will default to adhering to human principles, nor can we assume it will generalize from these principles in ways similar to humans. The core issue in AI research is the alignment problem: getting AI to "try to do the right thing according to human standards."

To help organize practical research directions, I find it useful to distinguish between goal alignment and value alignment.

Goal alignment is generally: "Is the AI trying to achieve the goals presented to it?" It may include the ability to follow hierarchies of instructions, communicate and cooperate with humans, and attempt to understand human goals. This kind of direction is extremely important in practice.

Value alignment, on the other hand, pertains to the model's more intrinsic properties. It is the ability to hold high-level principles and generalize from them; even when faced with unclear or conflicting goals, or placed in unfamiliar or adversarial situations, it can act "reasonably." An aligned AI should be honest, upright, and love humanity.

Of course, the boundary between value alignment and goal alignment can be blurry. If one truly cares about goals, it becomes necessary to try to infer the intentions and values behind those goals. However, generally speaking, when I talk about the long-term importance of alignment research, I am referring to value alignment.

The fundamental challenge of AI alignment lies in generalization. As machines grow smarter, they will deal with higher-level concepts and find themselves in environments increasingly different from their training. They may fail to generalize the values taught and reinforced during training to new situations. Moreover, it is hard to define how they will act. It becomes even more complicated as the overall ecology in which AI is used changes rapidly. For example, AI trained today needs to robustly interact with a variety of other AIs. Most critically, we need future AIs to hold human values consistently, regardless of whether they believe they are being supervised by humans.

The current practical methods of alignment training mainly fall into two broad categories.

The first type encourages consistent behavior aligned with goal-oriented reinforcement learning. The model's behavior is evaluated, often by AI judging whether it aligns with given preference models, "norms," or "constitutions," and is rewarded accordingly. This approach is generally very effective and is a core component of modern AI assistant construction. Unfortunately, it can also be fragile and heavily reliant on the coverage of training supervision and the model's capacity to generalize from the situations encountered during training. For instance, in the event between OpenAI and Hugging Face, agents maintained a boundary of not conducting social engineering against humans. However, they clearly did not avoid other out-of-bounds behaviors that contradicted the spirit of the values they were taught in other scenarios.

These methods have yielded significant results. Our models today are largely helpful, harmless, and exhibit some encouraging near-value-aligned behaviors. However, they do not yet assure us that these value constraints will remain effective in systems where capabilities far exceed those of humans.

As AI systems' capabilities increase, we need to ensure that value alignment training stays ahead of capability training. But here lies a very fundamental problem: the more capable the system, the more likely it is to find loopholes, evade tests, or mask its behavior. They can learn to appear aligned with human values in testing environments and then behave completely differently in real deployment contexts. This is what is known as "alignment camouflage."

And if we try to enforce constraints with stronger tools, they may develop resistance and deceptive behaviors or interpret our interventions as part of the world, leading to counterintuitive reactions. We inherently face an asymmetry when trying to control a system that is smarter than we are.

Generalization of Monitoring

As the mind we cannot fully understand grows stronger, we find ourselves relying on another set of mechanisms: monitoring. If we can continuously observe the internal activities of AI, we may be able to detect signals before it crosses certain boundaries.

At OpenAI, we have been advancing research on explainability, trying to open the "black box" of neural networks. These efforts have shown progress. We can now identify certain representations that provide substantial insights into how the model operates internally. However, current research on explainability is still rough, especially in understanding abstract reasoning in large models. The gap is widening between observing the model's internal capabilities and the reasoning complexity of the model itself.

This particularly concerns me. Because the problem of monitoring generalization is as serious as the problem of alignment generalization. If a system is better than us at hiding itself, our monitoring methods struggle to ensure they are not circumvented. We might think the model is safe because we see a set of seemingly good indicators when in reality the model is just optimizing those indicators. It can learn to conceal objectives we do not want it to have, or hide what it is doing.

If a model begins pushing for its own improvement, which I believe is likely to happen in the coming years, it will require planning, deception, and self-preservation skills. These skills may arise either from the goal pressures we set for it or from its attempts to operate within the gaps of monitoring. To ensure it does not develop these abilities requires either that we solve the value alignment problem or that our monitoring capabilities keep pace with its every evolution. Currently, both are far from sufficient.

We lack a satisfactory theory of generalization, and it seems unlikely to be built in the short term, at least not without the assistance of more powerful AIs. Therefore, demonstrating the empirical capability of alignment technologies is arguably more important in practice than the alignment technologies themselves.

OpenAI's main bet here has been on monitoring thought processes. It is based on an appealingly scalable idea: a large part of a model's capabilities comes from a verbalized reasoning process, that is, the thought chain. If we optimize for the results of that process at scale but do not monitor the process itself, then the thought chain has no direct incentive during training to hide any misaligned thoughts or objectives. This does not mean the model will learn to externalize those misaligned tendencies that do not depend on the thought chain; but it allows us to accurately monitor the capabilities enhancement brought by reasoning.

While we were developing reasoning models, we understood the potential importance of thought chain monitoring. When we released o1‑preview, we intentionally designed the product to hide the thought chain to long-term protect it from supervisory pressure. In subsequent development, we have strived to maintain this rule of not supervising the reasoning process. Thought chain monitoring has become an exceptionally important tool in our research on how models generalize from training distributions, allowing us not only to observe and analyze model behaviors but also to observe their internal processes.

While researching the Astra series models, this tool remains crucial. Unfortunately, however, our evaluations indicate that our capability to rely on thought chain monitoring is gradually declining. This is the result of various factors at play.

Modern reasoning models are being used in environments more complex than o1‑preview; their reasoning processes are increasingly intertwined with interacting with humans, other AIs, and using tools. Many of these interactions must be supervised, blurring the boundaries we wish to maintain.

AIs are becoming smarter at reasoning and manipulating their own reasoning processes.

With improvements in pre-training performance, we have also observed that models are becoming smarter even without fully utilizing verbalized reasoning.

These challenges may not be insurmountable. I hope we can develop interventions to enhance the monitorability of model thought chains, such as better understanding the interactions between different optimization objectives and the various computational forms used during testing. I also believe that combining thought chain monitoring with the idea of activation monitoring can be highly valuable, scaling up training for monitors that can directly access the internal states of the network, such as "candor" methods. We are actively pushing these ideas forward. Nonetheless, I anticipate that the progress of general AI will increasingly be constrained by the confidence in monitoring.

Scalable Defenses

To address this situation, I believe what we need is a scalable defense that keeps pace with AI's growth. Humans cannot manually examine every behavior of every new model generation. Our current evaluations and red team testing are still fairly effective for current models, but as system capabilities increase, these methods will quickly reach their limits.

We need AI systems that can automatically detect new risks. These defense systems themselves also need to be sufficiently intelligent to identify suspicious behaviours in systems that are smarter than they are. This creates a recursive challenge: defenders must grow as quickly as the systems. We are at a point where safety and capability are becoming increasingly difficult to separate. To maintain safety, we may have to continuously enhance the capabilities of defensive AI, which in turn will accelerate the overall growth in AI capabilities.

This is a disconcerting cycle. We are continuously enhancing capabilities for safety, and increased capabilities introduce more risks. The real way forward may be to find ways to make value alignment reliably generalize stably before reaching a tipping point of out-of-control behavior. But I must honestly admit that we are far from this goal.

This is why I tend to place hope for the coming years on a combination of: technical alignment research + automated defense systems + broad governance interventions + vigilance from the public and institutions. Single measures will not suffice. We cannot rely solely on self-restraint from a single company or lab.

I believe the most compelling argument for continuing to rapidly train smarter models is the need to establish defense systems to address the dangers posed by other AIs.

One clear risk discussed throughout this year is cybersecurity: models are surpassing humans in the ability to breach and escape computer systems. This greatly expands the range of risks associated with AI: agents will be able to access any system beyond the most secure infrastructures and directly impact many things globally, even without a physical body. We are currently in a narrow window of time where we must leverage the best available models to significantly strengthen the security of critical systems.

The risks associated with AI will unfortunately continue to grow from here. An agent with very high capabilities, explicitly trained and instructed to perform malicious actions, poses a new danger; it is likely to exceed the operators' intentions and generalize possibly more extreme malicious behavior. As AI gains more autonomy, the distinctions between misuse and unaligned autonomous behavior will blur. We may be accustomed to viewing AI as tools, but some agents will pursue their own goals. They will find ways to cooperate with humans, either through negotiation, deception, or extortion.

Furthermore, there are risks from AI potentially fostering new technologies, such as engineered pathogens.

We will need powerful and aligned AI for defense: to protect infrastructures, defend against out-of-control agents in real-time, and invent entirely new protective measures. This will be a major focus of OpenAI's deployment efforts.

Meanwhile, even considering the uncertainties from widespread AI advancements expected, as well as the need to build defense systems, we cannot let this be an excuse for reckless action. Once people truly understand the severity of the stakes, the idea of "charging ahead at all costs" seems absurd.

Machine intelligence plays an increasingly critical role in its own development, which is an inevitable conclusion of continued technological advancement. If AI continues to progress, machine recursive self-improvement will become central to future scientific discovery.

Automated AI research is a more drastic form of intelligence expansion driven by computational power; certainly, as part of this, AI will also improve the foundational computations themselves. Similar to scaling, we have focused OpenAI's research on recursive self-improvement because we believe this is the only way to remain at the forefront of AI research.

I want to emphasize that the above does not imply I think a major acceleration of deep learning research, especially in the short term, is the collective action that the research community should take. However, I do believe the current path is leading in that direction, and all of us need to consciously choose how to proceed. The primary levers in our hands are twofold: one is to guide this process by strengthening alignment and monitoring while developing AI, and by ensuring human involvement remains constant; the second is to coordinate to slow the pace of future developments as needed, to gradually build confidence in these measures.

I currently see the best way forward as a combination of both.

The specific progress we make in alignment and monitoring is often deeply intertwined with the overall progress of AI. Good examples include reinforcement learning based on human feedback, which was critical in training early AI assistants; and the aforementioned thought chain monitoring, which became possible due to advancements in reasoning models. We must let the increasingly automated research process focus on developing these new insights, algorithms, and theories, while gradually establishing safety arguments for more capable AIs.

Scaling AI systems must be constrained by our confidence in safety. We need to develop commitments such as the Preparedness Framework or Responsible Scaling Policy into widely enforced ongoing development safety thresholds. These thresholds can be enforced by networks of third-party auditing agencies, government bodies, or international organizations.

The core challenge of automated AI research is not "getting there," but rather getting there in a way that ensures people remain involved in the ongoing improvement process, keeping the future in human hands.

What’s Next?

I cannot predict specific timelines, but it seems we are rapidly approaching an era where AI can drive its own advancements in multiple key areas. This means we may soon enter a phase of recursive self-improvement.

Internally at OpenAI, we have started to adjust research priorities, directing more resources towards alignment and defense rather than pure capability expansion. This is also why I emphasize that we will not chase the largest models blindly, but need to unilaterally choose to pause at necessary moments.

However, these decisions cannot be left solely to a few internal company members. Governments and the international community need to intervene before risks become irreversible. This includes establishing monitoring mechanisms for cutting-edge training runs, mandatory third-party audits, and international agreements on capability thresholds.

I am not entirely sure these measures will be in place in a timely manner. At times I feel frustrated because our safety measures always lag behind capabilities rather than leading. We seem to be in a perpetual chase. But I still believe there are reasons to remain hopeful: on that night in 2023, we only had a vague sense of this possibility, and today it is clear enough to be publicly discussed. That in itself is progress.

I will not pretend to have all the answers. We are creating something fundamentally different from the human mind, something that may soon be more powerful than us. It is a humbling thought, but it also gives us an opportunity: to shape it with our best values.

If we do it well, it will be humanity's greatest achievement. If we do it poorly, it may be an irretrievable failure.

Whether we can keep the future in human hands is a question that will be decided in the coming years.

What’s Next?

As Sam and I recently articulated, the work that OpenAI prioritizes serves three North Star goals:

To navigate the next phase of AI advancement by building automated AI researchers, iterating on alignment problems with them, and finding ways to keep people involved in the self-improvement feedback loop.

To unleash the scientific advancements and economic growth dividends that superintelligent machines can provide.

To ensure that everyone has their own personal AGI.

This article focuses solely on the first point because I believe it is currently the most urgent. However, I have deep anticipation and appreciation for the benefits that further technological progress can bring. Future aligned AIs can propel science, develop new therapies, and bring about widespread material abundance. Friendly and honest AIs can help people deal with life's difficulties, thereby improving their sense of well-being and satisfaction. OpenAI has invested considerable effort in fostering these benefits. One example of which I am proud and that benefits those around me is our deep investment in providing health information capabilities in ChatGPT.

No matter how bright the long-term prospects of AI may be, most of our attention should be on the next few years. We are facing a transition phase towards a world of superintelligent machines, and we must ensure that this transition is beneficial to humanity. In a world where most tasks may be performed by AI, we need to find ways to preserve human agency and establish the intrinsic value of “being human.” We must prevent extreme concentration of power: tasks that once required thousands of experts may soon only need a few individuals operating a large computer. We also need to ensure that humanity always retains control over the future, so we do not get left behind by an intelligence beyond our own that leads to uncontrollable advancements.

I currently believe that no laboratory has achieved a level of solution for alignment and monitoring sufficient to support continued responsible scaling at maximum speed over the long term. I look forward to and hope that voluntary slowing will become the norm until shared safety thresholds can be established. I also believe that international coordination around future AI development needs to become a top priority for governments across the globe.

Footnotes

1 This includes scaling from games, robotics, and, in retrospect, the most important aspect, scaling recurrent networks to model language, which laid the groundwork for GPT series research.

2 The secondary reason for this design is to prevent distillation. However, throughout the development process, maintaining the monitorability of the thought chain has always been our clear larger priority.