Author: Ruxandra Teslo

Translation: Deep Tide TechFlow

Deep Tide Guide: As the forefront of artificial intelligence research surges forward in San Francisco at an astonishing pace, the arrival of the technological singularity, recursive self-improvement, and superintelligence (ASI) seems to have transformed from a science fiction projection into a predetermined agenda spoken of by laboratory engineers. However, in this fervent epicenter, a peculiar and fatalistic fatalism is quietly spreading among practitioners.

In this game concerning the future of civilization, not only is there a chilling extinction anxiety, but a “handoverism” — beautified as the best way out — has emerged: it posits that humanity is destined to lose control over ultimate intelligence and should relinquish sovereignty and labor value, retreating to become subjects nurtured by machine deities.

The author sharpens the analysis of this utopian illusion that permanently cedes civilizational sovereignty to machines by starting from the intellectual threads of Tocqueville, Arendt, and Popper, boldly stating that at this critical crossroads in history, we should not casually give up humanity's future and free will.

"This is not the outcome I was hoping for," an engineer working on cutting-edge models in a leading AI lab told me, "but once we achieve superintelligence, and since it is already unstoppable at this stage, I really can't see any other future for humanity."

The outcome he refers to is humanity permanently losing the power of self-control, and we will be jointly subjected to the absolute rule of a (hopefully benevolent) AI deity.

What's shocking is that he appeared unusually calm and casual when discussing this grand theme, as if he were naturally stating the rising and falling of tides.

After a brief pause, he added, "However, this is still much better than my actual greatest fear, which is that all of humanity will head towards extinction."

This statement is particularly striking because it does not come from fringe doomsayers or technophobes. Instead, it comes from a core developer at a major company directly building this system.

His words were filled with a fatalistic submission, as if the existential ultimate question of "What use is humanity really?" had already been decisively answered elsewhere by others. What lies before us seems to be only sitting back and awaiting our fated destiny.

I moved to San Francisco just three weeks ago. According to the prevailing consensus here, we are on the eve of a full-blown technological singularity. In this brief time, I have come to realize that this way of thinking is far from isolated within the circle: in fact, ceding humanity's ultimate control to a benevolent superintelligence has been seen by many as a rare and better outcome.

It needs to be heeded because the imminent singularity is showing an incredibly realistic texture. The most crucial watershed lies in the prospects of "recursive self-improvement" (RSI): AI systems are starting to learn self-iteration and evolution, with the next generation of systems birthing stronger systems, until they completely evolve into superintelligence that exceeds human control limits.

Within many social circles in San Francisco, people talk about RSI as if it were merely a few months away. Some even believe it could be shorter. At this pace, a person barely has time to choose a suitable set of furniture for their apartment.

Generally, when we still have some decision-making power over the direction of technological evolution, we should clearly outline what goals we want these machines to serve for human civilization. Since the developers themselves are repeatedly warning of the extreme risk of species extinction, people have natural reasons to expect a clearer and more promising vision.

The bleakest extreme scenario is already known to all: all of humanity or a significant portion of the population facing catastrophe. However, in my view, even in those optimistic proposals, the comfort they can offer to ordinary people is also pitifully scant.

One widely acclaimed, seemingly optimistic idea is called "handoverism": humanity voluntarily cedes the future's leadership to a deity-like super AI and then resides in a comfortable refuge where the course is no longer determined by themselves.

In such a future scenario, the substantive meaning of democracy will be completely dismantled, and the vast majority of humanity faced with such a systemic establishment may never have had the chance to express consent.

The permanent replacement of human labor is viewed here as a fait accompli. However, there has been no convincing answer provided about how humanity can continue to play a role and find meaning among each other. Even real relationships between people have become shaky.

I do not know whether the vision of handoverism will ultimately become a reality. Creators at the forefront of technological research believe it is feasible and even desirable, but this does not constitute any guarantee: being in the technology center often does not mean being able to accurately predict its profound impact on human society.

However, they are, after all, the closest people to this behemoth, and their thoughts and considerations should be taken seriously. If mainstream society foolishly dismisses their concerns as mere hype and chooses to stand by, it would be a highly risky dereliction of responsibility.

Some even calmly accept humanity's end

In current San Francisco, insisting on discussing "the flourishing of humanity" sounds almost misplaced, like complaining about the lack of refinement in desserts during a famine.

Within the core circles that truly grasp technological trajectories, the blind optimism has already vanished. Letting humanity barely survive amidst the impending technological tsunami has been defined as a rather noble, even arduous goal. Conversations about the probability of human extinction (p(doom)) are constant, with many estimates above 10% or even higher.

Outsiders might naively think that everyone has at least a fundamental consensus on the baseline of "avoiding human extinction." But the reality is far from this. Some groups have even given up the goal of survival, calmly accepting and even secretly welcoming the curtain call of humanity as a biological species.

For example, Ginevra Lily Davis's work "The Silicon Man" extensively investigates and documents the spread of this extreme philosophical stance: the belief that silicon-based life replacing carbon-based humanity is the inevitable final chapter of cosmic evolution.

For those who view human extinction as liberation and ultimate destiny, I have no intention of arguing. Once the value of human life is completely detached, any logic extrapolation based on humanism will lose its foundational cornerstone.

What I hope to engage in dialogue with are those creators and witnesses who still firmly believe humanity should endure, yet are gradually sliding into the mire of handoverism.

Even if humanity survives, it might lose all autonomy

Within the camp that recognizes humanity should continue to exist, there are also enormous disagreements and compromises regarding what kind of living condition humanity should possess in the future.

A recent long article by Scott Alexander on exploring future scenarios provides a great entry point. The core premise of the article is: as long as it can be confirmed that superintelligence will not destroy humanity, civilization can smoothly transition into a meticulously cared-for new phase.

This ideology has not yet formed a politically organized framework, yet in the elite circles of technology in San Francisco, the proportion of those who secretly hold or tacitly approve of handoverism has conservatively taken up a considerable portion.

The psychological motivations behind handoverism are similarly complex. One faction could be called the "technological saviors": they feel utterly despondent about the complicated realities of human politics, hence place their hope entirely on an ultimate arbiter free from human narrow biases.

The second camp is purely the "fatalistic submissives": they may not deeply desire to hand over sovereignty, but facing the recursive evolution of supercomputing, they conclude that humanity has no choice but to obediently pass over the keys.

Different factions have varied visions on what role humanity would be reduced to after relinquishing power. The most representative metaphor sees humanity as "house pets" of the future superintelligence: well-fed and cared for, yet with no participation or voice in significant decisions.

The public statements of Anthropic's CEO Dario Amodei represent another technocrat-style compromise. In a recent long article, he depicted a "cybernetic symbiosis" meant to retain some degree of human involvement under machine dominance.

However, this concept carries a distinctly Rousseauian romanticism: it assumes that once machines completely take over all hard-core issues, humanity will return to its essence and engage in pure artistic exploration and spiritual joy.

The recent public stance of OpenAI's CEO Sam Altman has also subtly shifted. Although he previously warned of superintelligence's erosion of human autonomy, his rhetoric has gradually shifted to emphasize the endless abundance brought by inclusive technology.

Other AI lab leaders, while clearly rejecting extreme expressions of handoverism in public, often can only provide vague and perfunctory responses when asked about specific implementation paths.

Tightly connected to this is the ultimate questioning of labor value. In a societal construct that completely eliminates human labor, universal basic income (UBI) has become almost the only universally proposed cure-all.

Yet this economically structured model, dispensed by machines, fundamentally evades a core question: how should humanity maintain mutual support and social dignity among one another? When all creative and constructive functions are stripped away, monetary compensation cannot fill the collapse of existential meaning.

Why is "handoverism" an extremely dangerous mistake?

It must be deeply recognized that permanently ceding human sovereignty and the future of civilization to machines, no matter how beautifully depicted, is fundamentally an extremely dangerous regression.

First and foremost, I firmly defend humanity's sacred right to self-determination. As Tocqueville profoundly revealed in "Democracy in America": freedom is not an abstract concept, but a virtue and capability continuously forged in the self-management, mutual association, and collective exploration of daily life.

Tocqueville foresaw a seemingly mild and benevolent paternalistic guardian rule that would quietly strip away people's independence of thought and life responsibility. The super AI rule envisioned by handoverism is undoubtedly the most extreme variant of such soft totalitarianism.

Secondly, even purely from a pragmatic and realistic outcome perspective, the governance vision of handoverism is also internally incoherent. The premise of handoverism is that AI alignment can become a permanent state that lasts forever.

However, all institutional frameworks in human history that have stably responded to demands have been built on centuries of trial, error, and dynamic checks and balances. Once humanity relinquishes all constraints, there is no mechanism to ensure that this absolute power remains aligned with human welfare over the long term.

Furthermore, entrusting all contradictions to ultimate authority for a one-size-fits-all resolution has always had extremely dangerous precedents in the history of political thought. Karl Popper incisively pointed out in "Utopia and Violence": arranging political organization around some ultimate blueprint will inevitably view normal real-world divergences as obstacles that must be eradicated.

Popper emphasized that solving specific problems through open criticism and gradual improvement is the reliable path for humanity's civilization to achieve genuine progress.

Finally, what deeply unsettles me is the absolute permanence brought about by this handover. Why should technological cognition in 2026 become the eternal constitution of future human civilization?

As Hannah Arendt said, the meaning of life lies in "action" and "natality": every new birth brings infinite possibilities to open a new chapter in the existing world. The reason human civilization continues to thrive is precisely that generation after generation can continuously reshape the future according to their own will.

To confine future generations in the amber defined by machine deities is equivalent to depriving them of their inherent rights as human beings. At this critical crossroads in history, we still hold the right to choose and should never easily hand over humanity's future.