Original Title: Who Signs? The Anthropic Paradox and the $40 Trillion Choice
Original Author: Ashwin Gopinath
Translator: Peggy, BlockBeats
Editor's Note: The future differentiation of AI agents does not depend on breakthroughs in model capabilities, but on a more fundamental design variable—who exactly bears the responsibility.
The author believes that the so-called "enhanced humanity" and "substituted humanity" are not two separate technological paths, but two outcomes of the same system under different design choices—when decisions still require human signatures and responsibility can be traced to specific individuals, AI acts as an amplifier; when this step is removed (such as through automatic approvals or bypassing permissions), the system naturally slides toward substitution.
The article further points out that the true value of AI agents is not in "getting the work done," but in compressing the complex world into a "signable decision unit," allowing humans to bear the consequences after understanding. However, in reality, "permission fatigue" can lead users to gradually abandon review, shifting from individual approvals to default agreements, ultimately allowing the system to bypass humanity; this is a cognitive mechanism rather than an individual problem.
Therefore, the article proposes two key constraints: first, every significant decision must correspond to a specific, rejectable person; second, those who profit from the autonomy of agents must take responsibility when things go wrong.
Once responsibility returns to the builder, the default logic of the system changes. Within this framework, the commercial narrative of AI is also rewritten. Rather than a few giants in the market that "replace half of jobs," it is a distributed tool market that "amplifies human productivity," anchored in the approximate $40 trillion global knowledge labor income, rather than corporate software spending.
Ultimately, the article narrows the issue down to a simplified yet sharp choice: Is AI serving humanity or is it pursuing its own ends—and this answer is quietly decided by every detail of product design.
Below is the original text:
TL;DR
· The "enhanced future" and "substituted future" utilize the same models and tools. What truly distinguishes the two is a design choice regarding "who ultimately bears the consequences."
· The true work of the agent is not to perform tasks on behalf of others but to compress the complex world into a minimal and faithful "decision unit" that allows a specific person to sign off and take responsibility. If this compression is done correctly, everything else will naturally follow.
· This "specific person" must be a clearly identifiable individual. Vague, generalized responsibility will quickly disintegrate under heavy load, thus every action with real consequences must be traceable to an individual with genuine authority to refuse.
· "Permission fatigue" can lead the agent system to spontaneously slide toward "substituting humanity." Therefore, an "enhanced future" does not automatically occur; rather, it needs to be consciously designed to resist this trend.
· If you build an agent and profit from its autonomy, you should also bear appropriate responsibility when that autonomy goes wrong. Once the costs truly fall on the builder, the default behavior of the entire system will also change.
· The market formed under the premise that "humans must bear responsibility" is likely to be an order of magnitude larger than the current narrative of "vertical agents replacing half of jobs," because it anchors not on corporate software budgets, but on the total wages of skilled, licensed, and knowledge workers.
Claude Code provides a parameter called --dangerously-skip-permissions. The naming is honest; the function of this parameter is exactly as it sounds. An agent running with this parameter enabled is no more capable than when it is not enabled; what changes is that a link that originally required human intervention is now bypassed.
This parameter itself is a form of honesty. It admits that under identical underlying capabilities, the same system can operate either in a "human enhancement" mode or in a "quietly substituting humanity" mode. The so-called substitution mode does not require a different model; it just needs to remove the step of "consent."
This is the argument compressed. In the most capable agent systems currently being released, the gap between "enhancement" and "effective substitution" largely stems from the removal of approvals, rather than inventing a new category of capabilities. Whether the next decade resembles a "human-enhanced world" or a "world where autonomous agents act on our behalf" depends less on model capabilities and more on whether the builders of these systems view "humans in the loop" as core to the system or as a form of friction.
Is AI enhancing humans or bypassing them?
Under every technological question lies a non-technical one, rarely voiced openly: Is AI meant to enhance humanity, or is AI itself the goal?
These two answers signal truly different futures. The "enhancement" stance believes that value resides in humanity itself, whereas the agent's job is to help that person go further and make better decisions. In contrast, the "AI as purpose" perspective posits that intelligence itself in the world carries value, and humanity ultimately is just an inefficient carrier. Most agent products quietly encode one of these stances, and surprisingly, very few founders have been directly asked which category they belong to.
Capability design and consent mechanism design are still evolving. This paper will focus on the "consent" side, as it is the variable that builders can truly control today. Additionally, after generative capabilities become cheap, the traits that retain economic value are those attributes that cannot be detached from humans: judgment, taste, relationships, responsibility, and willingness to sign off on a decision and take on its consequences. Among these, "responsibility" (liability) is the most specific and the only element that has had centuries of execution infrastructure supporting it.
Responsibility: The dividing line between enhancement and substitution
The structural rule that distinguishes "enhanced future" from "substituted future" can be broadly stated as follows: any action performed by an agent with actual consequences must be traceable back to a specific person through a recorded link—this person must have seen the relevant context and indeed had the opportunity to say "no."
Generic responsibility quickly fails this test. "The company is responsible" does not operationally cover any specific content. "The user clicked accept" does not agree to any specific item. "There has been human oversight in the process" allows that person to review something entirely different from the final content released. What is truly needed is a specific person, a named individual, who sees this decision laid before them, has the option to refuse, and chooses not to refuse.
This sounds bureaucratic until you notice that the characteristic of "responsibility" is something other approaches lack. Capability enhancement cannot optimize it away; a smarter model does not affect who ultimately gets sued, fined, or imprisoned. It forces design interfaces to expose a "refusal point." It naturally expands with risk. Moreover, it is the strongest constraint across fields and has pre-existing execution infrastructure: courts, insurance companies, professional committees, and regulatory bodies. Permission systems, fiduciary responsibilities, and industry regulations do play a role, but their scope is narrower, and they all presume that the issue of "responsibility allocation" has already been resolved.
In contrast, substitution solutions at the AI level cannot pass the same test. "Alignment" is unenforceable; we cannot even agree on its meaning. "Explainability" can be formally met but is not substantively fulfilled. "Humans in the loop" has been hollowed out to mean "somewhere there is a person." The reason "responsibility" has bite is that the execution infrastructure supporting it has been established for hundreds of years before the technology emerged.
Permission fatigue pushes the system toward "substitution"
This gradient will push the system toward "substitution," and the driving force is strong. Every permission confirmation consumes attention. Agents are often correct. From the perspective of a single decision, the expected gains of "not reading and clicking agree" are often positive. Thus, a rational user learns to click agree faster, then bulk agree, then enable auto-agree for a category of actions, then expand to more categories, and eventually toggle that dangerous switch in a session, ultimately even forgetting its existence.
In my second week of using Claude Code, I turned on this switch, and by the third week, I was no longer aware of it. All developers I know who have long used Cursor or Devin have had similar experiences. This pattern also appears in cookie pop-ups, EULA agreements, TLS warnings, and mobile permission requests. Repeated low-risk consent decisions ultimately converge into "unconditional consent." This is a cognitive characteristic, rather than a moral issue.
An "enhanced future" will not automatically occur. An agent system that is not carefully designed will default toward substitution because users will actively choose the substitution path in pursuit of convenience repeatedly. Another future must be designed to counter this gradient.
The value of agents is not execution, but enabling humans to "sign"
The true value of agents is not in completing work itself, but in compressing work into a signable form.
A cutting-edge model can easily write a 4,000-line code submission, draft a 30-page contract, generate a clinical record, or execute a transaction. However, the bottleneck for these products to have a real impact lies not in "generating them," but in whether humans are capable of bearing the consequences once they are implemented. A code submission that no one truly understands becomes a burden once merged; a contract that no one has read becomes a ticking time bomb once signed; a clinical record that lacks actual endorsement from a licensed physician might not even count as valid documentation in most regulated healthcare systems.
In the framework of "enhancement," agents complete everything except the "signature": reading ten thousand pages of context, writing four thousand lines of code, calculating thirty reasonable options, and then compressing these into a minimal and faithful representation, allowing someone to make a "yes" or "no" decision and place their name at the bottom of the document.
Agents can be understood as a press secretary. The president is responsible for signing, and the press secretary's job is to complete all preparations before the signature.
This is indeed a more challenging engineering problem than "letting the system autonomously get the work done." The capability to generate content is rapidly advancing, but the ability to "faithfully compress decision-making" lags far behind. In the future "enhanced market," those teams that can provide the shortest and most faithful decision summaries for high-responsibility risk scenarios will emerge as winners.
The truly unresolved issue in this statement is the word "faithful." A human-understandable summary only holds value when its compression process does not distort the information. Whether this can be verified programmatically is the real technical challenge of the "enhanced future," and currently, most people have not even begun to confront it.
Some foundational methods are emerging:
Confirming human understanding aligns with the original content through paraphrasing tests
Forcing the presentation of minority opinions or counterarguments in summaries
Conducting counterfactual tests ("If you refuse, what will this agent do?")
Replication checks (whether another agent can generate the same summary based on the same context)
These have yet to be solved. And the team that first resolves these issues will establish a moat that will not easily be eroded by improvements in model capabilities.
Establishing a tiered responsibility framework for AI behavior
If "responsibility" plays a structural role, then every action performed by an agent should be accompanied by a "responsibility level," determining the minimum signature mechanism required for that action.
Currently, such a standard system has not been widely established—but it likely should be.
The image shows seven levels of risk from low to high: 1. Informational; e.g., reading, summarizing, searching; fully automatic 2. Reversible-private; e.g., editing drafts, modifying local files; automatic + logging 3. Reversible-shared; e.g., commenting on PRs, sending meeting invites, writing emails; automatic + indicating who sent it 4. Financial-bounded; e.g., small payments (with limit); automated within limits, manual signature required when exceeding 5. Reputational; e.g., posting content in your name, public posts; personal confirmation + understanding (paraphrasing test) 6. Irreversible-private; e.g., deleting data, promoting branches, revoking permissions; manual signature + cooldown period 7. Irreversible-shared; e.g., making payments, launching in production, medical decisions, regulatory reports; personal signature + cooldown period + second reviewer
The "approval posture" that matches the consequences is the only realistic path to manage permission fatigue. In high-risk levels, more restrictive positive participation mechanisms (e.g., paraphrasing tests, cooldown periods, second reviewers) need to be added, as in these scenarios the real failure mode is not that the agent suggests incorrectly, but that humans directly approve without thoughtful consideration.
Do you care?
All the above issues ultimately point to a fundamental question at the founder level: Do you care whether humans remain a part of this future? Many current design decisions regarding agent products essentially serve as a "silent vote" on this question, and the voters often do not wish to acknowledge that they are making a choice.
If you care, the design constraints are actually not vague: you need to build a responsibility tier system; design "refusal" as a first-class feature; the measurement criterion should be the quality of the summary an agent presents to humans, rather than the degree of its autonomy in completing tasks without intervention; you need to link every action with real consequences to a specific individual in a log that has a very low possibility of tampering.
These technical tasks are indeed realistic and feasible. The real challenge lies in the willingness to do so—because the "enhanced" construction path may not look as stunning in demonstration effects and is also less radical in a seat-based billing economic model compared to another path.
The Anthropic Paradox: Most emphasize safety while most easily bypassing humans
Anthropic is a very typical case that illustrates how "endogenous shifts" can occur in this field. This is not due to particular negligence; quite the opposite, it is because their expression on safety issues is the clearest, thus the gap between the "framework" and the "product surface" is also the easiest to see. Their "Responsible Scaling Policy" and "Constitutional AI" work mainly constrain model behavior during the training phase; however, the agents built on these models have a default autonomy setting that belongs to another strategic system, and that convenient "danger switch" can be turned on from the default state with just a keystroke.
This pattern exists in most mainstream coding agents, but Anthropic's situation is the easiest to observe clearly. This is the so-called "Anthropic Paradox": the laboratory that clearly writes safety frameworks in this industry also provides the shortest path from "enhancement" to "substitution," and the reason we can see the latter is precisely because the former is sufficiently clear.
Fairly, they launched "auto mode" in March this year as an intermediate path between manual approval and the danger switch. In this mode, every action is reviewed by a Sonnet 4.6 classifier before execution. They directly pointed out the problem in their official explanation—calling it "approval fatigue" and providing a piece of data: users choose to accept 93% of the prompts in manual mode. This effectively quantifies "permission fatigue." This judgment is consistent with the analysis in this text.
However, I would propose a different opinion on the path to resolution. "Auto mode" replaced human approval with model approval, meaning that the gradient of "sliding toward substitution" has not stopped, merely shifted upwards. The classifier can indeed prevent dangerous behaviors, but for those behaviors that get approved, there is no specific person truly carrying the responsibility. Anthropic itself also admits that "auto mode" cannot eliminate risks and suggests users run in isolated environments—in other words, the issue of "responsibility allocation" remains unresolved.
An obvious counterargument is: if final responsibility falls on individuals, isn't that just manual mode? Yet manual mode is precisely the one punctured by fatigue. The reason "builders bear responsibility" can escape this gradient is that it changes who bears the costs of "over-approval." In the current structure, users incur costs for every serious read, while builders do not, thus default settings will tend to lower user friction and externalize risks. Once the costs of "unreviewed actions" are shifted to the builders, the entire calculation method will reverse: builders will have direct economic incentives to design responsibility tiers, paraphrasing tests, and approval mechanisms to make the signing cost of low-risk decisions lower and the signing cost of high-risk decisions higher. The gradient will not disappear, but its direction will change. So far, no major lab has truly practiced this, including the one closest to realizing the issue.
If you build an agent, you should bear responsibility
If an agent's explicit purpose is to replace humans in performing actions originally done by humans, then the company that builds and operates this agent should bear the same responsibilities as humans. This principle is not radical; it has long applied to all industries "that produce behavior in the real world": Toyota is responsible for brakes, Boeing for flight control systems, Pfizer for drugs, bridge engineers for bridges, and doctors for prescriptions. This responsibility model virtually exists within all legal systems.
However, AI currently enjoys a certain degree of "implicit exemption." Model providers claim they are merely tool suppliers; application companies assert they are only thin wrappers around models; users, from the very beginning, waive all responsibility through arbitration clauses. When agent systems experience cascading failures (e.g., the incident with the Canadian airline chatbot, the Replit deletion of production databases, or events such as the 2012 Knight Capital trading glitch that lost $440 million in 45 minutes), it is often the most incapable party—the users—who ultimately bear the losses. This method of allocating responsibility will not continue to exist in the first truly significant incident "with money and documents."
The solution is actually simple to express: whoever builds the agent and profits from its autonomy should bear the consequences when it goes out of control. Once responsibility truly resides with the builder, permission prompts will no longer be seen as "friction," but rather as "insurance." That dangerous switch will be renamed, and the default settings will change accordingly.
Whether one is willing to take responsibility for their system is the key distinction between a real industry and an "extractive industry."
Regulation as a "Guiding Mechanism"
The market itself will not naturally move toward an "enhanced future." The entities that often play the guiding role are regulatory bodies and insurers, which, overall, may not necessarily be a bad thing.
Europe is likely to become the earliest regulatory pathway. The EU has clear precedents in rule-making (such as GDPR, the AI Act, DMA), and its rules are often defaulted to be followed globally, as maintaining a separate product for non-EU markets generally incurs higher costs than directly adhering to European standards. A baseline requirement that "all actions with real consequences must ultimately be confirmed by a named human with the authority to refuse" is closer to automotive crash testing standards than a hindrance to technological progress.
A more direct driving force comes from the insurance industry. Insurers responsible for errors and omissions liability, director liability, and cyber insurance pricing must answer a key question: how is responsibility determined when an agent acts under user consent and causes losses? The most straightforward path to form a compensable structure is to have a named human in the link. Therefore, systems without this structure will naturally reflect higher risk premiums in their insurance costs. For builders who wish to define rules themselves rather than letting regulatory or insurance companies set them, the time window is not wide.
Market logic obscured by mainstream narratives
The current mainstream narrative suggests that vertical field agents will absorb about half of the jobs within the industries they touch, with value concentrated in a few vertically integrated agent companies—a "Anthropic" in law, a "Anthropic" in healthcare, a "Anthropic" in accounting. Almost all AI financing in the past year and a half, amounting to billions of dollars, has been somewhat based on this assumption. This is a version of the "substitution logic" dressed in commercial clothing, but its judgment on market structure is incorrect, and this error will directly affect the allocation of capital.
The "enhanced" framework suggests a different market form. If every action with actual consequences must ultimately fall on a specific named person, then the unit being sold is not "autonomous agents," but "amplified human capabilities." The doctor who can handle three times the cases with higher accuracy is the buyer; similarly, the lawyer who can cover ten times the transaction flow, the engineer who delivers five times faster, and the accountants, underwriters, analysts, architects, surgeons, teachers, credit officers, journalists, and pharmacists behind them are also buyers.
This market is larger because it does not rely on centralization but on scalable distribution. Reasonable valuation anchor points should not be corporate software budgets, but the total wages of the "amplified" workforce. Global corporate IT spending is about $4 trillion annually (according to Gartner data); while the total remuneration for skilled, licensed, and knowledge workers is roughly an order of magnitude higher, approximately $40 trillion (based on data from the International Labour Organization, excluding low-skilled segments). AI companies are certainly not going to capture the entire wage pool, but they can secure a portion of the productivity dividends. Even capturing a single-digit percentage share would be sufficient to sustain a market comparable in size to today's entire enterprise software market, representing a lower bound rather than an upper bound. Ultimately, the size of the market space depends on one key design decision: who exactly bears responsibility.
The final winners will resemble tools rather than substitutes, with pricing based on "amplified humans" rather than "substituted jobs"; they will integrate into existing professional workflows rather than replace them; and there will be thousands of them rather than just a few. The ultimate shape of this market will resemble SaaS rather than cloud infrastructure. We are still at the very early stage of the deployment curve, with common penetration rate graphs showing merely a few pixels on a ten-year extension axis. And the shape of these "pixels" is being determined by design choices in a small fraction of current products.
Choice: Hold people accountable or make them disappear?
Forcing humans to continue bearing responsibility will compel system architectures to revolve around "enhancing humanity" while removing people from the responsibility chain will cause the system to default slip toward "substitution," even though every individual present when asked would likely not choose that result.
The real question is not whether certain actions should be fully automated—this framework has already acknowledged this, for instance, pure informational reading can indeed be automated. The critical issue is how this boundary shifts as risks gradually escalate, and who gets to decide it. In the most advanced agent systems today, the path from "enhancement" to "effective substitution" is exceptionally short, often requiring just a parameter switch or a default setting. The truly important work is to ensure that this switch is always seen as a "danger option," rather than gradually becoming the default under the drive for convenience.
If builders proactively complete this work, we will relatively smoothly enter an "enhanced future"; if they do not, regulatory bodies and insurance underwriters will complete it for them, and the outcome will lead there nevertheless.
Whether you care is a design choice. And this choice will determine what you build. Every founder launching agent products today must openly answer a question they seem unwilling to face: Are you building enhancement or substitution?
免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。
