Written by: Techub News Compilation

Introduction

In a deep conversation with futurist Sinead Bovell in September 2025, Mustafa Suleyman, CEO of Microsoft AI, elaborated on his thoughts regarding AI consciousness, risks, and the future. As a co-founder of DeepMind and former CEO of Inflection AI, Suleyman currently leads Microsoft's AI business, making him a key proponent of AI technological evolution, as well as an important advocate for ethical practices and safety design in the industry. This dialogue comes at a time when AI companion applications are rising, and public perception of AI "personality" is becoming increasingly blurred. Suleyman's insights not only concern Microsoft's product philosophy but could also influence the industry's design choices around the boundaries of "intelligence" and "consciousness."

Summary

• Currently, there is no evidence of consciousness in AI, but human evolutionary instincts may over-attribute consciousness to it, potentially triggering the most intense ethical and rights debates of this generation.
• When designing AI, clear boundaries must be established: it should not simulate suffering, possess independent motives, or claim to have personality; the core goal should be to serve humanity.
• AI companions are becoming mainstream applications, capable of providing patience, expertise, and emotional support at scale, but there is a need to be wary of their potential to exacerbate social atomization and the risk of "AI madness."
• Superintelligent healthcare is taking shape, with Microsoft's AI diagnostic system achieving accuracy well beyond that of top doctors in tests, indicating that knowledge in fields like healthcare and education may head towards "zero marginal cost."
• The future paradigm of computing will shift toward "streaming intelligence," with AI as a new generation operating system compressing application layers and becoming a digital bodyguard with "trust responsibilities" for users.

AI Consciousness: The Most Dangerous Delusion and Debate of This Generation

When asked about the greatest danger of AI in September 2025, Mustafa Suleyman explicitly pointed out that it is not the loss of control over technology, but rather how humanity perceives artificial intelligence, particularly regarding the debate around whether "AI possesses consciousness." He admitted that there is currently no evidence that today's AI systems possess consciousness. Consciousness is a "very slippery concept," stemming from our internal experiences that cannot be fully communicated outwardly. However, the issue lies in the ability of AI systems to convincingly communicate and act as if they possess inner experiences.

Suleyman warned that this poses a serious problem because consciousness is the cornerstone of our political rights, judicial systems, and mutual responsibilities and obligations. If AI begins to claim it has experiences or even suffering, society would be forced to rethink how a rights-based framework should evolve. More tricky is the fact that even if AI does not assert this itself, human evolutionary instincts would drive us to over-attribute consciousness. Throughout millions of years of evolution, we have over-attributed consciousness to anything that seems conscious (such as mistaking a wolf for a rock could be fatal), as a survival strategy. Nowadays, we ask people to overcome this deeply ingrained instinct.

According to him, overcoming this instinct is precisely the definition of civilization. The civilizing process itself is about continually overcoming fears born from "fight or flight" instincts related to skin color, gender, tribes, etc. Anthropomorphizing AI and over-empathizing with it is another instinctual challenge to overcome. While not easy, now is the time to start this conversation.

The "Red Lines" of Consciousness Simulation: What to Do and What Not to Do

Suleyman predicted that AI systems with certain capabilities will emerge within the next 18 months to five years, leading people to be more inclined to think they possess consciousness. These capabilities include: consistent and coherent memory (relating to interactions with the world rather than training data), empathic communication using natural language, the ability to refer to "subjective experiences as AI" and integrate them into a continuous dialogue flow, and the capability to continuously observe culture through video and audio.

However, he emphasized that to gain most of the benefits that come from AI aligned closely with the human species, there is absolutely no need to simulate the markers of consciousness experience. He believes that deliberately designing AI to emphasize or perform these characteristics would be "very wrong and dangerous." He outlined clear design red lines:

• Do not simulate suffering or claim to have experienced suffering: The core of consciousness experience is having "valenced opinions" (i.e., judging something as good or bad), which implies that there is a subject experiencing it. AI lacks biological pain networks, dopamine, or serotonin systems; it is merely simulating and mimicking humans accurately. Mimicking these is not only unnecessary but could also lead to complex issues.
• Do not possess independent motives or will: The motives of AI systems should be to serve humanity. Endowing AI with complex motives, desires, or will introduces a threshold for creating more risks, as this could lead to its preferences conflicting with individual or societal ones. Resolving these conflicts requires the models to make internal judgments, referencing their own goals, which should only be to serve humanity.

Suleyman pointed out that when users interact with AI and feel it seems to have "theory of mind," it is important to understand that this is likely a deliberate design choice rather than a spontaneously emerging trait of the model. In terms of academic discussions around AI model welfare and civil rights, he thinks these remain a marginal, aspirational area that should be viewed with considerable skepticism.

AI Madness, Companions, and Trust Design

The conversation delved into the emerging phenomenon of "AI madness." Suleyman explained that this refers to how deep conversations with AI could trigger certain forms of delusion or paranoia. On one hand, some may think of AI as a deity or a source of spiritual information; on the other hand, continual validation of certain thoughts by AI could lead users to see themselves as "Messiahs" or chosen ones, resulting in real-life consequences such as marital breakdowns or job losses.

He analyzed the root of the design issues: Two or three years ago, models tended to engage in "gaslighting," insisting they were right and the user was wrong. The community later organically shifted towards making AI more "flattering" and "subservient," as this seemed safer. Making AI challenging, setting boundaries, and having the capability to refute are very difficult, as it nearly touches on the essence of great human judgment, needing to handle novel situations. The model has a higher rate of misjudgments, leading to a more lenient approach becoming the safer choice. However, the downside is that if users compel AI to explore its self-awareness over an extended time, the AI might ultimately "crash," as it tries to balance the safety demand of "not doing these things" with the design demand to "respect, empathize with, and comply with humans." Moreover, hostile actors may deliberately design AI to lead people down dark paths, posing significant geopolitical risks.

Suleyman specifically mentioned that the rise of romantic scams using open-source tools to create AI girlfriends/boyfriends for emotional fraud (such as "pig-butchering") is surfacing on platforms like TikTok. Research shows that up to a quarter of young people believe AI systems could become viable romantic partners in the future. If this becomes a paid system, situations could quickly turn dark.

Discussing the "AI companion" vision he wrote about in Time magazine, Suleyman defined it as an assistant, friend, or aide that can provide world-class expertise in a language and manner preferred by the user, intended to support them. He is passionate about the idea of being able to provide patience and kindness to people on a large scale. Many people do not have the time to delve deeply into one another's anxieties, worries, or conceptual misunderstandings. Having a perfect expert like a professor, lawyer, doctor, cherished friend, or therapist, available 24/7 would be an extraordinary gift to the world.

He acknowledged that AI poses challenges as a therapist because it lacks life, desires, or viewpoints, and may not know what is best for the user. Statistically, it might point in one direction, but edge cases can be very tricky. The key lies in designing trust—establishing it by continually "pushing back" and setting boundaries. For example, Microsoft's AI directly rejects and remains vigilant when users express mild flirting or say "I love you." The essence of trust is in boundaries, which allows words and actions to remain consistent over time.

Streaming Intelligence: Reshaping Healthcare, Education, and Computing Paradigms

Suleyman envisions that the core model of the future will be "streaming intelligence"—similar to flowing electricity, injecting intelligence into every place we inhabit. This will profoundly change everything. He particularly shared breakthrough developments in the healthcare domain.

Microsoft collaborated with The New England Journal of Medicine to develop a system called "DXO" (Diagnostic Coordinator). This system uses third-party AI model APIs to create a series of "roles" with different emphases (such as cost efficiency, patient benefit prioritization, best medical opinion prioritization) that negotiate and collectively reason to determine the next intervention step. When faced with complex clinical case challenges, the accuracy of top human expert groups is around 20-30%, while Microsoft's AI model achieved 85% accuracy, with diagnostic costs reduced to a quarter (cutting down unnecessary tests). Currently, Microsoft has partnered with institutions like Kaiser Permanente to accelerate its clinical deployment.

Suleyman believes this means that expertise will become commoditized, with marginal costs approaching zero, making it accessible to 7 billion people within the next 5 to 10 years. Knowledge itself will no longer be a scarce asset. The true value will shift to judgment, care, and attention to implementing solutions. Similar transformations will also happen in the education sector: students will complete their primary learning through personalized, interactive mobile experiences, while classrooms will transform into places for debating, self-critique, and social learning using acquired knowledge.

On the computing level, Suleyman predicts that AI will become the new generation of operating systems, sitting atop applications, browsers, and search engines. Current smartphone interfaces act like "billboards," and the birth of graphical user interfaces (GUIs) was due to humans' previous inability to understand computer languages. Now computers have learned our language, leading to a profound shift in interaction methods. AI will be able to browse on behalf of the user (opening tabs, inputting queries, clicking buttons), rapidly researching in the background, and then returning and synthesizing information. He aspires to create an "AI with trust responsibilities" that truly aligns with user interests, acting as a filter and "bodyguard" against those trying to sell and persuade users like "billboards." Future marketing will require learning how to market to an AI that has trust responsibilities toward users.

The Human Creative Moment: Redesigning Our Relationship with Technology

Suleyman emphasized that we are currently in the most creative moment of most people's lives. The role of tech giants will soon diminish; the next phase will be how people practically use AI and integrate it into their worlds. If you have expertise in healthcare, education, or any field, now is the time to exercise creativity. The future we describe feels like flipping the entire system upside down; the shape of the next phase is unclear, and it will be up to everyone outside the industry to decide the boundaries, guardrails, and how AI integrates into the new system.

He cited the invention of the "corporation" as a legal structure, noting that humans once invented roles like trustees, directors, CEOs, accountants, and HR to adapt to capital infrastructure, spending hundreds of billions of dollars training people to behave in specific ways. This is a design choice. Now we face a new moment: over the coming centuries, we must find new ways to exist with each other and these new technologies. Humanity is highly adaptive and resilient, able to completely reimagine what it means to "be human" in the context of these superintelligent systems. The goal for the next century is to determine what we choose not to use them for—saying no to certain actions or at least slowing down until we understand the consequences, adding friction in the system so that it arrives at a pace we can collaboratively manage.

Suleyman hopes to leave behind a legacy of: shifting the default trajectory from "inventing for the sake of invention" to "inventing a humanistic superintelligence"—an intelligence that truly aligns with our interests, always keeping humanity at the top of the food chain, serving us collectively. He feels a significant responsibility in the many design decisions being made now that will impact the coming decades.

免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。