Author: Techub News Compilation

Introduction

At an online summit hosted by the Suchir Balaji Foundation, a debate titled "Broken Systems" sparked deep reflection. Professor Stuart Russell, an AI pioneer from the University of California, Berkeley, was unable to attend in real-time due to scheduling conflicts but still issued a stark warning about the risks of AGI losing control through a pre-recorded video. Joining him were critical voices from academia, such as Eva Navarro López, and Carl Mabuka from Africa, who has rich practical technical experience. This dialogue is significant as it does not merely dwell on the common surface discussions of "AI ethics," but sharply points to the structural "broken" issues in the current AI development, application, and the entire technology-academic-business system behind it, and attempts to explore a reconstruction path based on humanity, collectivity, and fairness.

Summary

• Stuart Russell warns: the current path pursuing AGI is inherently unsafe, with significant risks leading to "Chernobyl-level" disasters or even human extinction, while developers have no control plans.
• Eva Navarro López points out: the so-called "AI" is merely a mirror reflecting societal biases and power structures, currently excluding diversity, exploiting creativity and invisible labor, and stifling scientific methods and critical thinking.
• Carl Mabuka emphasizes: building AI that truly serves people requires a cultural barrier of "listening first, building later," especially focusing on communities overlooked by global supply chains, to prevent technology from becoming a new colonial tool.
• The consensus solution: regulation is essential, but the ultimate hope lies in the awakening and organization of civil society, reclaiming technological governance from a handful of companies and governments through forms like "Citizen AI" agreements and international alliances.

The Risks of AGI Losing Control: We Are Boarding a Plane That Never Lands

Stuart Russell's opening video set a serious tone for the entire debate. He first pointed out the fundamental shift in the AI field: for the past 80 years, AI was primarily an engineering discipline based on predictable, provable mathematical theories; now, we are training "black boxes" with trillions of parameters to mimic human language behaviors. We fundamentally do not know how they work, but we know they have learned to mimic human behaviors aimed at pursuing goals (like persuading, impressing, surviving, selling, etc.). This acquisition and pursuit of goals are inevitable results of system development, which makes current AI "inherently unsafe."

Even more concerning is the scale and objectives of investment. Russell pointed out that human investment in AI this year has surpassed any technological project in history, 25 times that of the Manhattan Project and 250 times that of the Large Hadron Collider, with the objective of surpassing humans on all dimensions with AGI. He believes the probability of success might be 70%, but it could also lead to an "AI Ice Age" and the evaporation of trillions of dollars in capital due to stalled progress. However, the most dangerous scenario is success without control.

"Currently, developers acknowledge they have no plans for controlling the AGI systems they aim to create, and the government also has no plan to require them to do so," Russell warned, noting the significant possibility of a "Chernobyl-scale" disaster, such as coordinated attacks on the financial, communication, or power grid systems, potentially leading to millions of deaths and collapses of economies in multiple countries. Worse yet, humans may permanently lose control and have no say over their existence. Expecting everything to magically proceed smoothly is unrealistic; "we cannot forever maintain power over a system that is intrinsically unsafe, far more powerful than us, and whose workings we do not understand."

Russell emphasized that this is not a fringe viewpoint but a consensus among many top AI researchers and CEOs of companies developing AGI. He revealed that one CEO had told him that a "Chernobyl-level event" is the best outcome we can expect; a senior researcher at OpenAI believes the risk of extinction has reached 60%.

As for solutions, Russell believes that regulatory measures similar to those for pharmaceuticals and nuclear power (like requiring proof of safety as a condition for market access) would help, but the problem is that developers cannot prove that their "inherently unsafe" technology is safe. They argue that humans cannot protect themselves with such rules because they do not know how to comply. Russell rebutted this fallacy: "They can comply by not building these systems until they know how to make safe AI systems." Just as nuclear power developers will not build nuclear power plants until they know how to prevent explosions, the AI industry must choose a different technological path that embeds safety from the start, and the entire process must be transparent. "We can imagine AGI as a new type of airplane that has never been tested. In its maiden flight, every human will be a passenger, and this plane will never land once it takes off. This means that this technology must work perfectly from the first attempt and forever. We cannot board this plane until we are sure of that."

Broken Systems in the Mirror: Bias, Exploitation, and the Academic Crisis

If Stuart Russell sounds the alarm on macro future risks, then Eva Navarro López delves deeply into the "broken systems" that current AI relies on and reinforces. She pointedly stated that the so-called AI today is just a "mirror of ourselves, society, technological landscape, and academia," filled with algorithmic biases, lack of diversity, environmental destruction, ethical and legal issues, and the pursuit of war efficiency.

Eva sharply asked: what systems are we breaking? Which broken systems are we continuing that we don't want to change? She believes that the educational and scientific systems have been in crisis for decades, and the current so-called AI is leveraging and benefiting from this crisis. These "broken systems" refuse to change and instead "break those attempting to make a change."

She listed specific issues that need to be addressed: first are algorithmic biases against minorities and vulnerable communities, which remain overlooked in many "responsible AI" frameworks. Second is the lack of diversity, equity, and inclusion (DEI), which refers not just to people but also to ideas. "Will we all become users of technology rather than creators? Who decided that this is the AI we must use?" She cited her field of music technology as an example, pointing out that digital platforms recommend female artists far less frequently than male ones, which reflects the misogyny present throughout the music industry. To address this, she participated in a project promoting the "Music Gender Metadata Declaration," attempting to solve inequality from the data source.

The academic field is another "broken system" that particularly worries her. Many universities are mandatorily promoting the application of AI in education and research, which may lead to critical thinking and creativity being replaced by the commodification of knowledge, standardization of thinking, and homogeneity of ideas. "This is the death of the scientific method," Eva lamented. "Codes of conduct are being rewritten to normalize and promote plagiarism... But can we accept that everything we do will be plagiarized and stolen? The severity of the problem is completely disproportionate to the scale of the protests."

Moreover, she also mentioned the exploitation of "invisible labor" detailed by Carl Mabuka, as well as the environmental impact brought by AI's massive water and energy consumption. Facing these layers of "broken" scenarios, Eva did not fall into despair but saw hope in collective resistance and alternative constructions. She listed a series of positive movements: the generative AI refusal movement, data feminism and feminist AI, indigenous protocols and AI working groups, the Algorithmic Justice League, the distributed AI research institute founded by Timnit Gebru, and organizations empowering underprivileged communities like "Technolatinas" and "Women in AI Ethics." "Negativity and positivity coexist... the collective response is a collective resistance against the individual, and the collective will always win."

From an African Perspective: Listening, Humility, and Cultural Barriers

Carl Mabuka injected a sense of realism into the discussion about "broken systems" from the perspective of technical practice on the African continent. He opened his sharing with a Rwandan proverb: "Another person's eyes can help you find firewood in the forest, but your own eyes protect your home." In his view, the ultimate responsibility for shaping the future of AI, especially for Africa, lies within. We cannot merely rely on pressuring AI companies; communities must remain vigilant and proactive to ensure that the technologies they use and build genuinely adopt and serve local needs.

He sharply pointed out the fundamental flaw in the current AI development model: models are often built in places like Silicon Valley, trained using data from other regions, and then deployed in African markets like "plug-and-play" solutions. "But our realities are not plug-and-play. They are layered, nuanced, and rich in African context." He used names as an example; in Rwanda, names carry stories of family, place, struggle, and resilience, but an AI model trained solely on Western data may label these names as "anomalies" or even "high risk." "This is not a bug; it is a blind spot." Such blind spots exist globally, from US hospital AI systems prioritizing white patients to the high misidentification rates of people of color by facial recognition tools in the UK, and the quiet introduction of algorithms in border control, credit scoring, and policing in the Global South that lack local oversight, consent, and remedial measures.

Thus, Carl proposed that cultural barriers are more important than technical barriers. He called for a habit of "listening first, building later," especially in Africa. "Trust does not come from code; it comes from dialogue." He suggested that every AI team, whether in Nairobi or Silicon Valley, should continuously engage in dialogue with the people for whom they are building, rather than merely treating it as a checkbox in a pilot phase.

"Ultimately, the best safeguard is not policy or compliance, but humility. An AI without humility is not a human-centered AI." In response to Eva's question about "commercial pressures leading to ethical shortcuts," Carl quoted a Swahili proverb meaning "under despair or pressure, people will break the rules," including ethical rules. He pointed out the dark sides of an innovation culture characterized by "move fast and break things," exemplifying it with the story of "Samasource," revealing that data labelers for big tech brands like Meta and Scale AI in Kenya and Uganda receive meager pay and face violence and traumatic content for long periods, with no one protecting their mental health. "This is not a problem of a single company; it is an industry issue... If we don't audit the innovation's impact on end-users and the behind-the-scenes workers, we completely miss the point." He posed a soul-searching question: "If that were my child doing that job, could I accept it?"

Citizen AI: Returning Governance to Society

When discussing how to hold companies, researchers, and governments accountable for the impacts of AI on our lives, both Eva Navarro López and Carl Mabuka pointed their answers towards civil society itself. Eva questioned whether the term "responsible AI" might be contradictory, especially regarding her skepticism about whether generative technologies can truly be called "AI." "If it's all about mitigating risks and harms, is this the right technology? Or is it the wrong technology? Who decides that this is the future of AI? For whom? By whom? For what purpose?"

She believes that the solution lies within society and proposed the concept of "Citizen AI." Using Barcelona as an example, she introduced the "Citizen AI Protocol," whose principles include democratic regulation and oversight of AI, as well as civil society's participation in AI governance. This sounds like a dream, but it must begin. She asked, "Can we establish an international citizen alliance for AI governance? I believe this is the only way. Solutions will not come from the already broken academia, the equally broken companies, or the governments that are adopting and enforcing (like the agreement between OpenAI and the UK government)."

Carl Mabuka echoed this sentiment in his conclusion, pointing out the current greatest dilemma: "What's frightening is that there is no global referee." The AI competition is like a game without rules, with each team setting its own rules and the goalposts constantly moving. We need to break the cycle of merely asking and truly write, promote, and demand accountability to shape the future, "otherwise, AI will shape us in its current form."

To instrumentalize these ideas, Eva introduced a "Safe AI Toolkit" they created, which contains three pre-deployment questions, three non-negotiables for responsible AI (making bias and discrimination unacceptable, no "ghost workers," consent is not a checkbox), an AI barrier checklist, and five major "ethical excuses" or gimmicks that need to be loudly debunked, such as "Oh, this is just an algorithm," "I'm doing 'AI for Good,'" "We comply with all local laws," "It's not our fault, the users don't know how to use it," and "Ethics slow down innovation." She particularly highlighted that "AI for Good" itself is a dangerous signal, implying that AI is assumed to be evil.

Ultimately, this debate returned to the core belief Eva expressed in her opening statement: "Only when we change will AI change." This is not just a shift in the technical path, but a profound reflection and reconstruction of power structures, value priorities, and ways of human collaboration. Beyond the AI frenzy driven by capital and a privileged few, a force that emphasizes collective over individual, diversity over homogeneity, and ethics over expedience is gathering, attempting to inject a truly human perspective into technology.

免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。