(Click to listen to this episode👆)

Interview ｜ Yiwen, Hongjun

Text and Images ｜ Yiwen

In the recently submitted prospectus, Anthropic revealed nearly 100 pages of risk factors. In addition to common financial and market risks, it also included statements about the catastrophic risks posed by cutting-edge AI, which even threaten human survival. Anthropic stated that they cannot guarantee the effectiveness of their safety measures.

The prospectus also disclosed that after going public, the seven co-founders will control 50.1% of the voting rights for general matters through special shares; at the same time, there is an institution called the “Long-Term Interest Trust” that has the authority to elect a majority of board members. The goal of this arrangement is to ensure that the company considers humanity's long-term interests while pursuing commercial gains. However, media outlets have noted a detail: the founders can designate in advance who will replace their voting rights if they lose their ability to perform their duties. These candidates require internal approval, but the list is confidential. In other words, the public currently does not know who may participate in Anthropic's decision-making in the future.

This prospectus has once again highlighted the importance of AI safety to Anthropic. But at the same time, if their warnings come true, should we as the public have the right to know who can represent humanity in making decisions?

To understand Anthropic's emphasis on safety and the controversies surrounding trust and power, a recently recurring concept must be mentioned, which is Effective Altruism, abbreviated as EA.

This month, The Economist featured EA on its cover, calling it “the most significant idea of this century.” Some reports indicated that supporters of EA related to AI safety policies have clashed with the tech right mentioned in our podcast, and even the Trump administration has joined the debate — the U.S. Department of Defense directly tweeted a slogan: “Americanism, not effective altruism.” A concept that still sounds like a philosophical discussion has thus become the focus of the U.S. government's public delineation.

In this issue of "Silicon Valley 101," we have invited someone closely related to the EA circle: Ethereum founder Vitalik Buterin, and his friend, technology journalist Afra. Below is the first part of the interview, where Afra introduces the four stages of the development of effective altruism; the figure and affairs of its spiritual leader Will MacAskill; the differences in community temperament and action between EA and rationalism, e/acc (effective accelerationism); as well as how EA has dominated the AI safety narrative, gradually enlisting the smartest people in Silicon Valley.

Feel free to follow “Silicon Valley 101 Video Account” oraudio column to directly listen to this episode's podcast. If you enjoy our show, we highly recommend using an audio client to listen, as “Silicon Valley 101” is available on all major audio platforms (channels listed at the end of the article), and subscriptions are welcome!

The following is a selection of this conversation:

Yiwen: What topics do you think EA generally focuses on?

Afra: Actually, we need to divide the past decade or so of EA's global influence into roughly four stages, each with different focal points.

The first stage is the so-called “mosquito net era.” This stage focuses on using very detailed and quantifiable evidence to improve the efficiency of charitable efforts, influencing, for example, global health indices. It is a measurable good, supported by evidence and data. Because at the beginning of EA, many people wanted to do good and donate money through charity, but the efficiency of charity was very low, and their donations were not effectively utilized.

However, this group of people in EA wanted to use a quantifiable and scientific method to allocate their donations effectively and maximize their value. Thus, the origin of the mosquito net era — the mosquito net symbolizes the most effective method of “the amount you donate can save that many lives.”

Next, the next stage of EA is the so-called “long-term planning period.” This long-term planning period occurred around 2015, when this group began to disproportionately study existential risks and issues regarding future generations, and funding and institutions began to expand. They were no longer concerned about how many people a mosquito net could save, but rather about the quality of life and living standards for thousands or tens of thousands of future generations.

Thus, a significant amount of funding — including from SBF and many other wealthy individuals — entered the community during this stage. I think SBF might be the most extreme symbol of this era because he personally was very focused on long-termism issues.

I read an article about Will MacAskill in The New Yorker, and my judgment after reading it was: SBF might be one of the decisive factors for the entire EA’s turn towards long-termism.

Next came the age of community reputation collapse. A significant trust crisis emerged, prompting reflections on what issues the EA community faced. Meanwhile, ChatGPT was born, bringing about a turning point for AI — the previously discussed AI risks and survival issues among the rationalist community suddenly became concerns for a broader audience.

So this stage was marked by numerous frustrations but also significant exposure. It revealed many problems, such as Will MacAskill’s failures in judgment, the continuous criticism he faced, and his insufficient reflection on SBF, given that he had continually sheltered SBF.

However, this period was also one where EA truly became a philosophical school of thought that The Economist described as one of the most influential ideologies of the 21st century.

After that, we come to the AI safety era that we are now familiar with. I think during the AI safety era, the public began to hear the term “effective altruism” because AI safety became one of their core topics.

A significant number of members of the EA network entered frontier laboratories as employees, and large amounts of funding were allocated to AI safety, resulting in the intersection of the entire EA circle with policy circles, national security circles, and Washington circles. This also led, for example, to President Trump and the U.S. Department of Defense tweeting things like “We will definitely not become EA.”

I believe this current stage represents a new phase for EA, with new challenges, but it is also the time when it has entered public consciousness the most.

Hongjun: I wonder if you have recently noticed the relationship between Anthropic, some safety auditing agencies, and EA? After Jacob Coxon resigned from Anthropic in September, an independent researcher Kevin Bass analyzed the funding chains on X, looking at the independent safety auditing agency METR arranged by Anthropic: METR's funders, as well as other safety agencies, are also early investors in Anthropic — Dustin Moskovitz, founder of Coefficient Giving, holds a significant amount of Anthropic stock. Thus this so-called “independent evaluation” might actually be tied to Anthropic's commercial interests. This has led to numerous voices criticizing EA online, with many people beginning to question the funding sources of all safety agencies. What do you think about these criticisms?

Afra: After Jacob's tweet went viral, people linked all of EA's funding injections into AI safety and started weaving various conspiracy theories. For me, it is essentially a conspiracy theory.

Hongjun: Why do you think it's a conspiracy theory?

Afra: Many traditional media question: What are the purposes behind the flow of money to these organizations? What hidden agenda exists? I believe there is always some projection of agenda. But what is the agenda of these EA people? It is simply not wanting AI to become uncontrollable and cause consequences that humanity cannot bear. That is their agenda, and there is no hidden agenda.

Recently, former New York Times journalist and host of the Hard Fork podcast Kevin Roose wrote a book titled The AGI Chronicles, using some funding with EA backgrounds during the writing process. So many people questioned: Is the source of this money unclear? If one takes EA money, must their agenda be written into the book?

But I think the biggest question might be: When a thought movement has so much money, is it actually discovering the most important issues, or is it starting to participate in and decide what problems will be considered important?

We are now living in the consequences of that. They have been saying that AI safety is important and investing a lot of money into it for a long time; we are now living in that influence. I have seen some traditional think tanks in Washington struggle to recruit qualified AI policy analysts because young, capable talents have all flowed into EA-sponsored think tanks and organizations.

If you are a young researcher in your early twenties, just graduated from Stanford with a PhD wanting to study AI economics, you will find that the fields of AI alignment and AI safety have many researcher positions, a vast amount of research funding, and numerous research openings with decent salaries; these institutions just care about funding you and do not overly manage your research direction or require you to change beliefs. Other research institutions do not have so many resources or that much freedom. Hence, I have observed more and more young people being absorbed by these so-called “EA institutions” related to AI safety; I have also seen many exceptionally intelligent individuals become “safety-pilled” as a result of daily interactions and discussions with these people.

But in my view, being safety-pilled in this stage is quite normal. Even if you are not very concerned about AI safety issues, merely watching various ordinary news and television reports each day will also lead you to become safety-pilled.

Hongjun: I know that you often visit Berkeley and some EA organizations in San Francisco, or we can call them rationalist organizations. I have also heard that in San Francisco, like the METR organization we just mentioned — if you want to attend some events, say at Anthropic or with researchers, if you say “I am not an EAist,” they might respond — if you are not EA, many events in San Francisco might not even be open to you.

Afra: I find EA interesting. You ask some people, “Are you EA?” and they say, “Well, I’m not exactly EA, but I’m close to them.” No EAs are the ones who flaunt their EA identity every day. In fact, many of my EA friends are very EA, but they do not consider themselves EA members, which I find interesting.

Hongjun: What is the atmosphere like in Berkeley? I feel Berkeley is more inclined towards a rationalist community. I know there is an organization called Lighthaven where people often gather and hold events, which has a different atmosphere from San Francisco.

Afra: Yes, I think Berkeley has already formed its own system. There are many third-party auditing institutions for AI safety in Berkeley, or organizations like METR, and there are dozens of them, to be honest. Lighthaven is where Rationalists and some EA-sponsored events occur; it is a complex of about six or seven standalone houses forming an activity area in Berkeley.

There you can attend lectures, socialize freely, and roast marshmallows by the campfire at night. Lighthaven is viewed as a stronghold for AI safety, EA, and rationalism.

I sometimes go to Lighthaven to hold some AI-related meetings and meet people. Just two weeks ago, I returned from two events: one called The Curve and the other called the Progress Conference.

One is about AI's curve, exponential development related to cutting-edge AI research; the second event, the Progress Conference, advocates for societal progress, much like Ezra Klein's book Abundance, focusing on how to create a more abundant, future-oriented society using existing technologies. It discusses various cutting-edge scientific research topics, such as how to educate the next generation and develop greater confidence about the future.

Hongjun: Do you think these discussions, like pushing the progression of events, whether AI should accelerate development or slow down, hold significant influence on real outcomes?

Afra: I think it will. I remember that last year's The Curve, about cutting-edge AI, actually had many people from Washington and Congress, as well as many state legislators from California. The group dynamics brought about by The Curve were quite significant; it is a good Venn diagram of Washington and Silicon Valley. So I understand their influence.

Hongjun: And what about the rationalist community?

Afra: I think the rationalist community is a bit more laid-back. Rationalists don’t seek influence as much. So the people in the rationalist community still write blogs and research things they are interested in, without many ideas of the “4D Chess” claimed in Silicon Valley. They don’t have those calculations in their heads, like “Today I do A to achieve B, so I can ultimately reach goal Z, but how do I reverse-engineer this?”

Rationalists typically do not have grand plans; they are still quite charming, just as Vitalik mentioned, that Rationalists enjoy doing what they love, and many write fiction, blogs, or discuss very nerdy mathematical and AI problems.

Hongjun: I saw that you previously attended a program in London called GovAI Fellowship, where you met one of the spiritual leaders of EA we just mentioned, Will MacAskill. Did you have communication with them?

Afra: I first met Will MacAskill at The Curve. I spoke with him for about ten minutes, and I felt he encapsulated many of the ideological changes EA has undergone over the past decade. Gideon Lewis-Kraus from The New Yorker wrote an excellent profile about him in 2022. It described how Will, especially when founding these EA organizations, had a bit of the flavor of a prophet and ascetic. Gideon, as an author, mentioned that when you're around Will, there is a moral pressure to do good because Will's living standards are very simple, and he continually donates his income.

Philosopher, co-founder of the effective altruism movement William MacAskill Image source: Sam Deere / CC BY-SA 4.0 (Wikimedia Commons)

Hongjun: What do you think is his core influence on the entire EA circle?

Afra: He is considered a spiritual leader of EA and a person who established some core EA organizations at Oxford. When he was 22, he met an Australian professor, Toby Ord, who is also very focused on EA and long-termism.

The two clicked, and Will immediately committed to donating as much of his wealth as possible permanently. He was very willing to bind his future self to this idea, which is also a point of criticism by many: EA has an idea that you sign a contract with your richer future self to have them do what your current self wants to do.

I remember that Vitalik criticized this EA idea saying: Your future self will always betray your current self, because in 10 or 20 years, your thoughts will change, your behaviors will change, the friends around you, your circles will change, and what you want to do will change. How can you ensure that your future self, who becomes richer for doing good today, will stick to the promises made by today’s self? But that was Will at the time; he declared that he would donate all of his income.

I think there’s another background: Soon after EA was established, the Occupy Wall Street movement emerged. During the heat of Occupy Wall Street, the moral claim was that the Wall Street people were vampires, the evil ones causing many middle-class and underprivileged people’s lives to become more difficult, even destroying their lives.

However, during that time, Will gave a speech at the University of Oxford titled “Doctor, NGO worker, or what else? Which profession can benefit society more?” which helped Oxford graduates think about what profession they should pursue. That same year, he also founded the “80,000 Hours” project, aiming to provide some undergraduates with suggestions for “optimizing moral lives.”

He presented a viewpoint: For you elite students from Oxford, if you want to benefit this world, you can choose to be a doctor in a poor country, but over your entire medical career of 80,000 hours, how many people can you save? You might save around 140 lives. Or you can join "very evil Wall Street" or "evil consulting," and through wise donations, you might save ten times the number of lives than if you worked as a doctor in a poor country.

Because you can donate that much money, you can buy many mosquito nets, and each batch of mosquito nets might save numerous people. Therefore, when Wall Street was seen as the most morally tainted, notorious, and unattractive to elite students, some people listened to Will's moral suggestions and ventured into Wall Street.

I believe that the reason effective altruism attracted SBF and some traders from Jane Street is also that they are trying to use EA to bridge a certain moral dissonance. They often ask themselves: Is doing this really meaningful? Is making crazy, continuous quantitative trades, using leverage and various measures to make money — am I really doing well? However, if I donate it all, then what I do is good, right?

The claims of EA indeed provide an excuse for many people maximizing interests on Wall Street, that is, "I am here to do good." This excuse is something that can easily be manipulated and exploited by many.

In August 2015, EA Global was held at Google's headquarters, where Elon Musk discussed “AI existential risks” with Nick Bostrom, Stuart Russell, and others in a sub-forum. Image source: Robbie Shade / Wikimedia Commons / CC BY 2.0

Then in 2015, Will MacAskill was appointed as the youngest associate professor of philosophy at the University of Oxford, at only 28 years old. That summer, his 80,000 Hours project was also accepted by Y Combinator; the global summit of effective altruism (EA Global) was held at Google's headquarters, attended by many Silicon Valley billionaires, including Elon Musk for discussions. I think for Will MacAskill, it might have been a peak, because EA's philosophical ideas had already permeated the wealthiest, most affluent, and resource-rich group in Silicon Valley capable of driving society in some direction.

Hongjun: I noticed that EA originated from the University of Oxford, but its subsequent development, including you just mentioned, that this year Will MacAskill has moved to Silicon Valley — what does this indicate?

Afra: I think it indicates that the core claims of EA and what it wants to achieve has completely transformed into AI safety, which is quite different from before. Previously, issues such as preventing global diseases, improving the efficiency of doing good, and attending to other forms of existential risks — have almost disappeared. Essentially, all of EA’s focus and energy has now been disproportionately placed on AI safety.

Hongjun: I feel that EA is also chasing trends.

Afra: I think the trends are created by EA. I believe we are living in a state that EA already planned out many years ago. EA had announced long ago: We intend to spend this much money and create this much impact; we are now going to maximize our influence. And we are currently living in the world crafted by EA.

Hongjun: Would you say you are an EA?

Afra: I wouldn’t because I genuinely feel that I am not. I care about AI safety, but I’m not EA, for instance, I don’t donate money. I believe the things I’m doing right now do not reach the level where I consider, “Globally, what good should I do?” I am most interested in the philosophy of EA and their intellectual world, and I enjoy keeping track of these quirky groups.

Hongjun: Currently, the mainstream EA stance is to slow down AI development as much as possible, whereas e/acc (effective accelerationism), represented by a16z, advocates for acceleration. So do you think we should slow down AI development due to safety concerns?

Afra: First, I want to respond to the tension you just articulated between e/acc and EA. The e/acc people actually have a very unbalanced relationship with EA: e/acc has not formed a real power. The so-called e/acc people consider themselves effective accelerationists, yet it is not a very clear phenomenon — for many, e/acc is a meme, a topic tag online, it’s not a real action force. In contrast, EA is a genuine movement.

Secondly, regarding AI safety — after attending The Curve and the Progress Conference, I also chatted with over a dozen researchers from OpenAI and Anthropic, I feel I’ve gained a lot of new insights concerning AI safety. I have become more inclined to trust that the current RSI (recursive self-improvement) evolution curve is something that is a bit worrying to me.

Several researchers told me that The progress in AI over the past year has even exceeded their original expectations. I think the growth of frontier AI poses a serious epistemological problem. Thus, I believe that slowing down AI development, especially slowing down RSI development, might be a good thing — to give everyone time to sit down and discuss: what exactly should we do? What has happened? Can we clarify these matters in simple terms before proceeding with this frantic AI race?

The above is the first part of this interview, through Afra's perspective, we learned how EA has evolved from a discussion on “how to do good more effectively” to significantly influencing today's AI research, funding flows, and even political agendas. The thoughts that have generated today's impact are backed by substantial funding support.

In the second part of the interview, Silicon Valley 101 dialogues with a key donor — Ethereum co-founder Vitalik Buterin. He shares his journey from supporting and funding EA to beginning to question and reflect on it, analyzing different ideological camps' critiques of Anthropic's AI safety philosophy, and offering his thoughts on how humanity should respond to unknown AI risks. 'Dialogue with Vitalik: How to Start Reflecting on EA After Donating Hundreds of Millions?' will be published tomorrow, stay tuned.

免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到[email protected]，本平台相关工作人员将会进行核查。