Combining open-source AI with cryptographic technology can support the development of larger-scale models and drive more innovation, thereby creating more advanced AI systems.

Author: Daniel Barabander

Compiled by: Deep Tide TechFlow

Summary

• The current development of foundational AI is dominated by a few tech companies, characterized by closed systems and a lack of competition.

• While open-source software development is a potential solution, foundational AI cannot operate like traditional open-source projects (e.g., Linux) due to a "resource problem": open-source contributors not only need to invest time but also bear computational and data costs that exceed individual capabilities.

• Cryptographic technology has the potential to solve this resource problem by incentivizing resource providers to participate in foundational open-source AI projects.

• Combining open-source AI with cryptographic technology can support the development of larger-scale models and drive more innovation, thereby creating more advanced AI systems.

Introduction

According to a survey conducted by the Pew Research Center in 2024, 64% of Americans believe that the impact of social media on the country is more harmful than beneficial; 78% say that social media companies have too much power and influence in politics; and 83% think these platforms are likely to deliberately censor political views they disagree with. Discontent with social media has almost become one of the few consensus points in American society.

Looking back at the development of social media over the past 20 years, this situation seems to have been inevitable. The story is not complicated: a few large tech companies captured users' attention and, more importantly, controlled user data. Although there was initial hope for data openness, these companies quickly changed their strategies, leveraging data to establish unbreakable network effects and closing off external access. This ultimately led to the current situation: fewer than 10 large tech companies dominate the social media industry, creating an "oligopoly." Because the status quo is extremely favorable to them, these companies have little incentive to change. This model is closed and lacks competition.

Today, the trajectory of AI technology development seems to be repeating this scenario, but the implications are even more profound. A few tech companies have built foundational AI models by controlling GPU and data resources, while shutting off access to these models. For new entrants without billions of dollars in funding, developing a competitive model is nearly impossible. The computational costs of training a foundational model alone can reach billions of dollars, and those social media companies that benefited from the last wave of technological advancement are using their control over proprietary user data to develop models that competitors find hard to reach. We are retracing the steps of social media, heading towards a closed and competitive-free AI world. If this trend continues, a few tech companies will have unrestricted control over access to information and opportunities.

Open-source AI and the "Resource Problem"

If we do not want to see a closed AI world, what are our options? The obvious answer is to develop foundational models as* open-source software* projects. Historically, we have seen countless open-source projects successfully build the foundational software we rely on daily. For example, the success of Linux proves that even core software like operating systems can be developed through open-source methods. So, why can't LLMs (large language models) do the same?

However, the unique constraints faced by foundational AI models set them apart from traditional software, significantly undermining their viability as traditional open-source projects. Specifically, foundational AI models require enormous computational and data resources, which far exceed individual capabilities. Unlike traditional open-source projects that rely solely on people donating time, open-source AI also requires individuals to contribute computational power and data resources, which is known as the "resource problem."

Taking Meta's LLaMa model as an example, we can better understand this resource problem. Unlike competitors like OpenAI and Google, Meta did not hide the model behind a paid API but instead publicly provided the weights of LLaMa for anyone to use for free (with certain restrictions). These weights contain the knowledge the model learned during Meta's training process and are necessary to run the model. With these weights, users can fine-tune the model or use the model's output as input for new models.

While Meta's release of LLaMa's weights is commendable, it cannot be considered a true open-source software project. Meta controls the training process behind the scenes, relying on its own computational resources, data, and decisions, unilaterally deciding when to make the model available to the public. Meta did not invite independent researchers or developers to participate in community collaboration because the resources required to train or retrain the model far exceed the capabilities of ordinary individuals. These resources include tens of thousands of high-performance GPUs, data centers to store these GPUs, complex cooling facilities, and trillions of tokens (the text data units needed for model training). As noted in the Stanford University 2024 AI Index Report, "the sharp rise in training costs has effectively excluded universities, which have traditionally been strongholds of AI research, from the development of top foundational models." For example, Sam Altman mentioned that the cost of training GPT-4 reached $100 million, and this does not even include capital expenditures for hardware facilities. Additionally, Meta's capital expenditures increased by $2.1 billion in the second quarter of 2024 compared to the same period in 2023, primarily for servers, data centers, and network infrastructure related to AI model training. Therefore, while community contributors to LLaMa may have the technical ability to improve the model architecture, they lack sufficient resources to implement these improvements.

In summary, unlike traditional open-source software projects, open-source AI projects not only require contributors to invest time but also necessitate that they bear high computational and data costs. It is unrealistic to rely solely on goodwill and volunteer spirit to incentivize enough resource providers. They need further incentive mechanisms. Taking the open-source large language model BLOOM as an example, this model with 176 billion parameters brought together the efforts of 1,000 volunteer researchers from over 70 countries and 250 institutions. While BLOOM's success is admirable (I fully support it), it took a year to coordinate a single training session and relied on €3 million in funding provided by French research institutions (not including the capital expenditures for the supercomputer used to train the model). Relying on a new round of funding to coordinate and iterate the BLOOM process is too cumbersome to compete with the development speed of large tech labs. More than two years have passed since BLOOM's release, and there have been no reports of the team developing any follow-up models.

To make open-source AI possible, we need to find a way to incentivize resource providers to contribute their computational power and data resources, rather than letting open-source contributors bear these costs on their own.

Why Cryptographic Technology Can Solve the "Resource Problem" of Foundational Open-source AI

The core breakthrough of cryptographic technology lies in making high-resource-cost open-source software projects possible through an "ownership" mechanism. It addresses the resource problem of open-source AI by incentivizing potential resource providers to participate in the network, rather than requiring open-source contributors to bear these resource costs upfront.

Bitcoin is a great example. As one of the earliest cryptographic projects, Bitcoin is a completely open-source software project, with its code publicly available from the start. However, the code itself is not the key to Bitcoin. Simply downloading and running Bitcoin node software to create a local blockchain has no practical significance. The true value of the software is realized only when the computational effort to mine blocks far exceeds the capabilities of any single contributor: maintaining a decentralized, uncontrolled ledger. Similar to foundational open-source AI, Bitcoin is also an open-source project that requires resources beyond individual capabilities. While the reasons for the computational resource demands differ—Bitcoin requires computational resources to ensure the network's immutability, while foundational AI needs computational resources to optimize and iterate models—the commonality is that both rely on resources that exceed individual capabilities.

The "secret" that allows Bitcoin and any other cryptographic network to incentivize participants to provide resources for open-source software projects lies in offering network ownership through tokens. As Jesse described in the founding principles he wrote for Variant in 2020, ownership provides a powerful incentive for resource providers to contribute resources in exchange for potential gains within the network. This mechanism is similar to how startups address early funding shortages through "sweat equity"—by compensating early employees (such as founders) primarily in the form of company ownership, startups can attract labor they otherwise could not afford. Cryptographic technology extends the concept of "sweat equity" from focusing on time contributors to resource providers. Therefore, Variant focuses on investing in projects that leverage ownership mechanisms to build network effects, such as Uniswap, Morpho, and World.

If we want open-source AI to become a reality, then the ownership mechanism enabled by cryptographic technology is the key solution to the resource problem. This mechanism allows researchers to freely contribute their model design ideas to open-source projects, as the computational and data resources required to implement these ideas will be borne by resource providers, who will receive partial ownership of the project in return, rather than requiring researchers to shoulder the high upfront costs themselves. In open-source AI, ownership can take various forms, but one of the most anticipated is ownership of the model itself, which is also the solution proposed by Pluralis.

The approach proposed by Pluralis is known as Protocol Models. In this model, computational resource providers can train specific open-source models by contributing computational power and thus gain partial ownership of the future inference revenue of that model. Since this ownership is tied to a specific model and its value is based on the model's inference revenue, computational resource providers are incentivized to choose the optimal model for training, rather than fabricating training data (as providing useless training would directly reduce the expected value of future inference revenue). However, a key question is: how does Pluralis ensure the security of ownership if the training process requires sending the model's weights to the computational providers? The answer lies in using "Model Parallelism" technology to shard the model and distribute it to different workers. An important feature of neural networks is that even if a contributor only understands a tiny portion of the model weights, they can still participate in training, ensuring that the complete set of weights cannot be extracted. Additionally, since many different models will be trained simultaneously on the Pluralis platform, trainers will face a large number of different weight sets, making it extremely difficult to reconstruct the complete model.

The core idea of Protocol Models is that these models can be trained and used, but cannot be fully extracted from the protocol (unless the computational power used exceeds the resources required to train the model from scratch). This mechanism addresses a common criticism of open-source AI, which is that closed AI competitors might steal the labor of open-source projects.

Why Cryptographic Technology + Open Source = Better AI

At the beginning of the article, I illustrated the ethical and regulatory issues of closed AI by analyzing the control that big tech companies have over AI. However, in an era of pervasive powerlessness on the internet, I worry that such arguments may resonate less with most readers. Therefore, I want to present two reasons, based on practical effects, explaining why cryptographically supported open-source AI can truly lead to better AI.

First, the combination of cryptographic technology and open-source AI can coordinate more resources, thereby driving the development of the next generation of foundational models. Research shows that increases in both computational power and data resources help improve model performance, which is why the scale of foundational models has been continuously expanding. Bitcoin has demonstrated the potential of combining open-source software with cryptographic technology in terms of computational power. It has become the largest and most powerful computing network in the world, with a scale far exceeding the cloud computing resources owned by big tech companies. The uniqueness of cryptographic technology lies in its ability to transform isolated competition into collaborative competition. By incentivizing resource providers to contribute resources to solve common problems, rather than fighting alone and duplicating efforts, cryptographic networks achieve efficient resource utilization. Open-source AI powered by cryptographic technology will be able to leverage global computational and data resources to build models that far exceed those of closed AI. For example, Hyperbolic has already demonstrated the potential of this model. They have created an open market that allows anyone to rent GPUs at a lower cost, thereby fully utilizing distributed computing resources.

Second, the combination of cryptographic technology and open-source AI will accelerate innovation. This is because once the resource problem is solved, machine learning research can return to its highly iterative and innovative open-source nature. Before the emergence of foundational large language models (LLMs), researchers in the field of machine learning typically publicly released their models and their replicable design blueprints. These models often used open-source datasets and had relatively low computational demands, allowing researchers to continuously optimize and innovate based on them. It was this open iterative process that led to numerous breakthroughs in the field of sequence modeling, such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and attention mechanisms, ultimately making the Transformer model architecture possible. However, this open research approach changed after the launch of GPT-3. OpenAI demonstrated through the success of GPT-3 and ChatGPT that with enough computational resources and data, it is possible to train large language models with language understanding capabilities. This trend has led to a sharp increase in resource thresholds, gradually excluding academia, while big tech companies, in order to maintain competitive advantages, no longer publicly disclose their model architectures. This situation limits our ability to push the forefront of AI technology.

Open-source AI enabled by cryptographic technology can change this status quo. It can allow researchers to iterate on cutting-edge models again, discovering the "next Transformer." This combination not only addresses the resource problem but also reactivates the innovative vitality in the field of machine learning, paving the way for broader future developments in AI.

免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。