On March 5, 2026, the movie data website The Numbers suddenly went offline.

For a repository that had been running for 30 years, with about 2 million pages and accumulated data on 78,396 movies and 236,176 people, this was not a routine maintenance shutdown. A week later, on March 13, the website returned online in a simplified version, but historical charts, individual movie detail pages, and the core Report Builder feature were all removed.

According to a report by Stephen Follows quoting founder Bruce Nash, the direct cause of all this was that AI crawlers and agent traffic accounted for 90% of its total traffic, leading to constant server overload and crashes. Meanwhile, system logs also recorded malicious attacks targeting backdoors—the motive for the attacks may be related to predictive markets, as platforms like Polymarket use The Numbers' data as a settlement basis, and obtaining data in advance means a trading advantage. However, the specific technical details of the attacks and the identity of the attackers have not been disclosed to date, and Bruce Nash himself has not provided further elaboration on this matter.

When human visitors became zero, traffic turned from an asset into a liability. A core question surfaced: Do traditional websites face a reconstruction like The Numbers, and where is the boundary for this judgment?

Amputation for Survival: An 8-Day Power Outage History of a 30-Year Data Site

The Numbers is not a startup; it is a living fossil.

On October 17, 1997, Bruce Nash created this website on Geocities. At that time, the internet was still a wasteland, with very few web pages and search engines just starting out. The term "crawler" had not yet been assigned its today’s anxiety-inducing meaning. The Numbers grew from a personal project to an industry tool—it tracked box office, recorded release data, and provided authoritative numerical references for filmmakers and the media.

Before the onslaught of AI crawlers, The Numbers had an annual visitor count exceeding 8 million. For a vertical data website, this was quite a considerable scale. Its database held information on 78,396 movies, 178,375 release records, and 236,176 people, with a total of approximately 2 million pages and about 160,000 source files. These figures were valuable data assets in the era of human browsing, and were the core competitive advantage of the website.

But during those 8 days in March 2026, this longstanding site experienced a power outage-style survival crisis.

Bruce Nash revealed that in the old system era, the team spent 90% of their time maintaining the website's operation rather than developing new features or optimizing data. This figure alone indicates many issues: a system that has been running for 30 years has accumulated technical debt to a suffocating degree. The 160,000 source files represent countless dependencies, outdated frameworks, forgotten scripts, and "black box" modules that no one dared to touch. Fixing one bug could trigger three new crashes, and adding one feature could collapse an unknown subsystem.

When the peak traffic from AI crawlers arrived, this old architecture appeared utterly defenseless.

Here, a key technical detail needs to be understood: the architecture of traditional data-centric websites is generally designed for human browsing. A user visits a page, the server renders HTML once, and returns a response. After staying for a few seconds, the user leaves. In this mode, the server's concurrent pressure is manageable because human browsing has a natural "throttling" mechanism—people need time to read and will not request hundreds of pages in a second.

But AI crawlers do not have this throttling. They operate at machine speed, launching massive requests in a very short time, without needing "reading" time, without waiting for page rendering to complete, and even without adhering to reasonable request intervals. When a legacy architecture designed for human browsing encounters exponentially sourced requests launched at machine speed, the result is continuous server overload, exhaustion of the database connection pool, spiking response times, and ultimately crashes.

The Numbers’ old system, under this impact, had no room for repairs. You could optimize one query, but among the 160,000 source files, there may be countless inefficient queries hidden; you could add another server, but the old architecture might not support horizontal scaling; you could add a cache, but the cache hit rate for 2 million pages is extremely low in a long-tail distribution. The marginal costs of patching the old system far exceeded those of rebuilding on a new infrastructure.

The returned simplified version of the website was not a functional upgrade, but an amputation-style contraction. Cutting off historical charts and detail pages meant abandoning a substantial amount of long-tail content—those pages are precisely the targets that AI crawlers prefer to scrape because they contain structured, consumable data. Removing Report Builder meant eliminating the core paid tool attribute, sacrificing the product's commercial value to ensure the survival of the infrastructure.

This resolute scaling back reflects the cruel reality that the old architecture had no repair value in the face of new types of traffic. Abandoning 30 years of accumulation itself serves as the best sample for judging "the necessity of reconstruction."

1:38000 Plunder: The Broken Traffic Pact

To understand why The Numbers was overwhelmed, one must recognize the essential behavioral logic differences between AI crawlers and traditional search engine crawlers.

In the past internet ecosystem, there existed an unspoken value exchange pact between websites and search engines: websites allowed search engines to crawl their content, and in turn, search engines brought human visitors to those websites through search results. This pact was based on a relatively reasonable "exchange ratio." According to Cloudflare data, Google, for every 1 human visitor sent, would crawl about 5 pages. This ratio implies that the bandwidth cost paid by websites for allowing crawling could be monetized through the human traffic that flowed back from search engines—through ads, subscriptions, or brand exposure. This was a calculable account.

However, the emergence of AI crawlers completely broke this balance.

Also according to Cloudflare data, OpenAI sends over 1000 pages for every 1 visitor; while Anthropic's exchange ratio exceeds a staggering 1:38000. What does that mean? It means an AI crawler consumes your server resources while offering almost no equivalent human traffic in return. It takes your data for training models or generating search summaries, but users eventually consume that data not on your website—they ask questions in ChatGPT, view summaries in AI search engines, and your website becomes an invisible underlying supplier in the data supply chain.

This one-sided plunder-style scraping directly leads to an inverted traffic structure.

According to Cloudflare Radar data, as of June 2026, bots accounted for 57.5% of HTML page requests, with human traffic becoming a minority. HUMAN Security's 2026 report even pointed out that agent traffic grew by 7851% in 2025, AI-driven traffic as a whole grew by 187%, and the growth rate of automated traffic was 8 times that of human traffic. Thales' Bad Bot Report also provided similar judgments: in 2025, bots accounted for 53% of global web traffic.

These numbers point to a common conclusion: the main body of internet traffic has shifted from humans to machines.

In the case of The Numbers, 90% of its traffic came from machines. These machine visitors do not click on ads, do not purchase subscriptions, and do not generate any commercial value, yet they genuinely consume bandwidth and computational power. The traditional business logic of "allowing crawling in exchange for traffic" has completely failed against the 1:38000 exchange ratio. The foundational survival of data-centric websites—attracting human traffic with free content, and monetizing that human traffic— is being shaken by this unequal traffic structure.

What’s even more alarming is that this change in traffic structure is not temporary. As more AI applications rely on real-time data scraping—AI search, AI assistants, AI agents—the proportion of machine traffic will continue to rise. According to Cloudflare data, in the first half of 2026, 52.3% of AI crawler requests were used for training, 34.2% for mixed purposes, 10.1% for search, and only 2.6% were user-triggered. This means that the vast majority of AI traffic is "actively scrapped" rather than "passively responded," and it will not decrease due to a reduction in human users.

The Unaccounted Cost of Servers: The Real Cost of Machine Visitors

How much does machine traffic actually cost? The Numbers has not disclosed specific figures for bandwidth or server costs, but we can glean some insight from other similar cases.

The open-source documentation hosting platform Read the Docs once disclosed a shocking number in an official blog: a single AI crawler brought them 73TB of bandwidth consumption in just one month, costing over $5000. For a non-profit organization, this is an additional expense that could threaten its survival. Even more infuriating was that Read the Docs found that these crawlers did not always comply with the rules laid out in robots.txt—they could ignore the website's crawling limits and obtain data at their own pace and way.

The Wikimedia Foundation's situation is even more typical. According to its official blog, the bandwidth demand for multimedia content had increased by 50% since early 2024, and 65% of that costly traffic came from bots, despite these bots accounting for only 35% of page views. This means Wikipedia is paying disproportionately high infrastructure costs for machine visitors. Even more severe is that due to the popularity of AI search summaries, users increasingly do not need to click into Wikipedia pages to get answers—Wikipedia's human traffic decreased by 8% year-on-year between May and August 2025. Traffic costs are rising while the human traffic that can be monetized is declining, and this disparity is devouring content platforms' profit margins.

Wikipedia's situation reveals a deeper issue: AI is not only consuming the website's bandwidth but is also cutting off the human traffic sources. When AI search engines provide answer summaries directly on the search results pages, users have no reason to click on the original link anymore. This means that not only is the website paying bandwidth fees for AI crawlers, but also that it has lost the human visitors that would have flowed back due to the existence of AI search summaries. It is a double blow: rising costs, and declining revenues.

For small teams, the situation is even more desperate. Drew DeVault, the founder of the code hosting service SourceHut, angrily stated in his blog that he spends 20% to 100% of his operational time each week combating crawlers, causing his website to experience dozens of brief outages each week. His anger is direct and real: "Stop externalizing your costs onto me." This is not a technical issue but a fairness issue—AI companies are shifting the cost of data scraping onto content producers.

LWN editor Jonathan Corbet even directly characterized this crawler traffic as "DDoS attacks." Triplegangers, a nine-person e-commerce company, once experienced a website crash due to OpenAI's GPTBot scraping wildly during business hours, and its CEO stated this was "basically a DDoS attack." iFixit recorded that Anthropic's ClaudeBot initiated nearly 1 million requests in a single day against its website.

These cases demonstrate that machine traffic is not a virtual number; it is real bandwidth costs, real CPU usage, and real operational labor. When The Numbers' servers were continuously overloaded and crashed under 90% machine traffic, Bruce Nash was facing not a technical optimization issue but an incomprehensible survival crisis.

A rough estimation could be made: if The Numbers’ annual visitor volume exceeds 8 million and machine traffic accounts for 90%, then the machine request volume must be at least in the tens of millions or even higher (considering the crawling depth far exceeds human browsing). Even if the cost per request is very low, the accumulated bandwidth and computational power consumption would be enough to leave an independent team without enterprise-level infrastructure unable to survive. Bruce Nash chose to abandon the old system not because he didn't understand technology, but because under the old cost structure, the website was no longer sustainable.

Which Websites Are the Most Vulnerable? The Judgment Boundaries for the Necessity of Reconstruction

The fall of The Numbers does not mean that all websites must immediately reconstruct. The necessity of reconstruction depends on the type of website and its traffic structure.

The most vulnerable are data-intensive websites like The Numbers. They typically have several common characteristics.

First, they have many long-tail pages. The Numbers has 2 million pages, and each page is a target that AI crawlers can scrape. The more long-tail pages there are, the larger the crawling surface for crawlers, which means the more requests the server needs to respond to. A corporate website with only a few dozen pages, even if targeted by crawlers, won't generate significant bandwidth pressure.

Second, the content is easily scraped statically. The Numbers’ data pages are accessible without logging in, allowing crawlers to directly access structured box office data, actor information, and release records. This "naked" data exposure is the ideal data source for AI crawlers.

Third, they are highly dependent on monetizing human traffic. The Numbers’ business model is built on the logic of "attracting human visitors with free content → human visitors generating ad revenue or paid conversions." When 90% of the traffic turns into machine traffic, this business logic becomes ineffective—machines do not click on ads and do not buy subscriptions.

Box Office Mojo and IMDb, though similar movie data websites, have risk resistance significantly higher than the independently operated The Numbers. After being acquired by Amazon, Box Office Mojo is supported by enterprise-level infrastructure and can withstand traffic impacts far beyond independent sites; IMDb, also under Amazon, has login walls and paid tiers (IMDbPro) in which some core data is hidden behind authentication, preventing crawlers from scraping indiscriminately as they could with The Numbers.

Relatively safe are several types of websites.

Interactive websites, such as social platforms, have core content generated from user interactions, and a large amount of data is hidden behind logins, making it difficult for crawlers to acquire on a large scale. Even if crawlers scrape publicly available pages, they cannot replicate the dynamic networks of social relationships.

Real-time service websites, such as e-commerce and SaaS products. E-commerce websites have stock and price changes in real-time, making the data value of static scraping very limited and lacking timeliness; the core value of SaaS products lies in services rather than data, preventing crawlers from easily replicating their business logic and backend integrations.

Websites with paywalls or authentication mechanisms. Login walls are naturally a form of traffic filter, whereby unauthenticated crawlers can only access limited public content, and cannot conduct indiscriminate scraping as they can with The Numbers.

Therefore, assessing whether a website faces reconstruction pressure hinges on two key indicators: data exposure and dependence on traffic monetization. If a website's data can be easily scraped (no login required, static content, many long-tail pages), and its business model highly relies on the human traffic brought by this data (advertising or free to paid conversion), then it is the next target for AI crawlers. For such websites, reconstruction is not just a technical architecture upgrade, but a rewriting of the business logic: how to find new cost-sharing and value monetization mechanisms in a world dominated by machine visitors.

On the other hand, if a website has small data exposure (behind login walls, real-time changes, interactive generation), or its business model does not depend on monetizing human browsing (B2B services, API licensing, enterprise contracts), then it does not need to undergo an amputation-style reconstruction like The Numbers for the time being. What it needs to do is basic traffic monitoring and protection, but not to start over.

Defense Lines and Solutions: A Path for Website Owners

In the face of the wave of AI crawlers, what can website owners and developers do?

The most instinctive response is to modify the robots.txt. But in reality, this layer of defense has become a hollow shell. The case of Read the Docs shows that many AI crawlers can directly ignore the rules set out in robots.txt. The HUMAN Security report also pointed out that a significant number of crawlers even disguise themselves as legitimate identities to evade detection. According to Cloudflare data, currently only 7.9% of AI crawler requests are proactively denied by servers with a 403 status code. Solely relying on application-layer rules can no longer prevent the infrastructure-layer traffic impact.

The failure of robots.txt is fundamentally because it is a kind of "gentlemen's agreement"—it relies on crawler operators' self-discipline to comply. In the era of search engines, companies like Google had commercial incentives to comply with robots.txt because they needed to maintain good cooperative relationships with websites to secure a continuous supply of content. But AI companies have different motivations: they need as much data as possible to train models, and complying with robots.txt means giving up some data sources, which is a competitive disadvantage in the fierce model race. When the cost of complying with rules exceeds the cost of breaking them, the gentlemen's agreement collapses.

Real responses must start from the infrastructure layer.

In July 2025, Cloudflare launched a Pay-per-Crawl mechanism, allowing websites to charge for crawling by AI crawlers. According to their official announcement, starting September 15, 2026, Cloudflare will default to blocking unpaid "mixed-use" crawlers. This represents an important industry-level attempt to rebuild the value exchange mechanism—if AI companies need data to train their models or provide search summaries, they must pay for the bandwidth and computational costs involved in retrieving that data. The practical effect of this mechanism remains to be seen, but it represents the correct direction: transferring the cost of machine traffic from content producers back to data consumers.

For independent website owners and small startup teams, the response paths can be divided into several levels.

The first level is identification. By analyzing logs, clearly define the share and source of AI traffic, without overly simplifying server crashes to mere business growth. Many website owners respond to site slowdowns or crashes by thinking of upgrading servers or optimizing code, but if the underlying cause of excessive machine traffic is not identified, server upgrades merely delay the collapse at a higher cost without addressing the root cause. In The Numbers case, Bruce Nash realized that 90% of the traffic came from machines, and this recognition itself was the starting point for decision-making.

The second level is isolation. Utilizing edge computing and WAF (Web Application Firewall) rules to direct machine traffic to static caches or lightweight responses, thereby avoiding direct impacts on the database and dynamic rendering logic. For data-intensive websites, this means pre-rendering high-frequency request pages into static files, providing crawlers with cached HTML rather than real-time query results. This does not resolve the fundamental problem but can buy time for reconstruction.

The third level is reconstructing the business model. If 90% of the visitors are no longer human, then the advertising model based on human browsing must be adjusted. Possible paths include: paid API access (making AI companies pay for structured data interfaces), data licensing (directly licensing data as a product to AI companies), stronger login walls (moving core data behind authentication to reduce exposure). These directions are not simple technical adjustments, but redefinitions of the website's business model.

The Numbers paid the price of abandoning its 30-year-old system to sound the alarm for the entire industry. In the new era where AI is the dominant force of traffic, traffic is no longer simply an asset; it may become a liability that needs careful accounting. The reconstruction of traditional websites is no longer a question of whether to proceed, but how to find new survival boundaries after accounting for this change. For data-intensive, human traffic-dependent independent websites, The Numbers’ today may very well be their tomorrow. Meanwhile, for those websites with small data exposure and business models that do not rely on browser volume monetization, what they need to do now is remain vigilant, continuously monitor changes in traffic structure, and proactively lay out defenses before machine traffic truly becomes a burden.

The main body of internet traffic has already changed. It is time for website owners to reassess their accounts.

免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。