Patreon Stops Asking AI Bots Not to Scrape and Starts Blocking Them: Cloudflare Partnership Marks a Shift in How Platforms Defend Against AI Training

Patreon found that individual AI training crawlers were attempting to access its platform thousands of times per week even after it explicitly asked them to stop. Those numbers, revealed in the company's announcement on Thursday, capture exactly why the era of polite scraping defenses is over. Patreon is no longer asking AI bots not to scrape its creators' content. It is now actively blocking them, and it enlisted Cloudflare to do it.

The membership platform, which hosts paywalled content from over 250,000 creators, first put measures in place to deter AI crawlers in 2023. Those measures relied primarily on robots.txt files, the standard protocol that asks bots to follow rules about which parts of a site they can access. The problem, as Patreon discovered, is that robots.txt is an honor system. There is no enforcement mechanism. A bot that ignores the rules faces no consequences beyond the platform's ability to detect and block it after the fact.

Patreon's product chief Drew Rowny framed the shift plainly: "Creators deserve a meaningful say in how their work is used by AI companies. On most of the Internet, creators have to accept AI training on their work just to reach and grow an audience. Patreon has a different vision." The company is extending its existing relationship with Cloudflare to use the infrastructure provider's AI Crawl Control technology, which blocks scrapers at the network level before they can reach Patreon's servers.

The End of Robots.txt as a Viable Defense

Patreon's experience mirrors what every content platform is discovering. Robots.txt was designed for an internet where the primary bots were search engine crawlers that played by the rules. Those crawlers indexed content, linked back to the source, and drove traffic. The incentive alignment was clear: obey the rules, get access, send traffic, repeat.

AI training bots follow a different logic. They do not need to send traffic back to the source. They do not care about indexing for search results. They want content to train models, and the value of that content exists independently of whether the bot ever returns a human visitor to the site. The entire incentive structure of robots.txt collapses when the crawler has no reason to obey it.

Patreon's testing confirmed this directly. When the company activated Cloudflare's AI Crawl Control during testing, individual crawler attempts to access Patreon went from "thousands of attempts to zero." That is a data point worth sitting with. Thousands of attempts per week from individual crawler agents, all ignoring the robots.txt file that Patreon had in place since 2023. The scrapers were not confused. They were ignoring explicit instructions.

This is the reality that every platform with user-generated content now faces. Whether you run a forum, a marketplace, a publishing platform, or a community site, AI scrapers are hitting your servers and consuming your users' content for training. If your only defense is a robots.txt file, those scrapers are almost certainly ignoring it.

Cloudflare's Pay Per Crawl: A New Market for Training Data

The most interesting development in this story is not the blocking. It is what Cloudflare is building alongside it. The company now offers a "Pay Per Crawl" marketplace that lets websites charge AI companies for scraping access. Instead of a binary choice between open access and total blocking, Pay Per Crawl creates a third option: licensed access with a price tag.

Earlier this month, Cloudflare also changed its default policies so that mixed-use crawlers, those that both index for search and train AI models, are now blocked by default on any page that hosts ads. This policy change alone affects millions of sites running on Cloudflare's network, which handles roughly 20% of global web traffic. The collective effect is a rapid tightening of the access that AI companies have to training data from the open web.

For AI companies, the implications are significant. Training data that was freely available six months ago now has a paywall, a block, or a price tag attached to it. The cost of assembling a competitive training dataset is going up, and the number of sources that can be scraped without active defenses is shrinking. This is the beginning of a structural shift in how training data is sourced and priced.

What This Means for Founders Building Content Platforms

For founders running platforms where users create content, this story contains three actionable signals. First, robots.txt is dead as a scraping defense. If your platform hosts user-generated content and your only protection is a text file asking bots to behave, you are effectively unprotected. Implementing network-level blocking through Cloudflare, AWS WAF, or a dedicated bot management solution, should be a priority, not a nice-to-have.

Second, the Pay Per Crawl model opens a new revenue opportunity. If your platform has valuable content, you can now treat AI training access as a licensed asset rather than a leakage risk. Cloudflare's marketplace is the first infrastructure for this model, but it will not be the last. Founders should be thinking about their pricing strategy for AI training access before the market sets it for them.

Third, this shift creates a competitive advantage for platforms that move early. As more sites lock down, the remaining openly accessible training data becomes more valuable. But for platforms, being seen as a defender of creator rights against AI scraping is a positioning win. Patreon's announcement explicitly frames the company as standing with creators against extractive AI practices. That narrative has real brand value, especially as public awareness of AI scraping grows.

The Broader Pattern: AI Training Data Becomes a Controlled Asset

Patreon is not the first platform to make this shift, and it will not be the last. Reddit struck a $60 million licensing deal with Google for training access. Stack Overflow signed agreements with OpenAI and Google. The New York Times sued OpenAI over training data use. Each of these moves represents the same underlying trend: the era of free, frictionless training data access is ending.

What makes Patreon's move distinctive is that it combines blocking with a philosophical stance. Patreon is not just defending its own content; it is defending content that belongs to individual creators who use the platform. The company is effectively saying that creators should have a say in AI training, not just the platform they publish on. That is a different message from a licensing deal between two companies, and it resonates differently with the creator community that Patreon depends on.

For founders, the lesson is that the window for securing AI training data rights is closing. If your platform generates valuable content, the time to establish your policy, block, license, or some middle ground, is now, before the market decides for you. And if you are building an AI company that depends on web-scraped training data, budget for the fact that those costs are going up, and the access is getting harder.