Cloudflare today launched “Pay Per Crawl,” a landmark initiative to reshape the web’s economic model by empowering publishers against AI-driven content scraping. The company, which handles traffic for roughly 20% of the internet, will now block AI crawlers by default for new customers, according to its announcement. This allows content creators to charge AI companies for access, a direct response to the collapsing referral traffic that threatens the web’s foundational business model.
The move is a direct response to what Cloudflare CEO Matthew Prince has described as an “existential threat” to online publishers. In June 2025, Prince revealed stunning data showing a catastrophic collapse in traffic from AI search. For every 1,500 pages OpenAI’s crawlers scraped from a publisher, it sent only one visitor back—a dramatic decline from a 250-to-one ratio just six months prior. The data for other AI models was even more dire, with Anthropic’s ratio worsening to 60,000-to-one. This trend guts the ad-supported revenue streams that have sustained online content for decades.
Danielle Coffey, CEO of the News/Media Alliance, recently captured the industry’s frustration, stating “Links were the last redeeming quality of search that gave publishers traffic and revenue. Now Google just takes content by force and uses it with no return.”
To create a viable alternative, Cloudflare is proposing a new economic framework where content is directly compensated. This vision underpins the Pay Per Crawl system, which aims to provide a sustainable revenue stream for creators whose work is used to train and power large language models.
How Pay Per Crawl Works
At the heart of the initiative is the revival of the largely forgotten HTTP 402 “Payment Required” status code. The technical framework creates a machine-readable way for publishers to demand payment. When a registered AI crawler requests content, it can either receive a successful response or a 402 error that includes the price set by the publisher.
According to Cloudflare’s documentation, crawlers can interact with this system in two ways. In a “reactive” flow, a crawler makes a request and, upon receiving a 402 response, can retry with a header agreeing to the specified price. Alternatively, a “proactive” flow allows a crawler to include a `crawler-max-price` header in its initial request, signaling its willingness to pay up to a certain amount, which automates the transaction if the publisher’s price is within the budget. Cloudflare will act as the Merchant of Record, handling the transactions between AI companies and publishers.
The system is designed to give content owners full control over their monetization strategy. Publishers will have three distinct options for handling AI crawlers: allow free access, charge a configured domain-wide price, or block access entirely. This granular control allows creators to tailor their approach, whether they want to participate in the new economy, negotiate separate licensing deals, or simply keep AI bots out.
The initiative has already garnered significant backing from the publishing world, with companies like BuzzFeed, Fortune, TIME, Ziff Davis, and Search Engine Land’s parent company, Third Door Media, signing on as early adopters. Major media organizations have also voiced their support.
An Escalating Defense Against Scrapers
Pay Per Crawl is the culmination of a multi-year strategy by Cloudflare to arm publishers against unauthorized scraping. The company first launched a free, one-click tool to block known AI scrapers in July 2024, later expanding its capabilities and offering monitoring dashboards to all users in September 2024. These tools were a response to the widespread practice of AI bots ignoring the `robots.txt` protocol, a long-standing but unenforceable web standard.
In March 2025, the company escalated its tactics when it deployed “AI Labyrinth,” a sophisticated deception system. This tool lures non-compliant bots into a maze of fake, auto-generated content, designed to waste their resources and pollute their data sets while providing Cloudflare with valuable behavioral signals for its detection models.
Cloudflare CEO Matthew Prince has been vocal about the company’s determination to enforce these new rules, dismissing the idea that AI companies are too powerful to stop. He previously highlighted the company’s daily battles with state-sponsored hackers as evidence of its capabilities.
Ultimately, Cloudflare’s initiative represents one of the most significant attempts to date to rebalance the power between content creators and the AI giants that rely on their work. By creating a technical and financial framework for compensation, the company is betting it can safeguard the future of original content on a free and vibrant internet. The success of Pay Per Crawl could set a new standard for how AI interacts with and values the human-created information that fuels it.
