- Publisher Choice: Cloudflare’s new setting lets publishers keep Google and Apple search crawlers while requesting that their pages stay out of AI training.
- Bing Limit: Cloudflare’s setting does not yet send Bing a no-training instruction; a full crawler block would also cut off search.
- Payment Status: Cloudflare’s Pay Per Crawl remains in private beta, while CEO Matthew Prince’s wider payment plan is a proposal.
Cloudflare introduced a setting on September 15 that lets site owners keep Google and Apple search crawlers visiting their sites while sending the companies instructions not to use those pages to train AI models. For publishers that depend on search visitors, the change offers a narrower choice than turning away the same crawlers that help readers find their work.
Cloudflare calls it Disallow AI Training. Googlebot, Google’s web crawler, can gather pages for search and for other AI uses. The UK Competition and Markets Authority said in a January consultation that this overlap left publishers with too little choice over how Google used content gathered for search. A site that blocked Googlebot entirely could also lose its route into ordinary Google search results.
In a Decoder interview The Verge published on September 26, Cloudflare CEO Matthew Prince said the company could block Googlebot entirely if Google did not distinguish search from training. Cloudflare’s full Block option remains for a site owner who wants to stop the crawler, including from search.
Under Disallow AI Training, Cloudflare publishes a rule in the site’s robots.txt file, which tells a crawler operator that the pages should not be used for model training. It allows mixed-use search crawlers it designates Accountable to continue fetching pages, while blocking separate training crawlers. Accountable means the operator has the required controls or has committed to deliver them on a stated timetable. For Google and Apple, the training refusal depends on those companies honoring the published rule because Cloudflare still lets their search crawlers through.
Cloudflare offers Disallow AI Training as a changeable training preset for new ad-supported domains. New sites without ads are offered a preset that allows training. Existing sites that used Cloudflare’s older single switch for blocking AI bots move to Disallow, with search crawling still allowed. Sites that had configured search and training separately keep their search choice; a prior training block becomes Disallow.
Search Access Depends on the Crawler Operator
Google’s Google-Extended control is a robots.txt instruction against using pages to train future Gemini models. Googlebot can still fetch the pages for Search, and Google says the instruction does not change their inclusion or ranking there. Apple has a similar Applebot-Extended instruction. In both cases, Cloudflare can let the search crawler through while publishing the training preference for its operator to honor.
Bingbot still gets search access under Disallow AI Training, but Cloudflare cannot yet send Microsoft a no-training instruction through robots.txt. Cloudflare says Microsoft is targeting early 2027 for that support. Microsoft offers a separate page-level training control, which Cloudflare’s setting does not apply for the site owner. A full Block would stop Bingbot from visiting at all, including for search.
Payment Remains a Separate Project
Prince argued in the Decoder conversation that bots do not generate the ad clicks that help fund many websites. He proposed small payments when AI agents fetch pages and discussed work with Coinbase and Stripe on using HTTP 402, the web response for content that requires payment.
Cloudflare introduced Pay Per Crawl in 2025, allowing participating sites to set a price for crawler access. Its AI Crawl Control overview describes Pay Per Crawl as a private beta.
