Cloudflare is automatically blocking mixed-use web crawlers that simultaneously index sites for search engines and act as AI agents. This new default setting aims to give website owners more control over how their content is used, particularly for AI training. Previously an opt-in feature, Cloudflare now shifts to a more defensive stance, recognizing the prevalence of non-human traffic online.

Starting September 15, 2026, new Cloudflare customers and new sites will default to allowing search engine indexing but blocking AI training and agent use on pages with ads. This change impacts free accounts as well, requiring an opt-out before the deadline. The company is also updating its Pay Per Crawl feature to "Pay Per Use," compensating site owners when their content appears in AI chatbot answers.

This move rebalances the relationship between content creators and AI companies, addressing concerns that AI models can scrape data without consent or compensation. Cloudflare's announcement specifically targets companies like Google, whose crawlers often serve dual purposes for search and AI training without clear distinction for publishers. The company hopes this will encourage clearer intent from AI bots.