One year after the first Content Independence Day, Cloudflare revisited its AI traffic strategy. The original “Block AI Bots” option and Pay‑Per‑Crawl marketplace gave site owners a way to resist crawlers that took content without returning value. Feedback showed a need for finer‑grained controls, prompting a shift from a binary block to a nuanced taxonomy.

Today the platform lets owners manage three use cases: Search (indexing for answers), Agent (real‑time assistance), and Training (model‑building). New controls are available on all tiers, including Free. Starting September 15, 2026, new domains will have Training and Agent blocked by default on ad‑supported pages, while Search remains allowed. Multi‑purpose crawlers that combine Search with Training will follow the most restrictive rule, ensuring transparency.

To aid visibility, Cloudflare is launching Bot Base, a searchable database of known bots for Enterprise Bot Management. Bot Base will evolve into a control center for automated content. Customers can opt out of the upcoming defaults anytime before the change, and Cloudflare will send reminders so owners can tailor settings to their preferences.