HeadlinesBriefing favicon HeadlinesBriefing.com

AI Agents Beat Cloudflare with Open‑Source Browser

Hacker News •
×

Cloudflare began blocking AI crawlers on all new domains in July 2025, returning a 403 with a 27KB “Just a moment” body that bots read as valid content. The reply carries no error flag, so agents finish the request and summarize the block page, producing confident but wrong answers.

A test against twelve popular sites—Indeed, Glassdoor, Crunchbase, Upwork, Amazon, and others—showed every naive fetch returned the block body, never real data. In a benchmark of eight anti‑bot vendors, the plain fetch hit zero sites for real content, Fortress succeeded on five, and Tilion Cloud cleared all eight.

Fortress, an open‑source stealth browser, handles Cloudflare, Akamai, and Perimeter X challenges with a single headless fetch that completes JavaScript interstitials. Tilion Cloud adds warm, persisted profiles and fingerprint‑matching to tackle Data Dome and click‑walls that block even the fetch step. Both return a `blocked: true` flag so agents can abort before summarizing a fake page.

For developers, integrating Fortress via `tilion-mcp` lets agents detect anti‑bot screens, avoid misleading outputs, and reliably scrape data from sites that previously required paid anti‑bot services. The combination lowers cost while maintaining robustness, reshaping how automated agents handle protected web content.