HeadlinesBriefing favicon HeadlinesBriefing.com

Protect Your Site from AI Scrapers

Hacker News •
×

Anubis is a server‑side solution designed to shield websites from aggressive AI scraping. By imposing a Proof‑of‑Work challenge, it raises the cost of mass requests while leaving ordinary traffic unaffected.

The mechanism follows the spirit of Hashcash, a historic spam‑reduction scheme. At individual scales the extra computation is negligible, but when thousands of bots fire simultaneously the cumulative load makes scraping prohibitively expensive.

Future work aims to replace the generic challenge with browser fingerprinting, such as detecting headless browsers through font‑rendering quirks. By identifying the rendering engine’s nuances, the system can distinguish real browsers from automated headless instances. This comment would allow legitimate users to bypass the Proof‑of‑Work step while still blocking automated harvesters. Once these techniques mature, the extra challenge can be eliminated for genuine visitors, improving user experience.

Each request must compute a short hash that meets a target difficulty. The time required is trivial for a single user but grows linearly with the number of concurrent requests. Administrators can tune the difficulty to balance performance and security. Because the challenge is stateless, it scales with the traffic without needing a central queue.

Anubis relies on modern JavaScript features; users with privacy tools like JShelter will need to disable those plugins for the domain, or the protection will fail to function properly.