HeadlinesBriefing favicon HeadlinesBriefing.com

Anubis Proof-of-Work Blocks AI Scrapers

Hacker News •
×

Website administrators are deploying Anubis, a Proof-of-Work challenge inspired by Hashcash, to combat aggressive scraping by AI companies that causes downtime and resource exhaustion. The system presents a computational puzzle that individual users barely notice but becomes prohibitively expensive for botnets operating at scale, effectively raising the cost of mass data collection.

The implementation mirrors Hashcash's original design for email spam prevention: clients must compute a hash meeting a difficulty target before accessing content. This approach shifts the burden onto scrapers who must solve thousands of challenges per crawl, while legitimate traffic passes through with minimal latency. Anubis currently requires modern JavaScript features, which conflicts with privacy extensions like JShelter that disable the necessary APIs.

Developers position this as a stopgap. The roadmap focuses on fingerprinting headless browsers through behavioral signals such as font rendering discrepancies, which would allow the system to distinguish automated clients from humans without presenting challenges to every visitor. Several sites have already integrated Anubis as middleware, reporting reduced server load during peak scraping waves.

The technique represents a pragmatic escalation in the arms race between content owners and AI trainers. By making scraping economically inefficient rather than technically impossible, Anubis buys time for more sophisticated detection while imposing real costs on operators who ignore robots.txt and rate limits.