HeadlinesBriefing favicon HeadlinesBriefing.com

AI Crawlers Overwhelming git.kernel.org Resources

Hacker News •
×

AI crawlers are consuming massive resources on git.kernel.org, with 14 CPU cores across 5 geo-distributed nodes dedicated solely to rendering git commits as HTML for scrapers. This exceeds CPU usage for all legitimate access combined, including git clones. The kernel.org infrastructure hosts 1.48 million commits in linux.git and 922 forks, generating billions of duplicate URLs that crawlers systematically scrape instead of efficiently cloning repositories.

Crawlers target kernel.org because its open development history provides guaranteed pre-AI, LLM-free training data — "worth its weight in gold" since training on LLM-generated content causes "digital prion disease." Despite efficient cloning options, scrapers choose the "stupidest possible way" by rendering HTML commit-by-commit and parsing it, exploiting cgit's ability to generate diffs, patches, and arbitrary commit views.

Initial blocking via user-agent detection and fail2ban worked until bots spoofed browser user-agents. IP banning escalated to ASN blocking, but crawlers now operate from millions of residential and mobile IPs, making identification nearly impossible. This "background radiation" of system load permanently ties up capacity for single-purpose model training.