HeadlinesBriefing favicon HeadlinesBriefing.com

GitHub's Miasma: A Poison Pit to Trap AI Web Scrapers

Hacker News •
×

GitHub released Miasma, a tool designed to trap AI web scrapers by feeding them poisoned training data. Scrapers targeting public websites are redirected to Miasma's server, which serves deliberately flawed data alongside self-referential links, creating an endless supply of useless information. The server is lightweight, requiring minimal resources even under high load, making it practical for defenders. Installation is straightforward via Cargo or pre-built binaries, and configuration involves setting a link prefix and limiting in-flight connections to manage memory usage effectively.

To deploy Miasma, configure a reverse proxy like Nginx to route specific paths (e.g., /bots) to the Miasma server. Embedding hidden links pointing to these paths on your site ensures scrapers encounter the poisoned data without affecting human visitors. Miasma's memory usage scales directly with active connections, so setting a reasonable max-in-flight limit (e.g., 50) prevents resource exhaustion. The tool includes a poison source URL and supports compression options, offering a practical defense against data theft by large AI companies.

By deploying Miasma, developers can actively disrupt the data pipelines of AI companies that scrape public content without permission. This approach turns the tables, forcing scrapers to waste computational resources processing garbage data. While legitimate bots should be excluded via robots.txt, Miasma provides a scalable, low-overhead method to protect websites from becoming training fodder for competing models.