A developer who built sprinklz.io, a customizable feed reader that can deprioritize topics such as LLMs, posted the project to Hacker News. It drew about a hundred upvotes, and users seemed to enjoy filtering out AI content, which showed why a tool like this is useful.
Soon after, the website became painfully slow. The developer first assumed the extra attention was to blame, but the slowdown continued and even worsened several days after the post went live. Logging every endpoint revealed that most requests never tried to fetch news articles. Because the site is a single-page application, the majority of these requests did not even run the JavaScript needed to render a page.
NGINX logs showed that nearly half of the traffic came from one user, a service that describes itself as listening for changes to websites and summarizing them into AI-generated newsletters. It was querying the site many times per minute, and the developer blocked it. Several other sources were standard crawlers from Claude, OpenAI and Google's search indexer, which identified themselves. A few vulnerability scanners also appeared, including one probing for a WordPress file although the site does not run WordPress.
The developer updated robots.txt to discourage most AI scanners while allowing Googlebot and other search traffic, and added reasonable rate limits. The main lesson was the need for a faster, more scalable site. Over the following weeks, the developer made several infrastructure improvements that should yield substantial performance gains for all users.
Source: Hacker News · Summarized by HeadlinesBriefing