HeadlinesBriefing favicon HeadlinesBriefing.com

Meta's Facebook Bot Floods Robots.txt Files

Hacker News •
×

A self-hosted Forgejo instance owner reports Facebook's crawler has been hitting their `/robots.txt` file several times per second for the past four days. The requests come from Meta's IP ranges using the `facebookexternalhit/1.1` user-agent, which normally crawls content shared on Facebook, Instagram, or Messenger.

According to Facebook's documentation, this crawler gathers and caches information about apps or websites shared on Meta's platforms, including titles, descriptions, and thumbnail images. However, the owner notes they're receiving no other traffic besides AI bots crawling their repositories and occasional users of their Hex packages. The unusual behavior involves only accessing `robots.txt` repeatedly without requesting any other files.

This persistent crawling pattern raises questions about potential bugs in Meta's infrastructure, as such behavior would typically trigger monitoring alerts. While the traffic is currently benign compared to previous AI bot activity, the scale of this issue globally could represent significant wasted bandwidth and energy if similar patterns affect other servers. The owner speculates this might be a loop conditional error at Meta's end that somehow escaped detection.