HeadlinesBriefing favicon HeadlinesBriefing.com

How One GIF Broke ext4's Hardlink Limit in Discourse Backups

Hacker News •
×

A single reaction GIF of Jennifer Aniston from Friends caused a major backup failure in Discourse, a popular forum software. The issue emerged when developers discovered that one customer's 432 GB of uploads contained only 26 GB of unique content - the rest was duplicate files created by Discourse's secure upload feature.

Discourse treats each file context as unique, even when content is identical. This meant a popular GIF embedded across posts, private messages, and categories generated hundreds of thousands of copies. When developers implemented a deduplication strategy using hardlinks to reduce backup size from 377 GB to 26 GB, they hit an unexpected ext4 filesystem limit of roughly 65,000 hardlinks per inode.

The backup process initially succeeded, reducing downloads from 246,173 to just 25. But when encountering the 246,173 copies of the same GIF, the system hit the hardlink ceiling and began downloading duplicates individually. While still an improvement, this wasn't the dramatic reduction expected. The fix now includes automatic fallback to copying when the filesystem limit is reached, making the solution work across different filesystems without configuration.