HeadlinesBriefing favicon HeadlinesBriefing.com

Recall.ai's Rare Segfault Hunt

Hacker News: Front Page •
×

Recall.ai's meeting bots process millions of monthly hours of video, but a 1-in-36-million crash in their AAC encoder threatened data loss. The rare segfault originated in `libvo-aacenc.so`, deep in GStreamer's media pipeline. Losing even a handful of sessions was unacceptable for their stateful, ephemeral EC2 instances.

To diagnose the crash, engineers built a core dump collector named Garbage Truck. Using GDB with Debian's debug symbols, they traced the fault to a fixed-point math bug in `voAACEnc_pow2_xy`. The root cause was a decade-old patch that never reached downstream users, highlighting the perils of abandoned upstream libraries.

Rather than patching the legacy vo-aacenc library, Recall.ai replaced it with a modern AAC encoder. This approach eliminated the crashes and simplified their stack, proving that sometimes the best fix is a strategic upgrade, not a deep dive into legacy C code.