HeadlinesBriefing favicon HeadlinesBriefing.com

Harmonic's pbcc Fixes Python Protobuf Bottlenecks

Hacker News: Front Page •
×

Harmonic's infrastructure team built pbcc, a custom Protocol Buffers compiler for Python, to replace Google's standard library during high-performance workloads. The new tool generates specialized C++ code for Protobuf messages, allowing massive datasets to be processed in memory with a cleaner API. This move addresses specific scaling issues the company encountered with the official tools.

Engineers found the standard Python implementation slow and memory-hungry, often failing on messages larger than 2GB. Additionally, the interface lacked proper type checking support and used custom containers instead of native Python lists, creating friction. Harmonic prioritized correctness and ease of use by generating .pyi stub files for full IDE autocomplete and mapping fields directly to native types like lists and dicts.

pbcc uses ahead-of-time compilation, reading descriptors to generate C++ source code compiled into extension modules. This architecture encapsulates serialization complexity while maintaining standard Python object behavior. Harmonic open-sourced the tool on GitHub, offering a performant alternative for engineers who, like them, prefer solving core infrastructure problems in-house rather than settling for existing vendor solutions.