LinkedIn has detailed its transition to a next-generation service discovery system, replacing its decade-long reliance on Apache Zookeeper. The original system, built on Zookeeper, faced critical challenges with scalability, compatibility, and extensibility. These issues stemmed from a flat architecture that struggled to handle the volume of requests from thousands of microservices.

Zookeeper's architecture, using a custom D2 format, struggled with read storms and health check timeouts. The system was projected to reach capacity in early 2025. Furthermore, the D2 entities were incompatible with modern technologies like gRPC and Envoy. The lack of an intermediary layer also limited the development of centralized RPC management. These limitations prompted a complete overhaul of the service discovery infrastructure.

The new architecture separates write and read paths. Writes are handled by Kafka, a distributed streaming platform, processing service discovery URIs. Reads utilize a Service Discovery Observer, which consumes URIs from Kafka and writes them to its main memory. Application clients then open bidirectional gRPC streams to the Observer, using the xDS protocol for instant data updates. This approach overcomes the limitations of the previous system.

This new system marks a significant step forward for LinkedIn's infrastructure, designed to support its massive scale and ensure continued performance. By embracing a more modern, scalable, and extensible design, LinkedIn is better positioned to handle its ever-growing service demands. The shift signifies a move toward more efficient service management.