Reddit completed a massive infrastructure migration, moving over 500 Kafka brokers and more than a petabyte of live data from Amazon EC2 to Kubernetes without any downtime. The engineering team accomplished this feat by ensuring no client applications needed to change how they connected to Kafka, maintaining continuous operation throughout the entire process.

Before the migration, Reddit managed its Kafka fleet on EC2 instances using Terraform, Puppet, and custom scripts. As the fleet grew, manual operations became slow, error-prone, and expensive. Kubernetes paired with Strimzi offered a more scalable solution. The team established four strict constraints: Kafka had to stay online, metadata couldn't be rebuilt, client connectivity had to remain unchanged, and every step had to be reversible.

The migration involved multiple phases, including introducing a DNS facade layer and doubling the cluster size to free up broker IDs. Reddit created a fork of the Strimzi operator to enable Kubernetes brokers to join the existing EC2 cluster. Using Cruise Control, they incrementally moved partition leadership and replicated data from EC2 to Kubernetes brokers. The entire migration demonstrated how large-scale infrastructure transitions can be executed without disrupting critical services.