When Cloudflare last wrote about its DNS propagation pipeline in 2022, the Zone Builder received about 250 DNS record changes per second. Today it receives about 700, nearly three times as many, plus about 100 other events. Some zones now hold more than 20 million records.
Each event starts a build that reads the current database state, converts it into edge format, and writes changes to Quicksilver. In 2022, per-record builds were 150 times faster than full builds on average; on a one-million-record test zone, they were 4,250 times faster.
The team built a targeted build planner that chooses the smallest provably safe build, and a new scheduler that combines work for the same zone. Zone setting changes no longer rebuild zones with tens of millions of records. Together, they cut peak memory by 85%, and the extra headroom let them shorten waits the pipeline always needed for resilience, halving the time from a DNS change to its data being written to Quicksilver.
The DNS Kafka Aggregator combines events from several Kafka topics and sends them to the Zone Builder, which writes only what changed to Quicksilver. Unlike full builds that read and compare every record while holding the whole zone in memory, the new approach avoids rebuilding entire zones for setting changes.
Source: Hacker News · Summarized by HeadlinesBriefing