Aria is the firm's internal messaging framework and hosted system, processing multiple terabytes of data per day. Clients subscribe to Aria to receive a live stream of messages. As usage has rapidly grown, the Aria team has had to re-architect the system to keep pace with rising data volume and throughput. This summer, intern Theodor Totev focused on a specific use case: making it cheaper for clients to read just a subset of messages. His work using indexing and tree-splitting led to a 30% decrease in CPU usage on production workloads, while maintaining the high standard of correctness that such a critical system requires.
When Aria delivers messages over TCP, it keeps recent messages, known as the stream tip, in an in-memory ring buffer. Clients that fall behind can request recent messages from this buffer to catch up, a process called tip recovery. The challenge is that Aria stores the entire message stream, while clients often need only a small subset. Aria filters the stream by topic, and clients can subscribe to individual topics or to entire topic subtrees. The original approach was a simple linear pass over the stream. It was algorithmically inefficient but fast thanks to CPU cache friendliness. As the number of clients doing tip recovery grew, some servers reached 100% CPU utilization, causing clients to fall off the tip and fail to catch up.
An index for each topic was ruled out, since Aria instances can have almost a million topics. Instead, Theodor created an index for each topic partition, which groups all topics sharing the same two-segment prefix. Each partition's index records the location of its messages in the stream. When a client requests messages, Aria performs an n-way merge of the relevant indexes using a min-heap, reconstructing the in-order stream for only the requested partitions.
Benchmarking was central to the work. Theodor built a profiling tool to test variables such as the number of topic partitions, the degree of interleaving, and the number of readers. Profiling the initial implementation revealed that the min-heap was the biggest bottleneck.
Source: Hacker News · Summarized by HeadlinesBriefing