HRT's trading relies on a large research compute cluster, training models on years of exchange history and simulating trading scenarios across many CPUs and GPUs. As the company expanded storage and compute globally, including to distant places like Norway, programs increasingly ran far from the data they needed, and WAN bandwidth became a key constraint. When inter-site traffic overwhelmed the network, HRT discovered its observability was weak and could not easily determine which users and workflows were responsible.
The investigation began with a slow model training run. The program used GPUs in Chicago to analyze data from New York, mirrored through a dedicated cross-region service called Dupe Dog. The transfer stalled, and dashboards showed the transfer rate had cratered. Dupe Dog's own logs did not explain the slowdown, and the raw network switch counters revealed an unknown source of traffic completely saturating the capacity into Chicago. The traffic was not coming from Dupe Dog or from the distributed file cache Cashy.
HRT gives researchers wide freedom to choose their tools, so multiple services share the network and not all traffic passes through a single logged service. Jobs can communicate through sockets, storage cross-mounts, direct file copies, or custom web servers, which leaves many possible explanations for the unexplained load.
To connect users and programs to WAN usage regardless of the tool used, HRT turned to comprehensive capture with eBPF, joining service telemetry with network hardware byte counters to show which users and programs generate the most traffic.
Source: Hacker News · Summarized by HeadlinesBriefing