I built buildprof(Github), an open-source tracing tool that shows where the time goes when you compile software on Linux. Here’s a real-time video of it profiling a clean build of ripgrep: Watch the buildprof demo. Sometimes, builds are slow because there is simply a lot of code to compile.

But more often than not, there are fixable problems: poor parallelism, repeated work, dependency downloads or a huge compiler/linker invocation. buildprof makes all of this clearly visible, so you can see what’s worth investigating and optimizing. You run it by putting buildprof -- in front of any build command you already use: buildprof -- make -j16, buildprof -- cargo build, buildprof -- ninja -C out/target, buildprof -- just build, buildprof -- ./dev/custom-build-script.sh. buildprof records every process your build command launches, including their subprocesses (and their subprocesses…), and lays them out on one timeline. Time moves from left to right, bar width shows duration, and child processes appear beneath whatever launched them.

I made buildprof because this tweet from Jarred Sumner, chief architect of the Bun Java Script runtime, was living rent free in my head: Specifically, the claim that Bun’s new Rust build was >5× faster on Linux than its old Zig build really bothered me. In my experience, Zig projects had usually compiled much faster than Rust projects of similar complexity. That intuition was enough to make me feel there was a mystery to solve.

This was further compounded by another important, yet easily missed, detail in the tweet: the Zig build used Full LTO, while the Rust build used Thin LTO. Compilers normally optimize separate compilation units largely in isolation. Link-time optimization (LTO) lets them optimize across those boundaries.

Full LTO brings those units together into one large optimization job, while Thin LTO preserves more separation so much of the work can run in parallel. From past experience, this difference can have an enormous effect on build time. The tweet mentioned it in passing, but I wondered how much of the headline improvement it explained.

I started by trying to reproduce the numbers. The numbers reproduced. But now what? I checked out Bun 1.3.14 and Bun 1.4.0 and wrote some scripts to replay their Linux x64 CI builds on a 6-core, 12-thread Linux VM.

The scripts preserved the build steps and their dependencies, running everything on one machine. My timings were in the same ballpark as Jarred’s: Linux x64 build Zig era Rust era Bun’s reported CI median 30m06s 5m37s My single-machine CI-profile replay 24m24s 5m40s. OK, so the gap showed up on my machine too.

But a lot had changed between the two measurements besides the language; so what was actually responsible? Was it the Zig compiler that was taking all that extra time? Or maybe it was the Full LTO link? Or perhaps there was something else in Bun’s build I hadn’t even thought to look at? This is where my profiling and developer-tools brain kicked in. Usually, when I’m trying to understand why something is slow, I want a trace: what happened, when it happened and how long it took. It would be really cool to have that for these builds, to put them on a timeline and see where their time actually went.

But a build involves a lot of different tools, each with its own idea of what’s happening. What could I record that would let me see across all of them? Builds are process trees. When you type cargo build or zig build, it feels like you are running one program.

The build system works out what needs to be rebuilt, the ordering between those pieces and what can run in parallel. But generally, it does not perform all that work itself; it launches compilers, code generators, archivers, linkers and arbitrary scripts. Which can launch more programs which launch some more… Different build systems describe that work in different ways.

Cargo sees crates, Ninja sees build edges and CMake generates instructions for another build system. From the operating system’s point of view, however, they (mostly) look like p...