A few days ago I wrote about Rewind VM, a deterministic VM where every run of a Nix build is a pure function of its inputs, thread schedule included! I have been using it to find, reproduce and solve numerous race conditions in Nix builds, but it's been making me feel a little crazy. What is this superpower, and why is nobody else using it? The tool has quickly grown a source panel, stack frames, bookmarks, a "Compare" tab, a lot more gdb support, thread lanes, which show who held the CPU at every step, and "Check from here", which helps find the exact step where a race condition happens. I went into each new feature thinking it would be a big code-lift but I kept running into the same thing: the hard part of each feature was already done, and Nix had done it. A debugger for instance, needs the exact inputs of the program, its debug symbols, its sources, the sources of every library under it, and a way for someone else to get all of that on their machine. That's exactly what a derivation is, and Nix makes that easy.
Two tellers, one account
A classic simple example of a race condition is two threads depositing into one bank account. Each deposit reads the balance, writes a line to the ledger, then stores the balance plus the deposit. If one teller runs between the other's read and store, it writes a stale balance over the other's deposits. static void deposit(int teller, long amount){long seen = balance;char line[64];int n = snprintf(line, sizeof line,"teller %d: %ld + %ld\n",teller, seen, amount);if (write(1, line, n) != n)return;balance = seen + amount;}
A teller that runs between the other's read and store writes a stale balance over the other's deposits.
11
On my 16-core laptop, bank lost money in 396 of 1,000 runs. Pinned to a single core with taskset, it lost money in none of 1,000: one core alone rarely switches threads in the middle of a deposit, which is why Rewind perturbs the schedule.
Rewind's VM has one CPU, so its first run passes too. rewind check runs the build again under perturbed schedules, each asking the guest kernel to reschedule at different steps, and narrows the first failure down to one step:$ rewind check --where github:fzakaria/rewindvm#bank...step 3237 decides it: a reschedule there makes the run fail
passing: run 12feb5f832205a72, schedule 0
failing: run c2f1cfac0c288f3e, schedule 1 over steps 3237..3238
The two runs are the same machine, exit for exit, until step 3237, where only the failing one gets a reschedule.
Free thing one: the inputs
The argument to check is a derivation and can be a flake reference. The beauty of Nix is that it knows the inputs of a derivation, so that's all we need to make sure the run is reproducible. The derivation's inputs are the source, the compiler, the libraries, the kernel, and the VM's configuration. Rewind nix realises the derivation's inputs, packs the closure into a read-only erofs image, and boots the VM on it. The run's id is a hash of its inputs, the way a store path is, and rewind show prints the command that makes it again:$ rewind show 12feb5f8
rewind nix /nix/store/cr8rl40rjb9cmmcd9sxn9mpdrhvp53jj-bank-0.1.0.drv --epoch --clock branches --name bank-0.1.0 # 12feb5f832205a72
When the build succeeds, the guest reports each output's NAR hash, and rewind nix checks it against the host's copy and against every binary cache Nix substitutes from, by fetching only the .narinfo:$ rewind nix github:fzakaria/rewindvm#bank.../nix/store/27a4q7w90jzp4wg5kaaajvb64zll6yqg-bank-0.1.0 dbf0df3de84b7a84 matches your store This helps us validate that the build within the VM is the same as the build on my laptop, and that the VM's run is reproducible by Nix.
Source: Hacker News · Summarized by HeadlinesBriefing