Marc Brooker, an engineer at Amazon Web Services who works on agentic AI safety and policy, uses animations to help engineers build intuition for how distributed systems work and fail. He notes that animations can clarify behaviors that are hard to grasp from text alone.
One example shows that a Raft cluster can lose writes if even one member's storage layer falsely reports that fsync() has completed. Brooker's earlier Stability Sim tool illustrated metastable failures in the abstract, but engineers often doubt these behaviors occur in production-proven systems.
To answer that doubt, he presents traces from real systems. On Postgres 16 running in a Micro VM, a load spike fills memory with queries, pushing data off the page cache and onto disk, so a database that previously handled 20 queries per second can no longer keep up. In etcd, a write load spike causes overloaded peers to appear unavailable, triggering application-level resends that sustain the overload.
A further etcd example shows a garbage collection feedback loop: a read spike from 500 to 1000 reads per second increases GC time and reduces available CPU, and the system does not recover without intervention. Brooker suspects runtimes without GC could show similar behavior through allocator overhead, but has not yet reproduced it. All animations were generated with Kiro and Kiro Crew, using traces captured from a real test environment.
Source: Hacker News · Summarized by HeadlinesBriefing