Testing and developing for rack-scale hardware is difficult because complete racks are scarce; they are shipped quickly to customers' data centers. As a member of the Product Assurance Team at Oxide, the author describes the need to replicate diverse customer networking topologies, run the actual control plane, test up-to-date software, and represent hardware components including switches and microcontrollers, all without cutting special "test version" releases.
To meet these needs, Oxide uses "racklettes," smaller racks of four sleds and two switches each. Because they are real racks, they can be initialized and run just like production hardware. The company currently has four, with eight planned, and they can test any combination of hardware and software requirements. Oxide also runs simulated versions of its Omicron control plane, which can operate independently, in tandem, and at varying degrees of simulation.
The main tool is Virtual OXid E Labs (Voxel), the successor to the earlier a4x2 virtual labs. Voxel prioritizes fidelity, the "Prime Directive" that labs use stock Oxide software without hacks or modifications. Built on illumos in Rust, it emulates sled virtualization through Propolis, BGP or static networking with Debian and FRR, P4-powered switches through Oxide's Soft NPU, and microcontroller simulation for the Service Processor and Root of Trust.
Voxel runs in a small footprint. The author runs it on a Minisforum MS-A2 mini-PC with a 32-thread AMD Ryzen 9 9955 HX and 96GB RAM, spinning up six sleds in either a single rack or a multirack configuration. Launching takes about 10 minutes and requires only Oxide Helios and a fast NVMe drive. Because Voxel uses stock software, it also simplifies image creation: Oxide sleds are either normal sleds or "scrimlets" wired to sidecar switches.
Source: Hacker News · Summarized by HeadlinesBriefing