Google researchers introduced Decoupled DiLoCo, a breakthrough distributed architecture for training AI models across distant data centers. By dividing training into separate "islands" of compute with asynchronous data flow, the system isolates hardware failures while maintaining progress. This approach solves the synchronization challenges of traditional distributed training.

The architecture achieves remarkable results by training a 12 billion parameter model across four U.S. regions using standard internet bandwidth (2-5 Gbps). More than 20 times faster than conventional methods, Decoupled DiLoCo maintains the same benchmarked ML performance as traditional approaches while being resilient to hardware failures through self-healing capabilities.

A key advantage is the ability to mix different hardware generations like TPU v6e and v5p in a single training run, extending useful life of existing hardware. By enabling training at internet-scale bandwidth, the system can tap unused compute resources anywhere, turning stranded capacity into useful AI training power.