PyTorch Monarch has been ported to AMD Instinct GPUs via ROCm, extending single-controller distributed training beyond CUDA. Previous work achieved 96.16% scaling efficiency on a 1024-GPU MI325 cluster with DeepSeek V3-671B, but reliability at scale remains critical. Monarch introduces an actor-based runtime, process mesh abstraction, and hierarchical supervision trees that isolate failures, enabling healthy nodes to continue training while failed nodes recover in seconds.

Porting required three main paths: collective communications via hipify_torch and RCCL, GPU memory management with HIP equivalents, and RDMA integration preserving GPU-direct transfers. A Rust compatibility shim re-exports HIP symbols under CUDA names, avoiding platform-specific branches. All 1,171 tests pass on ROCm 7.0+, with contributions upstreamed.

Integrated with Torch Titan (training engine) and Torch FT (fault tolerance), Monarch orchestrates replica actors and a Lighthouse service for quorum coordination. In a four-replica demonstration, a GPU crash in one replica is detected by the Monarch supervisor; unaffected replicas continue training without global interruption, while the failed node recovers and rejoins. This checkpoint-less architecture minimizes wasted computation and cluster idle time.

The work represents a significant step toward stable, large-scale AI infrastructure on AMD hardware, combining elastic scaling with dynamic fault recovery across SLURM, Kubernetes, and multi-cloud environments.