Understanding how GPUs communicate across multiple nodes is crucial for efficient AI training. This deep dive explains the hardware infrastructure enabling multi-GPU communication, starting with the PCIe bus. PCIe generations offer varying bandwidths: Gen4 x16 provides ~32 GB/s, Gen5 x16 ~64 GB/s, and Gen6 x16 ~128 GB/s.

Modern servers typically support 8 GPUs, each requiring 16 lanes, making 128 lanes the practical limit. NVLink provides high-speed GPU-to-GPU communication within a server, bypassing the CPU. NVLink 3 (A100) delivers ~600 GB/s per GPU, while NVLink 4 (H100) reaches ~900 GB/s. NVSwitch acts as a non-blocking hub, allowing every Hopper GPU to communicate simultaneously at full NVLink speed, scaling to 256 GPUs. InfiniBand handles inter-node communication but is significantly slower due to network overhead. Key design principles include linear scaling goals and minimizing compute-communication overlap. However, scaling beyond 8 GPUs reveals a performance cliff due to slower inter-node communication via InfiniBand.