The foundational host-device paradigm defines how a CPU (host) manages discrete GPUs (devices) for AI workloads. The host runs the Python script and queues commands to the GPU via CUDA, enabling asynchronous execution where the CPU continues preparing data while the GPU computes. This separation of control and acceleration is critical for efficient multi-GPU programming.

Performance gains hinge on CUDA Streams, which are independent queues allowing concurrent operations. By default, PyTorch uses a single stream, but manually managing separate streams for computation and data transfer can overlap these tasks. Techniques like `non_blocking=True` transfers and CUDA Events for synchronization prevent GPU idle time, a method leveraged by features such as `DataLoader(pin_memory=True)`.

A major bottleneck occurs during Host-Device Synchronization, where the CPU blocks waiting for GPU results, as when printing a tensor's values. Scaling to multiple GPUs introduces the concept of a Rank, a dedicated process controlling one GPU. Effective distributed training requires minimizing synchronization points to keep all hardware busy, making this mental model essential for optimizing large-scale AI systems.