Large language models turn text into numbers and run billions of multiply‑add operations. A single forward pass through a 70‑billion‑parameter model consumes over 140 trillion FLOPs and 140 GB of weights. CPUs, built for branching logic, stall on the resulting memory‑bandwidth bottleneck, forcing engineers to turn to GPUs and TPUs for viable inference.

Graphics cards were born to rasterize pixels, a task that mirrors deep‑learning’s matrix math. NVIDIA’s H100 packs roughly 17,000 simple cores into SIMT warps, while Tensor Cores collapse a 4×4 multiply‑accumulate into a single cycle, delivering a 64× throughput boost. Stacked HBM supplies the bandwidth needed to feed these units without hitting the Von Neumann bottleneck.

Because parallel arithmetic, not raw clock speed, now dictates LLM performance, chip designers prioritize memory bandwidth and mixed‑precision support. Upcoming GPU generations and Google’s next‑gen TPU pods promise even larger HBM stacks and tighter integration with transformer workloads. Watching bandwidth‑centric roadmaps will reveal which hardware truly scales future AI models.