A new arXiv paper by David Patterson and Xiaoyu Ma argues that LLM inference is fundamentally harder than training due to its autoregressive decode phase. The authors pinpoint memory and interconnect as the primary bottlenecks, not raw compute, a shift from traditional hardware focus.

Their research highlights four key architecture opportunities for datacenter AI. These include using High Bandwidth Flash for 10X memory capacity, Processing-Near-Memory techniques, and 3D memory-logic stacking to boost bandwidth. Low-latency interconnects are also critical to speed up communication between components.

While the paper centers on datacenter-scale systems, the authors note these principles apply to mobile devices too. The work, accepted for publication in IEEE Computer, provides a roadmap for hardware designers tackling the memory wall in large-scale AI deployments.