The new Kimi K3 large language model, with 2.8 trillion parameters, presents significant hardware challenges. While NVIDIA's B200 and B300 GPUs are options, AMD's MI355X offers a compelling performance-per-dollar advantage. Despite initial software support hurdles common with AMD hardware, the MI355X, featuring 288GB of VRAM per GPU, demonstrates strong capabilities.

In benchmarks, the MI355X achieved 952 tokens/second/node and 118 tokens/second single stream, significantly outperforming a B200 deployment in aggregate throughput and single-stream decode, especially considering its lower cost. While B300 nodes offer higher aggregate throughput, the MI355X provides a better performance-per-dollar ratio.

Optimizations for speculative decode and prefill, including a fix for a 'Name Error' in the ROCm scheduler and a trivial head count adjustment for the AITER MLA prefill kernel, further enhanced the MI355X's performance. These improvements addressed critical areas like time-to-first-token, making the MI355X a viable and cost-effective solution for running large-scale AI models.