Researchers and developers can now experiment with frontier-class large language models privately on their own hardware using MLX, the open-source array framework for Apple silicon. The framework takes advantage of the unified memory architecture of Apple silicon, allowing anyone to run new inference and fine-tuning methods locally.
The latest MLX release makes it possible to run Muse Spark Open, a model with nearly a trillion parameters, from a desk setup. On a cluster of four Mac Studios, the model generates over 36 tokens per second, well above average human reading speed. Scaling begins on a single Mac Studio with up to 512GB of unified memory and extends across multiple Macs using the RDMA over Thunderbolt driver.
The benchmarks compare two generations of Mac Studio, the M5 Ultra and the previous M3 Ultra. The M5 generation includes a GPU Neural Accelerator that supports dedicated matrix multiplication, a capability essential for many machine learning workloads. Three models were tested: Muse Spark Open (816B parameters), Mistral Medium 3.5 (128B), and Qwen 3.8 (27B). Performance was measured for time to first token, generation throughput, and peak power usage across context sizes of 8,192 and 65,536 tokens.
Agentic coding and other private, agent-based applications often require large contexts, so the tests focus on those workloads. Qwen 3.8 and Mistral Medium 3.5 fit comfortably on a single 512GB Mac Studio, using up to 68GB and 158GB of RAM respectively at the larger context size. Time to first token is largely compute bound, which is where the M5 Ultra's Neural Accelerators provide their clearest advantage.
Source: Hacker News · Summarized by HeadlinesBriefing