Hypura solves the problem of running large language models (LLMs) that exceed a Mac's physical memory by streaming tensors from NVMe storage. This allows models like Mixtral 8x7B (30.9GB) or Llama 70B (31.8GB) to run on a 32GB M1 Max Mac Mini without crashing. The system intelligently places model tensors across GPU, RAM, and NVMe tiers based on access patterns and hardware capabilities, enabling inference speeds of 2.2 tokens per second for Mixtral and 0.3 tokens per second for Llama 70B.

Vanilla llama.cpp fails on both models due to naive memory loading.