I run a local LLM server on my M4 Pro Mac mini with 48 GB of RAM. It handles everything from my Hermes agent backend to quick chat queries on my phone. The whole thing takes about 30 minutes to set up. The stack includes Qwen3.6-35B-A3B-Opti Q-4bit as the main reasoning model, Gemma-4-E4B-it-Opti Q-4bit for lightweight tasks, MLX as the inference server, and Tailscale to link the Mac mini, iPhone, and MacBook.

The main reason to run local: cloud APIs are rented land. They can change pricing, hit usage limits, or swap models with no notice. I was regularly maxing out two $200/month subscriptions and getting inconsistent results. Data privacy is another issue, since you cannot control what third parties do with your data. AI sovereignty matters too, given how governments have limited various model rollouts. The only way to avoid that is to own your compute.

Other advantages include cost predictability (hardware plus electricity, flat), lower latency from no network roundtrip, offline capability, and no rate limits. The Mac mini stays on at my desk. Hermes runs locally as the agent backend, accessed through Telegram and the Hermes desktop app on my MacBook. I also use Apollo on iOS for quick chats, Raycast AI on Mac for random tasks, and Pi for coding.

The point is not to replace API-based models but to handle the 80% of requests that do not need GPT-5 or Claude Opus. Local just covers more of my day-to-day for free.