A new GitHub repository offers a detailed guide to building and running state-of-the-art large language models locally. The project, compiled by jamesob, addresses the desire for private, on-premises AI by outlining hardware configurations, from a $2,000 setup for Qwen and Whisper STT to a $40,000 system for near-Opus-level models.

The guide meticulously details hardware choices, including four RTX Pro 6000 GPUs for a $40k build, and emphasizes the use of PCIe Gen4 switches for direct GPU-to-GPU communication. This approach minimizes latency during tensor parallelism, a critical factor for efficient local LLM operation.

Configuration details cover BIOS settings, kernel parameters, and Docker setups for ready-to-run models. The author shares lesser-known techniques for optimizing performance, such as disabling ASPM and ACS override, to ensure optimal P2P traffic flow within the switch fabric.

This comprehensive resource provides a practical roadmap for developers and enthusiasts seeking to bypass cloud-based LLM services and gain full control over their AI inference.