Comma AI details its experience running its own data center, arguing against relying solely on cloud providers. They claim it fosters better engineering practices and offers cost advantages, especially for compute-intensive tasks like machine learning model training. They built their infrastructure to avoid being locked into expensive cloud services and to maintain control over their destiny.

Their data center utilizes approximately 450kW of power. Cooling is achieved through outside air, which helps reduce power consumption. The infrastructure includes 600 GPUs across 75 in-house built machines, along with Dell storage arrays. For networking, they use 100Gbps switches. This setup enables them to handle large-scale model training and other compute tasks efficiently.

They use Slurm for workload management and minikeyvalue for distributed storage. For training, Comma AI leverages PyTorch with `torch.distributed` FSDP. They also developed their own experiment tracking service. Comma AI's approach offers an alternative to cloud reliance, emphasizing self-sufficiency and control over resources.

Ultimately, Comma AI estimates they've spent $5M on their data center, compared to an estimated $25M+ in cloud costs. This highlights the potential cost benefits of owning your own infrastructure, especially for consistent compute needs. This strategy is attractive for companies that value control and cost optimization in their AI operations.