HeadlinesBriefing favicon HeadlinesBriefing.com

GitHub Outage August 17: Recovery and Reliability Plan

Hacker News •
×

On August 17, GitHub experienced a major outage lasting 7 hours and 47 minutes, disrupting services including github.com, GitHub Actions, APIs, and Copilot. This was the second significant incident in August, following an actions failure on August 6. The outage stemmed from a critical infrastructure component in GitHub's Central US data center failing to scale during peak traffic, causing cascading failures across multiple services.

GitHub's investigation revealed both incidents were capacity failures, not caused by code or configuration changes. Monthly commits grew from 1.4 billion to 2.9 billion since April, creating unprecedented system pressure. Recovery required coordinated efforts to reroute traffic, isolate affected infrastructure, and restore services in stages.

Since April, GitHub has added over 3 million CPU cores, 120 petabytes of storage, and significant network capacity while accelerating Azure migration. Azure now handles roughly 58% of GitHub's platform load. The company is implementing consistent retry limits, improving testing, and isolating critical systems to prevent future outages.

Written by Vladimir Fedorov, GitHub's CTO, the post emphasizes the company's commitment to reliability through architectural improvements and operational changes.