Bluesky suffered an eight‑hour partial outage on Monday, affecting roughly half of its users. The incident traced back to a new internal service that flooded the data‑plane RPC GetPostRecord with batches of 15‑20 k post URIs, overwhelming the memcached layer. Because this endpoint lacked the usual concurrency limit, thousands of goroutines opened TCP connections until the system ran out of ports.

Exhausted ports triggered a cascade: memcached errors flooded the logs, and each error generated a blocking write syscall. Millions of such syscalls forced Go to spawn far more OS threads, inflating stop‑the‑world garbage‑collection pauses and causing intermittent OOM kills. Restarting the data plane failed to obtain fresh memcached sockets because lingering TIME_WAIT sockets still occupied the address space.

Engineers broke the loop by assigning each memcached connection a random loopback IP, expanding the available ip:port combinations and immediately restoring service. After the fix, they reinstated the missing errgroup.SetLimit call on GetPostRecord, preventing unbounded goroutine creation. The episode underscores how a single missing concurrency guard can cascade into a full‑scale outage.

Jim, the system engineer who posted the post‑mortem, apologized and noted the status page had mistakenly blamed a third‑party provider. The team is now hiring and plans tighter per‑client observability.