HeadlinesBriefing favicon HeadlinesBriefing.com

Railway outage traced to Google Cloud account suspension

Hacker News •
×

On May 19 at 22:20 UTC, Google Cloud mistakenly suspended Railway’s production account, triggering an eight‑hour platform outage. The suspension knocked out the API, dashboard, and core networking hosted on GCP, causing 503 errors and preventing logins. Because Railway’s edge proxies depend on a control‑plane API in GCP, the failure quickly cascaded to workloads on its Metal and AWS environments, and forced developers to pause deployments.

Railway’s engineers detected the health‑check failures at 22:10 UTC, opened a P0 ticket with Google, and restored account access by 22:29. Persistent disks came online around 23:09, but networking remained down until 01:30 UTC, delaying edge traffic restoration. During recovery, GitHub rate‑limited Railway’s OAuth and webhook calls, temporarily blocking additional logins and builds, while the team posted status updates.

By 04:00 UTC the API, dashboard, and OAuth endpoints were confirmed operational, and the remaining workloads finished coming back online shortly after. The incident exposed a single‑point dependency on GCP for control‑plane routing, prompting Railway to reinforce its mesh network and add stricter monitoring of upstream provider actions, and will monitor similar risks. Service now runs without further interruption.