Over the past year, the size and demands on OpenAI's monorepo have increased at a staggering pace. It was clear that the organization needed to revamp its source control system to scale to the level of concurrent work that Codex makes possible. OpenAI built Pando, a new backend for Git repositories, without interrupting any engineering work or changing how its team uses Git.
Pando now powers all development in OpenAI's monorepo. Pando is designed to meet two ambitious goals: first, horizontally scalable writes, optimized for concurrent pushes to many branches. And second, maximize read throughput, so clients can download data fast enough to fully saturate their available network bandwidth.
In production, Pando cut median push latency by nearly 5x. It now delivers nearly a petabyte of Git data per day. OpenAI needed a new source control system because Codex has changed the scale and pace of development, enabling engineers to do far more work in parallel.
That means more branches and pushes, more merges, and more fetches and clones from development environments, research jobs, and CI. Starting around the end of last year, the systems supporting OpenAI's monorepo were struggling to keep up, suffering slow fetches and pushes along with failed merges, with reliability at times falling below 90%. Building on Git as the foundation meant improvements to the client could be useful beyond OpenAI and could be contributed to upstream.
Pando coordinates updates to each branch independently, allowing updates to unrelated branches to proceed in parallel. It reuses packfiles across fetches instead of generating an individual packfile for each client, letting OpenAI scale writes across branches and serve reads from caches.
Source: OpenAI Blog · Summarized by HeadlinesBriefing