HeadlinesBriefing favicon HeadlinesBriefing.com

PlanetScale Introduces Neki: Sharded Postgres for Scalable Workloads

Hacker News •
×

Neki is now available in platform preview. Neki is built from lessons we’ve learned over eight years of running some of the largest sharded MySQL clusters in the world. Thousands of production workloads with millions of queries per second for companies where even a few seconds of downtime is a very public event.

We know what it means to power the world’s biggest tier 0 workloads. When we released Planet Scale Postgres a year and a half ago, we knew we needed to do more. In that time we’ve onboarded several thousands of customers on Planet Scale, some of them rivaling the size of our largest MySQL customers.

Time and time again, we watched teams approach the ceiling of a single machine with Postgres. Metal bought them time, but with customers hitting the upper limit of what a single machine is capable of, we found there was no good option to hand them. Enter Neki.

What is Neki? Neki is sharded Postgres from Planet Scale. It lets you scale a Postgres database across many machines while keeping real Postgres on every shard. Your application connects to a Neki router over the standard Postgres wire protocol, so your existing drivers, ORMs, and connection string keep working.

Each shard is a full Postgres cluster with one primary and at least two replicas across 3 availability zones. There is no custom storage engine, so extensions, SQL support, and performance behave the way Postgres does. You choose the shard key and control how tables are grouped and distributed through a JSON data topology.

Schema changes, version upgrades, failovers, imports, and resharding all run as built-in fully online workflows. You also get the Planet Scale features you already rely on, including Insights, schema recommendations, branching, and MCP. You don't have to shard on day one.

Run Neki as a single primary with replicas, and when you outgrow one machine, resharding is a workflow you run against the cluster you already have. Why Neki? You already know the problems that come with fast-growing Postgres databases: tables too large to vacuum or index without affecting traffic, backups taking hours, connection limits, maintenance windows for schema changes, transaction wraparound and so much more. You can move to a bigger instance, but eventually you run out of big enough machines, and the problems don’t scale linearly as you add more cores and IOPS.

The existing answers each ask you to give something up. Application-level sharding pushes routing into your code. Postgres-“compatible” distributed databases hide the shard key from you, take away your extensions, and add complexity and latency which becomes difficult to handle and debug.

So we built Neki with a few principles, the biggest one being: stick to Postgres, don’t work around it, fake it, or turn away from it. How does Neki work? We architected Neki from first principles for Postgres, with real Postgres on every shard. There are four moving parts.

Neki routers Your application first connects to a Neki router. The router speaks the Postgres wire protocol so your existing drivers and ORMs keep working with a single connection string. A router has a full Postgres query parser, a distributed query planner, query buffering and more.

It parses your query, builds a plan that decides which shards should run it, sends the work out, and combines the results back into one stream. Routers can scale vertically and horizontally, so no single router becomes the bottleneck. Sharding and shard groups Every shard in Neki is real Postgres with 1 primary and at least 2 replicas, spread across availability zones.

There is no modified storage engine. Extensions, SQL support, and performance behave the way Postgres behaves, because it is Postgres. Shards are organized into shard groups, so different tables or workloads can live on different sets of shards.

Each shard uses a configuration profile that defines its instance size, replica count, storage, Postgres parameters, and extensions, so you can size each group for i...