HeadlinesBriefing favicon HeadlinesBriefing.com

pg_durable Brings Fault-Tolerant Workflow Execution to PostgreSQL

Hacker News •
×

Microsoft has open-sourced pg_durable, a PostgreSQL extension that enables durable execution of long-running SQL workflows directly within the database engine. The tool eliminates the need for external orchestrators like Airflow or Temporal by implementing checkpoint-based state management using standard PostgreSQL tables.

Engineers define workflows in pure SQL using composable operators like `~>` and `|=>`, then invoke them with `df.start()`. Each step executes with automatic checkpointing, so if the database crashes or a step fails, execution resumes from the last durable state rather than requiring manual reconstruction. This approach handles vector embedding pipelines, data ingestion, and scheduled maintenance tasks.

Traditional approaches scatter workflow logic across cron jobs, worker processes, status tables, and queue systems. pg_durable consolidates this into SQL-native definitions that survive restarts and provide auditability through standard database queries. The extension requires PostgreSQL 17 or 18 and runs as a background worker without external dependencies.

Installation uses Debian packages for supported PostgreSQL versions, with Docker images also available. Row-level security controls access to function instances, though the background worker needs superuser privileges to manage all users' workflows. The project represents Microsoft's push toward bringing compute closer to data storage.