HeadlinesBriefing favicon HeadlinesBriefing.com

Concurrent SQL on Remote DuckDB via Quack

Towards Data Science •
×

The Duck DB team released Quack, a database communication protocol enabling HTTP-based client/server interaction between Duck DB instances on different servers. Using Quack, a Duck DB database on one server can query or write to a database on a remote server. This article explores running parallel SQL statements across three remote Duck DB servers and gathering results on a coordinating server.

To test Quack, the author set up 3 AWS EC2 servers using a CloudFormation stack, each hosting a Duck DB database with one acting as a coordinator node. The setup includes Amazon Linux 2023 ARM64, Python 3.12, Duck DB 1.5.5, the Quack extension, and a custom quack_server.py. The coordination logic runs on Worker 1, using Python threads and barriers to synchronize query execution.

The coordination works by validating SQL statements into Query Fragments, launching threads per fragment, and using a barrier to release them simultaneously. Each fragment connects to its respective remote Duck DB via Quack, executes the query, and fetches results. Timing metrics like start offset and duration are recorded.

Test data consists of synthetic datasets with 10 million records each: sales on Worker 1, customers on Worker 2, and products on Worker 3. The experiment demonstrates concurrent reads and writes on remote Duck DB databases using Quack, though it's not a distributed query processor.