HeadlinesBriefing favicon HeadlinesBriefing.com

turbolite SQLite VFS achieves sub-250ms cold queries from S3

Hacker News •
×

Russell Romney has built turbolite, a SQLite VFS in Rust that delivers sub-250ms cold query performance directly from S3. The system uses B-tree introspection to group related pages, compresses them with zstd, and stores them as immutable objects in S3-compatible storage. Unlike traditional SQLite, which assumes local disk access, turbolite restructures data for cloud storage constraints.

Performance benchmarks show point lookups at 77ms on S3 Express One Zone versus 259ms on Tigris, with multi-table joins completing in under 200ms on the faster service. The design separates interior B-tree pages, index leaves, and data leaves into different page groups, then prefetches them based on query plans. A manifest file replaces SQLite's implicit page offsets with explicit pointers to enable atomic commits and garbage collection.

The project ships as a Rust library, SQLite loadable extension, and language packages for Python and Node.js. It defaults to 64KB pages to minimize S3 round trips and uses seekable compression so range GETs fetch only needed sub-chunks. While single-writer only and experimental, turbolite demonstrates that object storage has become fast enough to support embedded databases over cloud storage, potentially enabling database-per-tenant architectures without attached volumes.