HeadlinesBriefing favicon HeadlinesBriefing.com

DuckDB v2.0 Replaces SQL Parser with PEG-Based Engine

Hacker News •
×

DuckDB v2.0 introduces a new PEG-based SQL parser, replacing the PostgreSQL-derived parser used since the project's first commit in 2018. The previous Bison-generated LALR(1) parser became increasingly difficult to extend due to shift/reduce and reduce/reduce conflicts as Duck SQL evolved. The new parser, built using Parsing Expression Grammars, eliminates these conflicts by explicitly ordering alternatives and enables runtime extensibility.

The parser processes SQL queries through three stages: tokenizer, parser, and transformer. The tokenizer splits input into tokens like keywords and identifiers. The parser validates syntax against Duck SQL grammar, producing a parse tree. The transformer converts this into DuckDB's internal AST for the binder.

Development began with a research prototype that handled only a subset of SQL. It was incrementally expanded and introduced as an experimental feature in DuckDB v1.5. The complete grammar now supports all statement and expression types, correct operator precedence, proper keyword classification, and compatibility with existing AST structures.

While the implementation changes significantly, Duck SQL itself remains unchanged. The new parser also improves error reporting, providing better context for syntax failures. This transition aligns with similar moves by other systems like Python 3.9, which also adopted PEG-based parsing for greater flexibility.