Many years ago, if you wanted to store large amounts of data that could be sensibly queried, a database like Oracle or Postgres was your main choice. The next big advance was the data warehouse, followed by the data lake. Data lakes allowed organisations to store much larger volumes of raw structured, semi-structured, and unstructured data cheaply. Companies like Databricks, Snowflake, and AWS charge heavily, but you can develop an effective data lake for almost zero cost with Duck DB and the Duck Lake extension.

Both Duck DB and Duck Lake are MIT-licensed, open-source, and free to use. Duck Lake, developed by the Duck DB team, manages metadata in a relational database instead of files co-located with Parquet data.

The article walks through creating a local Duck Lake, examining metadata, time-travel queries, evolving a table schema, and using cloud-based S3 data with Duck Lake.