As a contract data engineer, I sometimes experience periods of inactivity. When browsing the online market for suitable roles, one of the most in-demand skills I keep seeing is experience with dbt. To give myself the best possible chance of obtaining work, I decided to learn as much as I needed to be confident in discussing it. This article distills that process and what I’ve learnt, with plenty of practical code and real-world examples. I have no affiliation with dbt or Duck DB; dbt Core is free open-source under Apache 2.0, and Duck DB is MIT licensed.

dbt provides a broad range of capabilities, but as an introduction, I focus on basics like models, sources, testing data, and documentation. If you’ve worked on analytics projects, you’ve probably ended up with a folder full of SQL scripts. Initially, everything feels manageable, but as the project grows, a column rename can break downstream reports or jobs. Traditional SQL is often treated as isolated scripts rather than a software project, and dbt offers a solution.

dbt (data build tool) was created in the mid 2010s by dbt Labs. It grew into an open-source CLI tool called dbt Core and a paid version called dbt Platform. dbt transforms data in databases or warehouses, and also handles testing data quality, documenting datasets, reusing SQL through macros, managing environments, and running transformations via jobs or CI/CD. It’s widely used with Snowflake, Big Query, Redshift, and Databricks, but my examples use local Duck DB.