HeadlinesBriefing favicon HeadlinesBriefing.com

From Loading to Usable Data with dbt

Towards Data Science •
×

I gave myself a 12‑month roadmap to move from data analyst to data engineer. Two months in, I had built two ETL pipelines from scratch: one pulling GitHub repo data into SQLite, the other pulling RSS articles into PostgreSQL with Docker and Kestra orchestrating the Alman. I celebrated the second pipeline running automatically every hour—data flowing on its own seemed like<IM> a milestone.

But when I queried the data, I couldn't sort articles by date or identify which blogs posted most. The published column was a TEXT string, not a timestamp, and every title followed the pattern “author: headline,” hiding the author in a single field. The data was loaded, but it wasn’t usable.

I considered fixing this in Python, but adding another ad‑hoc script would duplicate the untested, undocumented step I already had. dbt lets me write SQL transformations that are version‑controlled, tested, and documented in the same workflow. I learned that this is how industry teams handle the “T” in ETL.

Installing dbt‑postgres failed because I was on Python 3.14, which isn’t yet supported. Downgrading to 3.12 and creating a virtual environment solved the issue. I defined the raw articles table as a source, built a staging model that casts the date with to_timestamp and splits the title into author and article_title, then ran dbt to produce clean, analysis‑ready data.