The author has been documenting a transition into data engineering, having already built two pipelines: a scheduled GitHub ETL pipeline storing data in SQLite, and an automated RSS pipeline loading tech articles into PostgreSQL, orchestrated with Kestra and containerized with Docker for deployment to AWS.
For this new project, called Level 3, the goal is to build a pipeline that tracks the job market for the exact roles being pursued: data engineer, analytics engineer, ETL developer, and related positions. Every run checks whether employers are hiring for the kind of role the author wants. The project is a form of learning in public, building the tool the author wishes existed while still searching for a first data engineering job.
The design pulls from two sources, Hacker News and a job board, lands the data in a warehouse, orchestrates the work with a new tool, and uses dbt snapshots to track when listings open and close over time. Apache Airflow was chosen as the orchestrator because it is the name recruiters search for most often, replacing Kestra for this project.
BigQuery was the first warehouse choice, but Google Cloud requires a mandatory $30 prepayment to link a billing account in certain countries before its IAM and service account features unlock, even on the free tier. The author switched to DuckDB instead. dbt abstracts most warehouse-specific syntax, so the transformation and snapshot skills should transfer if the project later moves to BigQuery or Snowflake.
Source: Towards Data Science · Summarized by HeadlinesBriefing