Data analyst turned engineer documented his production push of a GitHub‑API ETL script. The original version extracted repo metadata, cleaned rows, and wrote a CSV, which sufficed for a learning exercise. When he reran it, the pipeline produced duplicate records and revealed that a simple script lacks persistence, queryability, testing, and the reliability expected of real data engineering.

Switching to a SQLite file gave the pipeline a true database layer, enabling SQL queries and record checks. He added an idempotency guard that deletes existing rows before inserting fresh data, eliminating duplicates on repeated runs. To survive Colab session resets, he mounted the database on Google Drive, ensuring the dataset persists beyond notebook lifecycle. He also indexed the URL column for faster lookups.

The final obstacle was automation: a notebook cannot run unattended at night. He recognized that production pipelines rely on schedulers such as Apache Airflow, Prefect, or cloud cron jobs to trigger jobs, handle failures, and log history. By moving from ad‑hoc scripts to orchestrated workflows, the project crossed from scripting into genuine data‑engineering practice. Such orchestration also provides alerting and metric collection for operational insight.