The author built a job market pipeline using RemoteOK and Hacker News data, enriched with Gemini Flash-Lite AI to extract skills, seniority, and Nigeria eligibility. The pipeline loads data into DuckDB, cleans it via dbt, and uses snapshots to track job openings. The AI enrichment step, implemented in `enrich_jobs.py`, asks the model for five structured fields per job: skills, seniority, years of experience, open_to_nigeria, and summary.
The Nigeria eligibility field is critical for the author, who is job hunting from Lagos, as many "remote" roles are country-restricted. Using a free Google API key avoided billing issues encountered with BigQuery. The script ensures data quality by preserving job descriptions, enforcing strict JSON output, and validating results with Python checks and dbt tests.
The article details the architecture, code, and real-world challenges faced during implementation.
Source: Towards Data Science · Summarized by HeadlinesBriefing