HackerRank open-sourced their hiring-agent ATS, which quickly gained attention on LinkedIn and Reddit. An engineer tested the tool with their resume and found a fundamental problem: identical resumes scored anywhere from 66 to 99 out of 100 across 100 runs. This wild variation exposes serious reliability issues in AI-powered resume screening.

The system parses PDF resumes and calls LLMs six times to extract structured information including work history, education, and skills. It pulls GitHub profiles and feeds all data into an LLM running gemma3:4b at temperature 0.1. Scoring allocates points across categories: 35 for open source contributions, 30 for personal projects, 25 for work experience, 10 for technical skills, plus up to 20 bonus points.

Technical skills score consistently because they're checklist-based, but project evaluations swing dramatically between runs. Experience scoring fails equally badly - everyone from interns to principal engineers receives 25/25. Even temperature 0 cannot eliminate this inherent non-determinism, revealing that LLMs struggle with subjective judgment calls required for hiring.

With 65% weighting on open source and projects, the tool cannot distinguish between candidates with vastly different qualifications. Engineering teams should avoid deploying this approach - unreliable AI screening doesn't filter for quality, it simply introduces arbitrary rejection rates that penalize qualified candidates.