HeadlinesBriefing favicon HeadlinesBriefing.com

Spec-Driven Test Automation: Real Run & Fix

Towards Data Science •
×

Part 2 follows a real test run using the qikly framework, where an independent testing agent catches a flaw the coding agent missed. The task: check radar headway and warn if under two seconds. The coding agent, working only from requirements, allowed a zero-metre gap—invalid per unseen test criteria.

After one failed test, it patched the code by changing one character, tightening the lower bound solely due to the hidden test, not explicit rules. This took under a minute and fewer than ten model calls. The first spec version ambiguously warned under 'the two-second rule', leaving exactly 2.00 seconds unresolved.

Across ten runs, only three converged; seven failed due to misinterpreted ambiguity. This proves that green suites can mislead if verifiers aren't independent—echoing GM's Super Cruise rule: builders must not verify their own work. Hidden criteria create false confidence; only split-agent workflows reveal true correctness.