We previously noted that while coding agents can easily meet quality bars using effective test techniques, software quality appears to be worsening, suggesting developer defaults may not work well. Here, we test whether simple instructions to agents to use specific test techniques or libraries improve implementation correctness, evaluating how effective agents are when guided by non-experts who’ve heard certain methods should be applied. We reuse the Zstd implementation eval from a prior comparison of agentic programming language effectiveness, now comparing different testing techniques and libraries when agents implement Zstd with various prompt addendums like 'Use test-driven development', 'Use Lean 4', or 'Use Quick Check'.

Additional evals on the IMAP RFC are briefly discussed. All implementations were in Rust. The 26 prompt conditions tested included ACL2, Alloy, 'Audit and fuzz risky areas', 'Audit first', Creusot, Default (no additional instructions), Differential testing, Fuzzing, Hegel, Insta, Judgement, Kani, Lean 4, 'Make no mistakes', Metamorphic testing, Mutation testing, Property-based testing, Proptest, Quick Check, rstest, Rust built-in test framework, SMT solvers (with Z3, cvc5, and Yices), Spin, TDD, TLA+, and Verus.

Four skills were also tested: Hegel with the official Hegel skill, the ECC Rust test skill (250k GitHub stars, 38k forks), the Trail of Bits property test skill, and a custom skill written by the author. Predictions were pre-registered: TDD will underperform (55% confidence), formal methods will not overperform (52% confidence), and 'Make no mistakes' will not outperform no instructions (95% confidence). The ECC test skill is expected not to outperform (65% confidence).

All predictions are low confidence due to uncertainty about agent behavior.