An AI agent passed every metric in the eval harness I published, yet the CFO shut it down because its cost per resolved ticket exceeded the human baseline. The mid‑market SaaS company built a customer‑operations agent to handle billing and account tickets. It met all twelve quality metrics—task completion, faithfulness, retrieval precision, tool‑call accuracy, latency, hallucination rate—yet each attempt cost about $3.40 in inference. With a 71% resolution rate, the agent’s total spend on 1,000 tickets was $4,618, 10% higher than the $4.20 cost of fully human resolution.

The harness was designed to answer whether an agent behaves correctly, not whether it pays for itself. Quality metrics mask the economic reality: a successful outcome must absorb the cost of failed attempts and the double‑payment that occurs when an agent escalates to a human. The only metric that determines survival is cost per successful outcome, which is harder to compute because it requires attributing failed‑attempt costs, accounting for escalation, and comparing against the human alternative.

In practice, teams track cost per call and see a comfortable number, but the true figure—$4.79 per resolved ticket—reveals the agent was actually 59 cents more expensive per success. The harness therefore needed a new metric: cost per successful outcome.

The CFO was right; the agent was accurate but economically underwater, and that difference decides whether an AI agent survives kres.