Prompt changes can silently break production behavior without detection, creating hidden failures that surface weeks later through user complaints. The author discovered this firsthand when adding document routing instructions caused negation queries to misclassify, despite overall accuracy appearing improved.

Most teams lack regression testing for prompts, treating them as static configuration rather than stochastic APIs that shift behavior across all query types with each modification. A prompt isn't just code—it's a contract that changes with every instruction added.

The proposed solution uses 40 golden queries across six intent categories, testing four prompt versions with deterministic validation. This approach caught a 66.7% negation classification collapse in v4 despite its 67.5% overall accuracy score—a False Improvement pattern where aggregate metrics hide catastrophic category failures.

Built in pure Python with zero external dependencies, the suite runs in under two seconds and validates outputs through schema checks, pattern matching, intent verification, and guard clauses. By avoiding LLM-as-a-judge scoring, it provides consistent, cost-free regression detection that treats prompt behavior as the contract problem it actually represents.