Replacing GPT-4 with a local small language model (SLM) initially seemed like a solution for unreliable CI/CD pipelines, but non-deterministic outputs created new challenges. The team’s document classification system, which extracts structured fields like methodology_type and dataset_source, faced failures due to GPT-4’s subtle formatting inconsistencies—such as camelCase vs. snake_case keys and markdown code fences. Despite temperature=0 settings and response_format constraints, failures persisted at 9 per six weeks, costing ~18 minutes per fix.

Attempts to enforce strict outputs included function calling with OpenAI’s schema contracts, which reduced failures to 23 down to 9 but introduced API dependency risks. A local parser cleaned messy outputs temporarily, but structural errors still surfaced three steps later. The breakthrough came when testing four SLMs: Qwen2.5-7B-Instruct achieved 90-95% accuracy across 50 documents with zero diffs in repeated runs, outperforming GPT-4’s inconsistency. Phi-3-mini struggled with long documents, while Mistral 7B and Llama 3.2 offered mixed results.

Qwen2.5’s deterministic outputs eliminated the need for retry logic, resolving the core issue. Though GPT-4 handled ambiguous cases better, the cost and unreliability of frontier models proved unsustainable. The team now prioritizes structured tasks for local models, reserving GPT-4 for nuanced reasoning. This shift highlights the hidden costs of probabilistic AI in mission-critical systems.

Key takeaway: For fixed-schema data extraction, local SLMs like Qwen2.5-7B-Instruct deliver reliability at scale, making them preferable to expensive, unstable frontier models in production environments.