When a data team tried to turn a hundred noisy compliance PDFs into structured JSON rules, they first fed the raw text to a large language model, expecting a single prompt to produce accurate output. Initial results looked valid, but manual sampling revealed overly broad rules, missed items, and lost nuances, exposing the fragility of a brute‑force LLM approach in production environments and scale.

The author reshaped the pipeline by shrinking the model’s task. They cached each PDF locally, stripped irrelevant metadata, and fed a single document at a time to a dedicated sub‑agent. Parallel workers logged progress, wrote reference IDs into every generated rule, and allowed the surrounding code to enforce schema, handle retries, and audit outputs against the original source for downstream processing.

By separating semantic reasoning from mechanical validation, the workflow became auditable and resilient. Reference IDs let engineers verify that each rule matched its source chunk, while lightweight evals on a sample batch confirmed coverage without a full golden set. The experience reinforced a lesson echoed at AI Engineer Singapore: treat LLMs as specialized components, not all‑purpose problem solvers for real.