Retrieval-augmented generation (RAG) answers questions using a document corpus. Typical evaluation sends a well-written question to a tidy corpus and checks if the retriever returns the correct passage. Production collections rarely stay tidy: old pages remain after policy changes, users type typos like "warehuse," OCR may read "SSO" as "SS0," and tables split across pages. Each problem can send the retriever to the wrong passage, giving the language model incorrect context.

A small retriever with fault injection—deliberately adding problems—tests system response. I added an outdated policy page, an OCR character swap, a query typo, and a divided table. The toy pipeline uses 4 short documents and a scoring function that splits queries and documents into lowercase tokens, giving 1 point per matching token. The highest score wins.

Stale documents contradict: returns policy changed from 60 days to 30 days in 2026, but ingestion added the new page without removing the old one. Both score equally, so list order returns the 2024 policy. A recency tiebreaker fixes this.

OCR errors hide matching words: "Enterprise" becomes "Enterpr1se" and "SSO" becomes "SS0." All documents tie at 0, returning the returns policy. A normalization function corrects known OCR substitutions before tokenization.