When a RAG system looks fine yet delivers wrong answers, the culprit is often retrieval failures or hallucinations. The only way to know and mitigate this is to evaluate it regularly. This guide walks through building an evaluation pipeline that starts with a simple manual pass, adds automated scoring with RAGAS, and extends to custom LLM judging and human‑in‑the‑loop monitoring.

Before any evaluation library, you need a *golden dataset* – a set of questions with known correct answers and source documents. Skipping this is the most common mistake; it prevents you from distinguishing retrieval failures from generation failures. A good entry includes the question, ground truth, and source doc; even a few dozen such entries can surface real signal.

Run the golden set through your pipeline and read the generated answers next to the ground truth. This manual pass catches broken prompts, empty retrievers, or models ignoring context. Automating scoring only makes sense once you’re confident the dataset and pipeline are sound.

With the basics in place, RAGAS evaluates on context precision, recall, faithfulness, and answer relevancy. A high faithfulness score can still hide a stale source, so add a custom LLM judge that checks numeric accuracy, recency, or domain‑specific constraints. Finally, deploy the process as a continuous pipeline, watch for drift, and involve humans when needed.