Your RAG Pipeline Is Missing a 1994 Algorithm
Embeddings understand concepts well but miss specific identifiers like error code ZX-4821. A customer-support assistant asks about this code, which exists in the knowledge base with a hardware failure explanation. However, the retriever returns general device error documents instead, missing the exact match. Semantically, the results discuss the correct product and problem type, but none contain ZX-4821. Lexical search misses the string that matters, while dense retrieval finds semantically similar content. The article traces the history of information retrieval from exact keyword matching through to dense vector retrieval and hybrid systems. It explains that exact matching preserves rare terms but cannot cross vocabulary gaps, while latent methods cross those gaps but may make exact identifiers less distinctive. This trade-off shows how retrieval evolved by sacrificing different capabilities rather than uniformly improving. The 1994 algorithm (BM25) survived not because it was the best, but because dense retrieval solved a distinct problem—semantic similarity. The core question is whether progress in retrieval has allowed us to stop choosing what information to lose.
Source: Towards Data Science · Summarized by HeadlinesBriefing