Enterprise teams building RAG systems often reach for cross-encoder rerankers when initial retrieval fails. bge-reranker-base seems like a logical upgrade—smaller than LLMs but smarter than cosine similarity. Teams wire in the reranker, send top-100 candidates, keep the top-10. Some previously broken queries work better, encouraging the approach.

But the magic fades quickly. When users ask for all clauses mentioning termination, the system returns exactly three instead of eleven. Domain-specific terms like non-employee labor get missed entirely. Negation failures persist—cross-encoders don't understand logical complementation any better than embeddings. Meanwhile, latency jumps to hundreds of milliseconds since cross-encoders can't precompute scores. Surprisingly, text-embedding-3-large alone often matches or beats the ada-002 plus reranker combination.

The classical retrieval funnel—bi-encoder narrowing millions to thousands, cross-encoder to dozens, LLM final processing—assumes each stage justifies its cost. In practice, the cost-performance gradient flattens or inverts. Teams lose months expecting miracles from individual components rather than auditing the entire pipeline.

The editorial position argues for architectural moves experts can understand: expert vocabulary, structure-aware retrieval, and question-specific pipelines. These earn more trust per dollar than stacking statistically distinct but opaque scorers.