HeadlinesBriefing favicon HeadlinesBriefing.com

Reproducing BM25, Dense Retrieval, SPLADE on 16GB MacBook

Towards Data Science •
×

Reproducing BM25, Dense Retrieval, and SPLADE on a 16GB MacBook using Castorini's Foundations of Retrieval onboarding path reveals practical gaps in official documentation. The path uses Anserini and Pyserini toolkits on MS MARCO (8.8M passages) and NFCorpus (3,633 documents) with exact expected scores.

Three setup issues appeared before any retrieval command ran: cloning full git history timed out (fixed with shallow clone), a submodule connection reset (fixed by raising git's HTTP buffer), and Java 21 requirement (default JDK was Java 11). None are exotic but likely hit anyone not configured like maintainers.

The first real exercise builds a Lucene inverted index over all MS MARCO passages for BM25. On a 16GB machine, the indexing command fails because `bin/run.sh` hardcodes a 192GB Java heap ceiling. The OS out-of-memory killer ends the process after 3.4M of 8.8M documents with only "Killed: 9" output.

For RAG systems, document search products, or internal knowledge bases, retrieval quality is the first thing to verify. Before changing prompts or swapping LLMs, you need to know whether the right documents are found. Reproducing these baselines gives a reference point: what BM25 can do, what Dense Retrieval adds, what learned sparse retrieval costs, and how to check your scoring pipeline.