HeadlinesBriefing favicon HeadlinesBriefing.com

RAG Pipeline Handles Diverse PDFs

Towards Data Science •
×

The article showcases a production RAG pipeline that stitches together four upgraded bricks—document parsing, question parsing, retrieval, and generation—into a single linear call, pdf_qa. A PDF enters, and a typed, cited answer exits, a process first tested on a 15‑page arXiv paper, Attention Is All You Need, with a simple question about positional encoding.

Each brick was refined independently. Parsing now yields a relational set with a TOC and a parsing_summary; question parsing transforms noisy input into a structured brief; retrieval consults the document’s own TOC via a small LLM and merges keyword hits; generation returns a typed answer with a single citable span per item and quality indicators.

The four bricks are wired into pdf_qa so that one call accepts a PDF path and a question, returning the answer and a full audit trail. Two side channels—parsing_summary feeding both LLM bricks and a feedback loop from generation to retrieval—enable dynamic adaptation without re‑training.

The same pipeline is run unchanged on three very different PDFs: the *NIST Cybersecurity Framework 2.0*, the original Retrieval‑Augmented Generation paper, and the World Bank Commodity Markets Outlook (April 2024), which ships with a broken table of contents, demonstrating its robustness across document styles.