Modern search apps must return a coherent set of results, not variations of one match. For "camping gear", users want a tent, sleeping bag, stove, and headlamp. Systems use query fan-out to break a broad prompt into sub-queries. But teaching an LLM database-aware query decomposition drains a massive thinking budget. Zero-shot LLMs are general text predictors, not optimized for a target corpus, so they need extended test-time computation to optimize set-level properties like diversity and coverage while staying grounded.

In the ICML 2026 paper "Efficient, Property-Aligned Fan-Out Retrieval via RL-Compiled Diffusion", we address this via a reward-to-data compilation framework. Instead of a large inference thinking budget, our Retrieve-for-Train framework uses offline reinforcement learning to discover reward-aligned fan-outs and compile them into supervision. By distilling these behaviors into a lightweight diffusion retriever, we enable efficient, single-pass query fan-out at inference.

Off-the-shelf LLMs face two challenges: paraphrastic collapse, where they generate redundant queries like "bohemian festival fashion" and "bohemian festival clothes" instead of fringe jackets or crochet dresses; and autoregressive latency bottlenecks, requiring hundreds of chain-of-thought tokens. This creates a latency floor at odds with sub-second search. Retrieve-for-Train treats training as an offline practice session.