The AI serving layer is shifting from simple model selection to on‑the‑fly collaboration. Researchers behind the vLLM Semantic Router and Sakana Fugu propose a router that treats a model call as a surface, then orchestrates bounded micro-agent to synthesize answers. By keeping the OpenAI‑compatible API ({"model":"vllm-sr/auto"}) unchanged, developers gain advanced routing seamlessly in production without rewriting client code.

Four looper patterns drive this capability: Confidence escalates from cheap to expensive models based on a confidence threshold; Ratings runs a capped parallel ensemble and aggregates results with quality scores; Re Mo M fans out multiple reasoning attempts, waits for a quorum, then synthesizes a contract‑compliant answer; Fusion gathers divergent model outputs as evidence for a judge model. Each pattern respects budget, latency, failure policies.

The router selects a looper automatically via “auto recipes” that map request signals—difficulty, risk, latency, cost—to the appropriate algorithm, preserving a single model identity. Early benchmarks show that no single loop dominates; GPQA‑Diamond benefits from Fusion, live‑code tasks favor Re Mo M, while SWE scenarios need Workflows. The approach lets operators embed safety and cost controls directly in the inference stack.