HeadlinesBriefing favicon HeadlinesBriefing.com

Deterministic Prompt Pruning Cuts LLM Token Usage 34%

Towards Data Science •
×

Long-running LLM conversations accumulate stale tool outputs, duplicate retrievals, and expired context that inflate prompts without adding value. The author built a three-pass architecture that prunes this redundancy before the model sees it, using only standard-library components — no embeddings, no external calls, no non-determinism. Pass 1 drops expired tool results by keeping only the newest call per key. Pass 2 normalizes whitespace and casing to eliminate duplicate retrieved chunks. Pass 3 restores any message that a later turn still depends on, using a lightweight DEFINE:/REF: tagging scheme the developer adds to messages.

Benchmarks across 15 configurations — plain chat, RAG assistant, and tool-heavy agent at five conversation lengths each — show workload-dependent savings: 2–4% for chat, 27–32% for RAG, and 33–34% for tool agents. All labeled required facts survived. Preprocessing stays under 50 milliseconds even at 2,000 turns and 131,000 tokens, and the pipeline reaches a stable fixed point after one pass. Two bugs discovered during development — a synthetic benchmark that masked real-world scaling and untested dependency restoration — are documented alongside fixes.

The complete implementation, 35 tests, and raw terminal output live in a public GitHub repository. This approach sidesteps the brittleness of positional truncation, which silently drops hard dependencies like user preferences stated dozens of turns earlier, while avoiding the unpredictability of model-based relevance scoring.