HeadlinesBriefing favicon HeadlinesBriefing.com

Context Windows Don't Know What's Still True - Validity Layer Benchmark

Towards Data Science •
×

I built a deterministic benchmark in pure Python to measure the cost of acting on stale context windows. The core problem: context windows remember what happened but don't know whether that information remains valid. Two executors perform identical work — one validates dependencies before acting, the other discovers invalidity only after failure. That small difference creates measurable waste in steps and failed tasks.

I ran a 96-configuration sweep expecting graph shape to drive wasted work, but size proved the real driver. I updated the experiment rather than forcing my hypothesis, yielding more precise results. The benchmark uses a flight pricing scenario: at 10:00 Flight A costs $420, a plan forms at 10:01, price jumps to $610 at 10:03, yet the baseline still books at 10:04 based on the stale $420 assumption. The validity-aware executor checks price validity before acting and replans immediately.

This isn't context loss — nothing was forgotten or cut off. The fact driving the wrong decision sits in the window but stopped being true minutes ago. The validity layer doesn't predict the future; it re-verifies facts right before actions rely on them. The runnable repo demonstrates this deterministic gap between executors with real numbers.