Harnesses significantly impact AI inference efficiency by managing prompt caching. Neuralwatt data shows uncached input varies from 1.6% to 11.1% across clients like Open Code and Pi. A 92% cache rate results in 2.7 times more prefill work than 97%. Cache breaks—caused by harnesses editing prior prompts—force full recomputation from the first change. Deep sessions amplify costs: re-reading over half a 128k-token prompt equals 600 normal turns in energy. Clients differ dramatically: deepseek-harness breaks cache ~1% of the time, while Open Code and Pi do so ~3%. Open Code 2.x improved in lab tests but not in production. Energy measurements confirm cache misses roughly double consumption; on GLM-5.3, misses cost 2.5x more. The harness, not just model or hardware, is a major efficiency lever.
Most agentic traffic sends 100k+ input tokens for minimal output. Servers reuse KV caches for unchanged prefixes, making caching the top efficiency source. Any edit by the harness—even a timestamp—invalidates the entire prefix. Neuralwatt’s cloud data over 30 days reveals the harness dictates a surprising share of GPU workload. Optimizing prompt construction could yield substantial energy and cost savings without changing models or hardware.
Source: Hacker News · Summarized by HeadlinesBriefing