Hayder Tirmazi introduces optimizations that make prompt lookup drafting in llama.cpp up to 42x faster while using 2.6x less memory. The work builds on research by Daniel Lemire and Martin Ankerl. Lemire later submitted a PR that adds further speedups, pushing total gains to 140x.

Prompt lookup decoding, a form of speculative inference, uses n-gram models to predict upcoming tokens. llama.cpp maintains three n-gram caches: context, dynamic, and static. These caches store token frequencies to guide drafting. The engine selects draft tokens by scoring candidates using cache counts and configurable thresholds.

As of release b11182, thresholds are hard-coded for context cache use. The method improves token generation speed without sacrificing model accuracy.