A researcher has developed an algorithm that computes optimal tokenizers in certain settings, tackling a problem that's theoretically intractable but practically solvable. The work connects tokenization to integer linear programming, representing datasets with color and edge variables to minimize token count while maintaining vocabulary constraints.

The approach builds on Tempus et al.'s framework, which formulates tokenization as an ILP with flow constraints ensuring valid encodings. While state-of-the-art tokenizers already achieve near-optimal results (often within 1%), the author applied cutting-plane techniques inspired by TSP solvers to tighten bounds and improve solutions.

However, the method has notable limitations. Optimal tokenizers on training data don't necessarily generalize better to test data, and inefficient tokenizers remain acceptable since vocabulary size can compensate. The current implementation also requires pretokenization, restricting solutions to being 'near optimal' under that pretokenizer.

By using Codex to search for valid cuts through auxiliary linear programs examining word pairs and triplets, the author found constraints that significantly improved both lower and upper bounds. The research demonstrates that integer linear programming can yield practical results for LLM tokenization, even when theoretical guarantees suggest otherwise.

Despite practical caveats, the work opens avenues for improving tokenizer optimization through algorithmic advances rather than accepting heuristic approximations.