Researchers compared classical hyperparameter optimization algorithms against LLM-based methods using autoresearch, a repository that enables LLM agents to optimize hyperparameters by directly editing training code. When tested on tuning a small language model under fixed compute budget, classical methods like CMA-ES and TPE consistently outperformed pure LLM approaches, particularly when avoiding memory failures mattered more than search diversity.

Allowing LLMs to directly edit source code narrowed but did not close the performance gap, even with frontier models like Claude Opus 4.6 and Gemini 3.1 Pro. The study found LLMs struggle to track optimization state across trials, while classical methods lack the domain knowledge that LLMs possess. This led to the introduction of Centaur, a hybrid approach combining CMA-ES's interpretable internal state with an LLM.

Centaur achieved the best results in experiments, with a 0.8B LLM sufficient to outperform all classical and pure LLM methods. Unconstrained code editing required larger models to be competitive with classical approaches. The research demonstrates that LLMs are most effective as a complement to classical optimizers rather than a complete replacement, highlighting the value of combining both approaches for hyperparameter optimization.