Frontier LLMs are surprisingly good at natively reading and writing neural network weights, a capability well beyond humans or conventional algorithms. This breakthrough impacts optimization and interpretability. Simply observing raw weights, LLMs can predict a network's trained function.
Furthermore, without environment feedback, LLMs can write small game-playing networks from scratch that outperform networks trained with PPO for 100 million steps. In a specific test, an LLM identified a reasoning loop issue in Qwen3.5-0.8B, warned about in the model card, and fixed it by natively editing 0.016% of weights. This correction raised MATH-500 scores from 7.9% to 48.1% and generalized to instruction following, raising IFEval by 3x.
This process, termed "Weight-Space Programming," allows general-purpose LLMs to read and write network weights directly around natural language goals, sometimes without environment feedback. Historically, deep learning relied on symbolic optimizers like Adam. This paradigm shift suggests neural network weights will soon be natively optimized by LLMs that reason about their structure.
The authors pose critical questions: If asked to handcraft a Go-playing network, where would one start? Today, weights are typically written by optimization or compilers. Now, LLMs are becoming eerily good at understanding and crafting weights for natural-language goals, signaling the arrival of reasoning-driven optimization for differentiable networks.
Source: Hacker News · Summarized by HeadlinesBriefing