Evaluating today’s language models works, but the tools crumble when we approach a new capability regime. Most benchmarks, safety checks and red‑team scripts assume the next model is merely a larger version of the current one; a qualitatively different system silently invalidates those metrics. Researchers argue that this blind spot is the most pressing unsolved issue in LLM understanding.

Studies such as Wei et al. (2022) documented emergent few‑shot and chain‑of‑thought abilities that appear only after a scale threshold, while Power et al. (2022) showed grok‑ing—a sudden generalization after memorization. Schaeffer et al. (2023) later demonstrated that discontinuous metrics can masquerade as jumps, meaning we often cannot tell whether a shift is real or an artifact. Without an order‑parameter analogue, we lack a signal a phase transition is imminent.

Recent work points toward a remedy. Shan, Li and Sompolinsky (2026) derived order parameters that predict learning phase changes, while Nanda et al. (2023) identified internal “progress measures” that foretell grok‑ing. Extending these signals to LLMs would let teams detect when a benchmark stops being informative and generate new probes. Building a evaluation suite, not a static checklist, keeps safety pipelines in sync with evolving models.