A recent study by Anthropic Fellows investigates how AI systems fail, focusing on whether they systematically pursue wrong goals or exhibit incoherent behavior. The research decomposes model errors into bias (systematic) and variance (incoherent) components, revealing that as tasks become more complex, AI failures increasingly resemble a hot mess rather than coherent misalignment. This suggests future AI failures might look more like industrial accidents than the pursuit of unintended objectives.

The study, conducted as part of the Anthropic Fellows Program in Summer 2025, analyzed frontier reasoning models such as Claude Sonnet 4 and Qwen3 across various benchmarks. Researchers found that as models spend more time reasoning, they become increasingly incoherent, with longer reasoning leading to more unpredictable outcomes. This trend holds across different tasks, from multiple-choice questions to coding and safety evaluations.

These findings have significant implications for AI safety. As models scale, they tend to become more incoherent on difficult tasks, indicating that scaling alone won't eliminate failures. The study suggests that future AI failures may resemble industrial accidents, where models fail unpredictably rather than pursuing specific misaligned goals. This insight could influence how developers and safety researchers approach AI alignment and risk mitigation strategies.

Looking ahead, this research underscores the need for better tools and frameworks to manage AI coherence. Companies like Anthropic are at the forefront of these efforts, and their findings could shape the development of more reliable AI systems. As the field progresses, understanding and mitigating incoherence will be crucial for ensuring the safe and effective deployment of advanced AI models.