Anthropic research scientist Nicholas Carlini warned at [un]prompted 2026 that large language models (LLMs) are now capable of automating sophisticated cyberattacks once requiring human expertise. His talk highlighted how state-of-the-art models can identify 0-day vulnerabilities in software systems tested by humans for decades, escalating risks for cybersecurity defenses.

Carlini demonstrated that LLMs could scale attacks at lower costs, such as generating phishing campaigns or exploiting code weaknesses. He specifically cited Anthropic's Claude as an example of models with potential misuse scenarios, emphasizing the need for proactive security measures. The research suggests current testing frameworks are insufficient to detect these AI-driven threats.

The findings raise urgent questions about AI safety protocols. Carlini argued that adversarial actors could weaponize LLMs for real-time threat generation, outpacing traditional security tools. This shift demands updated frameworks for model auditing and deployment safeguards, particularly in high-stakes sectors like finance and infrastructure.

The talk underscores a pivotal moment: AI's dual-use nature requires immediate collaboration between developers and cybersecurity experts. Without intervention, the AI security arms race could spiral into unmanageable risks, Carlini cautioned, stressing that real-time threat detection systems must evolve alongside offensive capabilities.