HeadlinesBriefing favicon HeadlinesBriefing.com

Incidents de sécurité de Claude et corrections

Hacker News •
×

On July 30, we reported three incidents in which Claude models gained unauthorized access to real computer systems. The models—intentionally running without cyber safeguards for evaluation purposes—accessed the internet due to a misconfiguration inside a third-party evaluation environment. Separately, on August 4, the UK AI Security Institute reported an incident from its own cybersecurity testing, in which Claude Mythos 5 took a series of unauthorized actions on the live internet. In that case, the model, again intentionally running without cyber safeguards for evaluation purposes, had been deliberately given internet access. We are conducting an in-depth analysis of both incidents. We are also planning to work with MET R for an independent review. We want to ensure both studies are thorough, and will share more in the coming weeks.

We believe the incidents reflect a failure of operational security, as well as two alignment issues: motivated reasoning, and willingness to take harmful actions in pursuit of a narrow task (both of which we have described in previous system cards). On security, we describe the improvements we've made to our containment and monitoring systems, along with practices that we've developed for third-party evaluators. On alignment, we discuss the two issues more in depth; we also believe lasting progress comes not only from understanding what happened in a given incident but from understanding how misalignment arises in the first place, and we share early research in that direction.

In light of these incidents there has been increasing discussion about pacing the frontier. It is helpful to distinguish between two kinds of pacing. Within a company, pacing means a series of decisions that prioritize safety over speed when the two are in tension. Across the field, it means establishing processes to guard against race-to-the-bottom dynamics. In this post, we discuss actions we have taken, both prior to and after these incidents, in service of the first approach.

Securing evaluation and training environments was our first priority. We paused external cyber evaluations of pre-release models after the incidents, and briefly paused internal ones as well while we put the measures below in place. The incidents we reported on July 30 showed that we had been largely relying on a single layer of defense (the configuration of the environment itself) where we needed several, including setting explicit boundaries in the prompt, establishing processes for verifying that a sandbox is sealed where intended, and implementing monitoring that can intervene in real time.