Anthropic has announced that it has begun giving employee-like access to a team from MET R, a nonprofit that evaluates frontier AI models. MET R will publish the results of its risk assessments. The move follows the company's commitment to stronger independent, third-party assessment of its models and safety practices, and its signing of the White House Accord on Super Intelligence, in which companies pledged to partner with independent external auditors or evaluators.
Over the past year, Anthropic has worked with MET R on several reviews of its Risk Reports. The company is also working with Secure Bio and Accenture as evaluators and is in early discussions with several other organizations, including Hugging Face. Anthropic says that as models have become more powerful, they require stronger safeguards and more challenging risk assessments, and that the company wants to be publicly accountable for its decisions. Because some safety information could be useful to malicious actors or is commercially sensitive, the company is instead giving third-party evaluators deep access and having them share evidence and reasoned conclusions.
To qualify, evaluators need expertise in assessing frontier models, independence from the companies they evaluate, enough access to meaningfully assess risks, and freedom to publish their findings. Anthropic compares the approach to bank supervision, in which regulators place resident examiners inside the largest banks.
Anthropic has onboarded six MET R employees with company hardware, office space, and access to internal tools roughly on par with its own risk assessment teams. Anthropic cannot veto publication, though it can require redactions in narrow, predefined categories such as customer data, privileged information, or model architectures. The company says the arrangement is not a way for other organizations to obtain its information or that of its customers and partners.
Source: Hacker News · Summarized by HeadlinesBriefing