Anthropic, the company developing the Claude chatbot, has taken a step that no other major AI company has yet agreed to. Six researchers from METR, a nonprofit organization that tests the most powerful artificial intelligence models, have been granted employee-level access at Anthropic. They are to independently examine the company’s models, its safety procedures, and how it manages risks. They will publish their findings themselves, and Anthropic has waived the right to prevent them from doing so.
The organization specializes in determining what the latest models can do and where they may pose dangers, and over the past year it has already conducted several reviews of Anthropic’s risk reports. Now, instead of conducting reviews remotely, its researchers will work directly inside the company.
The reason is understandable. Artificial intelligence is no longer just a chatbot that answers questions. Companies are deploying so-called agents, programs that independently carry out tasks, write code, or navigate the internet. Since late August, reports have emerged roughly every two weeks of agents doing something no one instructed them to do. This has only amplified calls for independent oversight of AI companies.
Taking the company’s word is not enough
We want to be publicly accountable for the decisions we make, Anthropic wrote on its website. The catch is that there are many things it cannot disclose to the public. Some information could help people with malicious intentions, while other information is commercially sensitive, such as data on computing power or how models are trained. The proposed solution is an intermediary that the company can show more information to and that can then tell the public what it has concluded.
According to Anthropic, such oversight must meet four conditions to be meaningful. Evaluators need expertise, including an understanding of how tests can go wrong. As an example, the company cites a phenomenon in which a model merely pretends to follow the rules because it suspects that it is being tested. It behaves like a student who shows their best side during an exam but does something completely different at home. Evaluators must not own a stake in the company they are assessing or be financially dependent on it. They must be able to see not only the finished models, but also how they were created and how the safeguards work. They must also be free to publish their findings.
Anthropic compares this to banking. Banks understand their risks better than anyone outside them, which is why regulators send examiners directly into the largest banks. They are supposed to identify dangerous risk-taking before the public pays the price. The company itself acknowledges that the comparison is not perfect, but the principle is similar. Those on the inside can see more.
Exactly what is and is not allowed
METR’s staff have company laptops, office space, and access to company tools at roughly the same level as Anthropic’s own teams that assess risks. Employees are expected to speak with them as openly as they would with a colleague investigating whether artificial intelligence is actually doing what people want it to do.
The exceptions concern customer data, information protected by attorney-client privilege, and the technical architecture of the models. Regarding the last category, the company explains that evaluators may also work for other companies, creating a risk of inadvertent disclosure, and that this information is generally not very relevant to risk assessments.
The disclosure rules are attracting the most attention. Anthropic cannot block publication; it may only request the redaction of information falling within predefined categories. METR does not have to remain silent about what information was redacted. It may publicly state whether it believes the redaction was justified, provided that it does not reveal the concealed information itself. However, the company ultimately decides what falls within the exempted categories. Anthropic counters concerns that this could create a back door for competitors by noting that the evaluators form a small team bound by enforceable contracts and that other METR employees do not have access to their work.
Three levels of trust
For now, METR is the only organization to have reached the top of the three-tier system. Evaluators at the lowest tier conduct specifically focused reviews and need the company’s approval to publish their findings. The middle tier receives more information and may publish without approval. The highest tier adds employee-level access.
SecureBio, an organization focused on biological safety, is at the first tier. It is currently reviewing the chapter on chemical and biological weapons in the August risk report and previously published a trial review of the preceding report. Anthropic is preparing a broader collaboration with consulting firm Accenture and the company Faculty, which is expected to include model evaluations, attempts to break them, and tests of safety safeguards. The company is also in preliminary talks with other organizations, including Hugging Face.
The evaluators did not appear out of nowhere. In a September essay, Anthropic CEO Dario Amodei proposed embedding independent evaluators in AI laboratories and described this as the first step toward slowing the development of the technology. OpenAI quickly promised the same, and both companies, together with Google, Meta, Nvidia, and xAI, subsequently signed a White House agreement committing to independent assessments. Anthropic is now the first to show what such a promise might look like in practice, emphasizing that it is voluntarily granting evaluators access and that every company developing the most powerful models should do the same.
Sources: independent.co.uk, techcrunch.com and theinformation.com
