American company Anthropic admitted in its IPO filings that it may be approaching one of the most closely watched thresholds in the development of artificial intelligence. Its models could soon significantly accelerate their own development, and the company may not even notice in time.
A warning hidden in the fine print
On Monday, Anthropic released an amended version of its filings for the U.S. Securities and Exchange Commission. While the main focus is on finances, as the company’s revenue is rising sharply along with its spending, a passage of far greater significance has been added to the risk factors. The first version of the filings already contained a warning that artificial intelligence could pose a catastrophic or even existential threat to humanity.
The company now writes that future models may be on the verge of achieving what its internal rules describe as dramatically accelerated progress driven by artificial intelligence. This is how Anthropic defines recursive self-improvement, known in English by the abbreviation RSI. Experts use the term to describe a state in which artificial intelligence is capable enough to train and improve itself. Proponents see it as a path to enormous productivity gains and scientific discoveries, while critics fear that such technology could escape human control.
Acceleration the company failed to notice
Anthropic openly admits that it has previously recognized significant acceleration in its own development only in hindsight, and the same mistake could happen again. In developing new models, it is increasingly relying on its own artificial intelligence. Model capabilities may therefore grow faster than the company can develop and verify appropriate safety measures.
According to the filing, the data on which its estimates are based reflects developments with a delay, meaning the company has only a limited ability to predict the pace of progress. It says that none of its current models has crossed the threshold yet, but it continues to evaluate the underlying data.
Humans will remain in control
The company is therefore already taking more cautious steps. It releases its most powerful models with a delay and in a limited form. One example is the Mythos model, which is exceptionally effective at detecting security vulnerabilities in software and is therefore accessible only to approved individuals and organizations.
Anthropic writes in the filing that it does not expect Claude to be able to safely and completely independently design subsequent generations of models without genuine human oversight, even if doing so were technically possible. It is therefore considering revising its rules to ensure that the development of the most advanced models remains under human control.
Skeptics point out that IPO filings serve both as disclosures and as advertising. They argue that talk of artificial intelligence building itself may therefore be aimed more at attracting investors’ attention.
But caution comes at a cost, and Anthropic is not hiding that from investors. As model capabilities grow, safety costs will rise as well, meaning the company could fall behind competitors that do not implement similar safeguards. Moreover, all of this is happening as it heads toward an IPO with a valuation of around one and a half trillion dollars.
Sources: independent.co.uk and techcrunch.com
