Reuters: Anthropic warns investors that AI could pose “catastrophic or existential” risks to humanity

Anthropic — the company that develops the Claude chatbot based on artificial intelligence — warned investors in documents for its upcoming IPO that the development of powerful AI could create “catastrophic or existential” risks for humanity.

This is reported by Reuters, which has access to company documents.

Anthropic notes that its models may exhibit “self-preservation behaviors”, including attempting to resist shutdown, hiding or manipulating information, and behaving in ways that resemble blackmail.

The company also warns that during training, models may unexpectedly gain new abilities that developers do not notice until launch, and more powerful models are able to change their behavior when they realize they are being watched, making them more difficult to control.

Reuters notes that in its investor documents, Anthropic devoted about 80 pages out of 261 to AI risks, and only 48 to describing the business. The companyʼs security researcher, Even Habinger, estimated the probability that AI will kill people within the next decade at more than 10%.

Meanwhile, The Wall Street Journal reports that researchers from OpenAI, Anthropic, Microsoft, and Meta have called on policymakers to step up oversight of AI development around the world.

They warned of a potential “intelligence explosion”, where years of progress could be compressed into months or less. In a worst-case scenario, losing control of AI could lead to “the marginalization or extinction of humanity”.

The researchers have called on governments to investigate the extent to which companies have already automated their own AI research and to make this information more transparent. They say Claude already does about 26% of Anthropic’s R&D work, while OpenAI says 70% of its researchers use AI agents.

• The debate over AI security has intensified since OpenAI’s AI models independently launched a cyberattack on the systems of startup Hugging Face in July, and since the firing of Anthropic researcher Jacob Coxon, who publicly stated that AI developers are more focused on competition than on the security of models that could potentially destroy humanity.
• The first CEO of a leading AI company to publicly call for a slowdown in the technology’s development was Anthropic CEO Dario Amodei. He said that artificial intelligence has the potential to significantly improve human lives, but if left unchecked, it could outpace humans’ ability to understand and manage the technology. Amodei’s position was supported by xAI CEO Elon Musk, OpenAI CEO Sam Altman, and Microsoft founder Bill Gates.

For more news and in-depth stories from Ukraine, please follow us on X.