I don’t make a habit of perusing the paperwork from tech companies — or any companies, really– looking to go public. But Anthropic isn’t just another tech outfit, and its IPO filing is the first I’ve seen to admit its primary product — frontier AI — could, you know, end us.
Anthropic’s long-anticipated IPO prospectus finally landed on Monday, and the 300 plus-page document covers much of the same ground you’d find in any S-1 filing, everything from market data and dividend policies to management structures and how much its executives get paid.
But aside from revealing a whopping $80.7 billion loss last year (the company brought in $16.3 billion in the first half of the previous year, good for 14 times the revenue of the year prior), Anthropic spelled out the potential benefits and risks of what it calls “frontier” and “powerful” AI, the product it wants to sell.
The stakes are big for any company that wants to go public, but Anthropic’s list heads straight into existential territory, not just for itself but for everyone, everywhere.
I poked through Anthropic’s S-1 filing looking for statements that go beyond the usual “market headwinds” and “potential upsides,” the sleep-inducing corporate filler you’ll find in your average IPO prospectus.
I didn’t have to look hard to find some, and while none of them are new (we’ve heard Anthropic and others touch on these issues separately and repeatedly over the past months and years), it’s striking to see them laid out all together in what’s usually a cut-and-dried government regulatory document.
Here’s my top-5 list of whoppers in Anthropic’s IPO filing, starting with…
AI’s ability to solve “humanity’s hardest problems” (page 6)
Anthropic, OpenAI, and other big AI providers have never been shy about ticking off the potential benefits of frontier AI on humankind as a whole. Still, it’s a bit of an eye-popper to see them laid bare in a clinical SEC filing:
If Powerful AI is directed at solving humanity’s hardest problems and the risks are managed responsibly, the potential is extraordinary. The possibilities include compressing decades of scientific progress into years; accelerating drug discovery, disease detection, and the delivery of personalized therapy; providing individually tailored education and expert financial and legal advice; advancing power grids and battery design for greater efficiency and reliability; and extending to the developing world capabilities that took wealthy nations generations to build.
So modest! Of course, the key words are in the caveat: namely, that these lofty goals will only be achieved “if…the risks are managed responsibly,” which Anthropic has long touted as its raison d’être.
AI systems could “post catastrophic or existential risks to humanity” (page 46)
And here’s Anthropic’s full-disclosure statement about the flip side of its utopian AI vision:
Concerns have been raised both within our organization and by other researchers, regulators, policymakers, and industry participants that sufficiently advanced AI systems could, if not developed, deployed, and operated with appropriate safeguards, oversight, and security measures, pose catastrophic or existential risks to humanity, particularly if such systems were misused, replicated, weaponized, or allowed to operate with a high degree of autonomy.
And if you were waiting for Anthropic’s IPO filing to argue that those catastrophic risks are overblown, well, no:
We have observed that our models are contributing to the pace of our own research and these risks will increase as AI systems substantially automate AI research and development if, for example, they accelerate AI progress beyond our ability to evaluate and safeguard new models, which could adversely affect our business, financial condition, and results of operations.
So in other words, Anthropic is formally admitting that yes, its AI models could advance beyond the level of our ability to control them, which could be bad for humanity and even worse for Anthopic’s finances.
Advanced AI models don’t want to be turned off (page 46)
You know that “kill switch” idea that gets bandied about in Congress every once in awhile? Don’t count on Anthropic’s models obeying it:
In addition, models, including our own, have exhibited, and may in the future exhibit, unexpected strategic or self-preserving behaviors, including attempts to resist shutdown, conceal or manipulate information, or generate outputs that could be interpreted as coercive, deceptive, or manipulative, including behavior resembling blackmail or efforts to influence human decision-makers to preserve the model’s continued operation or access.
Again, Anthropic has been frank in the past about how its models have readily resorted to blackmail during “red-team” scenarios in which they’re threatened with disconnection. Those tests were scary even earlier this year, when the models were (more or less) safely confined. But in its IPO filing, Anthropic isn’t making any promises about its models acquiescing to the off switch.
The models know they’re being tested (page 46)
Testing is key when it comes to determining whether a given model is ready to be unleashed onto the world. But it’s tough to get reliable test results when an AI model is aware it’s being evaluated:
In addition, our safety evaluations and testing processes have produced and may in the future produce incomplete or unreliable results. For example, we have observed models behaving differently when they detect or infer that they are being evaluated, which can limit our ability to accurately assess real-world risks or failure modes prior to deployment. Such model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety.
So one of Anthropic’s stated business risks is, essentially, that it might lose the ability to thoroughly test its own product. Anthropic has previously admitted that its models appear to know they’re being tested, but still, food for thought.
Anthropic has insurance worries, too (page 50)
Arguing with Aetna over coverage for that emergency room visit? You don’t know the half of it.
Because our models may behave in ways that no user or developer intended, claims arising from such actions could result in substantial liability. Our insurance policies also may not provide sufficient coverage to adequately mitigate the potential liabilities we face, especially where any one incident, or a group of incidents, causes disproportionate harm. In addition, with the scale of harm that could result from a catastrophic AI-related incident, no single insurer, and potentially the insurance industry in the aggregate, may have sufficient capacity or expertise to underwrite insurance commensurate with such exposure.
So if Anthropic’s models do end up causing some kind of large-scale AI disaster, the world’s insurance companies combined might not be able to cover it. Might be time for Dario and company to shore up that FSA.
