AI Safety Has a Business Model Problem
In late February, Anthropic refused Pentagon terms that would have allowed unrestricted use of its AI model Claude for any lawful military purpose. The company held two red lines: no fully autonomous weapons and no domestic mass surveillance. The Defense Department gave Anthropic until February 27 to drop those conditions. Anthropic declined. President Trump directed all federal agencies to cease using the company’s technology, and Defense Secretary Pete Hegseth designated Anthropic a supply chain risk. The move was extraordinary. Supply chain risk designations had previously been reserved for entities tied to foreign adversaries, not disputes with domestic companies over contractual limits.
OpenAI later announced it had secured a deal for the Pentagon to use its models in classified systems. According to Fortune’s reporting, OpenAI CEO Sam Altman said the agreement contained the same two limitations Anthropic had insisted on. But where Anthropic tried to spell them out explicitly in the contract, OpenAI agreed that the Pentagon could use its technology for “any lawful purpose” while claiming the restrictions were also embedded in the deal. How both of those things can be true at the same time remains unclear.
In the weeks since, the two companies have moved in opposite directions. A federal judge in California blocked the Pentagon’s supply chain designation, calling it “classic illegal First Amendment retaliation.” But last week, a federal appeals court in Washington denied Anthropic’s request to pause the designation in a separate case, meaning it remains in effect for military contracts while litigation continues. Anthropic is simultaneously excluded from Pentagon work and protected from broader enforcement, depending on which court you ask.
Then, Anthropic announced Claude Mythos Preview, a new frontier model the company described as its most powerful to date. According to Anthropic’s technical blog, the model’s previous generation had a near-zero success rate at autonomous exploit development. Mythos operates in a different category entirely.
In testing, the model found thousands of high-severity vulnerabilities, including some in every major operating system and web browser, with at least one flaw dating back 27 years. Rather than release the model publicly, Anthropic restricted access to a group of partner organizations through a program called Project Glasswing, committing up to $100 million in usage credits for defensive security work, a decision NBC News noted was the first of its kind by a leading AI company in nearly seven years. The company withheld a commercially valuable capability on safety grounds while its competitors prepared to release models with similar power to broader audiences.
This conflict is unfolding against a broader fragmentation of AI governance. As I covered in a recent post, California and the Pentagon are operating under increasingly incompatible safety standards, and Governor Newsom’s March 30 executive order now allows the state to independently review federal supply chain designations it considers improper.
The public responded to Anthropic’s stance by downloading Claude in record numbers. According to NPR, more than a million people signed up for Claude per day in the week following the designation, making it the top AI app in more than 20 countries. The public rewarded the company that said no. The government punished it. Investors, meanwhile, rewarded the company that said yes, with OpenAI’s valuation reportedly rising by $110 billion that same week.
The AI safety conversation has spent years focused on whether companies are sincere about their commitments. Anthropic’s decisions over the past six weeks suggest sincerity may not be what matters most. The company held its red lines with the Pentagon. It restricted access to one of the most powerful cybersecurity models ever built by a private company. Both decisions cost it something. The contracts, the procurement rules, and the investor incentives are not currently built to reward that. Until they are, the companies that hold their commitments may find themselves choosing between their principles and their ability to survive.