Why California's AI safety bill split OpenAI and Anthropic

California's SB 1047 would set new rules for high-risk AI models, including safety protocols, audits, incident reports, and whistleblower protections. OpenAI says frontier model regulation belongs at the federal level, while Anthropic says the bill is broadly useful despite flaws.

WTF Index TERMINATOR
◄ Terminator 2 Idiocracy 0 ►

The story centers on frontier AI systems posing catastrophic safety and cybersecurity risks, though mainly in the context of regulation.

Why California's AI safety bill split OpenAI and Anthropic

California's proposed SB 1047 has become a clear test of how leading AI companies want frontier models to be governed. The bill targets AI systems that could create catastrophic risks for people or cybersecurity, and it has drawn sharply different responses from OpenAI and Anthropic.

OpenAI opposes the measure and argues that national safety questions should be handled by federal policy. Anthropic, while raising concerns, says the bill is likely to help reduce serious AI risks.

What SB 1047 Would Require

The proposed law, called the "Safe and Secure Innovation for Frontier Artificial Intelligence Models Act" (SB 1047), would apply to certain high-risk AI models. Its central idea is that developers of the most capable systems should document, publish, and test how they plan to prevent severe harm.

Under the bill, companies developing covered models would need to create safety and security protocols (SSPs). These protocols would assess possible risks and explain mitigation steps. Companies would also need to publish their SSPs and undergo annual third-party audits starting in 2026.

The bill also requires developers to submit compliance statements and report AI safety incidents. It links company liability for AI-caused harm to the quality of safety protocols, creating an incentive to make those plans meaningful rather than symbolic.

SB 1047 also includes whistleblower protections for people who report risks to authorities. In addition, it promotes research on AI risk mitigation and includes plans for a public cloud computing cluster called "CalCompute" to support safe and ethical AI development.

Which AI Models Are Covered

The proposal does not apply to every AI system. It defines a category of frontier models using compute and cost thresholds.

The bill covers models trained with more than 10^26 integer or floating-point operations and costing more than $100 million. It also covers models created by fine-tuning those systems with at least 3x10^25 operations and costing more than $10 million.

A new "Board of Frontier Models" would update these definitions over time. The source article also notes that the law includes detailed rules on scope, obligations, oversight, and penalties.

In plain terms, SB 1047 focuses on the largest and most expensive AI systems rather than ordinary software products. Its supporters frame that focus as a way to address the most serious risks without regulating every AI tool in the same way.

OpenAI Wants Federal Rules Instead

OpenAI, the company behind ChatGPT, has argued against SB 1047 in a letter to California Governor Gavin Newsom. Its position is that frontier AI model regulation should be shaped at the federal level because the issues involve national safety, U.S. competitiveness, and national security.

Jason Kwon, OpenAI's chief strategy officer, wrote: "The broad and significant implications of AI for U.S. competitiveness and national security require that regulation of frontier models be shaped and implemented at the federal level. A federally-driven set of AI policies, rather than a patchwork of state laws, will foster innovation and position the U.S. to lead the development of global standards,"

OpenAI also warns that the law could weaken California's role in AI development and push companies out of the state. The company's objection is not presented as opposition to AI safety rules in general, but as opposition to this state-level approach.

Senator Scott Wiener, the bill's author, rejects that argument. He says OpenAI does not address specific provisions of the bill and instead asks for regulation to be left to Congress, which has yet to act.

Wiener also rejects the claim that companies would leave California, calling it a "tired argument." He notes that SB 1047 would apply to all companies doing business in the state, regardless of where they are headquartered.

Anthropic Supports the Bill With Reservations

Anthropic, creator of Claude, takes a different position. In its own letter to Governor Newsom, the company broadly supports SB 1047 while also identifying problems it believes should be addressed.

Anthropic says the bill responds to real concerns about fast-moving AI capabilities. It also says AI progress can bring major economic potential for California while creating significant dangers. The company warns that serious AI misuse could occur within one to three years.

Anthropic highlights several parts of the bill it views as useful:

  • Requirements for safety and security protocols (SSPs)
  • Public transparency around those protocols
  • Liability incentives tied to protocol quality
  • Support for research on AI risk mitigation

At the same time, Anthropic says the bill has issues, including overly prescriptive oversight and broad authority for regulators. Even with those concerns, the company says the benefits outweigh the drawbacks.

Anthropic's stated priorities are transparency in safety practices, incentives for effective safety plans, and avoiding unintended consequences. That puts the company in a more nuanced position than simple support or rejection.

Former OpenAI Staff Enter the Debate

Former OpenAI employees William Saunders and Daniel Kokotajlo have also criticized OpenAI's stance. In an open letter, they say they left OpenAI after losing confidence in its commitment to safe, honest, and responsible AI development.

They allege that OpenAI ignored safety protocols and silenced internal critics while publicly claiming otherwise. They also criticize OpenAI's opposition to regulation and point out that CEO Sam Altman had previously publicly supported it.

Saunders and Kokotajlo contrast OpenAI's approach with Anthropic's response. They say Anthropic shared specific concerns, suggested changes, and concluded that the bill was likely to be positive overall and not too difficult to comply with.

The source article also notes that several former OpenAI security staffers have recently joined Anthropic, including Jan Leike, OpenAI's former head of Super AI alignment. OpenAI co-founder Jan Schulman also joined Anthropic, saying he wanted to focus on safety research.

SB 1047 now moves to the State Assembly for deliberation and voting. If it passes, Governor Newsom will decide whether to sign it into law or veto it.