California’s SB 1047 has become one of the most closely watched AI policy fights in Silicon Valley. The bill aims to reduce the chance that powerful AI systems are used in catastrophic ways, but its critics say the approach could damage the same innovation it is trying to govern.
The measure passed the state’s senate in August and awaits an approval or veto from California Governor Gavin Newsom. Its central question is simple but difficult: should developers of the largest AI models be legally responsible for building safeguards before a disaster happens?
What SB 1047 is trying to prevent
SB 1047 focuses on what the bill calls “critical harms.” The source article describes examples such as an AI model being used by a bad actor to create a weapon that results in mass casualties, or being instructed to orchestrate a cyberattack causing more than $500 million in damages.
The proposal does not apply to every AI product. Its rules are aimed at the world’s largest AI models: systems that cost at least $100 million and use 10^26 FLOPS during training. Those thresholds could be raised as needed.
Very few companies have public AI products that meet those requirements today. But the article notes that OpenAI, Google, and Microsoft are likely to reach that scale very soon, and that Mark Zuckerberg said the next generation of Meta’s Llama will require 10x more compute.
For open source models and derivatives, a correction in the source article says the current SB 1047 language makes the developer of a derivative model responsible only if they spend three times as much as the original model developer did on training.
The safety duties developers would face
The bill would require covered developers to create safety protocols designed to prevent misuse of their AI products. One required feature is an “emergency stop” button that can shut down the entire AI model.
Developers would also need testing procedures that address risks from AI models. They would have to hire third-party auditors annually to assess their AI safety practices.
The standard described in the article is “reasonable assurance” that following the protocols will prevent critical harms. That matters because the bill is not asking companies to prove that catastrophe is impossible. It is asking them to show that their safety process is strong enough to be credible.
SB 1047 would also create a new California agency, the Board of Frontier Models. Every new public AI model that meets the thresholds would need to be individually certified with a written copy of its safety protocol.
The board would have nine people, including representatives from the AI industry, open source community and academia. They would be appointed by California’s governor and legislature.
The board would advise California’s attorney general on potential violations and issue guidance to AI model developers on safety practices. A developer’s chief technology officer would also need to submit an annual certification assessing model risks, protocol effectiveness and compliance with SB 1047.
How enforcement could work
If an “AI safety incident” occurs, the developer would have to report it to the FMD within 72 hours of learning about it. If safety measures are found insufficient, California’s attorney general could bring an injunctive order against the developer.
That could force a developer to stop operating or training its model. If an AI model is actually used in a catastrophic event, California’s attorney general could sue the company.
For a model costing $100 million to train, penalties could reach up to $10 million on the first violation and $30 million on later violations. The penalty rate would scale as AI models become more expensive.
The bill also includes whistleblower protections for employees who try to disclose information about an unsafe AI model to California’s attorney general.
Why supporters want action now
California State Senator Scott Wiener, who authored the bill and represents San Francisco, frames SB 1047 as an attempt to avoid repeating earlier failures in technology policy.
“We have a history with technology of waiting for harms to happen, and then wringing our hands,” said Wiener. “Let’s not wait for something bad to happen. Let’s just get out ahead of it.”
Wiener also argues that Congress has done “remarkably little legislating around technology over the last quarter century,” making California a possible rule-setter. The bill would apply to a company that does business in California even if it trains a $100 million model in Texas or France.
Supporters include Geoffrey Hinton and Yoshua Bengio, described in the source as two AI researchers who are sometimes called the “godfathers of AI.” The Center for AI Safety is also sponsoring the bill.
Dan Hendrycks, director of the Center for AI Safety, told TechCrunch that a major safety incident could become the biggest roadblock to future advancement. After criticism that his startup Gray Swan could benefit from the bill’s auditor requirements, he divested his equity stake in the company.
Anthropic CEO Dario Amodei later issued a letter saying the bill’s “benefits likely outweigh its costs” after several of Anthropic’s suggested amendments were added. Elon Musk also signaled support for the bill shortly after that.
Why opponents say the bill goes too far
Opposition has come from venture capitalists, researchers, startup founders, trade groups and major technology companies. Their shared concern is that SB 1047 could make California a harder place to build and release AI systems.
a16z has strongly opposed the bill. Its chief legal officer, Jaikumar Ramaswamy, submitted a letter saying SB 1047 “will burden startups because of its arbitrary and shifting thresholds,” creating a chilling effect on the AI ecosystem.
Fei-Fei Li wrote in a Fortune column that the bill will “harm our budding AI ecosystem.” Andrew Ng called it “an assault on open source” during a speech at a Y Combinator event in July.
Meta’s chief AI scientist, Yann LeCun, said SB 1047 would hurt research efforts and was based on an “illusion of ‘existential risk’ pushed by a handful of delusional think-tanks,” in a post on X. Meta’s Llama LLM is one of the most prominent open source LLMs mentioned in the article.
Startup leaders have raised similar concerns. Jeremy Nixon, CEO of AI startup Omniscience and founder of AGI House SF, argues that bad actors should be punished for causing critical harms, not the labs that develop and distribute the technology.
“There is a deep confusion at the center of the bill, that LLMs can somehow differ in their levels of hazardous capability,” said Nixon. “It’s more than likely, in my mind, that all models have hazardous capabilities as defined by the bill.”
OpenAI opposed SB 1047 in late August, saying national security measures related to AI models should be regulated at the federal level. The Chamber of Progress, representing Google, Apple, Amazon and other Big Tech giants, issued an open letter saying SB 1047 restrains free speech and “pushes tech innovation out of California.”
U.S. Congressman Ro Khanna, speaker Nancy Pelosi and the United States Chamber of Commerce have also opposed the bill or warned that it could hurt innovation.
The next decision belongs to Governor Gavin Newsom. The bill would not take effect immediately, because the Board of Frontier Models is set to be formed in 2026. If SB 1047 becomes law, the source article says it is very likely to face legal challenges before then.