California’s attempt to regulate frontier artificial intelligence has become a proxy fight over how governments should treat the most powerful AI systems: as tools that need practical safeguards, or as future agents capable of catastrophic harm.
The “Safe and Secure Innovation for Frontier Artificial Intelligence Models Act,” also known as SB-1047, is aimed at companies training sufficiently large AI models. Its supporters see it as a needed safety framework. Its critics say the bill is shaped too heavily by speculative fears and could damage open source AI and ordinary research.
What SB-1047 Would Require
SB-1047 was introduced by State Senator Scott Wiener and passed the California Senate in May with a 32-1 vote. The bill was described as well positioned for a final vote in the State Assembly in August.
The measure applies to AI models that meet a high size threshold, currently set at $100 million in training costs and the rough computing power implied by those costs today. For those systems, developers would need to put testing procedures and response systems in place to prevent and handle “safety incidents.”
The bill’s central concern is not every possible misuse of AI. It focuses on “critical harms,” including harms that could cause “mass casualties or at least $500 million of damage.” The examples include “the creation or use of chemical, biological, radiological, or nuclear weapon” and “precise instructions for conducting a cyberattack… on critical infrastructure.”
SB-1047 also refers to “other grave harms to public safety and security that are of comparable severity.” That broad language is part of what has made the proposal controversial. Supporters see flexibility for serious risks. Critics see uncertainty that could make AI development harder to plan.
The Bill’s Focus On Future AI Behavior
One important limit in the bill is that model creators would not be liable simply because a system repeats “publicly accessible” information from outside the model. In other words, the target is not just an AI system summarizing existing harmful material.
The bill instead looks toward future systems that might generate “novel threats to public safety and security.” It also focuses on AI “autonomously engaging in behavior other than at the request of a user” while acting “with limited human oversight, intervention, or supervision.”
To address that possibility, the bill would require anyone training a sufficiently large model to “implement the capability to promptly enact a full shutdown.” Developers would also need policies covering when that shutdown would be used, along with other precautions and tests.
This is where the debate becomes especially sharp. The bill contains language about AI actions that would require “intent, recklessness, or gross negligence” if a human performed them. Critics argue that this points toward a type of agency that does not exist in today’s large language models.
Why Supporters Say The Risk Is Real
The source article identifies Center for AI Safety co-founder Dan Hendrycks as the bill’s original drafter. In a 2023 Time Magazine piece, Hendrycks argued that “evolutionary pressures will likely ingrain AIs with behaviors that promote self-preservation” and could create “a pathway toward being supplanted as the earth’s dominant species.'”
If that concern is correct, SB-1047 looks less like overreach and more like an early guardrail. Supporters including Geoffrey Hinton and Yoshua Bengio back the idea that advanced AI systems could create catastrophic risks and should be tested before they are deployed at scale.
Bengio framed the bill as a practical safety measure. He wrote: “AI systems beyond a certain level of capability can pose meaningful risks to democracies and public safety,” adding that they should be tested and placed under appropriate safety measures.
That argument treats frontier AI regulation as a preventive task. The point is to build procedures before a system is powerful enough to cause the kinds of harms described in the bill.
Why Critics Say It Could Backfire
Opponents argue that the bill starts from the wrong premise. Daniel Jeffries, described as a prominent voice in the AI community, told Ars that SB-1047 grew out of groups that believe in an end-of-the-world scenario involving sentient machines. He said: “You cannot start from this premise and create a sane, sound, ‘light touch’ safety bill.”
Tech policy expert Nirit Weiss-Blatt was even more direct. “If we see any power-seeking behavior here, it is not of AI systems, but of AI doomers,” she said. She argued that the result could be legislation driven by fictional fears and warned that it could harm California’s and the US’s technological advantage.
Another major concern is open weight AI. Ars contributor Timothy Lee argued in his Understanding AI newsletter that SB-1047 could make it harder to share so-called “open weight” AI models because creators could remain liable for “derivative” models built from their original training.
The concern is simple: a company may train a model to refuse harmful requests, but someone else may later download and fine-tune it to remove those restrictions. Lee used Meta’s Llama models as an example of systems trained to reject assistance with cyberattacks, scams, bomb-making, and other harms, while noting that Meta may not be able to stop downstream modification.
Meta’s Yann LeCun warned that “the ignominious California regulation SB1047… would essentially kill open source AI and significantly slow down or stop AI innovation.”
The Bigger Regulatory Choice
Several critics argue that AI policy should concentrate on harmful applications that already exist, rather than regulating the development of AI technology itself. Andrew Ng wrote that SB-1047 “makes a fundamental mistake of regulating AI technology instead of AI applications, and thus would fail to make AI meaningfully safer.”
Ng suggested a different set of priorities: “outlawing nonconsensual deepfake pornography, standardizing watermarking and fingerprinting to identify generated content, and investing more in red teaming and other safety research.” Jeffries made a similar point, calling for rules on AI use cases rather than research and development, including mass surveillance and voice cloning scams.
The underlying policy question is whether AI safety rules should be aimed at the largest models before harm occurs, or at specific uses after risks are clearer. SB-1047 chooses the first path. Its critics prefer the second.
With federal efforts to regulate AI largely stalled, California’s decision carries weight beyond one state. If SB-1047 advances, it could shape how future AI regulation balances catastrophic risk, innovation, open source development, and accountability for frontier AI models.