New York moves to set guardrails for frontier AI models

New York state lawmakers passed the RAISE Act, a bill aimed at preventing frontier AI models from contributing to severe disaster scenarios. If Governor Kathy Hochul signs it, the measure would create legally mandated transparency standards for major AI labs and allow civil penalties of up to $30 million.

WTF Index TERMINATOR
◄ Terminator 3 Idiocracy 0 ►

The story centers on frontier AI safety rules aimed at preventing catastrophic misuse, loss of control, or mass-harm scenarios.

New York moves to set guardrails for frontier AI models

New York is closer to creating a new oversight framework for frontier AI models after state lawmakers passed the RAISE Act on Thursday. The bill targets the largest AI labs and is focused on preventing extreme harms, including scenarios involving the death or injury of more than 100 people, or more than $1 billion in damages.

The measure now goes to New York Governor Kathy Hochul. She can sign it into law, send it back for amendments, or veto it altogether.

What the RAISE Act would require

If signed into law, the RAISE Act would require the world’s largest AI labs to publish detailed safety and security reports for frontier AI models. It would also require companies to report safety incidents if they occur.

Those incidents could include concerning AI model behavior or a bad actor stealing an AI model. The bill gives New York’s attorney general the power to bring civil penalties of up to $30 million if companies do not meet the required standards.

The bill is written to apply to companies whose AI models were trained using more than $100 million in computing resources and are being made available to New York residents. The source article notes that this threshold is seemingly higher than any AI model available today.

That means the bill is aimed at the largest companies rather than ordinary startups, academic researchers, or smaller developers. The article names OpenAI, Google, Anthropic, DeepSeek, and Alibaba as examples of companies whose frontier models are central to the debate.

Why supporters say timing matters

The RAISE Act is being treated as a major moment for the AI safety movement. The source article describes it as a win for advocates who have lost ground in recent years as Silicon Valley and the Trump administration have prioritized speed and innovation.

Supporters include Nobel laureate Geoffrey Hinton and AI research pioneer Yoshua Bengio, both of whom have championed the bill. If it becomes law, the RAISE Act would establish America’s first set of legally mandated transparency standards for frontier AI labs.

“The window to put in place guardrails is rapidly shrinking given how fast this technology is evolving,” said Senator Gounardes. “The people that know [AI] the best say that these risks are incredibly likely […] That’s alarming.”

The policy argument is straightforward: the most capable models are advancing quickly, and lawmakers want transparency requirements in place before a severe incident occurs. The bill does not claim to stop every possible AI risk. Instead, it focuses on reporting, safety documentation, and accountability for companies developing the most expensive models.

How the bill differs from California’s SB 1047

The RAISE Act shares some goals and provisions with California’s controversial AI safety bill, SB 1047, which was ultimately vetoed. But New York State Senator Andrew Gounardes, a co-sponsor of the RAISE Act, told TechCrunch that he deliberately designed the New York bill to avoid chilling innovation among startups or academic researchers.

That criticism was often directed at SB 1047. According to Nathan Calvin, the vice president of State Affairs and general counsel at Encode, the RAISE Act was written to address criticism of earlier AI safety bills. Calvin worked on this bill and SB 1047.

Several differences stand out in the source article:

  • The RAISE Act does not require AI model developers to include a “kill switch” on their models.
  • It does not hold companies that post-train frontier AI models accountable for critical harms.
  • Its transparency requirements are tied to models trained with more than $100 million in computing resources and made available to New York residents.

Those choices are central to the bill’s political strategy. Supporters are trying to frame the RAISE Act as a narrower transparency bill, not a broad attempt to control the entire AI ecosystem.

Industry pushback and open questions

Silicon Valley has pushed back significantly on the bill, according to New York state Assemblymember and co-sponsor Alex Bores. Bores called that resistance unsurprising, while arguing that the RAISE Act would not limit innovation of tech companies in any way.

“The NY RAISE Act is yet another stupid, stupid state level AI bill that will only hurt the US at a time when our adversaries are racing ahead,” said Andreessen Horowitz general partner Anjney Midha in a Friday post on X.

Andreessen Horowitz and startup incubator Y Combinator were among the fiercest opponents to SB 1047. The same general concern is now being raised in New York: that state-level AI regulation could slow companies down or make them less willing to release advanced models in the state.

Anthropic has not taken an official stance on the RAISE Act, according to co-founder Jack Clark in a Friday post on X. Clark said it could help to give more context, but he also expressed concerns about how broad the bill is, noting that it could present a risk to “smaller companies.”

When asked about Anthropic’s criticism, Senator Gounardes told TechCrunch he thought it “misses the mark,” saying the bill was designed not to apply to small companies.

OpenAI, Google, and Meta did not respond to TechCrunch’s request for comment.

Could AI companies leave New York?

One criticism of the RAISE Act is that AI model developers might simply choose not to offer their most advanced AI models in New York. The same argument was made against SB 1047, and the source article says that a similar pattern has largely played out in Europe because of the continent’s tough technology regulations.

Assemblymember Bores argued that the bill’s regulatory burden is relatively light and should not require companies to stop operating their products in New York. He also pointed to New York’s third largest GDP in the U.S. as a reason companies would not take withdrawal lightly.

“I don’t want to underestimate the political pettiness that might happen, but I am very confident that there is no economic reason for [AI companies] to not make their models available in New York,” said Assemblymember Bores.

For now, the RAISE Act remains a bill awaiting action from Governor Kathy Hochul. Its next step will determine whether New York becomes the first state to impose this kind of legally mandated transparency standard on frontier AI labs, or whether the proposal joins SB 1047 as another high-profile AI safety bill that does not become law.