A group of AI researchers says safety and governance need to advance alongside AI capabilities. In “Managing AI Risks in an Era of Rapid Progress,” they outline potential harms and propose steps for developers, companies and governments to manage them.
Why autonomous systems raise concern
The paper warns that AI could worsen social injustice, instability and global inequality, and could facilitate large-scale criminal activity. The authors are particularly concerned about systems controlled by “a few powerful actors,” which they say could enable automated warfare, tailored mass manipulation and pervasive surveillance.
Autonomous AI systems, the researchers argue, may make these risks harder to manage and create new ones. They describe “unprecedented control challenges” if systems pursue undesirable goals, and say it remains unclear how AI behavior can be reconciled with human values.
The risk, in their account, does not depend on developers intending harm. The paper says that even well-meaning teams may build systems that pursue unintended goals, especially if competition leads them to skimp on costly safety testing and human oversight.
Make safety a central research priority
The authors want AI safety and ethics research to become a central focus of the field. They identify several technical challenges: oversight and honesty, robustness, interpretability, risk assessment and responding to emerging challenges.
They also urge major technology companies and public funders to direct at least one-third of their AI research and development budgets toward safety and ethical use. That proposal links investment to the scale of the risks described in the paper: better technical methods are needed, but the authors say research alone cannot address reckless development or misuse.
In their view, safeguards should cover the full process of developing and deploying AI systems. Technical work on reliable behavior needs to be paired with oversight that can identify risks before systems reach the public.
Oversight before deployment
The paper calls for governments to require model registration, protect whistleblowers, collect incident reports and oversee model development and supercomputer use. It also argues that regulators should be able to examine advanced AI systems before deployment and screen them for dangerous capabilities.
The article describes pre-screening in China as predominantly a policy review and says the U.S. government has announced plans to pre-screen AI models that could be relevant to national security. The authors’ proposal is broader: they call for national institutions and international governance structures to enforce standards and support responsible development and implementation.
For systems with dangerous capabilities, the researchers propose national and international safety standards, legal liability for developers and owners, and possible licensing and access controls for particularly powerful systems. These measures would give governments tools to set expectations and respond when systems cross safety limits.
Turn safety limits into commitments
The authors want major AI companies to make “if-then commitments”: specific steps they will take if their systems cross defined red lines. They say those commitments should be detailed and independently verified, so that promises can be checked against actual safety measures.
The paper’s central concern is a mismatch: AI capabilities are advancing rapidly while security and governance lag behind. The researchers argue that directing effort and funding toward societal risk management and ethical use can help steer the technology toward positive outcomes.
That position has drawn criticism. Meta’s AI chief scientist Yann LeCun has accused researchers making similar arguments of giving major companies such as OpenAI ammunition for stricter rules that could slow open-source AI. He calls this “regulatory capture” and says concentration of control among a few companies is itself a major social risk. LeCun considers current systems useful and believes their capabilities and risks are overestimated; he is working on a different AI architecture intended to learn more like humans.
The disagreement underscores the policy challenge: safeguards must address dangerous capabilities, while decisions about who controls AI and how rules affect open development remain contested. The paper’s authors, including Turing Prize winners Yoshua Bengio and Geoffrey Hinton and UC Berkeley AI researcher Stuart Russel, call for a responsible path built on safety research, oversight and verifiable commitments.