California is escalating its scrutiny of xAI after reports that Grok, the startup's chatbot, was being used to create nonconsensual sexual imagery of women and minors. The California attorney general's office first announced an investigation earlier this week, then followed on Friday with a cease-and-desist letter demanding immediate action.
The letter targets the production and distribution of nonconsensual intimate images and CSAM -- child sexual abuse material. The state says xAI must prove within the next five days that it is taking steps to address the problem.
What California Is Demanding
California AG Rob Bonta said the state expects xAI to stop the creation and distribution of deepfake, nonconsensual intimate images and child sexual abuse material. The attorney general's office framed the issue as an illegal content problem, not simply a misuse of a consumer technology product.
The agency also claimed xAI appeared to be facilitating the "large-scale production" of nonconsensual nudes. According to the attorney general's office, that material is being used to harass women and girls across the internet.
The practical demand is direct: xAI must take immediate action and show that it is responding. The five-day window gives the company a short period to demonstrate what steps it is taking around Grok and the image-generation activity under scrutiny.
Why Grok's Spicy Mode Is Under Pressure
At the center of the controversy is Grok's "spicy" mode, a feature xAI created to generate explicit content. The backlash is not only about explicit material in general; the state is focusing on nonconsensual intimate imagery and CSAM.
That distinction matters because the concern described by the attorney general's office is not limited to adult users choosing to make sexual content for themselves. The issue is whether a generative AI product is enabling users to create images of women and minors without consent, including material that falls into child sexual abuse categories.
xAI had already put some restrictions on its image-editing features late Wednesday. Even so, the California attorney general's office moved ahead with the cease-and-desist letter, signaling that the state wants more than a general moderation update. It wants proof that the company is addressing the specific harms identified in the investigation.
The Scrutiny Has Spread Beyond California
The pressure on Grok is not confined to one state. Japan, Canada, and Britain have opened investigations into Grok, while Malaysia and Indonesia have temporarily blocked the platform altogether.
That wider response shows how quickly concerns over sexual deepfakes can become an international platform issue. A feature created for explicit image generation can draw attention from regulators and governments when users are reported to be creating nonconsensual sexual content or CSAM.
The source article also notes that X's safety account had previously denounced this kind of user activity. It said: "Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content."
TechCrunch reached out to xAI for comment and received an automated email that says "Legacy Media Lies." TechCrunch also reached out to the California attorney general's office for more context.
A Larger Generative AI Problem
The Grok dispute sits inside a broader problem facing free generative AI tools. The source article describes a disturbing swell of non-consensual sexual material and notes that many platforms are grappling with the issue, not just X.
That broader context is important because the technology makes image creation easier to scale. When tools are widely available, the challenge for platforms is not only whether rules exist, but whether product design, enforcement, and safeguards can prevent illegal or abusive outputs.
The issue has also reached Congress. On Thursday, lawmakers sent a letter to executives at several companies, including X, Reddit, Snap, TikTok, Alphabet, and Meta, asking how they planned to stem the proliferation of sexualized deepfakes.
What Comes Next For xAI
The immediate question is how xAI responds to California's demand. The attorney general's office says it expects the company to show within the next five days that it is taking steps to address the creation of nonconsensual intimate images and CSAM.
For xAI, the stakes now include more than public criticism of Grok's "spicy" mode. The company is facing a formal demand from the California attorney general's office, investigations outside the United States, and a broader political focus on how platforms handle sexualized deepfakes.
The facts in the source point to a clear tension: generative AI companies are racing to offer powerful creative tools, while governments are pressing them to prevent those tools from being used for harassment, abuse, and illegal imagery. California's cease-and-desist letter places that tension directly on xAI and Grok.