Open source AI image generators sit at the center of a difficult tradeoff. They make powerful creative tools available to researchers, artists, academics and developers, but that same openness can be turned toward harassment, nonconsensual images and deepfake porn.
The problem is not simply that image models can produce realistic visuals. It is that open source software can often be copied, modified, tuned and used outside the reach of the people or platforms that first released it.
Creative freedom and harmful use now share the same tools
Reuven Cohen, a Toronto-based consultant who helps companies develop AI tools, uses AI-generated images to draw attention and explore art, design and video. He also warns that the technology can be trained for harmful purposes, including explicit images used to harass women.
That dual use is the central issue. Open source image generation has helped fuel experimentation, but experts cited in the source also connect it to fake image abuse and nonconsensual pornography. Henry Ajder, who has spent years researching harmful use of generative AI, says open source image generation software has become a foundation for deepfake porn.
Some tools are built directly for abusive use, including apps that digitally remove women’s clothes in images. Others have legitimate creative or entertainment uses but can also be used by bad actors. A popular open source face-swapping program is one example: it is used in entertainment, while also being described by Ajder as a favored tool for nonconsensual deepfakes.
Why guardrails are easier to bypass in open systems
The Taylor Swift case showed how quickly nonconsensual images can spread on a major platform. After such images recently spread on X, Microsoft added new controls to its image generator. But open source tools pose a different challenge because they can be downloaded, customized and run with fewer limits.
Stable Diffusion, the high-resolution image generator developed by Stability AI, is claimed to have more than 10 million users. It has guardrails intended to prevent explicit image creation and policies barring malicious use. At the same time, the company open sourced a customizable version of the image generator in 2022, and online guides describe ways to bypass built-in restrictions.
Smaller AI models known as LoRAs add another layer. They can tune a Stable Diffusion model toward a style, concept or pose, including a celebrity’s likeness or certain sexual acts. These models are widely available on marketplaces such as Civitai, where users share and download AI models.
One creator of a Taylor Swift plug-in on Civitai urged others not to use it for NSFW images. But once a model has been downloaded, the creator no longer controls how it is used. That is the practical limit of many voluntary warnings in open source AI image generation.
Marketplaces and forums show the moderation challenge
The source describes 4chan pages devoted to nonconsensual deepfake porn, made with openly available programs and AI models dedicated to sexual images. WIRED also observed 4chan users sharing workarounds for NSFW images using OpenAI’s Dall-E 3.
Adult image message boards include AI-generated nonconsensual nudes of real women, including porn performers and actresses like Cate Blanchett. That activity has led some users in AI image-making communities on Reddit and Discord to push back against malicious and pornographic images.
Reddit's policies prohibit all AI-generated “nonconsensual intimate media.” Some creators also encourage users to report images involving minors on Reddit and model-hosting sites. These efforts show that parts of the community are trying to set boundaries, even while the broader tool ecosystem remains difficult to police.
Civitai says it prohibits depictions of real people and minors in a mature context. In December, it encouraged reporting of violations. A spokesperson for Civitai said images flagged by its moderation system as potentially meeting the site’s definition of mature content are sent for human review. Policy violations can lead to image removal, suspension of access to Civitai’s onsite image generator or a platform ban. Real people featured in models can also send Civitai a removal request.
Still, the source notes that 404 Media reported in November that users openly asked others on Civitai to create nonconsensual images, mostly of women. That gap between stated rules and user behavior is one of the clearest signs of the problem.
New face tools lower the barrier further
Creators including Cohen have also raised concerns about InstantID, a method published in January by researchers at Peking University and Chinese social media company Xiaohongshu. The method can swap faces in images using just a single example, which means it requires less preparation and processing.
The team behind InstantID acknowledged in the paper introducing the model that it could be used for “offensive or culturally inappropriate imagery” involving human faces. But a YouTube channel with more than 143,000 subscribers promoted the technique as enabling “Uncensored Open Source Face Cloning.”
Cohen’s concern is straightforward: easier face cloning makes it simpler to create a fake image of someone in a compromising position. If such tools become easy to plug into a browser, the barrier to misuse could fall even further.
What can still make a difference
Some open source AI creators are trying to discourage malicious use. David Widder, an AI ethics researcher and postdoctoral fellow at Cornell Tech, interviewed the people behind an open source deepfake tool. They said they did not want the software used for porn of any kind, consensual or nonconsensual, but felt unable to stop misuse.
Other groups are building friction into the ecosystem. Researchers at Hugging Face have promoted ethical AI tools, including image guarding, which they say protects images from generative AI editing. Hugging Face also lets developers control access to models uploaded to the platform.
Widder says release-gating, open source licensing and contractual obligations for commercial platforms may not stop all abuse, but could prevent some misuse. He also argues that community norms matter because they shape what users consider acceptable.
Law and platform policy are also moving. Sharing AI-generated intimate images without consent was made illegal in the UK through online safety legislation enacted in January. The Taylor Swift scandal has added fuel to calls for similar federal laws in the US, and at least 10 states have deepfake-related laws.
Tech companies and social platforms are exploring AI watermarking from prominent tools, and tech companies signed an accord to crack down on AI during elections. But the source notes an unresolved question: it is unclear how much those measures would affect images made with niche models.
The result is a technology landscape where useful openness and harmful flexibility are tightly connected. Open source AI image generators can expand access to creative tools, but without stronger norms, better platform controls and practical barriers to abuse, the same systems can keep enabling damage after they leave their creators’ hands.