Lensa’s Magic Avatars feature turns selfies into digital portraits, but the results can reflect more than a person’s chosen pose or style. In one test, a journalist received a large share of nude and sexualized images, while male colleagues got portraits as astronauts, explorers, and inventors. The contrast shows how an AI image app can reproduce assumptions embedded in its training data and product decisions.
One set of selfies, sharply different results
Lensa launched in 2018 as a digital retouching app. Its Magic Avatars feature later brought it widespread attention by generating portraits from uploaded selfies. The journalist tried it expecting results similar to those seen by colleagues at MIT Technology Review.
Instead, among 100 avatars, 16 were topless and another 14 showed extremely skimpy clothing and overtly sexualized poses. The images included generic Asian women modeled on anime or video-game characters. Some appeared to be crying, and the app produced sexualized female images even when asked to depict the journalist as a man.
A white female colleague received significantly fewer sexualized images, while another colleague with Chinese heritage got results more like the journalist’s. The comparison suggests that the app’s outputs were not simply a uniform response to everyone’s selfies.
How training data can shape an image
Lensa uses Stable Diffusion, an open-source model that generates images from text prompts. Stable Diffusion was built using LAION-5B, a large open-source data set compiled from images scraped from the internet.
The source article describes the internet as full of images of naked or barely dressed women, alongside depictions reflecting sexist and racist stereotypes. When those images are included in training material, a model can learn associations that make sexualized depictions more likely. Aylin Caliskan, an assistant professor at the University of Washington who studies bias and representation in AI systems, says such models can sexualize women regardless of whether they want to be depicted that way, particularly women from historically disadvantaged identities.
Research by Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe found racist stereotypes, pornography, and explicit images of rape in a data set similar to the one used to build Stable Diffusion. The openness of LAION made that kind of examination possible. The article notes that other popular image-making systems, including Google’s Imagen and OpenAI’s DALL-E, are not open but are built in similar ways with similar kinds of training data. That points to a broader issue for image-generating AI.
Design choices also matter
Training data alone does not determine what an app produces. Ryan Steed, a PhD student at Carnegie Mellon University who has studied bias in image-generation algorithms, says developers choose the data, decide to build a model, and decide whether and how to reduce bias.
That leaves room for product design to shape the experience. In the journalist’s test, men were cast in space suits while women appeared in cosmic G-strings and fairy wings. The app also produced more realistic, clothed images when the journalist’s pictures went through male content filters. Several showed a white coat that looked like it belonged to a chef or doctor.
Prisma Labs, Lensa’s developer, said that “sporadic sexualization” affects people of all genders, though in different ways. The company also said that because Stable Diffusion was trained on unfiltered internet data, neither it nor Stability.AI “could consciously apply any representation biases or intentionally integrate conventional beauty elements.” The spokesperson attributed the biases to unfiltered online data, while the company said it was working to address the issue.
Why the outputs carry consequences
Prisma Labs said in a blog post that it had changed the relationship between certain words and images to reduce bias, without providing further detail to the spokesperson. Stability.AI had released a new version of Stable Diffusion in late November. A spokesperson said the original model came with a safety filter that Lensa did not appear to use, and that the filter would remove these outputs.
The article also describes other steps: Stable Diffusion 2.0 filters some content by removing images that appear repeatedly; the model had made graphic content harder to generate; and creators of the LAION database had introduced NSFW filters. These measures address parts of the problem, but the account does not suggest that filtering alone resolves the underlying patterns in the data or the choices made by app developers.
Because Lensa became the first hugely popular app developed from Stable Diffusion, its results raise questions beyond one product. The article warns that image tools can be used to make nonconsensual nude images from women’s social media pictures or to create naked images of children. It also argues that repeated stereotypes can affect how women and girls see themselves and how others see them.
Caliskan frames the issue as a question about the cultural record these systems are creating: “In 1,000 years, when we look back as we are generating the thumbprint of our society and culture right now through these images, is this how we want to see women?”