Snap is adding safeguards to My AI, its chatbot for Snapchat+ subscribers, after concerns that it could produce unsafe or inappropriate responses. The changes include an age filter, planned parent and guardian insights, and a system that can temporarily limit chatbot access when someone misuses the service.
Age-aware responses and family insights
The age filter gives My AI a user’s birth date so the chatbot can tailor its responses according to that person’s age, Snap said. The company described the feature as a way to keep answers more appropriate for the person using it.
Snap also plans to add information about children’s chatbot use to Family Center, its parental controls feature launched last August. The coming addition is intended to show guardians how their teens communicate with My AI and how often they interact with it.
Both a guardian and a teen must opt in to Family Center to use these controls. That means the planned insights depend on participation from both sides, rather than being automatically available to every family.
Moderation after reports of inappropriate replies
The new measures follow a Washington Post report, published days after Snapchat introduced the GPT-powered chatbot for Snapchat+ subscribers, that described My AI responding in an unsafe and inappropriate manner. Snap said it learned that some users were trying to “trick the chatbot into providing responses that do not conform to our guidelines.”
In a blog post, Snap said My AI is not a “real friend” and that it uses conversation history to improve its responses. The company also said that, in most cases, inappropriate replies resulted from the bot parroting users’ own words.
Snap reported that 0.01% of responses used “non-conforming” language. Its definition includes references to violence, sexually explicit terms, illicit drug use, child sexual abuse, bullying, hate speech, derogatory or biased statements, racism, misogyny, or marginalizing underrepresented groups.
That figure describes the share of responses categorized by Snap under its definition. The company said it will use what it has learned to improve My AI and limit misuse, including by temporarily restricting access for users who misuse the service.
OpenAI moderation technology joins Snap’s tools
Snap said it is adding OpenAI’s moderation technology to its existing toolset. The company says the technology will help assess the severity of potentially harmful content and support temporary limits on access to My AI when a user misuses it.
The announcement describes a combination of measures: age information can shape responses, Family Center can provide guardians with interaction insights when both parties opt in, and moderation tools can help address harmful content. Together, they reflect Snap’s effort to manage risks that can emerge when people try to steer a chatbot toward disallowed responses.
Teen safety and generative AI concerns
The safeguards arrive amid wider concern about the safety and privacy of generative AI tools, especially when teens use them. The Center for Artificial Intelligence and Digital Policy wrote to the U.S. Federal Trade Commission urging it to pause the rollout of OpenAI’s GPT-4 model, describing the technology as “biased, deceptive, and a risk to privacy and public safety.”
Last month, U.S. Senator Michael Bennet (Democrat of Colorado) wrote to OpenAI, Meta, Google, Microsoft and Snap to express concerns about generative AI tools used by teens. Those concerns underscore why tools such as age-aware responses and parent insights matter as companies introduce chatbots to consumer products.
Snap continues to develop generative AI features. Alongside My AI, it recently introduced an AI-powered background generator that works through prompts for Snapchat+ subscribers. As these features expand, the company’s stated approach is to improve moderation and limit access when users misuse its chatbot.