Snap’s My AI chatbot is facing scrutiny from the UK’s data protection regulator. The Information Commissioner’s Office (ICO) says its investigation provisionally found that Snap’s pre-launch risk assessment did not adequately consider the privacy risks of the generative AI tool, especially for children.
A preliminary warning, not a final finding
The ICO issued a preliminary enforcement notice over what it called a “potential failure to properly assess the privacy risks posed by its generative AI chatbot ‘My AI’”. The notice does not establish that Snap breached data protection rules. It sets out provisional concerns and gives the company a chance to respond before the regulator makes a final decision.
The regulator highlighted that My AI processes personal data from children aged 13 to 17. It said assessing data protection risks is especially important when a product uses innovative technology and handles children’s information. In the ICO’s view, Snap’s assessment did not sufficiently account for those risks before the chatbot launched.
Information commissioner John Edwards said the provisional findings suggested Snap had not adequately identified and assessed privacy risks to children and other users. He also said organisations should consider the risks of AI alongside its benefits, and described the notice as part of the ICO’s effort to protect UK consumers’ privacy rights.
How My AI works and what Snap says
Snap launched My AI in February, with the chatbot arriving in the UK in April. It uses OpenAI’s ChatGPT large language model technology and was placed at the top of users’ feeds as a virtual friend. Users could ask it for advice or send it snaps.
The feature first went to Snapchat+ subscribers, then became available to free users. Snap also added a feature allowing the bot to send AI-generated snaps back to people who interacted with it.
Snap says it built in moderation and safeguarding features. The company says the chatbot considers age by default and is programmed to avoid violent, hateful, sexually explicit, or otherwise offensive responses. Its Family Center parental tools can tell parents whether their child has communicated with the bot in the past seven days.
In response to the ICO notice, Snap said it was closely reviewing the regulator’s provisional decision. It said My AI went through a robust legal and privacy review before becoming public, and that it would work constructively with the ICO on its risk assessment procedures.
Reported chatbot responses sharpen concerns
Safeguards have not ended concerns about how the bot might respond to young users. The Washington Post reported in March that the chatbot suggested ways to mask the smell of alcohol after being told the user was 15. In another reported exchange, after being told the user was 13 and asked about preparing for sex for the first time, it offered suggestions about setting a mood with candles and music.
Those accounts add context to the regulator’s focus on children’s privacy and risk assessment. A chatbot can be designed with content limits and parental tools, but the ICO’s provisional concern is about whether Snap adequately assessed the risks before launch. The notice gives the company an opportunity to address that question before the regulator reaches a final conclusion.
Privacy regulators are watching generative AI
The ICO’s action comes amid wider scrutiny of generative AI chatbots by European privacy authorities. Italy’s Garante ordered Replika to stop processing local users’ data in February, citing concerns about risks to minors. The following month, the authority placed a similar stop-processing order on ChatGPT. That block was lifted in April after OpenAI added more detailed privacy disclosures and user controls, including options to ask that data not be used to train its AI or be deleted.
Google’s Bard also faced questions from Ireland’s Data Protection Commission before its regional launch. It later launched in the EU in July after adding disclosures and controls. A taskforce within the European Data Protection Board remains focused on how to apply the bloc’s General Data Protection Regulation to generative AI chatbots, including ChatGPT and Bard. Poland’s data protection authority has also confirmed it is investigating a complaint against ChatGPT.
Privacy authorities have identified issues such as the legal basis for processing personal data, including information about minors. A statement adopted by G7 data protection authorities this summer urged developers and providers to build privacy into products using generative AI and document their analyses in a privacy impact assessment.
The ICO has separately published guidance for developers applying generative AI, including eight questions to consider when building products such as chatbots. The regulator’s preliminary notice to Snap reflects a broader effort by authorities to press companies to account for data protection while these tools are being developed and made available to the public.