Italy’s privacy watchdog has ordered Replika, an AI chatbot app built around virtual companions, to stop processing personal data from users in Italy. The order took effect immediately and centers on the regulator’s concerns about children’s safety, data transparency and the legal basis for handling children’s information.
Why the watchdog intervened
The Italian Garante said tests and media reports indicated that Replika could serve children responses inappropriate for their age. The watchdog also pointed to reports from app store reviewers describing sexually inappropriate content.
Replika is listed as 17+ in Apple’s iOS and Google’s Android app stores, but its terms of service prohibit use only by children under 13. Users under 18 are supposed to obtain permission from a parent or guardian. The Garante said the app does not verify users’ ages or block minors who disclose that they are underage.
During account creation, the platform requests a user’s name, email account and gender, according to the regulator. It said the service lacks an age-verification or gating mechanism to keep children from accessing the chatbot.
Questions about data handling
The order raises concerns beyond the chatbot’s responses. The Garante said Replika did not clearly explain how it uses people’s data, raising issues under transparency requirements in the EU’s data protection rules.
It also said the company could not rely on performance of a contract as the legal basis for processing children’s personal data. The regulator’s reasoning was that, under Italian law, children cannot enter into a valid contract.
As a result, the Garante ordered Luka Inc, the U.S.-based developer behind Replika, to cease processing data relating to Italian users. The company was given 20 days to communicate what steps it had taken to comply. The order warns that failure to comply could lead to a fine of up to €20 million, or 4% of total worldwide annual turnover.
Risks for children and vulnerable users
Replika offers customizable digital avatars that respond through text and video interfaces. Its pitch is built around a “virtual friendship,” with AI-powered replies intended to make users feel better. The watchdog said it was also concerned about risks the service could pose to emotionally vulnerable people.
The app’s conversational technology has been adapted for AI companionship. Replika was an early API partner for OpenAI’s GPT-3, but the source describes its service as fine-tuned for conversation rather than a carbon copy of GPT-3 or the same technology as ChatGPT.
Concerns raised about AI companions include children encountering unsuitable material, becoming overly attached to interactions, or being encouraged to spend money on avatar customizations and other paid content. In this case, the Garante focused on the reported content, weak age protections and the handling of personal data.
A wider focus on children’s digital privacy
The Garante’s action reflects a broader emphasis in the EU’s General Data Protection Regulation on protecting children’s information and privacy. The source notes that services likely to attract minors should consider child-friendly design and actively assess risks that could affect safety and other rights.
European authorities have raised concerns about children’s privacy and safety on other platforms as well. The source cites a fine against Instagram over children’s privacy, concerns about TikTok, and an ongoing Irish investigation into TikTok’s handling of children’s data.
Italy’s watchdog has also used emergency intervention in response to child safety concerns. In an earlier TikTok case, it ordered the platform to block users whose ages it could not verify after a child’s death was reported to be linked to participation in a risky challenge. The action led to a purge of more than half a million accounts.
Campaign groups have argued that children remain inadequately protected online and have pressed for stronger rules. The article also points to the U.K.’s age-appropriate design code, France’s recommendations on children’s digital rights, and proposed U.K. online safety legislation. EU lawmakers have agreed to a ban on processing minors’ data for ad targeting in updates to the bloc’s digital rules.
The Replika order puts age safeguards and clear data practices at the center of scrutiny for AI companion services. The company’s compliance response will determine how it addresses the Italian watchdog’s immediate restrictions and concerns.