San Francisco-based creator of AI chatbots, replica — which operates a freemium “virtual friendship” service based on customizable digital avatars whose “personalized” responses are powered by artificial intelligence (and designed, according to its pitch, to make human users feel better) — has been ordered by the Italian privacy watchdog to stop processing local users’ data.
The Guarantee said it is concerned that Replika’s chatbot technology poses risks to minors – and also that the company does not have a proper legal basis for processing children’s data under EU data protection rules.
In addition, the regulator is concerned about the risk that the AI chatbots may pose to emotionally vulnerable people. It also accuses Luka Inc, the developer behind the Replika app, of failing to comply with regional legal requirements to clearly communicate how it uses people’s data.
The to order stopping the processing of data from Italians is effective immediately.
In a press release the watchdog announced its intervention, saying: “The AI-powered chatbot, which generates a ‘virtual friend’ using text and video interfaces, will not be able to [the] provisional personal data of Italian users. The Italian imposed a provisional restriction on data processing Guarantee to the US-based company that developed and operates the app; the restriction takes effect immediately.”
“Recent media reports along with tests from the SA [supervisory authority] conducted on ‘Replika’ showed that the app poses actual risks to children – primarily the fact that they get answers that are definitely inappropriate for their age,” it added.
Replika was an early API partner for OpenAI’s text-generating large language model technology, GPT-3 – although the service doesn’t run on a copy of GPT-3 (nor is it the same technology as OpenAI’s bustling ChatGPT). On the contrary, the startup claims it “refinedGPT-3, using a networked machine learning model trained on dialogue, to hone the generative technology for its specific use case: conversational (and it claims “empathetic”) AI companions.
However, concerns have previously been raised about the risks the technology may pose to children – ranging from concerns that children will be exposed to inappropriate content to more general concerns that they will become addicted to the interactions or simply be encouraged to spend a lot of money on their customize avatars or access other paid content. But the Italian watchdog appears to be the first regulator to take formal action on child safety.
The Warranties order notes that several user reviews of the app report that it offers sexually inappropriate content. It also notes that while the app is listed as 17+, on Apple’s iOS and Google’s Android app stores, the developer’s terms of service prohibit use by under 13s only. And while consent from a parent or guardian is required for those under 18, the watchdog points out that the app does not attempt to verify users’ ages, nor does it block minors from providing information about their age – hence the view that Replika is failing to protect children.
“There is basically no age verification mechanism: no child gate mechanism, no blocking of the app if a user declares to be underage. While creating an account, the platform only asks for the username, email account and gender,” it notes. “And the chatbot’s ‘answers’ are often in clear conflict with the enhanced safeguards that children and vulnerable persons are entitled to. Several reviews in the two main App Stores contain comments from users flagging sexually inappropriate content.
“’Replika’ violates the EU data protection regulation: it does not meet the transparency requirements and it processes personal data unlawfully as the performance of a contract cannot be relied upon as a legal basis, even implicitly, as children are not able to to enter into a valid contract under Italian law,” de Guarantee added, saying it has ordered its US-based developer to stop processing data related to Italian users – giving it 20 days to communicate measures taken to comply with the order.
Failure to comply with the order risks a fine of up to €20 million, or 4% of total global annual turnover, it further notes.
Replika has been contacted for comment on the Warranties order.
The EU’s General Data Protection Regulation (GDPR) puts a strong emphasis on protecting children’s information and privacy — suggesting, for example, that services likely to have underage users should think about incorporating child-friendly design and be proactive in conducting risk assessments to ensure they spot potential security and other rights issues.
Watchdogs in the region have shown a willingness to pay attention to violations in this area.
Last fall, for example, Instagram was fined nearly $440 million for violating children’s privacy. Consumer protection authorities in Europe have also raised concerns about the safety of children on TikTok – although an investigation is still ongoing in Ireland into TikTok’s handling of children’s data.
Italy’s data protection watchdog has shown itself particularly sensitive to child safety in recent years — launching an emergency intervention two years ago to order TikTok to block users it couldn’t verify age in response to the death of a child who was reported to have taken part in a risky challenge on the platform. That led to a cleanup of more than half a million accounts.
Despite some enforcement of the GDPR (and consumer protection laws) regarding child safety issues, campaign groups have argued that children are still not properly protected – and continue to push for stricter laws. So the restrictions will likely only get tighter.
In the UK, an age-appropriate draft code came into effect in autumn 2021, aimed at protecting minors from security and privacy risks. While the French data protection watchdog also has a set of recommendations to ensure that children’s digital rights are protected.
The UK is also working to pass the Online Safety Act, focusing on child safety, in response to public concerns about what children are being exposed to online.
In recent months, EU lawmakers have also agreed on a total ban on processing minors’ data for ad targeting in a pair of flagship updates to the bloc’s digital rulebook: the Digital Services Act and Digital Markets Act, due later this year. become applicable. .
