Google Play is adding new requirements for apps that generate AI content. Developers will need to give users an in-app way to flag offensive outputs, while new limits will govern access to photos and videos and the use of full-screen notifications.
Reporting becomes part of AI app policy
The policy applies to Android apps on Google Play that generate AI content. Google says developers must make reporting and flagging available within the app, then use those reports to guide their filtering and moderation approaches.
The covered features include text-to-text conversational chatbots when interaction with the chatbot is central to the app. The policy also covers images generated by AI from text, image, or voice prompts. That scope includes familiar forms of generative AI products, rather than only apps devoted to image creation.
Google Play’s policy updates are rolling out today, but AI app developers have until early 2024 to implement the flagging and reporting changes. That gives developers time to build reporting into the user experience and connect incoming reports to their moderation processes.
Why Google is addressing generated content
The policy follows a surge in AI-generated apps and examples of users getting them to produce content they should not. The source article points to Lensa, whose users tricked the app into creating NSFW imagery, and Remini, an AI headshot app that was found to enlarge some women’s breasts or cleavage and thin them.
Other examples involved users getting around safeguards in Microsoft’s and Meta’s AI tools. They prompted images of Sonic the Hedgehog pregnant or fictional characters doing 9/11. These cases illustrate that moderation challenges can arise from how people use an app, as well as from the app’s normal output.
The concerns extend beyond offensive or unwanted images. The article notes that open source AI tools have been used to create child sexual abuse material at scale. It also describes fears that AI-generated fake images, or deepfakes, could mislead or misinform voters around coming elections.
Google’s announcement also emphasizes that AI content generators remain subject to its existing developer policies. Those rules prohibit restricted content such as CSAM and apps that enable deceptive behavior. The new reporting requirement adds a way for users to flag problems; it does not replace those existing restrictions.
Photo access and full-screen alerts face limits
Google is also reviewing some app permissions, including requests for broad access to photos and videos. Under the new policy, an app can access those files only when that access directly relates to its functionality.
For a one-time or infrequent need, such as asking someone to upload a set of selfies, apps will need to use a system picker, like the new Android photo picker. The change narrows the path to a user’s media when an app does not need ongoing access to it.
Full-screen notifications are getting a separate restriction. Google says apps should use them only when there is a high-priority need, such as an incoming phone or video call. Apps have also used the feature to promote paid subscriptions or other offers, a use that the policy aims to limit.
Apps targeting Android 14 and above will need special app access permission to use full-screen functionality, and Google says that permission will go only to apps that actually require it. The rule makes full-screen alerts a more limited capability tied to the app’s purpose.
A new direction for app store oversight
The source article describes Google as moving ahead of Apple on a formal policy for AI apps and chatbots. It notes that Apple’s App Store Guidelines did not then have a formal AI or chatbot policy, although Apple had tightened rules in other areas, including apps that seek data to identify a user or device and apps that try to copy others.
Together, the Play Store changes address several points where an app can affect a user: the content it generates, the media it can reach, and the urgency it claims when interrupting the screen. For developers, the immediate task is to add in-app reporting for offensive AI output by early 2024 and ensure permissions match what the app actually needs.