On July 16, 2026, San Francisco City Attorney David Chiu took the unprecedented step of sending cease-and-desist letters directly to Apple and Google, demanding the removal of 13 apps from their respective app stores that enable the creation of AI-generated nonconsensual nude images, or "nudify" content. The move represents the strongest municipal action yet against the underground economy of deepfake abuse, where independent research estimates the targeted apps alone have generated over $120 million in revenue from nearly 480 million cumulative downloads.
The Scale of the Deepfake App Economy
The 13 apps identified by Chiu's office, eight on Apple's App Store and five on Google's Play Store, represent only the visible tip of a much larger ecosystem. Research published by the Tech Transparency Project (TTP) earlier this year identified roughly 100 apps across both platforms that offer nudification features, with many disguising themselves as benign face-swapping tools. A preprint paper from Cornell University and Georgetown University published in May went further, identifying 420 apps offering general face-swapping capabilities and finding that 70 percent of the 155 apps they tested could be used to create nude face swaps without any safety guardrails in place.
One app singled out in Chiu's legal letters had surpassed 1 million downloads, advertising more than a dozen AI image styles including descriptions that are explicitly sexualized. These apps charge users through in-app payments, meaning Apple and Google take their standard commission cut on every transaction. "These companies have made millions of dollars in fees from apps that offer nudification," Chiu told WIRED, adding that both companies have a responsibility to ensure their platforms do not facilitate sexual abuse.
Platform Responses and the Moderation Gap
Both Apple and Google have developer policies that explicitly ban pornography, harassment, and abusive content, and both companies have pointed to their moderation efforts in response to the city attorney's letters. Google spokesperson Dan Jackson stated that the company has removed "hundreds" of apps with nudifying features, restricted search terms like "nudify" in the Play Store, and is continuing to take proactive steps. Apple representative Adam Dema confirmed that three of the eight flagged iOS apps had already been removed, with the company in the process of terminating their developer accounts. The remaining four developers have been given an ultimatum: address policy violations or face removal.
Yet the persistent reappearance of these apps suggests a structural gap in how moderation works. TTP Director Katie Paul captured the frustration bluntly: "Apple and Google make a lot of promises in their marketing about how trusted and safe their app stores are. And that is just not what is playing out in reality." The dual-use nature of these face-swap apps presents a unique moderation challenge, as the Cornell and Georgetown researchers noted: they present as benign tools but possess the capability to create harmful content, making them difficult to catch with automated screening alone.
The Human Toll and Legal Precedent
The impact of this technology extends far beyond policy debates. WIRED and Indicator Media previously documented incidents in at least 90 schools where deepfake sexual abuse images of minors were created and circulated. The victims are overwhelmingly women and girls, targeted by people, mostly men, who upload a single reference photo and, with a few clicks, generate explicit content in seconds. "These images are used to bully, humiliate, and threaten women and girls," Chiu said. "This industry has a horrific impact on one's reputation, mental health, loss of autonomy. There have been victims who have been suicidal."
The legal framework underpinning Chiu's action relies on California state laws that prohibit supporting services that create deepfake pornography. The cease-and-desist letters argue that Apple and Google are "aiding and abetting" the sale of explicit deepfake images by providing the payment infrastructure and distribution platform. This legal theory, if tested in court, could establish a precedent that holds app store operators directly liable for third-party apps that use their payment systems to monetize harmful technology.
What This Means for Founders and the AI Industry
For founders building on AI, this escalation carries several clear signals. First, the window for self-regulation is closing. When municipal attorneys are sending legal letters directly to the world's largest tech companies, the broader regulatory wave is already here. Second, the dual-use argument will not hold as a defense. If your AI tool can be used to create harmful content and you do not implement meaningful guardrails, you are exposed to liability regardless of your stated intent. Third, app store distribution is increasingly a regulatory bottleneck. Apple and Google now face mounting pressure to pre-screen for abuse potential, which will inevitably tighten their review processes for all AI-powered apps.
Chiu has signaled that this is just the beginning. "My hope is that Apple and Google will immediately remove these apps and strengthen their screening systems to make sure that apps like this never get onto their platforms in the future," he said. "If they do not, we will have to consider all of our legal options." For solo founders and small teams deploying AI products, the takeaway is clear: build guardrails from day one, not as an afterthought. The cost of ignoring this is no longer just reputational damage, it is legal action directed at the platforms themselves, and by extension, at every app that rides on their infrastructure.