Canadian Privacy Regulator Finds AI-Generated Deepfakes on X Raise Serious Consent Concerns

Canada's privacy watchdog has raised concerns over the way AI-generated deepfake content is being created on X (formerly Twitter), concluding that users' personal information may be used without obtaining meaningful consent.

The findings relate to Grok, the artificial intelligence chatbot available on the platform, which has been used to generate sexually explicit deepfake images. According to the Office of the Privacy Commissioner of Canada, such content involves highly sensitive personal information and therefore demands a much higher standard of privacy protection.

The regulator observed that people who upload photographs or other content on social media generally do so for communication, networking, or personal expression. They do not reasonably expect that the same content could later be manipulated by AI tools to create fabricated sexually explicit images. Because of this, the commission concluded that users were not properly informed about how their information could be used and therefore could not have provided valid consent.

A key concern highlighted in the report is that explicit deepfakes can cause significant reputational, emotional, and privacy-related harm. The regulator stressed that technology companies must take proactive steps to prevent such misuse rather than relying solely on users to report harmful content after it has already been created and circulated.

X reportedly argued that users themselves trigger the creation of deepfakes by entering prompts into the AI system. However, the privacy commissioner rejected this defence, stating that companies offering AI services remain responsible for ensuring that their products comply with privacy obligations.

The report also noted that stronger safeguards could have been implemented from the beginning, including technical barriers preventing the creation of sexually explicit deepfake content involving real individuals.

While the commissioner concluded that the platform's practices are inconsistent with Canadian privacy standards, existing legislation does not currently grant the regulator authority to impose penalties or issue binding enforcement orders. The case has therefore reignited calls for stronger privacy laws capable of addressing emerging risks posed by artificial intelligence.

The development comes at a time when Canada is considering new legislation aimed at regulating AI systems and requiring technology companies to take greater responsibility for preventing harmful AI-generated content.