Deepfakes: how synthetic media is made, detected and regulated
Deepfakes are synthetic media generated or modified by AI. Detection is challenging, but benchmarks like FaceForensics++ help evaluate detectors. Content provenance standards such as C2PA and regulatory obligations like the EU AI Act aim to increase transparency and accountability.
What synthetic media is and how it is made
Synthetic media is audiovisual content that is generated or modified by artificial intelligence. The Partnership on AI's responsible practices notes that AI-generated content has moved from research labs to become a major engine of digital content creation, encompassing audio, visual, and multimodal forms.
Deepfakes are a prominent type of synthetic media, typically involving manipulated facial images or videos. The NIST's synthetic content report explains that approaches for handling such media include authenticating content, tracking provenance, labeling with watermarking, and detecting synthetic content.
Detecting manipulated media
Detecting deepfakes is difficult, both for automated systems and human observers. The FaceForensics++ benchmark was created to standardize evaluation of facial manipulation detection methods, including DeepFakes, Face2Face, FaceSwap, and NeuralTextures.
The benchmark provides a database of over 1.8 million manipulated images, which is more than an order of magnitude larger than comparable public forgery datasets. The paper shows that data-driven forgery detectors using domain-specific knowledge achieve high accuracy even under strong compression, and they clearly outperform human observers.
Provenance and authentication standards
Content provenance standards help establish the origin and edit history of digital content. The Coalition for Content Provenance and Authenticity offers an open technical standard called Content Credentials, which functions like a nutrition label for digital content, giving anyone access to its history at any time.
The NIST report examines various technical approaches, including tracking provenance data and watermarking synthetic content. These methods aim to make manipulation detectable and to provide verifiable information about how a piece of content was created and altered.
Regulatory and ethical frameworks
The EU AI Act Article 50 sets transparency obligations for providers and deployers of AI systems. It requires providers of AI systems that generate synthetic audio, image, video, or text content to mark outputs in a machine-readable format as artificially generated, unless the system performs only standard editing or does not substantially alter input data.
The Partnership on AI has developed Responsible Practices for Synthetic Media, a framework offering recommendations for builders, creators, and distributors. It is based on consent, disclosure, and transparency, and asks organizations to collaborate to counter harmful uses and pursue mitigation strategies.
Ethical considerations also come from academic philosophy. The Stanford Encyclopedia's AI ethics entry discusses concerns about human autonomy and manipulation, which are directly relevant to the deceptive potential of deepfakes.
| Framework | Type | Key focus |
|---|---|---|
| NIST report | Technical overview | Authentication, provenance, labeling, detection |
| C2PA | Open standard | Content Credentials for origin and edits |
| EU AI Act | Regulation | Transparency obligations for synthetic content |
| Partnership on AI | Guidance | Responsible practices for creators and distributors |
Frequently asked questions
What is a deepfake?
A deepfake is a type of synthetic media, usually a manipulated facial image or video, created with artificial intelligence. It can be used for creative purposes but also to spread false information.
How can deepfakes be detected?
Detection can use machine learning models trained on large datasets of manipulated images. Benchmarks like FaceForensics++ help evaluate detector accuracy across different manipulation techniques.
What is C2PA?
C2PA, or the Coalition for Content Provenance and Authenticity, provides an open technical standard called Content Credentials that documents the origin and edits of digital content.
What does the EU AI Act require for synthetic content?
Under Article 50, providers of AI systems that generate synthetic audio, image, video, or text must mark outputs in a machine-readable format as artificially generated, unless the system does only minor editing.
What are the Partnership on AI's responsible practices?
The framework gives recommendations for builders, creators, and distributors, based on consent, disclosure, and transparency, and includes collaboration to counter harmful uses.