AI and Misinformation: Synthetic Content and Influence Operations
Large language models can produce realistic text that is hard to distinguish from human writing, which malicious actors can use to automate influence operations. Standards bodies and research groups propose technical transparency methods such as watermarking and provenance tracking, along with stakeholder responsibilities. Combined technical, institutional, and societal measures are needed to reduce these risks.
Synthetic content and its risks
Synthetic content refers to audio, visual, or multimodal material that is generated or modified by artificial intelligence. NIST's publication on synthetic content explains that such content can be misused to deceive people or to produce harmful imagery, while also offering creative opportunities. The same technology that enables realistic images and voices can be turned to disinformation when it is used to fabricate events or attribute statements to real people.
As synthetic media becomes more accessible and sophisticated, its potential for both beneficial and harmful impact grows. Partnership on AI's responsible practices for synthetic media assigns responsibilities to technology builders, creators, and distributors to prevent the spread of misleading content. Harms range from individual reputation damage to broad erosion of trust in news and institutions.
Misinformation and disinformation are not new, but AI makes them cheaper and faster to produce at scale. The Harvard Kennedy School's Misinformation Review covers research on how people encounter and respond to false claims, including studies on fact-checking and media literacy. Such research informs policy about platform moderation and public education.
Language models and influence operations
A report from Georgetown's Center for Security and Emerging Technology, OpenAI, and the Stanford Internet Observatory describes how generative language models can automate the creation of convincing and misleading text for influence operations. The arXiv paper on automated influence operations explains that these models produce realistic outputs that are difficult to distinguish from human-written content, lowering the barrier for malicious actors.
The same report proposes a framework that maps the stages from building a language model to the formation of beliefs in the public. The stages include model construction, model access, content dissemination, and belief formation. Mitigations can target each stage, such as controlling who can use advanced models, detecting AI-generated content before it spreads, or strengthening public resilience to false information.
No single mitigation can fully prevent AI-enabled influence operations, but a combination of measures at different stages can make an important difference.
Technical approaches for transparency
Technical methods for digital content transparency include authenticating content and tracking its provenance, labeling synthetic content through watermarking, and detecting synthetic content after distribution. Each approach has strengths and limitations, and many are still in early development.
Provenance tracking involves recording the history of a piece of content, such as the device that captured it and any edits made. Watermarking embeds a hidden or visible mark that indicates AI generation. Detection algorithms attempt to classify whether a given image, video, or text was created by a model, but they can be evaded or misled.
The NIST report also outlines the need for testing and auditing these transparency tools, as well as maintaining them over time. Standards for interoperability between systems are part of the effort to create a trustworthy content environment.
Stakeholder responsibilities and cooperation
The Partnership on AI's Responsible Practices for Synthetic Media offers recommendations for three groups: builders of technology and infrastructure, creators, and distributors and publishers. The framework centers on consent, disclosure, and transparency, and it asks organizations to collaborate on countering harmful uses and to pursue mitigation strategies when harm occurs.
The International AI Safety Report, produced by over 100 experts and backed by more than 30 countries, identifies that technical safeguards such as watermarking are advancing, but there are gaps. The International AI Safety Report notes that sophisticated attackers can often bypass current defenses, and the real-world effectiveness of many safeguards remains uncertain. The report stresses the need for international cooperation and sharing of best practices.
Research covered in the Harvard Kennedy School Misinformation Review shows that experts disagree on how to fight AI disinformation, yet they agree that solutions must differ between domains such as health and politics. This insight underlines the need for flexible, evidence-based policies that address specific harms rather than applying a single universal rule.
Effective mitigation depends on coordinated action across model developers, platforms, fact-checkers, governments, and educators. Each actor has a role, and none can solve the problem alone.
| Approach | Purpose | Example method |
|---|---|---|
| Provenance tracking | Record content's origin and edits | Create and store a provenance data track |
| Watermarking | Label synthetic content | Embed a visible or invisible mark in media |
| Detection | Identify synthetic content after publication | Use classifiers trained on AI output |
| Authentication | Verify content's source | Cryptographic signatures for capture devices |
Frequently asked questions
What is synthetic content?
Synthetic content is audio, visual, or multimodal material generated or modified by AI, such as deepfake videos or convincingly realistic text.
How can AI spread misinformation?
Language models can produce large volumes of plausible text that is hard to distinguish from human writing, enabling automated influence operations and propaganda.
What technical methods help detect synthetic content?
Provenance tracking, watermarking, and detection algorithms are three approaches described by NIST for increasing transparency and identifying AI-generated material.
Who is responsible for mitigating AI-related misinformation?
The Partnership on AI assigns responsibility to technology builders, creators, and distributors, and emphasizes cooperation among all stakeholders, including governments and civil society.
Why is international cooperation important?
The International AI Safety Report notes that technical defenses have gaps and real-world effectiveness is uncertain, so shared knowledge and coordinated policy are needed.