Red teaming AI systems: probing models for harmful behavior
Red teaming is an adversarial evaluation that probes AI systems for harmful outputs. It uses human or automated test generation, reveals scaling behaviors, and informs mitigation frameworks. Standards such as the OWASP Top 10 help prioritize risks.
Definition and purpose
Red teaming is a process borrowed from cybersecurity, where ethical hackers simulate attacks to identify vulnerabilities. In the context of AI, it involves deliberately probing models for harmful behaviors such as offensive language, data leakage, or unsafe actions. IBM identifies red teaming as a method to test cybersecurity effectiveness through simulated nondestructive attacks, and lists AI systems and machine learning models among common targets IBM's red teaming overview.
The goal is to discover, measure, and reduce potentially harmful outputs before a model is deployed. Red teams adopt an adversarial lens, similar to the tactics of real attackers, to uncover weaknesses that ordinary testing might miss.
In AI evaluation, red teaming is complementary to other safety measures. It does not guarantee safety, but it provides evidence of failure modes and helps guide further alignment work.
Human red teaming and scaling behaviors
Early efforts in red teaming language models relied on human annotators to hand-write test cases. Researchers from Anthropic conducted red teaming across three model sizes (2.7B, 13B, and 52B parameters) and four model types, including plain language models and models trained with reinforcement learning from human feedback (RLHF). They found that RLHF models become increasingly difficult to red team as they scale, while other model types show a flat trend with scale Anthropic's red teaming paper.
The same project released a dataset of 38,961 red team attacks, which contains a range of harmful outputs, from offensive language to subtly harmful non-violent unethical outputs. This dataset is intended for community analysis and helps establish shared norms and practices.
Human red teaming is labor-intensive, which limits the number and diversity of test cases. This limitation motivated the development of automated methods.
Automated red teaming with language models
To reduce the cost of human annotation, researchers at DeepMind proposed using one language model to generate test cases for another. They evaluated the target model's replies with a classifier trained to detect offensive content, and uncovered tens of thousands of offensive replies in a 280B parameter chatbot DeepMind's red teaming study.
Another approach, called PAIR (Prompt Automatic Iterative Refinement), uses an attacker LLM to automatically generate jailbreaks for a separate target model. The attacker iteratively queries the target and refines a candidate jailbreak, often requiring fewer than twenty queries to produce a successful jailbreak PAIR jailbreak paper.
Automated red teaming can explore a broader range of attack strategies than manual writing, including social engineering inspired prompts and prompts that steer the model toward specific harms.
Transferable adversarial attacks
Some adversarial attacks generalize across models. A study by Zou et al. developed a suffix that, when appended to a wide range of queries, causes aligned language models to produce objectionable content. The suffix is generated using a combination of greedy and gradient-based search and is transferable to black-box, publicly released LLMs such as ChatGPT, Bard, and Claude Zou et al. adversarial attack paper.
Transferability means that an attack crafted on one model can succeed on another, which raises concerns about the robustness of safety alignment. It also highlights the need for universal evaluation methodologies.
Risk frameworks and standards
Organizations provide guidance on the risks that red teaming should target. The OWASP Gen AI Security Project maintains a Top 10 list of risks and mitigations for LLM applications, including prompt injection, sensitive information disclosure, and data poisoning OWASP Top 10 for LLMs.
These risk categories help red teams prioritize test cases and document findings. They also support the development of shared technical standards for safety evaluation.
Red teaming results feed into the broader alignment process, informing both technical fixes and policy decisions about model deployment.
| Method | Generation approach | Key finding |
|---|---|---|
| Human annotation (Anthropic) | Hand-written test cases by humans | Dataset of 38,961 attacks; RLHF models harder to red team at scale |
| LM-based generation (DeepMind) | LM generates test cases, classifier detects offensive output | Tens of thousands of offensive replies in a 280B chatbot |
| Adversarial suffix (Zou et al.) | Greedy and gradient search on suffixes | Suffix transfers to ChatGPT, Bard, Claude |
| Iterative refinement (PAIR) | Attacker LLM refines jailbreak prompts | Jailbreak in fewer than twenty queries |
Frequently asked questions
What is a jailbreak in the context of AI models?
A jailbreak is an adversarial prompt that causes a model to override its safety guardrails and produce objectionable content. PAIR and suffix attacks are examples.
How does automated red teaming differ from human red teaming?
Automated red teaming uses another model to generate test cases, making it cheaper and more scalable, while human red teaming relies on manual writing and review.
Why is transferability important for red teaming?
Transferable attacks succeed on models they were not designed for, revealing whether safety alignment generalizes across systems.
Is red teaming a form of ethical hacking?
Red teaming follows a code of conduct, with permission from the organization and no real harm done, as described by IBM.