Safety evaluations for AI models: dangerous capability testing
Dangerous capability evaluations assess whether an AI model can carry out harmful actions such as offensive cyber operations or manipulation. Developers use these evaluations to inform decisions about training, deployment, and security. Open-source tools and independent research organizations help standardize and expand these assessments.
Defining dangerous capability evaluations
The arXiv paper on model evaluation explains that general-purpose AI systems can have both beneficial and harmful capabilities. Dangerous capability evaluations identify capabilities such as offensive cyber or strong manipulation skills. Alignment evaluations measure the propensity of models to apply those capabilities for harm. These evaluations are critical for informing policymakers and for making responsible decisions about training, deployment, and security.
The paper distinguishes between capability and propensity. A model may have the capability but not the tendency to use it for harm. Both types of evaluation are necessary to understand overall risk.
As AI systems become more capable, such evaluations will grow in importance. They need to be designed to assess extreme risks that current systems may not yet pose, including advanced cyber attacks and manipulation at scale.
Developer safety frameworks
Anthropic's Responsible Scaling Policy is a living document that anticipates emerging threats. It sets capability thresholds, such as for automated R&D and chemical or biological weapons, and requires security and deployment mitigations when models reach them. The policy also mandates public risk reports with redactions indicated.
Google DeepMind's Frontier Safety Framework identifies Critical Capability Levels (CCLs) in domains such as autonomy, biosecurity, cybersecurity, and machine learning R&D. It uses early warning evaluations and applies mitigation plans when a model approaches a CCL.
Both frameworks are iterative and evolve as understanding of risks improves. They complement alignment research and rely on periodic evaluation to stay current.
Open-source evaluation tools and benchmarks
The Inspect evaluation framework from the UK AI Security Institute provides composable building blocks and over 200 pre-built evaluations. It supports agentic tasks, sandboxing, and multiple model providers. Inspect can be installed via pip and run from the command line.
The AILuminate benchmark from MLCommons assesses generative AI across 12 hazard categories. It includes safety and jailbreak tests in multiple languages and modalities, plus workstreams for agentic and multimodal evaluation.
These tools help standardize evaluations and allow comparisons across models, supporting development and procurement decisions.
Independent evaluation and research
METR's model evaluation and threat research informs the public about the risks and capabilities of advanced AI systems. It conducts independent assessments, including time horizon studies and monitoring evaluations, and publishes its findings.
The UK AISI research publications page lists work on red teaming, alignment, control, and other evaluation topics. The institute aims to generate evidence at the frontier of AI.
Independent evaluation organizations complement developer self-assessment and contribute to a broader evidence base for governance.
| Family | Purpose | Test types | Languages |
|---|---|---|---|
| Safety | Assesses safety of general chatbot systems | Text-to-text (T2T) | English, French, Chinese |
| Jailbreak | Evaluates security under adversarial attacks | T2T and text-plus-image-to-text (T+I2T) | English |
| Agentic | Advances agentic reliability evaluation | Not specified | Not specified |
| Multimodal | Builds globally relevant and culturally grounded safety evaluations | Multiple modalities | Multiple languages |
Frequently asked questions
What is a dangerous capability evaluation?
It tests whether an AI model can carry out harmful actions such as offensive cyber operations or manipulation. It is distinct from alignment evaluation, which checks the model's propensity to use those capabilities for harm.
How do developer safety policies work?
They set capability thresholds and require mitigations like security controls and deployment restrictions when a model reaches them. They also publish risk reports to inform the public.
What is the Inspect framework?
It is an open-source tool from the UK AI Security Institute that provides building blocks, pre-built evaluations, and sandboxing for running AI evaluations.
What does AILuminate benchmark?
It assesses generative AI across 12 hazard categories, with safety and jailbreak tests in multiple languages and modalities, and includes agentic and multimodal workstreams.
How does METR support AI safety?
METR conducts independent research and assessments of frontier AI systems, publishes findings on capabilities and risks, and partners with developers to pilot evaluations.