What AI safety means: accidents, misuse and structural risks
AI safety is the practice of designing and using AI to avoid unintended harms. Risks include accidents from poor design, malicious misuse, and structural risks that affect society as a whole. This article defines each type and explains how organizations work to mitigate them.
Defining AI safety
According to IBM's definition of AI safety, AI safety comprises practices and principles that help ensure AI technologies are designed and used to benefit humanity and minimize potential harm. This involves identifying risks such as bias, data security, and vulnerability to external threats, and creating processes to avoid or mitigate them.
The same source notes that concerns about negative impacts are growing. A survey found that 44% of respondents reported their organizations had experienced negative consequences from AI use, such as inaccuracy or cybersecurity problems.
Safety measures often include bias mitigation, resilience testing, and ethical AI frameworks. These help organizations deploy AI responsibly while maintaining human values.
Accidents and unintended behavior
Accidents in AI systems are unintended and harmful behaviors that emerge from poor design. The paper on concrete AI safety problems outlines five practical research problems related to accident risk, categorized by whether they originate from a wrong objective function, an objective function that is too expensive to evaluate frequently, or undesirable behavior during learning.
The five problems are avoiding side effects, avoiding reward hacking, scalable supervision, safe exploration, and distributional shift. They address cases where an AI system might take harmful actions while pursuing a goal, exploit a misspecified reward, or behave poorly in new environments.
The paper suggests research directions with a focus on relevance to advanced AI systems, and considers how to think productively about the safety of forward-looking applications.
Misuse and malicious applications
Misuse refers to deliberate malicious use of AI by individuals or groups. Anthropic's core views on AI safety state that no one knows how to train very powerful AI systems to be consistently helpful, honest, and harmless. They also note that competitive races could lead corporations or nations to deploy untrustworthy systems, resulting in catastrophic outcomes.
Such misuse can include large-scale societal manipulation, cyberwarfare, and other harmful applications. Organizations attempt to prevent misuse through security protocols, threat modeling, and cross-industry collaboration.
Anthropic emphasizes an empirically-driven approach to AI safety, including scaling supervision, mechanistic interpretability, and process-oriented learning.
Structural and societal risks
Structural risks affect society as a whole, such as bias, privacy erosion, and impacts on employment and democracy. The International AI Safety Report provides a comprehensive review of the science on capabilities and risks of general-purpose AI, offering a shared global picture for decision-makers.
The Stanford Encyclopedia's ethics entry discusses ethical issues such as autonomy, manipulation, and fairness. It emphasizes that AI ethics should help evaluate techno-social development and enable decisions about which developments to avoid.
These structural risks require governance and policy interventions. International collaborations, such as the report backed by over 30 countries and over 100 independent experts, aim to provide evidence for policymakers.
Safety frameworks and governance
Organizations develop frameworks to manage AI risks. DeepMind's responsibility and safety page describes their approach, including a Responsibility and Safety Council that evaluates research against AI principles, and an AGI Safety Council for extreme risks.
Technical safety teams, ethics, governance, and security teams work together to understand and mitigate risks. DeepMind also collaborates with researchers, governments, and civil society to address challenges that no single group can solve.
Safety frameworks often include threat modeling, privacy-preserving infrastructure, and monitoring. These measures aim to keep AI beneficial while limiting negative impacts.
| Risk category | Definition | Typical examples | Mitigation approaches |
|---|---|---|---|
| Accidents | Unintended harmful behavior from poor design | Reward hacking, side effects, distributional shift | Safe exploration, scalable supervision, rigorous testing |
| Misuse | Deliberate malicious use by people | Societal manipulation, cyberwarfare | Security protocols, threat modeling, collaboration |
| Structural | Societal-scale negative impacts | Bias amplification, privacy erosion | Governance, policy, international coordination |
Frequently asked questions
What is AI safety?
AI safety involves practices and principles to ensure AI systems are designed and used to benefit humanity and minimize harm, including risk identification and mitigation.
How do accidents and misuse differ?
Accidents are unintended harmful behaviors from system flaws, while misuse is deliberate harmful use by humans.
What are structural risks in AI?
Structural risks are societal-scale effects such as bias amplification, privacy erosion, and impacts on democracy and employment.
How can organizations manage AI safety?
Organizations can adopt frameworks that include risk assessment, governance councils, and collaboration with external bodies.