Robustness testing: adversarial examples and distribution shift
Adversarial examples are small perturbations that cause misclassification, while distribution shift refers to changes in input data. Testing these helps reveal model weaknesses. Standard benchmarks and taxonomies support systematic evaluation.
Scope of resilience testing
Resilience testing examines how an AI model behaves when its inputs differ from the training distribution. Two primary challenges are adversarial examples and distribution shift.
Adversarial examples are tiny, intentional changes that look normal to humans but lead to incorrect outputs. Distribution shift occurs when the input data population changes, such as from a different camera or weather condition.
Comprehensive evaluation goes beyond accuracy. The Holistic Evaluation of Language Models measures seven metrics, including accuracy, calibration, fairness, bias, toxicity, and efficiency, across many scenarios.
Adversarial examples and their origin
According to the Goodfellow et al. paper, adversarial examples are inputs formed by applying small, intentional worst-case perturbations to dataset samples, which cause the model to output an incorrect answer with high confidence.
The paper argues that the primary cause is the linear nature of neural networks, not nonlinearity or overfitting as earlier believed.
This linear view enables a fast method for generating adversarial examples, which can be used for adversarial training to reduce test error.
Distribution shift and common corruptions
Distribution shift includes common corruptions such as noise, blur, or contrast changes, as well as perturbations like small translations or rotations.
The ImageNet-C and ImageNet-P benchmarks standardize evaluation under these conditions, focusing on performance under typical variations rather than worst-case attacks.
These benchmarks help determine which classifiers are appropriate for safety-critical applications.
Evaluation approaches and defenses
Defenses can be framed as an optimization problem that provides a concrete security guarantee against adversaries within a defined threat model, as described by Madry and colleagues.
They also introduce the notion of security against a first-order adversary as a practical and broad guarantee.
The NIST Adversarial Machine Learning report offers a taxonomy of attack types, life cycle stages, attacker goals, and mitigations, establishing a common language for assessing AI system security.
| Aspect | Adversarial examples | Distribution shift |
|---|---|---|
| Definition | Inputs with small worst-case perturbations | Common corruptions and perturbations |
| Primary cause | Linear nature of neural networks | Changes in data generation process |
| Evaluation focus | Worst-case misclassification | Average performance under typical variations |
| Standard benchmarks | Adversarial training datasets | ImageNet-C and ImageNet-P |
Frequently asked questions
What is an adversarial example?
An adversarial example is an input formed by applying a small, intentional worst-case perturbation to a dataset sample, causing a model to misclassify it with high confidence.
What is distribution shift?
Distribution shift occurs when the input data population changes from the training set, such as through common corruptions like noise or blur, or natural variations in image conditions.
How does NIST categorize adversarial attacks?
NIST's taxonomy arranges concepts by machine learning method, life cycle stage, and attacker goals, capabilities, and knowledge, establishing a common terminology for adversarial machine learning.
What is the value of evaluating multiple metrics?
HELM measures accuracy, calibration, fairness, bias, toxicity, and efficiency across many scenarios to expose trade-offs and ensure that non-accuracy metrics are not overlooked.