Bias in AI systems: where it comes from and how it is measured
Bias in AI systems stems from skewed training data, flawed algorithms, and human biases embedded during development. Measuring bias involves auditing data and predictions with statistical fairness criteria. Standards bodies provide frameworks for identifying and managing bias across the AI lifecycle.
Sources of bias
According to the NIST publication on identifying and managing bias, bias is pervasive across AI technology processes and can cause harm even when there is no intention to bias the system. NIST's standard on AI bias
The arXiv survey on bias and fairness lists various sources of bias, such as historical bias, representation bias, and measurement bias, which can affect AI applications. the arXiv survey on machine learning
Types of bias
IBM describes several common types of AI bias, including measurement bias, sample or selection bias, confirmation bias, and stereotyping bias, each of which can distort outputs and cause harm. IBM's article on AI bias
Other types include exclusion bias, when important data is left out, and recall bias, which occurs during data labeling when labels are inconsistently applied. These biases often interact and compound each other.
Measuring bias
Google's fairness module advises that before putting a model into production, it is critical to audit training data and evaluate predictions for bias. Google's fairness module
The Gender Shades study measured error rates in commercial gender classification systems and found that darker-skinned females were misclassified with error rates up to 34.7%, while lighter-skinned males had a maximum error rate of 0.8%. the Gender Shades study
Standards and governance
The textbook Fairness and Machine Learning discusses how United States anti-discrimination law applies to automated decision making and how discrimination tests can be applied to algorithmic systems. the FairMLBook
Organizations can adopt governance policies that direct, manage, and monitor AI activities to ensure responsible development and use, balancing benefits for businesses, customers, and employees.
| Bias type | Description | Example |
|---|---|---|
| Measurement bias | Incomplete data fails to include the whole population | A college model for graduation includes only graduates, missing dropouts |
| Sample or selection bias | Training data is not representative or large enough | All school teachers consulted have the same qualifications, so model requires identical qualifications |
| Confirmation bias | Model relies too much on pre-existing beliefs | A hiring model favors candidates similar to past hires |
| Stereotyping bias | System reinforces harmful stereotypes | A translation system associates genders with certain professions |
Frequently asked questions
What causes bias in AI systems?
Bias can come from skewed training data, biased algorithms, or human biases embedded during development, such as measurement bias or sample bias.
How is AI bias measured?
AI bias is measured by auditing training data and evaluating model predictions across demographic groups using fairness metrics like demographic parity or equalized odds.
What is an example of AI bias?
The Gender Shades study found that commercial facial analysis systems misclassified darker-skinned women at rates up to 34.7%, while lighter-skinned men had error rates below 1%.
Are there standards for managing AI bias?
NIST has published a special publication titled Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, which addresses bias across the AI lifecycle.