AI Bias and Discrimination: How Algorithms Fail Us

In 2014, Amazon began building an experimental tool that would rank job applicants from one to five stars. It was trained on about ten years of résumés the company had received, and because the technology industry has long been male-dominated, most of those résumés came from men. By 2015 the team had found that the system preferred male candidates for technical roles. It penalized résumés containing the word "women's," and it marked down graduates of certain all-women's colleges, according to reporting summarized by MIT Technology Review in October 2018.

Engineers edited the system to treat those terms neutrally, but they lost confidence that it was not finding other routes to the same result, and the project was abandoned. Nothing in the software was broken. It had learned, accurately, what the past looked like, and the past was the problem. That pattern, a system faithfully reproducing an unequal history, sits behind most of the cases in this article.

Key takeaways

• AI bias usually starts in training data, in the labels, or in a poorly chosen proxy for the thing the system is meant to measure.
• Removing a protected attribute such as race from the inputs does not remove bias, because other variables can stand in for it.
• Several common definitions of fairness cannot all be satisfied at once when groups have different base rates, a result proved mathematically in 2016 and 2017.
• Documented harms include wrongly flagged defendants, underserved patients, higher error rates in face analysis, and a wrongful arrest.
• Law is catching up unevenly: the EU AI Act's high-risk rules for hiring and credit now apply from December 2027, and New York City has required bias audits of hiring tools since July 2023.

What "AI bias" actually means

In everyday speech, bias means prejudice. In machine learning it has a looser meaning: a systematic pattern in a system's outputs that disadvantages some group in a way that is unrelated to the task, or that cannot be justified by it. A model does not need any intent to be biased, and nobody has to write a discriminatory rule. A statistical system that learns patterns from data can pick up unfair patterns as easily as fair ones.

The U.S. National Institute of Standards and Technology (NIST) makes the same point in its 2022 publication on the subject, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence. Its abstract says bias remains endemic across technology processes and can cause harm regardless of intent, and that even unintentional harm makes it harder to build public trust in AI.

Think of a model as a very diligent student who has only read one textbook. If the textbook has a skewed view of the world, the student will pass every exam based on it and still give wrong answers about everything it left out. The student is not lying. The student is limited by the source.

Where bias enters the machine

Researchers often sort sources of bias into a handful of types. The labels vary by author, but the mechanics are consistent.

Historical bias happens when the data accurately reflects an unfair world. Amazon's résumés are the example. The data was a true record of who had been hired, and using it as a template for who should be hired next copied the imbalance forward.

Representation bias happens when some groups are missing or thin in the data. The 2018 paper Gender Shades, by Joy Buolamwini and Timnit Gebru, looked at the benchmark datasets commonly used to test face analysis. It found that the IJB-A benchmark was 79.6 percent lighter-skinned subjects and the Adience benchmark was 86.2 percent. A system can score well on such a benchmark while having seen few darker-skinned faces, and the benchmark would never reveal the weakness.

Measurement bias happens when the thing that is measured is a poor stand-in for the thing that matters. A recidivism model that uses arrests as a measure of crime is measuring policing as much as behavior, because arrests depend on where officers look. A health model that uses spending as a measure of illness, discussed below, makes the same mistake.

Feedback loops appear when a model's decisions change the data it will be trained on next. The following is an illustration, not a documented case. Suppose a lender's model rates applicants from certain postal codes as risky and declines them. Those residents then build less credit history and fewer assets, so in the next training cycle they look even riskier. The model's earlier output becomes its next input, and a small initial skew compounds.

Proxies: why removing race does not remove racial bias

A common reaction to these cases is to delete the sensitive column, so that the model never sees race or gender. This rarely works. Postal code, school name, shopping habits, the vocabulary used in a résumé, and the hours someone works can each correlate with a protected characteristic. A model that is rewarded for prediction will use any variable that helps, including ones that carry that signal in disguise. The Amazon tool is a small example: it did not need a "gender" field to learn that the word "women's" was associated with candidates who had not been hired.

The same dynamic is visible in language. A 2017 paper by Aylin Caliskan, Joanna Bryson, and Arvind Narayanan, available on arXiv, applied a standard word-embedding model (GloVe) trained on web text and found that it reproduced human biases measured by the Implicit Association Test and other psychology studies. Some of the associations were harmless, such as insects with flowers. Others involved race and gender. The authors argued that exposure to everyday language may be enough to produce such biases in a learning system.

The myth of the neutral algorithm

There is a persistent belief that replacing human judgment with a computer removes prejudice, since the machine has no feelings about anyone. The belief confuses consistency with fairness. An algorithm applies the same rule to everyone, but if the rule is skewed, it applies the skew to everyone, instantly, and at volume.

A hypothetical shows the scale effect. Suppose a human loan officer unconsciously turns down a slightly higher share of qualified applicants from one group. A model trained on that officer's decisions would reproduce the pattern in every application it sees, automatically and without the occasional inconsistency or second thought that a human might have. This is a thought experiment, not a measurement, but it captures why researchers worry about automation of judgments that were already flawed.

Every algorithm also embeds choices. Someone decided what to optimize, which data to include, how to label outcomes, and where to set the threshold between "approve" and "reject." Those are value judgments, even when they are presented as arithmetic. A study of an online ad system shows how this plays out in ordinary products. In a 2013 paper, Latanya Sweeney examined ads served for searches of 2,184 racially associated names and found statistically significant discrimination in ad delivery: names given mostly to Black babies produced arrest-suggestive ad text in 81 to 86 percent of searches on one site and 92 to 95 percent on the other, against 23 to 29 percent and 0 to 60 percent for names given mostly to white babies. The ads appeared whether or not the named person had an arrest record in the company's database.

Real-world harm across domains

Criminal justice: COMPAS and two kinds of fairness

In 2016, ProPublica published an analysis of COMPAS, a risk-assessment tool made by Northpointe and used in U.S. courts. Looking at more than 7,000 people arrested in Broward County, Florida, in 2013 and 2014, the investigation reported that Black defendants who did not go on to reoffend were nearly twice as likely as white defendants to have been labeled higher risk (44.9 percent against 23.5 percent). White defendants who did reoffend were more often labeled lower risk (47.7 percent against 28.0 percent). ProPublica said the gap persisted after accounting for prior crimes, age, and gender.

Northpointe disputed the analysis and, according to ProPublica, does not publicly disclose its calculations. Race itself was not one of the 137 survey questions behind the score, but the article notes that factors correlated with race, such as poverty and joblessness, may be included.

The dispute was later recognized as a clash between two reasonable definitions of fairness. One asks whether error rates are equal across groups, which is the test ProPublica applied. The other asks whether a given score means the same thing for every group. Mathematics shows that those definitions pull against each other.

Why you cannot have every kind of fairness

Two papers, by Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan and by Alexandra Chouldechova, both published in 2016 and 2017, proved the conflict formally. Kleinberg and colleagues showed that three reasonable fairness conditions (calibration, and balance for the positive class and for the negative class) cannot hold at once except in highly constrained special cases. Chouldechova reached a matching result for recidivism prediction: the criteria "cannot all be simultaneously satisfied when recidivism prevalence differs across groups."

A worked example with invented numbers makes the logic concrete. Suppose a tool is equally precise for two groups: when it flags someone, it is right 70 percent of the time. Suppose it also catches 70 percent of the people who really will reoffend, in both groups. Now let the true rates differ. In Group A, 50 percent will reoffend, and in Group B, 20 percent will. Applying those rates to 1,000 people in each group gives the following (the arithmetic is ours, not a published case):

	Group A (50% reoffend)	Group B (20% reoffend)
People who will reoffend	500	200
Correctly flagged (70% caught)	350	140
Wrongly flagged to reach 70% precision	150	60
People who will not reoffend	500	800
False positive rate	30%	7.5%

The tool meets two fairness tests at once: its flags are equally reliable, and it catches the same share of reoffenders in each group. Yet a person in Group A who will not reoffend is four times as likely to be wrongly flagged as an equivalent person in Group B. Equalize the false positive rates instead, and the flags can no longer be equally reliable. No clever training fixes this, because the conflict lives in the base rates, not in the code. A society has to decide which kind of error it will tolerate, and that is a values decision.

Health care: spending is not sickness

In 2019, Ziad Obermeyer and colleagues published a study in Science of a commercial algorithm that health systems used to decide which patients should receive extra care. According to the study's abstract, at the same risk score, Black patients were considerably sicker than white patients. The cause was the target the model was trained to predict: future health care costs, not illness. Because Black patients have less access to care and less money is spent on them, the algorithm read lower spending as better health. The authors estimated that correcting the bias would raise the share of Black patients flagged for extra help from 17.7 percent to 46.5 percent.

The case is a clean example of measurement bias. The model was accurate at predicting spending. It was spending that was a poor stand-in for need. For the wider picture of how algorithms are used in medicine, see our overview of AI in healthcare.

Lending: the same pattern in mortgage approvals

In 2021, The Markup analyzed 2019 mortgage data and found that, after accounting for 17 financial factors, lenders were more likely to deny applicants of color than similar white applicants: about 40 percent more likely for Latino applicants, 50 percent for Asian and Pacific Islander applicants, 70 percent for Native American applicants, and 80 percent for Black applicants. Black applicants earning $100,000 or more with the same debt ratios were denied more often than white applicants with more debt.

Two caveats matter. The Markup says Fannie Mae and Freddie Mac require lenders to use Classic FICO, a credit model built on 1990s data, and that their underwriting software is proprietary, so applicants and regulators cannot see how decisions are made. Industry groups criticized the analysis for leaving out credit scores and government-backed loans, though they did not identify errors in the calculations. A gap in outcomes is evidence that something is wrong, but, as our piece on correlation and causation explains, the data alone does not show where in the process it arises. For a closer look at credit, see our explainer on AI loan approval and lending discrimination.

Face recognition: unequal error rates and a wrongful arrest

Gender Shades evaluated three commercial gender classification systems on a new dataset balanced by gender and skin type. Darker-skinned females were the most misclassified group, with error rates of up to 34.7 percent, while the maximum error rate for lighter-skinned males was 0.8 percent. That study tested gender classification, which is a different task from matching a face to a person, so it should not be read as a measure of every face recognition product.

For matching, NIST ran a large evaluation in 2019. According to NIST's summary, it covered 189 algorithms from 99 developers and 18.27 million images of 8.49 million people. For one-to-one matching, Asian and African American faces had higher false positive rates than Caucasian faces, often by a factor of 10 to 100 depending on the algorithm. For one-to-many search, false positives were higher for African American women. NIST also found wide variation between algorithms, and the most equitable algorithms also tended to be among the most accurate, which means the problem is not a fixed law of the technology.

A false match has human consequences. In January 2020, Detroit police arrested Robert Williams outside his home. According to the ACLU's account of the case, he was held for about thirty hours, in what the ACLU describes as the first publicly reported wrongful arrest based on a false face recognition match. A blurry surveillance still had been run through a face recognition search, which returned his expired driver's license photo, and police then showed his photo in a lineup to a witness who had not been at the store. In a settlement on June 28, 2024, Detroit agreed that police must back up face recognition results with independent, reliable evidence before making an arrest, train officers on the technology's dangers, and audit cases since 2017 in which it was used to obtain an arrest warrant.

Hiring: from Amazon to age filters

Hiring is the domain where the failure is easiest to picture, because the algorithm is often a filter that decides who gets seen at all. In addition to the Amazon case, the U.S. Equal Employment Opportunity Commission (EEOC) announced a settlement in which iTutorGroup agreed to pay $365,000. The EEOC alleged that the company's application software automatically rejected women aged 55 and older and men aged 60 and older, affecting more than 200 U.S.-based applicants. That case involved a rule that was explicit rather than learned, a reminder that "algorithmic discrimination" covers simple code as well as machine learning.

What researchers are doing about it

Technical fixes fall into a few families, and each has a cost.

• Better data. Collecting more examples from underrepresented groups, or reweighting what exists, addresses representation bias. It does not repair labels that encode a past judgment.
• Better targets. The health case shows the value of asking what a model is really predicting. Changing the target from cost to a direct measure of illness is a design choice, not a statistical trick.
• Fairness constraints. Moritz Hardt, Eric Price, and Nathan Srebro proposed in 2016 a criterion called equality of opportunity and showed how to adjust a trained predictor to satisfy it after the fact. Their framing shifts the cost of poor classification from disadvantaged groups to the decision maker, who then has an incentive to improve accuracy. As the worked example above shows, a constraint always trades against another definition of fairness.
• Audits and monitoring. Testing a deployed system against real outcomes, broken down by group, catches problems that benchmarks hide. This is a close cousin of model monitoring in production.
• Interpretability. Tools that show why a model made a decision help reviewers spot a bad proxy, though they do not stop harm on their own. Our article on mechanistic interpretability describes the research frontier.

None of these is complete. Adding diverse data helps, but it does not erase biased labels. A fairness constraint works only if the chosen definition matches the harm, and the COMPAS dispute shows there is no definition everyone accepts. Interpretability aids oversight but cannot, by itself, prevent a harmful deployment.

A related practical point concerns evaluation. A model that scores well overall can fail badly on a subgroup, in the same way that a high accuracy number can hide a failure on a rare class, a problem covered in our article on precision and recall under class imbalance. The solution is to report performance for each group, not only on average.

Law and regulation: a patchwork

European Union. The AI Act entered into force on August 1, 2024. According to the European Commission's summary, it classifies uses such as recruitment, worker management, and credit scoring as high-risk, and it requires high-quality training datasets to reduce the risk of discriminatory outcomes, along with logging, documentation, and human oversight. The timetable has moved. The Commission page reports that the AI Omnibus, which entered into force on July 27, 2026, pushed the high-risk rules for these uses to December 2, 2027. Readers should confirm current dates before relying on them, since the schedule has already changed once.

New York City. Under Local Law 144, according to the city, employers and employment agencies may not use an automated employment decision tool unless it has undergone a bias audit within the previous year, information about the audit is publicly available, and candidates have been given notice. Enforcement began on July 5, 2023. The law requires audits, not outcomes: it does not prohibit a tool that fails one.

United States, federal level. The sources used for this article do not point to a comprehensive federal statute written for algorithmic discrimination. Existing civil rights laws are what regulators have invoked: the iTutorGroup settlement, for example, rested on the Age Discrimination in Employment Act.

The deeper problem: accountability and consent

Technical and legal fixes are necessary but not sufficient, because several structural features make harm easy to produce and hard to attribute.

Diffuse responsibility. When an algorithm discriminates, who is at fault? It might be the team that picked the training data, the vendor that built the model, the company that deployed it, or the manager who followed its output. This is sometimes described as an accountability gap, and it makes it easy for everyone to point somewhere else.

Invisibility. People affected by an algorithmic decision often do not know that a machine was involved, what data it used, or how it was tested. The Markup's mortgage analysis shows what this looks like in practice: proprietary software, a legacy credit model, and no way for an applicant to see why a loan was denied.

Incentives. Organizations that profit from automated decisions have limited reasons to look hard for bias in them. Independent testing, such as NIST's evaluation or ProPublica's investigation, has been decisive in the cases above precisely because it did not come from the builders.

AI bias is a social problem that technology has made faster and harder to see. Systems are now judged by whether they reproduce the world as it was, when the question that matters is whether they should.

Common misconceptions

"Bias is a data problem, so more data fixes it." More data helps with representation, but not with a flawed target such as spending in the health algorithm, or with labels that record past discrimination.

"If the model is accurate, it is fair." The COMPAS dispute showed that a model can be equally accurate across groups and still produce very different error rates. Which error matters more is a value judgment.

"Leaving out race makes the model colorblind." Proxy variables carry the signal. The lending, hiring, and language examples above all involve systems that never had to see a protected attribute to reproduce its effects.

"Bias only matters in high-stakes systems." The Sweeney study concerned ad text shown to people searching for names, and the Caliskan study concerned word associations. Lower-stakes systems can still shape what people see and believe, and they often feed larger ones. Related questions arise in how large language models are trained, where choices about data and feedback shape what the finished system treats as normal.

Frequently asked questions

Q: What is AI bias? A: AI bias is a systematic pattern in an AI system's outputs that disadvantages certain groups in ways unrelated to the task. It usually arises from training data that reflects historical inequality, from labels or proxies that measure the wrong thing, or from groups being underrepresented in the data.

Q: Can an AI system be biased without using race or gender as inputs? A: Yes. Variables such as postal code, school, spending, or vocabulary can correlate with protected characteristics. The 2019 health algorithm study found racial bias in a model that predicted health care costs, and the Amazon hiring tool learned to penalize the word "women's."

Q: Why can't we make an algorithm that is fair by every definition? A: Mathematical results from 2016 and 2017 show that popular fairness criteria, such as calibration and equal error rates, cannot all hold when groups have different base rates, except in special cases. Choosing a definition means choosing which errors to accept, which is a decision about values.

Q: Which laws address algorithmic discrimination? A: Existing anti-discrimination laws apply to automated decisions, as an EEOC settlement over an age-filtering hiring tool shows. New York City requires bias audits of hiring tools, and the EU AI Act treats hiring and credit scoring as high-risk uses, with those rules now scheduled to apply from December 2, 2027.

Q: Does face recognition work equally well for everyone? A: Not always. NIST's 2019 evaluation of 189 algorithms found false positive rates for Asian and African American faces that were often 10 to 100 times higher than for Caucasian faces in one-to-one matching, though results varied widely by algorithm and the most equitable ones were also among the most accurate.

Q: Who is responsible when an algorithm discriminates? A: The answer is unsettled. Responsibility can lie with the developer, the organization that deployed the system, or both, and the split between data, model, and deployment choices makes attribution hard. That is why many researchers argue for independent audits and transparency requirements.

The takeaway

Algorithmic bias is rarely a bug that can be patched. It is what happens when a system learns from an unequal record, measures the wrong thing, or optimizes one definition of fairness at the expense of another. The best evidence of where it occurs has come from outsiders, among them journalists, academic researchers, and government labs, who tested systems the builders had not. The practical lessons are to ask what a model is really predicting, to check performance by group, to expect trade-offs between fairness definitions, and to decide in advance who answers for the result. The technology can be improved, as NIST's finding that the most equitable face algorithms were also among the most accurate suggests, but only if someone is looking.

Sources

• Buolamwini, J., and Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of Machine Learning Research 81.
• Obermeyer, Z., Powers, B., Vogeli, C., and Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science 366. Europe PMC record.
• ProPublica (2016). Machine Bias.
• Kleinberg, J., Mullainathan, S., and Raghavan, M. (2016). Inherent Trade-Offs in the Fair Determination of Risk Scores.
• Chouldechova, A. (2017). Fair prediction with disparate impact.
• Hardt, M., Price, E., and Srebro, N. (2016). Equality of Opportunity in Supervised Learning.
• Caliskan, A., Bryson, J. J., and Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases.
• Sweeney, L. (2013). Discrimination in Online Ad Delivery.
• NIST (2019). NIST Study Evaluates Effects of Race, Age, Sex on Face Recognition Software.
• NIST (2022). Towards a Standard for Identifying and Managing Bias in Artificial Intelligence.
• The Markup (2021). The Secret Bias Hidden in Mortgage-Approval Algorithms.
• ACLU. Williams v. City of Detroit.
• U.S. EEOC (2023). iTutorGroup to Pay $365,000 to Settle EEOC Discriminatory Hiring Suit.
• European Commission. AI Act regulatory framework.
• NYC Department of Consumer and Worker Protection. Automated Employment Decision Tools.
• MIT Technology Review (2018). Amazon ditched AI recruitment software because it was biased against women.

---

Read next:

• AI Ethics: The Key Debates Shaping How AI Gets Built
• Machine Learning Bias and Fairness: Ensuring Equitable AI
• AI Safety and Alignment Explained