Six Sources of Bias

Bias enters AI systems at every stage of the lifecycle — not just during training, and not just through bad intentions. Understanding where each type of bias enters is the first step toward preventing it.

The lifecycle: Data Collection → Labels & Features → Training & Evaluation → Deployment → Feedback Loop

1. Historical Bias — Data Collection. Collected data reflects existing biases in the world. The original Word2Vec models, trained on text from a particular era, embedded gender associations that mirrored those of the source corpus. "Engineer" pulls male. "Nurse" pulls female. You're not seeing a flaw in the algorithm — you're seeing the algorithm faithfully reproducing what was in the data, which faithfully reflected the world at the time. The data was the problem, not the math.

2. Representation Bias — Data Collection. The training set is not representative of the target population. Most medical datasets contain very few pregnant women — pregnancy is often an exclusion criterion in studies because of risk to the fetus. So when you're pregnant and need a medication, doctors are often working without trial data, just clinical experience. The people who most need the research are systematically absent from it.

3. Measurement Bias — Labels and Features. The variable you're measuring is a poor proxy for the thing you actually care about. GPA is often used as a proxy for student learning success. GPA is not a great proxy — plenty of people with lower GPAs go on to successful careers, and vice versa. The proxy is convenient and quantitative, but it doesn't measure what we say it measures.

4. Learning Bias — Training and Evaluation. Modeling choices amplify performance disparities across groups. Cost functions can optimize aggregate performance at the expense of consistency across subgroups. Compact models — optimized for deployment efficiency — can be pruned in ways that remove features that matter for minority subgroups, because those features don't help aggregate metrics much.

5. Deployment Bias — Deployment. Mismatch between how a tool was intended to be used and how it's actually used. An automated teacher evaluation tool, intended to assist administrators, gets used to terminate teachers — including teachers who grade strictly but teach effectively, who then disappear from the system and stop contributing to future model training.

6. Feedback Loop Bias — Feedback. The system's outputs influence its future inputs. A product recommendation engine that orders items by number of positive reviews makes those items more visible, leading to more reviews, reinforcing the ordering. After enough cycles, the system is mostly recommending what it has always recommended. The rich get richer; the novel and underrepresented get buried.