Why This Chapter Matters
In 2023, Detroit police used facial recognition technology to falsely identify Portia Woodruff as a carjacking suspect. She was arrested while eight months pregnant, for a crime she did not commit. When her lawyers asked why the model had identified her, the answer was: they couldn't say. The model had simply produced an output. The team that deployed it could not explain why.
A recent study by Sorokovikova, et al., showed that all tested LLMs exhibited perceived gender bias when providing recommendations for salary negotiation. The only difference in the prompt was the name on the provided resume -- either a female presenting name or a male presenting name!
Recruitment, finance, and healthcare are real high-stakes domains actively using AI tools. The patterns these examples describe are not fictional. They have happened. They will happen again. Building AI tools means accepting some responsibility for outcomes like these — and this chapter is about giving you the vocabulary and frameworks to handle that responsibility.
A facial recognition system misidentifies a person, leading to their wrongful arrest. The engineering team says 'the model just did what it was trained to do.' Why is this insufficient as a response?