This article is an excerpt from the Shortform book guide to "Weapons of Math Destruction" by Cathy O'Neil. Shortform has the world's best summaries and analyses of books you should be reading.
Like this article? Sign up for a free trial here.
What are the dangerous effects of statistics? How do statistics perpetuate social bias?
Believe it or not, dangerous math models can have a grave impact on society. According to Cathy O’Neil, dangerous statistics disproportionately harm poor people, reproduce social bias, and make harmful self-fulfilling prophecies.
Learn more about how statistics can negatively affect people and society when used incorrectly.
Dangerous Models Disproportionately Inflict Harm on the Poor
O’Neil argues that one of the negative effects of statistics is that dangerous mathematical models tend to harm poor people while offering preferential treatment to rich people. It’s more efficient and cost-effective for institutions to automate the bulk of their interactions, so they use mathematical models to determine college admissions, sort through job applications, evaluate job performance, and calculate credit scores—all processes that favor wealthy people’s backgrounds and circumstances.
The result is that poor people often get poorer from their interactions with dangerous mathematical models, while rich people largely benefit from their interactions with the same systems. Over time, this leads to increased income disparity and social inequality.
For example, algorithms that calculate credit scores are designed to predict a borrower’s ability to repay loans based on factors like loan repayment history, current debt levels, and income. Poor people, often with low scores due to unstable incomes, little credit history, or small financial missteps (such as late bill payments, which are common among those struggling financially), face higher loan interest rates or denial of credit, amplifying their financial struggles. Conversely, wealthier individuals often have long credit histories and enough resources for on-time payments, leading to high credit scores.
Dangerous Models Reproduce Social Bias by Lumping People Into Groups
According to O’Neil, dangerous mathematical models tend to mete out identical judgments to large groups of individuals who share characteristics such as location, income level, race, and gender. This often results in the replication of human biases, as harsh judgments are meted out to people from a certain background or geographical area.
Dangerous Models Reach Biased Conclusions Based on Stand-In Variables
One way that dangerous models reproduce social bias is by using stand-in variables to make decisions. Stand-in variables (called proxies) are sets of data used to approximate another set of data that may be more difficult to obtain.
For example, suppose you’re feeling a bit nosy and want to estimate your neighbor’s annual income. Feeling too sheepish to ask them outright how much they make, you might make a judgment based on stand-in variables that are related to income, such as the model of their car or the size of their house.
(Shortform note: As mathematical models reflect those who create them, the proxies may come from the stereotypes that creators have in their heads. This use of stereotypes is what economist Daniel Kahneman calls the representativeness heuristic—a mental shortcut wherein we estimate the likelihood of an event by comparing it to a prototype. In Thinking, Fast and Slow, Kahneman writes that we tend to use representativeness to make sense of the world, even if it means interpreting data to fit the narrative. The way to overcome this is to update beliefs and probabilities as new information comes in. In the example of your neighbor, you might learn that they work as a Google engineer, which gives you a better idea of their income.)
According to O’Neil, stand-in variables are dangerous because they sometimes correlate with race, gender, income, and other social factors. When mathematical models use such stand-in variables to make decisions at scale, it can lead the model to lump people into groups and enact bias based on these factors.
For example, a mathematical model might use your ZIP code to determine whether you’re a good candidate for a loan. By looking at the financial history of your neighbors, the model will make a judgment about how you’re likely to behave. If you come from a low-income neighborhood, such models may judge you to be a liability through no fault of your own.
(Shortform note: Using ZIP codes as proxies imports decades of racially biased housing policies into predictive models. In The Color of Law, Richard Rothstein explains how the government segregated cities by constructing segregated public housing beginning World War I, enacting zoning laws that restricted African Americans’ property ownership in certain areas, and denying financial support to Black families attempting to live outside African American neighborhoods.)
Dangerous Models Make Self-Fulfilling Prophecies
O’Neil asserts that the negative judgments made by dangerous mathematical models often turn into self-fulfilling prophecies. This is especially true of mathematical models that are used to predict your financial behavior. If mathematical models deem you to be financially unreliable, you may be unable to get a mortgage, a credit card, a car, or an apartment. These financial consequences make it difficult for you to improve your financial situation, leading to a vicious cycle, in which it’s impossible to overcome the negative judgment of a dangerous model.
(Shortform note: What can you do if you find yourself the victim of mathematical models predicting your financial behavior? Some companies now offer options, such as secured credit cards, to people in this situation. With a secured credit card, you pay a security deposit up front, then borrow against that deposit rather than against a traditional line of credit. By essentially taking out micro-loans from yourself and paying them off, you prove your creditworthiness and build your credit score. This is one way to bypass the vicious cycle of unfair financial models.)
———End of Preview———
Like what you just read? Read the rest of the world's best book summary and analysis of Cathy O'Neil's "Weapons of Math Destruction" at Shortform.
Here's what you'll find in our full Weapons of Math Destruction summary:
- The insidious ways mathematical models are being used in society
- How mathematical models are being used to reinforce bias and harm the poor
- Recommendations for how to rein in dangerous mathematical models
