PDF Summary:Weapons of Math Destruction, by Cathy O'Neil

In Weapons of Math Destruction, data scientist Cathy O’Neil details the insidious ways mathematical models are being used to determine everything from interest rates to prison sentences. She contends that while mathematical models—simulations of real-world events—can efficiently sort through vast amounts of information, they can produce dangerous results when used broadly, opaquely, and without incorporating feedback. Specifically, O’Neil argues that dangerous mathematical models treat poor people worse than rich people, reinforce bias by lumping people into groups, and produce judgments that become self-fulfilling prophecies.

In our guide, we’ll explore the difference between benign mathematical models and dangerous ones and describe the negative effects that dangerous mathematical models produce. Then, we’ll consider O’Neil’s recommendations for how to rein in dangerous mathematical models. We’ll also discuss other research and perspectives on data science and digital ethics.

(continued)...

By contrast, dangerous mathematical models are deployed at massive scale, often far beyond the contexts they were originally designed for. O’Neil says that when used in such broad contexts, even low rates of inaccuracy cause harm to many people, as even small fractions of massive numbers often represent sizable groups of people.

For example, imagine that researchers discover a slight correlation between favorite color and credit—people who prefer blue are found to be slightly more likely to pay their bills on time than those who prefer red. In response, banks start using the favorite color model to set interest rates for mortgages. Nationwide, people start paying different amounts for the same services, all because their favorite color supposedly corresponds to their reliability. As a result of being deployed at scale, far beyond its intended context, the favorite color model becomes dangerous.

Making Mathematical Models Antifragile

Mathematical models that become harmful when deployed at scale are what Nicholas Nassim Taleb might call fragile systems—predictive models that fall apart under big changes such as scaling. In Antifragile, he writes that one way to minimize the harmful effects of models used at a massive scale is to perform an acceleration of harm test: Take the model and ask, “What if it’s wrong?” Change key assumptions incrementally and assess how it affects the results. If negative changes outpace positive ones, then the model ideally shouldn’t be used at scale. In other words, stress-test a model by pushing its assumptions to extreme or unexpected scenarios in order to see if the model breaks down or leads to harmful outcomes.

For example, if a bank were to use mathematical models for loan approvals, it could test three assumptions: 1) the borrower’s income remains stable over time, 2) the economy remains stable and unemployment rates remain low, 3) credit scores accurately reflect a borrower’s creditworthiness. The bank would take each of these assumptions and ask “What if it’s wrong?” (For instance, what if the borrower’s income doesn’t remain stable over time?) By stress-testing each of these assumptions, the bank would be able to identify the vulnerabilities and limitations of the model and determine whether it’s a good model to use at scale.

Dangerous Models Produce Dangerous Effects

Now that you understand the differences between good mathematical models and dangerous ones, let’s consider the impact dangerous models have on society. According to O’Neil, dangerous models disproportionately harm poor people, reproduce social bias, and make harmful self-fulfilling prophecies.

Dangerous Models Disproportionately Inflict Harm on the Poor

O’Neil argues that dangerous mathematical models tend to harm poor people while offering preferential treatment to rich people. It’s more efficient and cost-effective for institutions to automate the bulk of their interactions, so they use mathematical models to determine college admissions, sort through job applications, evaluate job performance, and calculate credit scores—all processes that favor wealthy people’s backgrounds and circumstances.

The result is that poor people often get poorer from their interactions with dangerous mathematical models, while rich people largely benefit from their interactions with the same systems. Over time, this leads to increased income disparity and social inequality.

(Shortform note: While O’Neil argues that dangerous mathematical models disproportionately harm the poor, Matthew Desmond argues that we all play a part in perpetuating poverty. In Poverty, by America, Desmond argues that poverty in America persists because it benefits many Americans to keep some of their fellow Americans poor: As consumers, we demand cheap goods and expect quick service and delivery of those goods, we buy stock in companies that exploit their workers, and we support housing rules that trap people in poor communities and, consequently, generational poverty.)

For example, algorithms that calculate credit scores are designed to predict a borrower’s ability to repay loans based on factors like loan repayment history, current debt levels, and income. Poor people, often with low scores due to unstable incomes, little credit history, or small financial missteps (such as late bill payments, which are common among those struggling financially), face higher loan interest rates or denial of credit, amplifying their financial struggles. Conversely, wealthier individuals often have long credit histories and enough resources for on-time payments, leading to high credit scores.

US Credit Scores Perpetuate Racial Inequality

Credit scores were created to prevent the sort of harm that O’Neil writes about—in theory, using mathematical models to crunch cold, hard numbers rather than having humans evaluate who’s creditworthy is a more objective approach. However, the current system—which gives people a score ranging from 300-850 (with numbers above 700 generally considered good)—has been shaped by long-standing patterns of systemic bias and punishes people of color.

The credit scoring system awards more points for things like home ownership and credit history—things that more often tick the boxes of wealthier, white individuals. For instance, the homeownership rate for white Americans is at 71.7% versus the homeownership rate for Black Americans at just 47%. This means people of color have lower credit scores, making it harder for them to obtain credit to buy homes and build a credit history to begin with. One study found that Black and brown borrowers were given higher interest rates for loans and that the denial rate for Black mortgage applicants in 2022 was 84% higher than that of white borrowers.

Experts recommend a couple of ways to improve the credit scoring system. First is to factor rent payments into credit histories—this could boost renters’ credit scores by nearly 60 points. Another way is to consider cash flow (how money goes in and out of someone’s bank account during a specific period) to get a clearer picture of someone’s financial standing. This would be especially helpful to young people and immigrants who are just starting to build their credit histories.

Dangerous Models Reproduce Social Bias by Lumping People Into Groups

According to O’Neil, dangerous mathematical models tend to mete out identical judgments to large groups of individuals who share characteristics such as location, income level, race, and gender. This often results in the replication of human biases, as harsh judgments are meted out to people from a certain background or geographical area.

Dangerous Models Reach Biased Conclusions Based on Stand-In Variables

One way that dangerous models reproduce social bias is by using stand-in variables to make decisions. Stand-in variables (called proxies) are sets of data used to approximate another set of data that may be more difficult to obtain.

For example, suppose you’re feeling a bit nosy and want to estimate your neighbor’s annual income. Feeling too sheepish to ask them outright how much they make, you might make a judgment based on stand-in variables that are related to income, such as the model of their car or the size of their house.

(Shortform note: As mathematical models reflect those who create them, the proxies may come from the stereotypes that creators have in their heads. This use of stereotypes is what economist Daniel Kahneman calls the representativeness heuristic—a mental shortcut wherein we estimate the likelihood of an event by comparing it to a prototype. In Thinking, Fast and Slow, Kahneman writes that we tend to use representativeness to make sense of the world, even if it means interpreting data to fit the narrative. The way to overcome this is to update beliefs and probabilities as new information comes in. In the example of your neighbor, you might learn that they work as a Google engineer, which gives you a better idea of their income.)

According to O’Neil, stand-in variables are dangerous because they sometimes correlate with race, gender, income, and other social factors. When mathematical models use such stand-in variables to make decisions at scale, it can lead the model to lump people into groups and enact bias based on these factors.

For example, a mathematical model might use your ZIP code to determine whether you’re a good candidate for a loan. By looking at the financial history of your neighbors, the model will make a judgment about how you’re likely to behave. If you come from a low-income neighborhood, such models may judge you to be a liability through no fault of your own.

(Shortform note: Using ZIP codes as proxies imports decades of racially biased housing policies into predictive models. In The Color of Law, Richard Rothstein explains how the government segregated cities by constructing segregated public housing beginning World War I, enacting zoning laws that restricted African Americans’ property ownership in certain areas, and denying financial support to Black families attempting to live outside African American neighborhoods.)

Dangerous Models Make Self-Fulfilling Prophecies

O’Neil asserts that the negative judgments made by dangerous mathematical models often turn into self-fulfilling prophecies. This is especially true of mathematical models that are used to predict your financial behavior. If mathematical models deem you to be financially unreliable, you may be unable to get a mortgage, a credit card, a car, or an apartment. These financial consequences make it difficult for you to improve your financial situation, leading to a vicious cycle, in which it’s impossible to overcome the negative judgment of a dangerous model.

(Shortform note: What can you do if you find yourself the victim of mathematical models predicting your financial behavior? Some companies now offer options, such as secured credit cards, to people in this situation. With a secured credit card, you pay a security deposit up front, then borrow against that deposit rather than against a traditional line of credit. By essentially taking out micro-loans from yourself and paying them off, you prove your creditworthiness and build your credit score. This is one way to bypass the vicious cycle of unfair financial models.)

How to Defuse Dangerous Models

In response to these negative effects, O’Neil proposes strategies industries and governments can take to limit the harm caused by dangerous mathematical models. O’Neil recommends monitoring and measuring the impact of mathematical models, regulating industry use of mathematical models, and setting more positive goals for mathematical models as opposed to targeting profitability above all else.

Measure Impact

O’Neil argues that you need to measure the effects of dangerous mathematical models before you can mitigate them. As we’ve learned, good mathematical models incorporate feedback, sharpening their algorithms based on previous results. Measuring the impact of dangerous models creates feedback that can be used to improve those models.

When measuring a mathematical model's impact, consider whether the model’s judgments are fair. Ask whether the model treats all individuals equitably or whether it produces judgments that either favor or punish some groups or individuals unfairly.

(Shortform note: To determine whether a mathematical model’s judgments are fair, we can take a page from best practices in machine learning, the branch of artificial intelligence that trains computers to mimic intelligent human behavior. These practices include having a third party conduct regular audits of input data and output decisions, and having collaborative work teams to address potential blind spots and have a more holistic view of a mathematical model’s impact.)

For example, if your company uses a mathematical model to sort through job applications, evaluate whether it’s favoring men or those with typically white names. Then consider refining the model so that it ignores names altogether and instead zeroes in on qualifications. (Shortform note: Here, too, is an opportunity to employ machine learning best practices. Companies that use a mathematical model to sort through job applications should start with having diversity in the hiring team and the team designing the model, as well as diversity in the data used.)

Regulate Industry

O’Neil’s second recommendation for limiting the damage done by dangerous models is to impose regulations on industry use of mathematical models. Since data science is a relatively new field, there are few government strictures on how companies are allowed to use customer data. O’Neil argues that increased regulatory oversight will help prevent companies from misusing mathematical models.

Specifically, O’Neil supports mandatory transparency for companies that use algorithms. Transparency allows consumers to understand the criteria that they’ll be judged on. This increases fairness, as those customers will better understand what measures they can take to improve their standing.

Transparency also helps bring to light patently unfair processes, such as models that discriminate based on race or gender.

(Shortform note: O’Neil calls for mandatory transparency but doesn’t go into detail about how much transparency is enough. Some experts note that too much transparency can actually be detrimental: It can open a system up to exploitation, or it can further confuse consumers who can’t make sense of the lines of code or the huge amounts of data models are trained on. One way companies may be able to hit the transparency sweet spot is by using explainable AI (xAI)—systems that enable consumers to understand the output of algorithms. In the case of mathematical models, xAI can provide insights into how models arrive at specific decisions and give individualized feedback to consumers.)

O’Neil also proposes increased regulation around which data organizations are allowed to collect and sell. She notes that in Europe, many countries have adopted an opt-in model regarding data collection. Under such a model, none of your personal data can be collected and sold unless you opt-in to a given program.

(Shortform note: In 2018, two years after Weapons of Math Destruction was published, the European Union enacted the General Data Protection Regulation, which requires websites to get users’ consent before collecting their data. Since then, most websites ask visitors whether they would like to accept “cookies”—small pieces of information that a website uses to remember information about users and their activity on the site.)

Set Different Goals

O’Neil writes that in addition to measuring impact and increasing regulation, organizations should optimize their models for fairness and social good and incorporate feedback to increase fairness over time. She says companies should build mathematical models with values and ethical standards included in their decision-making processes. Many of the problems produced by dangerous models occur because companies optimize models for profit alone.

(Shortform note: When did companies start prioritizing profits above all else? In What Money Can’t Buy, Michael J. Sandel traces it back to the political shift to the right in the 1980s, when British Prime Minister Margaret Thatcher and US President Ronald Reagan were in power. Sandel argues that the two leaders were the leading proponents of neoliberalism—believing that reducing taxes on the wealthy and reducing the state’s role in the economy would lead to prosperity. This gave rise to free-market capitalism, which Sandel contends corrupts our values by putting goods and services—such as health care and education—up for sale.)

For example, people who are strapped for cash and search online about food stamps might be served ads for high-interest payday loans. While these ads generate profits for the advertiser and the search engine, they do this at the expense of vulnerable consumers. In this case, regulation should look beyond the bottom line and factor in the ethics of such predatory practices.

(Shortform note: In addition to being targets of predatory ads, low-income households are also easier targets for digital thieves. These thieves steal people’s data from benefit cards used by participants of the Supplemental Nutrition Assistance Program. These benefit cards aren’t legally protected from fraud in the way that credit and debit cards are, so benefit cardholders typically don’t get reimbursed.)