This article is an excerpt from the Shortform book guide to "Weapons of Math Destruction" by Cathy O'Neil. Shortform has the world's best summaries and analyses of books you should be reading.
Like this article? Sign up for a free trial here.
What causes big data discrimination? How do dangerous math models hurt the poor?
In Weapons of Math Destruction, Cathy O’Neil argues that dangerous math models disproportionately inflict harm on the poor. This is because they show favoritism towards rich people.
Let’s dive deeper into big data’s discrimination problem.
Breaking Down Privilege in Big Data
O’Neil argues that dangerous mathematical models tend to harm poor people while offering preferential treatment to rich people. It’s more efficient and cost-effective for institutions to automate the bulk of their interactions, so they use mathematical models to determine college admissions, sort through job applications, evaluate job performance, and calculate credit scores—all processes that favor wealthy people’s backgrounds and circumstances.
The result is that poor people often get poorer from their interactions with big data discrimination, while rich people largely benefit from their interactions with the same systems. Over time, this leads to increased income disparity and social inequality.
(Shortform note: While O’Neil argues that dangerous mathematical models disproportionately harm the poor, Matthew Desmond argues that we all play a part in perpetuating poverty. In Poverty, by America, Desmond argues that poverty in America persists because it benefits many Americans to keep some of their fellow Americans poor: As consumers, we demand cheap goods and expect quick service and delivery of those goods, we buy stock in companies that exploit their workers, and we support housing rules that trap people in poor communities and, consequently, generational poverty.)
For example, algorithms that calculate credit scores are designed to predict a borrower’s ability to repay loans based on factors like loan repayment history, current debt levels, and income. Poor people, often with low scores due to unstable incomes, little credit history, or small financial missteps (such as late bill payments, which are common among those struggling financially), face higher loan interest rates or denial of credit, amplifying their financial struggles. Conversely, wealthier individuals often have long credit histories and enough resources for on-time payments, leading to high credit scores.
| US Credit Scores Perpetuate Racial Inequality Credit scores were created to prevent the sort of harm that O’Neil writes about—in theory, using mathematical models to crunch cold, hard numbers rather than having humans evaluate who’s creditworthy is a more objective approach. However, the current system—which gives people a score ranging from 300-850 (with numbers above 700 generally considered good)—has been shaped by long-standing patterns of systemic bias and punishes people of color. The credit scoring system awards more points for things like home ownership and credit history—things that more often tick the boxes of wealthier, white individuals. For instance, the homeownership rate for white Americans is at 71.7% versus the homeownership rate for Black Americans at just 47%. This means people of color have lower credit scores, making it harder for them to obtain credit to buy homes and build a credit history to begin with. One study found that Black and brown borrowers were given higher interest rates for loans and that the denial rate for Black mortgage applicants in 2022 was 84% higher than that of white borrowers. Experts recommend a couple of ways to improve the credit scoring system. First is to factor rent payments into credit histories—this could boost renters’ credit scores by nearly 60 points. Another way is to consider cash flow (how money goes in and out of someone’s bank account during a specific period) to get a clearer picture of someone’s financial standing. This would be especially helpful to young people and immigrants who are just starting to build their credit histories. |
———End of Preview———
Like what you just read? Read the rest of the world's best book summary and analysis of Cathy O'Neil's "Weapons of Math Destruction" at Shortform.
Here's what you'll find in our full Weapons of Math Destruction summary:
- The insidious ways mathematical models are being used in society
- How mathematical models are being used to reinforce bias and harm the poor
- Recommendations for how to rein in dangerous mathematical models
