This article is an excerpt from the Shortform book guide to "Weapons of Math Destruction" by Cathy O'Neil. Shortform has the world's best summaries and analyses of books you should be reading.
Like this article? Sign up for a free trial here.
How can you use mathematical models ethically? What are ways to regulate industry use of math models?
Cathy O’Neil proposes strategies industries and governments can take to limit the harm caused by dangerous mathematical models. She recommends monitoring math models, regulating them, and setting more positive goals for math models.
Keep reading to learn more about ethics in mathematics.
1. Measure Impact
O’Neil argues that you need to measure the effects of dangerous mathematical models before you can mitigate them. Good mathematical models incorporate feedback, sharpening their algorithms based on previous results. Measuring the impact of dangerous models creates feedback that can be used to improve ethics in mathematics.
When measuring a mathematical model’s impact, consider whether the model’s judgments are fair. Ask whether the model treats all individuals equitably or whether it produces judgments that either favor or punish some groups or individuals unfairly.
(Shortform note: To determine whether a mathematical model’s judgments are fair, we can take a page from best practices in machine learning, the branch of artificial intelligence that trains computers to mimic intelligent human behavior. These practices include having a third party conduct regular audits of input data and output decisions, and having collaborative work teams to address potential blind spots and have a more holistic view of a mathematical model’s impact.)
For example, if your company uses a mathematical model to sort through job applications, evaluate whether it’s favoring men or those with typically white names. Then consider refining the model so that it ignores names altogether and instead zeroes in on qualifications. (Shortform note: Here, too, is an opportunity to employ machine learning best practices. Companies that use a mathematical model to sort through job applications should start with having diversity in the hiring team and the team designing the model, as well as diversity in the data used.)
2. Regulate Industry
O’Neil’s second recommendation for limiting the damage done by dangerous models is to impose regulations on industry use of mathematical models. Since data science is a relatively new field, there are few government strictures on how companies are allowed to use customer data. O’Neil argues that increased regulatory oversight will help prevent companies from misusing mathematical models.
Specifically, O’Neil supports mandatory transparency for companies that use algorithms. Transparency allows consumers to understand the criteria that they’ll be judged on. This increases fairness, as those customers will better understand what measures they can take to improve their standing.
Transparency also helps bring to light patently unfair processes, such as models that discriminate based on race or gender.
(Shortform note: O’Neil calls for mandatory transparency but doesn’t go into detail about how much transparency is enough. Some experts note that too much transparency can actually be detrimental: It can open a system up to exploitation, or it can further confuse consumers who can’t make sense of the lines of code or the huge amounts of data models are trained on. One way companies may be able to hit the transparency sweet spot is by using explainable AI (xAI)—systems that enable consumers to understand the output of algorithms. In the case of mathematical models, xAI can provide insights into how models arrive at specific decisions and give individualized feedback to consumers.)
O’Neil also proposes increased regulation around which data organizations are allowed to collect and sell. She notes that in Europe, many countries have adopted an opt-in model regarding data collection. Under such a model, none of your personal data can be collected and sold unless you opt-in to a given program.
(Shortform note: In 2018, two years after Weapons of Math Destruction was published, the European Union enacted the General Data Protection Regulation, which requires websites to get users’ consent before collecting their data. Since then, most websites ask visitors whether they would like to accept “cookies”—small pieces of information that a website uses to remember information about users and their activity on the site.)
3. Set Different Goals
O’Neil writes that in addition to measuring impact and increasing regulation, organizations should optimize their models for fairness and social good and incorporate feedback to increase fairness over time. She says companies should build mathematical models with values and ethical standards included in their decision-making processes. Many of the problems produced by dangerous models occur because companies optimize models for profit alone.
(Shortform note: When did companies start prioritizing profits above all else? In What Money Can’t Buy, Michael J. Sandel traces it back to the political shift to the right in the 1980s, when British Prime Minister Margaret Thatcher and US President Ronald Reagan were in power. Sandel argues that the two leaders were the leading proponents of neoliberalism—believing that reducing taxes on the wealthy and reducing the state’s role in the economy would lead to prosperity. This gave rise to free-market capitalism, which Sandel contends corrupts our values by putting goods and services—such as health care and education—up for sale.)
For example, people who are strapped for cash and search online about food stamps might be served ads for high-interest payday loans. While these ads generate profits for the advertiser and the search engine, they do this at the expense of vulnerable consumers. In this case, regulation should look beyond the bottom line and factor in the ethics of such predatory practices.
(Shortform note: In addition to being targets of predatory ads, low-income households are also easier targets for digital thieves. These thieves steal people’s data from benefit cards used by participants of the Supplemental Nutrition Assistance Program. These benefit cards aren’t legally protected from fraud in the way that credit and debit cards are, so benefit cardholders typically don’t get reimbursed.)
———End of Preview———
Like what you just read? Read the rest of the world's best book summary and analysis of Cathy O'Neil's "Weapons of Math Destruction" at Shortform.
Here's what you'll find in our full Weapons of Math Destruction summary:
- The insidious ways mathematical models are being used in society
- How mathematical models are being used to reinforce bias and harm the poor
- Recommendations for how to rein in dangerous mathematical models
