This article is an excerpt from the Shortform book guide to "Bad Science" by Ben Goldacre. Shortform has the world's best summaries and analyses of books you should be reading.
Like this article? Sign up for a free trial here.
What qualifies as scientific misconduct in research? How does misinformation about science spread to the public?
Many people trust scientists to give them facts about health, new discoveries, and more. However, their trust can be broken when scientific misconduct in research takes place.
Keep reading to learn how healthcare industries overstate the effectiveness of their treatments, according to Ben Goldacre’s book Bad Science.
Bad Claims
As Goldacre notes, it’s important to identify and call out scientific misconduct in research because companies sometimes use it to push ineffective, dangerous treatments to market. When you’re able to evaluate research results yourself, you can make more informed decisions and inform others, too.
The main strategies companies use to make their treatments seem more effective are selective publishing, cherry-picking positive results, and exaggerating based on surrogate outcomes.
1. Publish the Good, Bury the Bad
Goldacre argues that in order to make their treatments seem more effective, companies only publish results when their treatments perform well, and they actively conceal negative results. This is commonly referred to as publication bias. For instance, if a company conducts three trials for a new drug, and the drug only performs well in one of them, that company could choose not to publish the other two studies. By hiding these crucial pieces of data, the company makes its treatment seem more effective than it really is.
As an example, Goldacre notes that when selective serotonin reuptake inhibitors (SSRIs) were being researched as a treatment for depression, pharmaceutical companies buried multiple studies in which SSRIs were shown not to work better than placebos. By hiding these lukewarm studies, the pharmaceutical industry was able to make sure SSRIs made it to market as quickly as possible.
In addition to omitting mediocre results, drug companies sometimes hide results that show that their products cause harm. In particular, Goldacre references Vioxx, a painkiller that was marketed from 1999 to 2004. The company that manufactured Vioxx hid evidence that Vioxx caused an increased risk of heart attack and pushed the drug to market anyway. Goldacre estimates that Vioxx caused tens of thousands of heart attacks in its brief time on the market. It’s important to understand and identify bad science, because, as illustrated here, when drugs are pushed to market irresponsibly it can cause tremendous harm.
2. Cherry-Picking Positive Results
In addition to trying to control which research gets published, companies cherry-pick results, referencing only positive results and ignoring or dismissing negative ones.
Goldacre notes that cherry-picking is especially common in alternative medicine. For example, while the vast majority of medical literature states that homeopathic treatments are no more effective than placebos, homeopaths tend to cherry-pick from the few studies that support their practices. When confronted with studies that contradict their claims, cherry-pickers will dismiss them, claiming that those studies used flawed methods. By contrast, they’ll often choose to ignore the methodological flaws in the studies that support their treatments.
As an example, Goldacre describes how companies and government officials used cherry-picking to promote ineffective treatments during the height of the AIDS epidemic in South Africa. According to Goldacre, a vitamin company led by Matthias Rath made widespread claims throughout South Africa that vitamins were more effective for the treatment of AIDS than antiretrovirals (the type of drug typically prescribed to treat AIDS). These claims were based on unscientific, unpublished trials conducted by Rath himself, and they contradicted the overwhelming scientific consensus that antiretroviral drugs are an effective treatment for AIDS patients.
Due to their own biases, South African government officials cherry-picked Rath’s results and used them to justify the decision not to roll out antiretrovirals across South Africa. Instead, Rath’s ineffective vitamin treatments were promoted nationwide.
What was profitable for Rath was tragic for the people of South Africa—Goldacre estimates that hundreds of thousands of people died due to lack of access to antiretrovirals. This tragedy illustrates why it is so important to be able to call out bad science. When we allow companies to use misleading techniques to promote their treatments, real people suffer for it.
Meta-Analysis Counteracts Cherry-Picking
When evaluating treatments, scientists also compare research results, using a tool called meta-analysis. A meta-analysis collects, compares, and summarizes all the published studies about a particular topic. By looking at all the data at once, meta-analysis can often reveal patterns that aren’t apparent in individual studies.
For example, suppose that multiple small studies show that a new cold medicine offers mild benefits. In individual studies, the benefit appears so slight that it’s not taken to be significant. However, if you combined the sample sizes of all these studies, you’d be able to get a clearer, more reliable picture of the medication’s effects, based on a larger body of evidence.
If you’re worried about cherry-picking, try to find a meta-analysis of the treatment you’re investigating. The many sources included in a meta-analysis ensure that a few strategically chosen studies don’t overpower the rest.
3. Overstating Results Using Surrogate Outcomes
The last form of scientific misconduct in research is when drug companies exaggerate the effectiveness of their treatments based on surrogate outcomes. A surrogate outcome is an experimental result that implies a treatment may work, but doesn’t indicate conclusively if the treatment works or not. It’s important to identify surrogate outcomes because they’re often used to push unproven and potentially dangerous drugs to market.
For example, suppose you’re testing a new drug designed to treat infection. You find that in petri dishes, your drug successfully kills bacteria. Based on this finding, it’d be easy to jump to the conclusion that your drug can be used to fight bacterial infections. However, to be sure, you’d also want to measure whether or not real, human patients receiving your treatment actually experienced improved outcomes. In this case, the petri dish test result is the surrogate outcome—it suggests that your drug works as intended, but it isn’t conclusive on its own.
Spotting Surrogate Outcomes
You can better assess a company’s claims when you’re able to identify surrogate outcomes. Specifically, you should look out for claims that reference blood test results, experiments done with cells in petri dishes, and experiments performed on lab animals. These are all surrogate outcomes. When you come across them, take them as a suggestion that a treatment might work, and not proof that the treatment actually does work. By contrast, if a company offers results in terms of real-world outcomes, such as heart attacks or deaths, you can take the study’s findings more seriously.
(Shortform note: When you notice that a study’s findings include surrogate outcomes, it can be helpful to look up whether they’re considered acceptable stand-ins, so that you know how seriously to take them. The FDA has produced a comprehensive table that includes all the surrogate outcomes that it has allowed researchers to use as evidence for their drugs. If a study’s surrogate outcomes are listed in this table, it’s a good sign that you can take them seriously.)
———End of Preview———
Like what you just read? Read the rest of the world's best book summary and analysis of Ben Goldacre's "Bad Science" at Shortform.
Here's what you'll find in our full Bad Science summary:
- The strategies researchers, corporations, and journalists use to mislead the public
- The tools you need to identify and call out shady science when you see it
- Why media outlets have an ethical obligation to publish retractions
