This article is an excerpt from the Shortform book guide to "Merchants of Doubt" by Naomi Oreskes and Erik M. Conway. Shortform has the world's best summaries and analyses of books you should be reading.
Like this article? Sign up for a free trial here .
How does peer review in science work? How does the scientific community reach a consensus?
Peer review in science is based on principles established during the Scientific Revolution. The process, comprising of three steps, is the test that a scientific claim must pass in order to be considered legitimate.
Read more to learn about peer review in science.
The Peer Review Process
What does peer review in science look like? The basic system has been in place since the 1600s. When scholars first became interested in learning new information (in medieval times, learning was about studying ancient texts and preserving what was already known), they realized that there had to be some sort of way to check new ideas for validity, or people could just make things up and be taken seriously. To prevent this, they built a system in which claims had to be backed up by evidence, and both the claim and evidence needed to pass the judgment of other scientists to be considered legitimate. If a claim failed the process, it was just a claim, not science.
Today, the process is called peer review, and no science is considered legitimate until it passes peer review. Typically, the process works like this:
Peer Review in Science: Step #1
A scientist comes up with an idea, collects evidence to support it, writes a paper, and then submits it to a scientific journal for publication.
Peer Review in Science: Step #2
The journal sends the paper to three other scientists to review. The reviewers aren’t paid—since everyone’s work has to be peer-reviewed to be considered science, someone else will return the favor later.
Reviewers must be subject matter experts and mustn’t have a close relationship with the paper’s author. They look for bad science and mistakes such as:
- Made up, tweaked, or manipulated data
- Cherry-picked data—choosing to use only certain (favorable) data and leaving other numbers out
- Claims that aren’t testable
- Small sample sizes (Shortform note: If a sample is too small, results might be due to chance)
- Unclear analysis (the reviewer can’t understand how the author got the data or made calculations with it)
- Conclusions that don’t match what the results found
- Conclusions drawn from too little or inconsistent data
- Reputation of the author (if the author has a history of bad methods or unreasonable claims, reviewers will consider all of the above with more scrutiny)
If the reviewers provide conflicting comments, the journal might send the paper to more scientists or the editor might provide notes.
Peer Review in Science: Step #3
The journal sends the reviewers’ notes back to the paper’s author to address. The author may write multiple drafts as she implements and receives new feedback. If the author ultimately fails to revise well enough, the journal won’t publish the paper. The author has to start over or try a different (less prestigious) journal.
Papers that are presented at conferences don’t have to go through the same process and aren’t considered serious (they aren’t taken into consideration for tenure).
Peer review in science is an indispensable part of the scientific endeavor.
———End of Preview———
Like what you just read? Read the rest of the world's best book summary and analysis of Naomi Oreskes and Erik M. Conway's "Merchants of Doubt" at Shortform .
Here's what you'll find in our full Merchants of Doubt summary :
- How doubt-mongering techniques are used to discredit those who threaten a person or company's agenda
- The 10 most common doubt-mongering techniques
- Steps you can take to protect yourself from doubt-mongering
