Natural Experiments: Simple Explanation & Examples

This article is an excerpt from the Shortform summary of "Freakonomics" by Steven Levitt and Stephen Dubner. Shortform has the world's best summaries of books you should be reading.

Like this article? Sign up for a free trial here .

Natural Experiments and Economics

Economics is not an easy field in which to conduct randomized controlled experiments. Since much of economics focuses on quantities that are very large (national GDP, effects of fiscal policy on employment) and involves the study of individual decisions made by billions of people, it’s rare for economists to be able to test their hypotheses in a lab.

For this reason, economists must observe events in the real world that happen create “variable” and “control” groups that can substitute for the classical design of a laboratory experiment. These real world events are called natural experiments. Throughout Freakonomics, we’ve relied on several such naturally occurring datasets:

The CPS test scores indicating teacher cheating
The Chicago home sales data illuminating real estate agent incentives
The post-Wade v. Roe crime statistics

Raw public data sets are one of the best resources available to economists.

(Shortform note: a good example of this approach within the economics field is a famous 1994 study on the effect of minimum wage on unemployment. Conventional wisdom held that increasing minimum wage would increase unemployment, by raising the price of hiring employees. This would be hard to run an experiment at scale. Instead, two economists studied a natural experiment: New Jersey increased its minimum wage, while neighboring Pennsylvania didn’t. New Jersey thus acted as the variable group and Pennsylvania as the control group. The result: New Jersey’s employment rate did not suffer relative to Pennsylvania’s, even after New Jersey hiked its minimum wage.)

Can People Be Harmed By Their Names? A Public Study Sheds Some Light

In 2004, an economist studied a massive trove of public data, published by the state of California, consisting of birth certificate information for every child born in the state since 1961. The dataset, which encompassed over 16 million births, contained invaluable information for each birth, including:

Standard information like name, gender, race, and birthweight
Parents’ marital status
Economic status indicators like zip code, parents’ means of paying the hospital bill, and parents’ level of education

He used this natural experiment to explore whether or not having a distinctively “black” name had a negative impact on an individual’s life prospects. In other words, would naming your son DeShawn lower his life outcomes compared to naming him Connor, even if you changed nothing else? The point of the natural experiment was to unpack the socioeconomic disparity between blacks and whites—might black culture cause economic differences between races? In the case of names, having a blacker-sounding name might unfortunately cause more discrimination in job applications.

Given the impossibility and unethical nature of conducting a randomized controlled experiment on this topic (i.e., forcing two groups of parents to give their children black and non-black names, respectively, and then analyzing the children’s life outcomes), the dataset and natural experiment was the next best thing.

So what did the data show?

Over time, black and white parents diverged in the types of names they gave their children. In 1970, the typical girl in a black neighborhood received a name that was only twice as common among blacks as whites—by 1980, a girl in the same neighborhood would receive a name twenty times as common among blacks. By the 2000s, 40 percent of black girls born in California in a given year would receive a name that not one of the 100,000+ white girls born in the state that year would have. A further 30 percent of black girls receive names that no other babies (white or black) would have.

As a standard for comparison, white, Asian-American, and Hispanic-American parents differ a bit, but nowhere near as much as the white-black difference.

The economist also saw points of commonality between the mothers who were most likely to give their children distinctively black names. They tended to be:

Teen mothers
Unmarried
From low socioeconomic backgrounds
Have low educational attainment
Have distinctively black names themselves

This last data point was critical, since it enabled researchers to build another natural experiment.

Combing through the data, they could track the full life trajectory of those mothers who had distinctively black names andcompare them against a control group of mothers who didn’t have distinctively black names. Using regression analysis, they could control for other factors that might have influenced these women’s life trajectories like (education, income, or criminal history) and measure the impact of their names.

Natural Experiments: Simple Explanation & Examples