The Limitations of Regression Analysis in Stats

This article is an excerpt from the Shortform book guide to "Naked Statistics" by Charles Wheelan. Shortform has the world's best summaries and analyses of books you should be reading.

Like this article? Sign up for a free trial here .

What are the uses and limitations of regression analysis in statistics? What are some things you should keep in mind when reporting the results of regression analysis?

Regression analysis is a powerful research tool as it allows researchers to isolate variables of interest. However, regression analysis can be dangerous if its results are misinterpreted or misreported.

Here are a few pointers to keep in mind when interpreting regression analysis results.

Regression Analysis Pitfalls

Since we use regression analysis to study some of society’s most pressing problems, Wheelan explains that using regression appropriately can be a matter of life and death. Therefore, Wheelan emphasizes the importance of keeping the limitations of regression analysis in mind. To that end, he gives the following reminders for obtaining and interpreting reliable regression analysis results.

Reminder 1: Even when our regression analysis shows that our independent variable is a strong predictor of our dependent variable, we still can’t report that one has caused the other. For example, there may be a highly significant regression coefficient and large R² for the relationship between white-tailed deer populations and rates of Lyme disease. But this doesn’t mean that the deer are causing the Lyme disease (as we know, the ticks that feed on the deer are).

The Power of Hindsight

Sometimes it takes the power of hindsight to realize that an independent variable isn’t actually causing a change in a dependent variable. For example, a company called Sensa Products made $364 million between 2008 and 2012 by selling weight loss powder that users sprinkled on their food. The powder was meant to enhance the smell and taste of food, with the idea that users would feel satisfied with less food, and thus lose weight with little effort. The Federal Trade Commission called Sensa’s marketing misleading and scientifically unfounded and fined the company $26.5 million dollars in 2014.

As Sensa Products earned millions of dollars before being outed by the FTC, surely a percentage of users lost weight using the product. For those users, there was a correlation between using Sensa and losing weight. Finding out that Sensa’s claims weren’t scientifically valid doesn’t invalidate people’s actual weight loss, and the correlation remains. In hindsight, however, those who lost weight on the product might be able to identify other variables that led to their weight loss. For example, perhaps using Sensa inspired them to simultaneously adopt other healthy habits like more exercise or decreased sugar intake.

Reminder 2: We should be careful not to invert our independent and dependent variables. For example, if you wanted to study the link between money and happiness, you would have to be very thoughtful about which variable you selected as the independent variable. If money is your independent variable, you would be studying whether rich people tend to be happier than less-rich people. In contrast, if you select happiness as your independent variable, you would be studying whether happy people tend to be richer than less-happy people. While the two questions may sound similar, these are quite different studies from a research perspective.

Using Statistics to Assess Whether Money Can Buy Happiness

A 2010 study from Princeton University used two statistical methods Wheelan covers in Naked Statistics to study the question of whether money can buy happiness. First, the study used polling to gather data from 450,000 responses to a Gallup survey about day-to-day emotions and overall life evaluation (how people rate their life in the “big picture”). Next, researchers used multivariate regression analysis to analyze whether a high income is correlated with increased emotional wellbeing and better life evaluation.

The results of the study suggest that money can buy happiness to a point. Income was positively correlated with life evaluation in general (people had a more favorable opinion of their lives overall if they made more money) and positively correlated with emotional well-being up to an income of $75,000. Beyond $75,000, income no longer predicted emotional well-being.

In this case, income was the independent variable and emotional well-being and life evaluation were dependent variables (each dependent variable was analyzed separately). If the researchers were to invert the variables and use emotional well-being and life evaluation as independent variables, and money as the dependent variable, their study would be answering a different question.

For example, instead of suggesting that money makes people happy, inverted variables might suggest that high life evaluation or emotional well-being somehow leads people to make more money; perhaps they are more effective at work, more ambitious, more likely to be promoted, and so on. While the data may be the same, the implications of the research are different. In this case, additional studies might focus on how to leverage the traits of “happy” people to be more successful at work.

Reminder 3: Our results will be misleading if they omit a critical independent variable. For instance, say we published a hypothetical paper highlighting a large and significant regression coefficient for the amount of time people spent at the beach and rates of skin cancer. Our results imply that just being at the beach might somehow give a person cancer when in reality it would be the sunburns in unprotected beachgoers that lead to cancer.

The Placebo Effect

Recent insight into the effectiveness of placebo treatments has shown that the power of placebos may highlight missing independent variables in medical research. Since a placebo is generally thought of as “no treatment,” we might logically expect patients receiving a placebo “treatment” to experience no medical benefits in a clinical trial. However, in one study, participants who took a placebo pill experienced 50% as much migraine relief as participants taking the actual drug.

The results of the above study aren’t an anomaly. Research has shown that placebos help ease symptoms from pain to anxiety, even when people know that the “drug” that they’re taking is a placebo. Research on why placebos ease symptoms is ongoing, but possible explanations include increased body awareness, the release of dopamine and endorphins when taking a placebo, as well as the healing act of engaging in a self-care ritual (simply taking a pill, even a sugar pill).

From a research and statistics perspective, the knowledge that placebos can actually act as treatments suggests that there may be several independent variables at work in “control” groups that are unaccounted for.

Reminder 4: We shouldn’t extrapolate our regression analysis results to a population for whom our study was never intended. For instance, say we ran a regression analysis that showed a large and significant negative relationship between the number of miles that healthy 1- to 5-year-old dogs walked each day and behavioral problems. We shouldn’t use these results to make a blanket statement that all dogs will behave better when they are walked as far as possible, as this would likely be bad advice for many senior dogs and dogs with certain health conditions.

Grapefruit Juice and Medication

Reminding ourselves to interpret the results of regression analysis as they were intended is a good way to prevent alarmist reactions to research findings. For example, research shows that grapefruit juice can have health benefits for some and be a health hazard for others, as it contains a chemical that binds to digestive enzymes in our gut and prevents intestinal absorption of certain medications. When these medications can’t be absorbed in our gut, they instead travel to our bloodstream, which can lead to dangerous side effects. A sudden blood increase of calcium channel blockers used to treat high blood pressure, for instance, can lead to a dangerously slow pulse or low blood pressure.

However, it would be incorrect to infer that all people should abstain from grapefruit juice for fear of its “dangerous effects.” This caution only applies to populations taking specific prescriptions. For other people, grapefruit juice contains valuable vitamin C, potassium, and fiber.

The Limitations of Regression Analysis in Stats