This article is an excerpt from the Shortform book guide to "The Voltage Effect" by John A. List. Shortform has the world's best summaries and analyses of books you should be reading.
Like this article? Sign up for a free trial here.
Are you skilled at knowing when data is misleading? Can you recognize false positives?
When you test your ideas in various markets, you get data that can be invaluable. But, inaccurate data will send you off in the wrong direction. In the context of market research, economist John A. List discusses the importance of accurate data and shares strategies on how to recognize false positives.
Read more to learn how to get more accurate data in market research.
The Importance of Accurate Data
To determine whether your idea will have an audience at scale, you need to figure out who’s in your current audience. The best way to do this is to test your idea in multiple markets that include a wide variety of demographic groups. Testing your idea as broadly as possible will give you the data to determine who your product appeals to, and why.
List explains the importance of accurate data in this process. When determining whether your idea has a big enough audience to scale, remain alert to misleading results. Misleading results, or “false positives,” as List refers to them, are data that overstate the appeal of your idea.
(Shortform note: To avoid falling for misleading results, consider sources outside your organization alongside your own data. By reading economic reports and checking database information, you gain an outside perspective on the condition of the market. If your internal data matches up with this, your data is probably correct. However, if your estimation of your idea’s appeal drastically differs from outside evaluations of the market, you may be experiencing misleading results.)
It can be tempting to believe in misleading results because of confirmation bias, a phenomenon in which people interpret new events in a way that reinforces their prior beliefs—if you believe in your product’s appeal, confirmation bias can make it more difficult to see the product’s flaws.
(Shortform note: In The Art of Thinking Clearly, Rolf Dobelli offers a strategy for combating confirmation bias. He recommends seeking out perspectives that contradict your own beliefs and reevaluating your beliefs in the context of this new information. In the context of scaling, if you have data that suggests your idea will scale, seek out perspectives that suggest that it won’t. This could mean reconsidering economic factors and marketplace competitors, or following any other line of thinking that could expose a flaw in your idea.)
Misleading results occur for a variety of reasons. Often, misleading results arise from failing to sufficiently test your idea. List recommends that you replicate your tests to double-check their results, in addition to testing your idea in diverse markets. By replicating tests, you ensure that initial positive results aren’t flukes that occurred randomly or due to an unforeseen factor.
(Shortform note: To get the most accurate results from your repeated tests, take care to use proper experimental controls in every iteration of the test. In Bad Science, Ben Goldacre defines experimental controls as features of an experiment designed to prevent outside factors from influencing your research. For example, if you were testing a new soft drink with a focus group, you’d want to ensure that the test took place in a location with a moderate temperature and not an especially warm room in which a cold beverage would seem more appealing. When retesting the drink’s appeal, you’d want to implement identical experimental controls, to ensure that any change in your results wasn’t due to a change in temperature.)
If testing indicates that your idea is likely to scale well, you can retest the idea by introducing it in just a few locations before fully scaling up production. If the initial result was accurate, and your idea is scalable, it’ll perform well in these markets. If your idea doesn’t perform well in these new markets, it’s a sign that your initial results may have been misleading and that your idea may not scale.
(Shortform note: When testing your idea in new markets, ensure that your test period is long enough to gain accurate data. Experts note that test data can be impacted by factors such as initially ineffective advertising and “honeymoon periods” in which sales of new products are temporarily inflated. To account for these factors, new market tests should be at least six months long, and ideally even longer.)
———End of Preview———
Like what you just read? Read the rest of the world's best book summary and analysis of John A. List's "The Voltage Effect" at Shortform.
Here's what you'll find in our full The Voltage Effect summary:
- How to take ideas from the small scale to the big stage
- The red flags that signal you may have trouble scaling up
- Strategies designed to increase your idea’s chances of success
