This article is an excerpt from the Shortform summary of "Thinking, Fast and Slow" by Daniel Kahneman. Shortform has the world's best summaries of books you should be reading.
Like this article? Sign up for a free trial here .
What is the illusion of validity? Why is it an illusion, and why is it a problem?
The illusion of validity is the tendency to be overconfident about our skills of prediction. We’re especially vulnerable to the illusion of validity when faced with data points that seem to form a coherent narrative.
Learn what the illusion of validity is and why it happens, with examples.
The Illusion of Validity
The illusion of validity is the result of overconfidence and our tendency to form narratives around random details and events. With all the heuristics and biases described above working against us, when we construct satisfying stories about the world, we vastly overestimate how much we understand about the past, present, and future.
We desire a coherent story of the world. This comforts us in a world that may be largely random. If it’s a good story, you believe it. This is one factor of the illusion of validity.
Insidiously, the fewer data points you receive, the more coherent the story you can form. You often don’t notice how little information you actually have and don’t wonder about what is missing. You focus on the data you have, and you don’t imagine all the events that failed to happen (the nonevents). You ignore your ignorance. This is why the illusion of validity is an illusion.
The ultimate test of an explanation is whether it can predict future events accurately. This is the guideline by which you should assess the merits of your beliefs. This will help you avoid the illusion of validity.
The Illusion of Validity and the Hindsight Bias
The hindsight bias is related to the illusion of validity. Once we know the outcome, we connect the dots in the past that make the outcome seem inevitable and predictable.
Insidiously, you don’t remember how uncertain you were in the past—once the outcome is revealed, you believe your past self was much more certain than you actually were! It might even be difficult to believe you ever felt differently. In other words, “I knew it all along.” You rewrite the history of your mind.
Professional forecasters (eg experts who show up on talk shows) perform no better than chance in predicting events. This might also be because they’re hired for their charisma and vocalness, not accuracy.
Hindsight bias is a problem because it inflates our confidence about predicting the future and contributes to the illusion of validity. If we are certain that our past selves were amazing predictors of the future, we believe our present selves to be no worse.
Shortform note: antidotes to hindsight bias, and therefore the illusion of validity, include:
- Rewarding people based on the decisions they make at the time with the information they had, before the outcomes come out. Don’t reward people who took outlandish risks but got lucky.)
- Keeping a journal of your current beliefs and what you estimate the outcomes to be. In the future, once the outcomes are known, reflect on your beliefs at the time to see how accurate you were.
The Illusion of Validity and the Narrative Fallacy
The illusion of validity is also related to the narrative fallacy.
We desire packaging up a messy world into a clean-cut story. It is unsatisfying to believe that outcomes are based largely on chance, partially because this makes the future unpredictable. But in a world of randomness, regular patterns are often illusions. This leads to the illusion of validity.
- History is presented as an inevitable march of a sequence of events, rather than a chaotic mishmash of influences and people. If the past were so easily predictable in hindsight, then why is it so hard to predict the future?
- Management literature also tries to find patterns to management systems that predict success. Often, the results are disappointing and not enduring.
- The correlation between a firm’s success and the quality of its CEO might be as high as .30, which is lower than what most people might guess. Practically, this correlation of .30 suggests that the stronger CEO would lead the stronger firm in 60% of pairs, just 10% better than chance.
Even knowing about the illusion of validity, you might still be tempted to write a narrative that makes sense—for example, successful companies become complacent, while underdogs try harder, so that’s why reversion to the mean happens. Daniel Kahneman says this is the wrong way to think about it—the gap between high performers and low performers must shrink, because part of the outcome was due to luck. It’s pure statistics.
Antidotes to Narrative Fallacy
Understanding how to counter the narrative fallacy will help you avoid the illusion of validity.
Be wary of highly consistent patterns from comparing more successful and less successful examples. You don’t know lots of things—whether the samples were cherrypicked, whether the failed results were excluded from the dataset, and other experimental tricks.
The Illusion of Validity: Are Smart Predictions Always Good?
Daniel Kahneman notes that absence of bias is not always what matters most. Relying too much on statistics would avoid the prediction of rare or extreme events on shaky information.
For example, venture capitalists make their money by correctly predicting the few companies that will make it big. They lose only 1x their money on a bad investment but make 1000x their money on a Google, so it’s important not to miss out on the next Google. However, using the type of quantitative analysis above might paralyze some investors, if they start with the baseline failure rate of startups (which is very high) and have to adjust upward from that anchor. For some people prone to paralysis, having distorted overoptimistic evidence might be better. There are benefits to the illusion of validity.
Similarly, sometimes the evidence is against you but your choice feels right, like when you know the high divorce rate as you’re about to get married. In these cases, you might be happier deluding yourself with extreme predictions—”our marriage is going to defy the odds.” Listening to our intuitions is more pleasant and less hard work than acting consciously against them. But at the least, be aware of what assumptions you are making and understand how much you know.
———End of Preview———
Like what you just read? Read the rest of the world's best summary of "Thinking, Fast and Slow" at Shortform . Learn the book's critical concepts in 20 minutes or less .
Here's what you'll find in our full Thinking, Fast and Slow summary :
- Why we get easily fooled when we're stressed and preoccupied
- Why we tend to overestimate the likelihood of good things happening (like the lottery)
- How to protect yourself from making bad decisions and from scam artists
