Dan Gardner's book Future Babble: Why expert predictions fail and why we believe them anyway (or alternatively subtitled How to stop worrying and love the unpredictable) is a terrific book.
(In fact, anything by Dan Gardner is great, including his book Risk and his collaboration with Philip Tetlock, Superforecasting: The art and science of prediction.)
In this article, I'm summing up some of Gardner's points. This is no substitute for reading the entire, entertaining book.
We all predict and make bets on the future.
The topic of prediction is important because we're always predicting the future and making bets on it. Scheduling an appointment in your calendar involves making predictions about the future.
As Gardner says, "Almost everything we do is influenced by an explicit or implicit judgement about the future, whether that future is five seconds or five decades from now."
When we make long-term, impactful decisions, like choosing a job or a career, choosing a partner, deciding to buy a home; or deciding to have children; we are making important predictions about what the future will hold for us.
The future is uncertain and uncertainty feels really bad.
We have a "hard-wired aversion to uncertainty". Uncertainty is so potent, that it's one of the key tools in a torturer's toolbox.
Gardner quotes Daniel Gilbert (of Stumbling on Happiness fame): "An uncertain future leaves us stranded in an unhappy present with nothing to do but wait".
Quoting Gilbert again: "People feel worse when something bad might occur than when something bad will occur."
Because of this, we're vulnerable to "seers".
We want certainty. So we look to others to make predictions. We want to believe anyone who can give us a sense of certainty.
But in many cases, peddling certainty is a confidence art.
Gardner refers to Scott Armstrong's 'seer-sucker' theory: "No matter how much evidence exits that seers do not exist, suckers will pay for the existence of seers."
Gardner explains that human beings appear to have a "confidence heuristic"- "If someone's confidence is high, we believe they are probably right; if they are less certain, we feel they are less reliable".
Experts are especially effective at being confident and convincing about their predictions. Not necessarily because they have any special predictive ability, but because they are exquisite rationalisers:
"Not only are [experts'] brains stuffed with information about the subject at hand, giving them more raw material to work with, but experts are experienced at generating a hypothesis, assembling arguments,and making a case. If they do not restrain their thoughts with self-scrutiny and reason, they can easily spin failure until it seems meaningless, or very close to a success, or even a triumphant vindication."
Trends end. Surprises happen.
"Trends end, surprises happen". The difficulty is that many forecasts are based on a "straight-line projection of current trends into the future". "The danger of [doing this] is notorious."
This is important. As Gardner explains, "when the road is straight, anyone can see where it is going. It's the curves and corners that cause crashes. And so, in general, predictions are most likely to be right when they are least needed and least likely to be right when they are essential."
The difficulty with prediction isn't that it's hard, but that it's essentially impossible.
Hindsight bias be damned. The future is always uncertain
The past feels so inevitable. But it's not. Survey a sample of people before a major event about what they think will happen, and observe the change in their certainties after the event has happened.
Gardner points out that "The future is always uncertain, whether it is the future we face right now or the future people faced a century ago. But when we look back, hindsight bias causes us to see much less uncertainty than we do today."
Because of this, "we think that the future we face now is much more uncertain than the future we faced in the past."
This can make people nostalgic. Gardner uses the example of his grandfather. "At the end of his life, my grandfather looked ahead at the world I would live in and he saw nothing but uncertainty and darkness. So many things could go so wrong. He worried for me. But his life? It had been sweet. This man who had seen his family's prosperity swept away, who had lived through two world wars, who had raised a young family in the worst depression in history - he was convinced he had lived in he best of times. It was the future that frightened him. This illusion is pervasive and yet it is seldom recognised."
An antidote: Think of the future in pluralistic rather than singular terms
One of the best antidotes against false certainty is taking a pluralistic stance towards the future.
Discussing an article by James Fallow, Gardner says "The 'only sensible answer' to the question of whether the United States is Rome in the waning days of the empire... is 'maybe'. To a mind craving certainty, that's not an answer at all. But it is the correct answer."
"Uncertainty is omnipresent.... Thus, the most that we can ever hope to do is distinguish between degrees of probability with reasonable accuracy. That job is relatively easy when we're deciding whether to cross the road or calculating life insurance premiums. But [for many types of predictions, especially] about large-scale, highly complex, highly uncertain social phenomena over long periods of time - reliable forecasting is a challenge on par with climbing Mt Everest barefoot. And that's when it's even theoretically possible, which it usually isn't."
Gardner explains that "accurate prediction often isn't needed in order to make good decisions. A rough sense of the possibilities and probabilities will often do."
"While our decisions have to be made on the basis of what we think is going to happen, we must always consider how our decisions will fare if the future turns out to be very different. A good decision is one that delivers positive results in a wide range of futures."