Phil Tetlock spoke at Greenwood Investors investor day in New York last week. Tetlock, as you probably know, wrote a bestseller titled “Superforecasting: The Art and Science of Prediction.” My friend Steven Wood (Greenwood’s founder) sent me a copy of the book, which I read before Greenwood’s event. I want to thank Steven for inviting me to his excellent event and for sending me the book.
Tetlock’s is a good read in a Malcolm Gladwell sort of way. Lots of compelling stories and interesting nuggets… It’s based on research from the “Good Judgment Project,” which you can read about online. The basic idea is that thousands of volunteers engaged in forecasting tournaments over a period of years, with the results being compiled and analyzed by researchers. The results show some people seem to be remarkably good forecasters. They are “superforecasters.”
While I enjoyed the book, there was something about the tournaments that bugged me. Something critical seemed to be missing… And Steven is undertaking a bold project that will fill in that missing element. More on that below, but first what’s my objection?
If I could put the question to Tetlock, I would say the following:
“What would you say to the criticism that you’ve not proven that people can make good forecasts; instead you’ve proven that some people can be good at playing forecasting games?
In other words, you’ve set up this artificial situation where people are making multiple forecasts on things in which they have no stake in the outcome.
My analogy would be a poker tournament, or series of tournaments, where everyone is given $100,000 in play money. And then we run another tournament, or series, where people have to put up $100,000 of their own money. I would contend that people would play quite differently in each case. I would say the two contests are not similar – at all.”
To me, this seems an insurmountable hurdle. People behave differently when they have skin in the game than when they don’t. (In Tetlock’s experiment, the forecasters are even anonymous! No social pressures, etc.)
I wouldn’t say someone was a good investor because they had great returns in a play money account over a period of years that they ran anonymously. I’d say they were good at play money investing. Real money – especially managing other people’s money – is a whole ‘nother ballgame! Not even close.
This is where Steven’s project comes in. As he wrote in a recent blog post titled “A Trial Whose Time Has Come”:
“[Tetlock acknowledges the potential weakness in his research]. From page 207 of Superforecasting, ‘I can already hear the protests from my colleagues in finance that the only reason the superteams beat the prediction markets was that our markets lacked liquidity: real money wasn’t at stake and we didn’t have a critical mass of traders. They may be right. It is a testable idea, and one worth testing.’
Yesterday, with Dr. Tetlock at our investor day, we accepted this challenge. We will be sharing clips from our discussion with Dr. Tetlock in the coming weeks, and the full video replay will be available in the coming days for investors who weren’t able to join us.
Because many of our investors are business builders and fund managers, or some cross between the two, we believe our investors could be something pretty interesting: a superteam with skin in the game. Our superteam has equity on the line, a wealth of experience, well over $100 billion of assets under management, and an eagerness to win. And they are already very well informed through the research we publish.”
Now that’s pretty interesting and I hope to participate in it. Read Steven’s full blog post here.
We Don’t Know the Odds
There are other ideas in Tetlock’s book that I wonder about…
For example, Tetlock wants us to put numbers on our forecasts. He wants us to say, for example, there is a 60% chance of X happening by Y date. The idea is to create a forecast that is track-able, or that can be scored.
But the conceit of putting percentages on events seem too great to me. For one thing, putting a percentage on a forecast assumes you know the odds. You don’t. And you can’t. You can with a deck of playing cards. We know the deck has 52 cards. We know what all the cards are. (Assuming the deck is legit). We know there are only four kings, one of each suit, etc.
In real life, we’re playing a game where the number of cards is seemingly infinite. It is possible to have six kings. All of the same suit. Weird stuff can happen. It is possible to draw a Princess of Spades. And when that happens, you’ll have your Tetlocks come by and explain how it was not inconceivable to draw a Princess of Spades. After all, there is already a King, a Queen and a Jack. Logically, there could be a Princess. And then he’ll cite references where people have talked about adding another face card, etc.
But that would be hindsight bias. That would be creating a narrative to explain something after the fact, to make it seem plausible.
And Tetlock sort of does this when talking about 9/11. To me, that event was unpredictable, or at the very least extremely unlikely. Not, it seems, to Tetlock. He cites a 1994 plot to hijack a jet and ram into the Eiffel Tower. He cites a 1998 FAA exercise in which they examined a scenario where a terrorist uses hijacked FedEx cargo planes and crashes them into the World Trade Center. He cites 2001 government official who said that he suspects some terrorist group is planning to use an aircraft as a bomb. All of this is to show that 9/11 perhaps should not have been such a surprise.
However, I bet you could find hundreds of such examples from cities from all over the world, involving all kinds of possible terrorists attacks. But they didn’t have a 9/11, New York City did. And in a very specific way that had never been done before.
Hard To Know When Forecasts Are Right – Or Wrong
As I say, Tetlock wants us to score our forecasts. But, as Tetlock points out, if I make a forecast that something has a 60% chance of happening and it doesn’t happen, it doesn’t mean my forecast was wrong. After all, I did say there was a 40% chance that it wouldn’t happen. Likewise, if I say something has a 60% chance of happening and it happens, it doesn’t mean I made a good forecast.
Tetlock admits this, but seems to commit the same mistake. For example on page 250, he writes about how his superforecasters “aced” a question about Scotland leaving the UK, because they correctly forecast Scotland would stay. But again, any forecast, with any set of numbers short of 100% certainty, could still claim to be “right” regardless of what happened.
Beyond this, you have to be really careful crafting the questions in the first place. Re: 9/11, Tetlock writes “we may have no evidence that superforecasters can foresee events like those of September 11, 2001, but we do have a warehouse of evidence that they can forecast such questions as: Will the United States threaten military action if the Taliban don’t hand over Osama bin Laden? Will the Taliban comply?”
I would beg to differ. What does “threaten” mean exactly? I can envision scenarios where people disagree whether certain statements constitute threats. What does the “the United States” mean? Does it mean the President? Does it include members of Congress? And what does the word “comply” mean? Certainly, there are scenarios we can imagine where compliance is murky indeed.
To me, 9/11 was a black swan, a near unimaginable event of enormous consequence.
Taleb and Black Swans
Speaking of black swans, Tetlock does bring Taleb into his discussion. (Some of my favorite pages are in this section). Taleb, of course, popularized the idea of a black swan event. As Tetlock writes, summarizing (some) of Taleb’s ideas: “What matters can’t be forecast and what can be forecast doesn’t matter.”
This, to me, is a strong critique I find it hard to get past.
My own views are more in line with Taleb’s as far as the unpredictability of big, consequential events. And this affects the way I invest. I like companies with strong balance sheets, for example. I like businesses with contractual cash flows. I am careful about the price I pay. All of these are efforts to build in a margin of safety against the unpredictable.
In any event, I invite you to read the book and come to your own conclusions. I thank Tetlock for his talk and book and for giving me something to think about. And I thank you for reading this far.
***
Published September 9, 2019
See our disclaimers