Bishop Hill Superforecasting
Dec 14, 2015 Books 
I'm sure for many readers here, Philip Tetlock's famous studies of the predictions of experts need little introduction. His most famous finding - that specialists are considerably worse at making predictions than generalists - is now quite well known, although less so among the general public and media than it should be.
His more recent work has been a project looking at forecasts in the realm of geopolitics and the results are summarised in his latest book, Superforecasting. The project asked panels of volunteers to make forecasts and then tested how accurate they turned out to be in reality. Much of the book focuses on "superforecasters" - those people who turned out to be noticeably better at it.
Suffice it to say that there was little by way of expertise that made these people stand out from the crowd. They were bright people, but not off-the scale geniuses; numerate but not those inclined to conjure with arcane mathematics. It was more about independence of mind, the ability to constantly recalibrate and to question assumptions, the ability to think in terms of probabilities rather than in black and white terms.
And although the book barely mentions climate, and indeed the focus of the project was on short-term predictions not long-term ones, it was hard while reading it not to keep wondering "do climatologists think like this?"
Co-authored by pop-sci writer Dan Gardner, the text is as light as a feather, making it readily accessible to nearly everyone. A good Christmas pressie for the geek in your life. Buy it here.
Reader Comments (27)
So - a book effectively on why sometimes planes fall out of the sky but mostly they don't.
And nothing on climate change nor its supposed modelling weakness?
Ho hum.....
Boy the trolls are keeping a sharp eye on the Bish these days. Trolls first up again. Must be getting under their skin!
Take 100 people.
Get them to make binary predictions on 5 topics.
On topic 1, 50 get it right.
On topic 2, 25 of them get it right
On topic 3, 12 of them get it right
On topic 4, 6 of them get it right
On topic 5, 3 of them get it right.
You now have three 'superforecasters'. Write a book about how they got it right. Under absolutely no circumstances allude to the fact that out of any random 100 people making five binary decisions, a random three of them will get them all right.
TBYJ, if the result is the same as the coin-toss result you are correct. If the choice is one where judgement could have an influence, one expects some folks to beat the coin toss. If we run the test again and the same folks beat the toss, we have a meaningful result. Of course none of this has any predictive effect on the NEXT prediction they make. Or does it?
Lady luck has no memory.
Also as has so often been stated - modelling (including climate modelling) often doesn't do "predictions".
Start talking "projections" instead (you do know the difference?) and maybe you'll start to get the point.
TBYJ, don't tell me you don't see the difference. This is not a coin toss. It is not a 50-50 ball. The events used in the test must be predictable. And of course a cascade test as you describe wouldn't be a good way to find anyone who can beat the general population consistently. It isn't roulette, it's poker.
I get the point. Being wrong all the time doesn't disqualify you from being called an expert because in an obscure part of the paper you wrote 'suggest' and 'may' and it's not your fault if the university press dept. turns your guesswork into a probable or that gormless journalists then extrapolate that probable to a fact or that sanctimonious, faux-green rent-a-drones endlessly repeat this false fact in spite of numerous subsequent papers refuting it's methodology and assumptions.
"It was more about independence of mind, the ability to constantly recalibrate and to question assumptions, the ability to think in terms of probabilities rather than in black and white terms."
That is a sceptic!
Usually older, well educated, working in science or industry, independently minded, numerically literate, a lifetime experience of seeing how easily it is to make bad decisions from bad assumptions.
I don't see sceptics as a monobloc. Some of us are like that.
YouKnowNothing,, do climate projections have the ability to go down aswell as up, because climate predictions only ever go up. If this is the case, then projections may have about a 50% chance of not being as inaccurate, which would be a definite improvement.
Why didn't climate scientists think of this before? It seems so obvious to anyone who does not start with a false presumption.
projection (prəˈdʒɛkʃən) n
6. a prediction based on known evidence and observations
Or perhaps also relevant:
8. Psychology
a. The attribution of one's own attitudes, feelings, or suppositions to others:
In climate science, a "projection" is an estimate of the future which if actually comes true is hailed as a victory but if is nowhere close to reality is discarded and buried. E.g. claims of the Arctic being ice-free.
In real life I find my ability tp predict is inversely related to how much I am involved with the subject and the outcome. The more dispassionate you are the more likely you can predict in a sensible manner. When it's your own future it's more unlikely you'll get it right. Every 'expert' fails this test if they have any involvement, financial or reputational.
How many alarmists predict the fiery death of the planet because they want to be the Cassandra who saw it coming?
How many scpetics predict nothing much will happen because they instinctively reject every alarmist story, because they can't contemplate ANY disaster?
YouKnowNothingBishHill,
Fairly sure that I know what the IPCC means when it uses the term "projection". It's a simulation of the climate that could come true and so id testable.
A "prediction" is some incomplete aspect of the climate that can never be tested.
Strangely, "projections " are not to be used in making policies. Which is a bit of a blow for the people who just had a jolly in Paris.
But, please enlighten me, what do you think the difference is?
YouKnowNothingBishHill, please provide your definition of "Projection".
http://www.theguardian.com/business/2015/dec/14/spain-energy-giant-abengoa-collapse-predicted-by-17-year-old
@ golf Charlie
Yes of course they do. What on earth are you thinking? Depends on input conditions...
@ MikeC
Honestly can't believe you had to look that up! Plus the definition is partial and not complete......
YouKnowNothingBishopHill
40 percent youth unemployment in Spain after joining the Euro Nigel predicted that.
Here is most of a preliminary review by a statistician/economist involved in forecasting research himself, Frances Diebold.
The same chap has made some comments on a recent paper on Climatology and Predictive Modelling.
YouKnowNothing, so after all the disastrous predictions, when do we get to hear of some valid projections? It just seems as though the predictions/projections of disaster, repeated ad nauseam in Paris, do not seemed based on any evidence at all.
Could you give a breakdown of what are valid projections, so it is easier to know what can be disregarded as speculative guesswork like Mann's Hockey Stick.
You are aware that all the ridiculous claims started with the global warming fantasists aren't you?
youknownothingtroll:
And this is just English semantics. Think about it. When COPout21 came to reduce the thousands of words of pre-amble into the 31 pages of press release, how many languages - with their own nuanced semantics - did they have to cover? Or was everything carried out in (UK) English? (Can't see that being the case - remember COPout was in Paris, France (for the benefit of our colonial cousins who might have thought it was in Texas) - and the French are very protective of their language. So, what I'd like to know is, what is the difference, if any, between "prédiction et projection" for them?For US readers (who are well aware that Paris is in France, having made two 20th Century excursions there in large numbers to pull UK nuts from the fire):
http://www.amazon.com/Superforecasting-Prediction-Philip-E-Tetlock/dp/0804136696/ref=sr_1_1?ie=UTF8&qid=1450122816&sr=8-1&keywords=superforecasting
Jorge.....Tongue and Cheek. No offence to my friends in the US.
Paris is indeed in Texas. I went there earlier in the year on a pleasant Sunday evening. It was shut.
Rhoda, if Paris Texas is still shut, they may have derived some economic benefit from holding a climate party there. It would be just as destructive for the rest of the world.
Golf, I think the significant word in my comment is Sunday.
Wasn't that the film with those folks in an ELdorado? The guy had snakeskin shoes.
"Jorge.....Tongue and Cheek. No offence to my friends in the US." --Harry Passfield
None taken, actually. I've never been to Paris, TX, but have been to Swinney Switch. Unlike either Paris, not many people can say that. It's not easy to tell whether Swinney Switch is open or closed, since it looks pretty much the same either way. Odds are at any given day or time, it's closed, or might just as well be. However, it's not far from George West, the county seat, which has a stop light, and is just open enough on Sundays to serve food to truck drivers going to or from Corpus Christi on their way from or to San Antonio. It's untrue that people in George West gather at the stop light to watch it change color. I once read their entire phone book--five minutes shot to hell, but there wasn't much else to do there. Then I went looking for that stop light.