Bishop Hill Significance doing the rounds
Dec 10, 2014 Climate: RC Climate: Statistics Climate: Surface I'd like to commend to readers a couple of postings on the subject of statistical significance in the temperature records.
Last week a little visited website called Real Climate had an article by climatologist Stefan Rahmstorf, which addressed many of the issues discussed here in recent months. What I found interesting was that there was a measure of agreement:
...the confidence intervals (and claims of statistical significance) do not tell us whether a real warming has taken place, rather they tell us whether the warming that has taken place is outside of what might have happened by chance.
You will hear no dispute of that sentiment from this quarter. However, as readers here know, it is hard to demonstate that the post-industrial warming is outside the range of natural variability, a subject that the Rahmstorf article rather glosses over, instead preferring to focus on the pause:
...the question the media love to debate is not: can we find a warming trend since 1998 which is outside what might be explained by natural variability? The question being debated is: is the warming since 1998 significantly less than the long-term warming trend? Significant again in the sense that the difference might not just be due to chance, to random variability? And the answer is clear: the 0.116 since 1998 is not significantly different from those 0.175 °C per decade since 1979 in this sense.
It goes on to look at a changepoint analysis of the data performed by Niamh Cahill, a postgraduate student from Dublin.
The optimal solution found for the global temperature data is 3 change points, approximately in the years 1912, 1940 and 1970. There is no way you can get the model to produce 4 change points, even if you ask it to – the solution does not converge then, says Cahill. There simply is no further significant change in global warming trend, not in 1998 nor anywhere else.
And so if she is to be believed we are just seeing long-term warming with natural variability superimposed. However, Ms Cahill's conclusions have been challenged at WUWT, where reader Jeff Paterson seems less than impressed:
[Cahill's] basic thesis is that since a CPA analysis detects no significant recent change in the slope of the GISS dataset there is no pause. Unfortunately, the analysis is of no value because, as is commonly known, the CPA cannot be used on auto-regressive time series.
I am insufficiently expert to be able to say who is right here. Ms Cahill lists time series analysis as one of her areas of expertise so I am dumbfounded if she really has made the error Paterson says she has.
Do any readers here know the answer?
Reader Comments (26)
Googled 'change point analysis autoregressive time series' and got as first reference
http://www.variation.com/cpa/tech/changepoint.html
'Change-Point Analysis: A Powerful New Tool For Detecting Changes' by Dr. Wayne A. Taylor
About halfway down that page:
'Data not appropriate for a change-point analysis and control charting include autoregressive time series data such as stock prices.'
Seems Jeff Paterson may have a point!
I know nothing, but the first Google hit for 'cpa analysis time series' is this:
https://sites.google.com/site/changepointanalysis/
and it immediately states in its Assumptions:
CPA assumes that the process (time series) must be DISTRIBUTED IDENTICALLY, and the observations must be INDEPENDENT (at least there is no strong autocorrelation)
so there do seem to be limitations to the method. I would have thought that historical temperatures aren't independent, but as I said, I know nothing.
Worryingly, that page also says
'There are numerous approaches to performing a change-point analysis.'
Another hit from that google is:
http://link.springer.com/chapter/10.1007/978-3-7908-2604-3_50
is entitled 'Fourier Methods for Sequential Change Point Analysis in Autoregressive Models', so one would need to know more about the methodology used by Ms Cahill, I suppose.
"The optimal solution found for the global temperature data is 3 change points, approximately in the years 1912, 1940 and 1970"
///////////////////////////
But according to the IPCC there was little anthropogenic CO2 prior to the 1950s, so tow of those change points (ie., 1912 and 1940) would not appear to be the result of manmade CO2 emissions.
If 2 of the 3 changepoints were not the result of manmade CO2 emissions, why would the third be the result of manmade Co2 emissions? This is particuolarly important since one probable explanation for the 1970 change point is that that is when we see the effects/corruption/contamination of data caused by UHI, station drop outs and/or inappropriate homogenisation/adjustments.
When considering that point, one should not forget that the land based thermometer record shows warming from tyhe mid/late 1970s through to say 1997, whereas the satellite data shows no warming between 1979 (the date it first streamed back data) and the run up to the Super El Nino of 1998 such that the satellite data suggests that there is some warming bias in the land based thermometer record at any rate as from 1979, and it would not surprise me that the warming bias took place earlier and explains the apparent change point of 1970.
I am with Ernest Rutherford, 1st Baron Rutherford of Nelson, OM FRS on this:
I remain convinced that we actually know beggorah all about what is happening, or has happened, in the climate - which is probably why we always seem to know beggorah all about what might happen in the future.
The point is moot. The projections of the earlier IPCC ARs have diverged from reality. That is more significant than the (lack of) warming trend since 1998.
Since they spent the next five years worrying about global cooling it is quite obvious that "change points! can only be identified well after the event. Perhaps it would be as well if they bore that in mind when drawing assumptions.
On the basis of a warming (~1912 to 1940) and a cooling (1940 to 1970) perhaps it is time they started looking for a change point around 2000-2005.
I presume that honest researchers are doing that. Anyone of them care to comment?
So being "insufficiently expert", and instead of asking Ms Cahill herself for any supporting information, or maybe contacting your local university to see if any statisticians are able to explain things in more depth; you read/link to an article on one of the most inept, anti-science websites on the internet, where a dodgy bloke performs some dodgy "analysis" that completely ignores any underlying physics; then tops it all off with some dodgy curve fitting baloney, and tells us that ONE "cycle" of unknown physical origin amounts to long term cyclical behaviour!
If they were asked to peg their salaries to either the "pause" data or the "pre-pause" data, then we would certainly see a mass outbreak of statistically significant honesty.
Time and again I hear nonsense from people saying climate signals are, or are not, "significant".
Unfortunately, this is a meaningless concept unless you have an understanding of the type of variation present.
And it is particularly meaningless applying the standards tests, intended for when the variation is independent from measurement to measurement. We know we have long periods of warm (medieval warming period) and long periods of cold (little ice-age) so right away we can categorically rule out the standard statistical tests. You just cannot use them.
Or to put it in simple terms: unless you understand what is "normal" you cannot know what is "abnormal".
I've got a few articles on the subject: Lies, damned lies, and statistical significance of climate trends
I've got a demo you can play with to start understanding the type of noise and why it doesn't fit standard statistics at uburns.com and some info on the generator at: About uburns.com
I've got some results showing how variance increases: Statistics of 1/f noise – implications for climate forecast
And I've even collected together a few examples of how 1/f noise causes errors in interpreting data even by sceptics: Natural habitats of 1/f noise errors.
I am with Quentin. Why any of you take information from Real Climate beats me too.
RC won't and can't improve as long as they run after more and more ex-post-facto excuses and data manipulations in order to keep up an appearance of being always right.
Had Rahmstorf made any of his arguments before the "pause", or explained years ago why change point analysis was going to be thing to do, or or or, his opining now would carry some weight. As it stands it's like open fishing season on the literature looking for something, anything to be the Deus Ex Machina that resolves every problem with the touch of a single article.
Just to clarify something which is important. Above I've explained why we cannot make any assertions about statistical significance of the real climate. So how can we say there is a "pause"? The answer is that the pause isn't really a pause in warming, but a discrepancy between the projected warming and actual data. So, the test for the pause is very very simple: is the global temperature lower than the range the IPCC gave for it. The IPCC didn't provide a statistical model but instead gave a very simple range which was 0.14 to 0.58C/decade.
So, the only test that matters is whether the current temperature is within the range given by the IPCC.
No other test is meaningful.
You don't suppose that those highly paid and experienced people at Real Climate are losing their blind faith in the all powerful CO2 molecule?
Just because none of their predictions became true, is no reason why they should not continue to enrich themselves at everyone else's expense.
As I am looking at the sea surface temp data from the 1900s to 1950s (bucket bias stuff - on a discussion thread) it might be worth noting that the Met Office noted a discontinuity around 1940 due to what is believed to be a switch to engine inlet temp measurements rather than bucket hauling. There was a war on after all. Some corrections were made to the temp anomaly record but it may still have a residual discontinuity. I wonder has this been taken into account ?
Micky: "As I am looking at the sea surface temp data from the 1900s to 1950s (bucket bias stuff - on a discussion thread)"
Steve Mc did a few posts on the subject a few years back which are probably worth checking on.
I couldn't find anything I'd written which was an introduction to 1/f noise and the climate to go along with the links, so I've quickly written one: Introduction to 1/f climate noise
The conclusion is also relevant: "Therefore, given that the 20th century changes are very typical within the temperature record, this means that if we were only looking at these temperature records, then we would be forced to conclude that the whole 20th century change is most likely due to natural variation."
Rahmstorf appears to be someone who is easily entranced by new shiny math thingys that he has no understanding of,
""Unfortunately, the analysis is of no value because, as is commonly known, the CPA cannot be used on auto-regressive time series." I am insufficiently expert to be able to say who is right here.[...] Do any readers here know the answer?"
It depends on what statistical model it uses. The standard basic change point analysis found in undergraduate text books assumes a non-autocorrelated model, and is therefore invalid if the data is autocorrelated.
However, it is entirely possible to construct a change point analysis test assuming an autoregressive model - indeed, Ross McKittrick did this in his paper discussed here.
Ross said: "Fitting a linear trend through a series with a positive step-change in the middle will bias the slope coefficient upwards. When I asked Tim if the VF method could be used in an application allowing for a suspected mean shift, he said no, it would require derivation of a new asymptotic distribution and critical values, taking into account the possibility of known or unknown break points. He agreed to take on the theoretical work and we began collaborating on the paper. Much of the paper is taken up with deriving the methodology and establishing its validity. For readers who skip that part and wonder why it is even necessary, the answer is that in serious empirical disciplines, that’s what you are expected to do to establish the validity of novel statistical tools before applying them and drawing inferences. Our paper provides a trend estimator and test statistic based on standard errors that are valid in the presence of serial correlation of any form up to but not including unit roots, that do not require the user to choose tuning parameters such as bandwidths and lag lengths, and that are robust to the possible presence of a shift term at a known or unknown break point."
I gather from reading WUWT that the RealClimate analysis used the standard algorithm with the non-autocorrelated model, rather than anything particularly clever. But I haven't checked, so I recommend that anyone interested (who understands what the hell I'm talking about!) should do so. I'm not sufficiently curious to look. However, it's not generally true that CPA cannot do autocorrelated series, it can be modified to do so.
By the way, I'm also not endorsing Ross's approach as unarguably the right method either. All statistical significance testing relies on assuming a statistical model, for which you need externally-sourced, independent reasons for thinking is a sufficiently good approximation to the physics. You can't do it just by looking at the data, and while Ross's approach is a lot more plausible, I'm not convinced that his model can be validated from the physics either. This is the same issue that Doug Keenan raised.
To say anything at all about the significance or non-significance of anything to do with the global temperature record, you need to have a validated statistical model of the normal background variation, preferably founded in the physics, and we simply don't have one. For that reason, the RealClimate post is just yet more hot air.
"a little visited website called Real Climate"
Ouch!
Micky / DaveJR - I came across Steve's page on this a while ago, here's the link:
sea temperature adjustments - http://climateaudit.org/2007/03/17/buckets-and-engines/
Lapogus
I've read these in the past and at the time. Shows how long I've been reading about this stuff.
My take on the bucket adjustments especially in Folland 1995 is a bit more fundamental. Or at least in the sense that I'm a bit confused on the approach taken once a model of buckets has been validated to some extent. I'll update my discussion thread later.
I thought the magic number for changepoint years would have been "17" ... Or else is that now disavowed as the key line for understanding a trend that moves beyond the bounds of the expected?
I bet if that same trendline graph is run with 17-year lines instead of 30, there may well be a recent changepoint. I do see it reasonable that 30 can be chosen, but also 17, based on climate scientists' own words.
I really don't care how close statistical correlations track reality. I care that climate scientists understand causality well enough to predict future results and try to do so while continually trying to falsify their own predictions. Alarmist climate science has given itself a free pass to ignore this type of rigor. Rather than trying to disprove their hypotheses, they try to prove them. This malpractice is written right into the charter of the IPCC. And with so much data available, with a little careful cherry picking, or tendentious modeling, one can find "evidence" for any kind of hypothesis.
I'm a little late to this discussion, but I saw Jeff Patterson's post when it first went up, and it is terrible. Patterson has no idea what he's talking about. I commented on his post to point this out when it first went up. I won't quote the whole comment, but my conclusion was:
To make things worse, Patterson responded to me by saying things which were completely untrue, and, quite frankly, were ridiculous. It's sad such mind-numbingly bad criticisms can get posts on WUWT.
For those who are just interested in this quote:
Know it is completely wrong. There are many different ways to try to detect change points. Some cannot handle autocorrelation. Some can. Patterson's claim it "is commonly known" none can is a sign he has absolutely no idea what he's talking about.