Mark
You cannot criticise (in depth) something you haven't read, and expect to be taken seriously.

I quoted from a text that I have read. A (maybe the) concept of the HSI is a proposed conspiracy to get rid of the MWP using a reconstruction starting after the MWP has ended. If you can give a plausible explanation for why such a conjecture is credible, I’ll endeavor to find a full copy and read it.

I’m intrigued to know how you judge me to be losing hands down, seeing as you don’t claim expertise in what is being discussed. Looking at my last post, it is clear that replacing missing values is common (google “replacing missing time series data” and then tell me that it is not), NiV’s 5 point example was not sensible, and he is confidently quoting stuff about central England summer temperatures using graphs of the wrong data set. And the obsession with the HS is clearly with skeptics; nobody else cares about it, probably not even Mann.

——

NiV, did you look at how common it is to replace missing data? You said it was unequivocally an error, and yet it is so common there is an R package for it. Does that not make you wonder at what people have been telling you?

I don’t know what I am supposed to make of your new series. Apart from oscillating between -0.1 and 0.1 without ever hitting zero, the only thing that seems clear is that it contains two probably bad values at the end of series 3. Anyone interpolating the first value of series 3 would use 0.1 (or -0.1, but as two consecutive values are never -0.1, that seems a bad choice. 0 might be a better choice but it never appears in the data set so probably not). To replace the final missing value one would need to know what the data represents. If it is physically plausible that the signal rises nearly 2 orders of magnitude in 2 samples then one might estimate 10, 20, whatever. If not the 0.1 might be better. Whatever is estimated has no effect on the earlier part of the dataset.

Now, I've "reconstructed" temperatures from random noise - and it looks pretty good!

In what way does it look good? To be pedantic, it is not random noise, but, I think, high persistence (0.9) red noise like Wegman/McIntyre used. But even so, all you’ve done is make a pretty pattern that is roughly stable for 400 samples then dives to -50 and rises to +50 over the next 100 samples. And that big-dipper corresponds to the ramp you used to correlate. What that means, if anything, I don’t know, but it seems to have nothing obvious to do with temperature. And notice that the shaft of this curve is flat - and it is the shaft that is contentious as everyone knows 20th C temps have risen.

So, are you really willing to tell me that I've just made "choices" processing "imperfect" data and this is all perfectly legitimate and scientific…

You don’t have any data, perfect or imperfect, so the question is moot. In your simulation I expect you could replace points 495-500 of a quarter of the 100 traces with the value of point 494 and have little effect on the lower part (points 1-400) of the result. Your choices are of course perfectly legitimate, since you are just playing with R for fun trying to make a hockey stick. But I doubt that the computation does anything that has physical significance in relation to processing proxy data. Do you think it does?

…or is it actually an error to claim this is a valid method for reconstructing global temperatures back in 1500?

The ramp you generated is arbitrary or are you suggesting that Mann has a similar ramp in his calculations?

Raff

Your continued refusal to read the book is perverse and baffling. I'm not sure what you have read (because you haven't made it sufficiently clear, to me at least - an online version via kindle of the introduction, or someone else's summary?).

I'm not sure whether it was you or NiV who started this discussion thread originally, about the hockey stick, but you seem determined not to let it go, and to defend it to the death. That's fairly strange behaviour for someone who claims: "And the obsession with the HS is clearly with skeptics; nobody else cares about it, probably not even Mann.".

Given your stubborn refusal to read the book (an alarming tendency on the part of alarmists is a refusal to read things that disagree with their preconceptions, while apparently feeling free to criticise what they haven't read) I give up. I still respect the sincerity of your views and the vigour with which you argue your point, and I will continue to observe the debate. But you have lost me on this one. I cannot accept that your position here is rational, and ultimately perhaps that is the difference between alarmists and sceptics. :-) Have a nice day.

Mark, I quoted from the actual book. Go to the Amazon page for the HSI and just above the picture of the front cover are the words "Look inside" and an arrow. Click the words and you get a popup window that shows the two prefaces, Notes on Usages, the entire Chapter 1 with the Lamb diagram and the suggestion of conspiracy to get rid of the MWP using a reconstruction that starts after the period had ended, and part of Chapter 2, down to fig 2.3. If you, anyone else, or even the author can give a plausible explanation for why that conspiracy conjecture, which seems to be the theme of the book, is credible, I will endeavor to find a full copy and read it. But if the underlying theme is unsupportable, why would anyone?

Raff and His Selective Reading:

It is quite clear from the context that Montford is talking about the start of a process that was completed the following year when Mann extended his study back 400 years in time.

This whole discussion is very funny. You can imagine Raff defending a guy who was seen stabbing a man with a vegetable knife by 5 witnesses and CCTV.

1) Ignore the witnesses, they are shills for Big Oil
2) The CCTV is usually switched off at 5 o'clock. Since this alleged crime took place at 5.30, the CCTV cannot have been operating and the images are fake.
3) The knife came in a package that says it is to be used for peeling and cutting vegetables. Therefore it cannot have been used to stab the victim.
4) The finger prints could have been applied to the handle at any time.

etc etc ad nauseam

.
5) The premise of the prosecution is that something bad happened. It is not a disinterested analysis.
6) The murder happened five years ago
7) The police, the CPS and the prosecution have an obsession about my client.

diogenes, the book went on:

Enter the Hockey Stick.

The paper
The Hockey Stick paper made its grand entrance in an article published in Nature on 23 April 1998.

Note the singular tense. Not the "first HS paper", but "the paper", meaning to me that the paper was indeed the "major study, presenting unimpeachable evidence that the Medieval Warm Period was a chimera". MBH98 seems, in Montford's telling, to be the paper to destroy the MWP. He does mention MBH99 later, in a way that implies it came about because of the success of the first, not because of the initial conspiracy. And he makes clear how tentative the conclusion of the 99 paper was given very little extra data before 1400. It is a strange conspiracy that publishes a paper that doesn't do what it aims to achieve and then adds a tentative extra bit because the first was well received. But I can see you might prefer to view it that way. And the cover of the book shows the 99 graph, supporting your view.

Martin, thanks for the offer. I'll bear it in mind.

"A (maybe the) concept of the HSI is a proposed conspiracy to get rid of the MWP using a reconstruction starting after the MWP has ended. If you can give a plausible explanation for why such a conjecture is credible, I’ll endeavor to find a full copy and read it."

1) The HSI explains this - the first IPCC report discussed the MWP in some detail, giving the Lamb graph you mention. The next report removed that discussion and replaced it with the new narrative: flat back to 1400, and no reliable data prior to that. This was, as HSI put it, "the first step".

2) Much of the message conveyed by the graph was in the *flatness* of the graph from 1400-1900. Even without going back before 1400, the message was that climate didn't normally vary much, and the modern rise therefore represents a sharply defined and distinctive change in behaviour.
Had a lumpier-bumpier line been apparent back to 1400, it would have been less 'obvious' that the modern rise was unusual, or that earlier peaks would be out of character.

3) MBH98 was followed by MBH99 which goes back to 1000 AD.

4) While we climate junkies know about the MWP, the politicians and public didn't. The new message was "flat as far back as the data goes" - the general public would not know that it didn't contradict the MWP (being too late) because nobody was telling them about the MWP any more. It's not what you say, it's what you don't say. In politics, spin and presentation are everything.

"And the obsession with the HS is clearly with skeptics; nobody else cares about it, probably not even Mann."

Jolly good. Get Mann and the IPCC and all their supporters to admit in public it's wrong and they knew it all along, and we can all move on then.

If nobody else cares about it, then why not?

"NiV, did you look at how common it is to replace missing data? You said it was unequivocally an error, and yet it is so common there is an R package for it."

It's a valid technique for some purposes and not for others. When the accountant tells you that adding is wrong, you need to subtract, it's no argument to point to all the places in accountancy where addition is done. Likewise, just because there's a software package for it doesn't mean the technique is valid to apply. There are a lot of people get taught in basic university courses how to press the buttons to apply a method and get an answer, but it's rarer for them to be taught how to check that the conditions and assumptions the method is based on apply.

Interpolation is based on assumptions regarding the smoothness of the function - or equivalently, the fall-off of the high-frequency part of the Fourier spectrum. If the value is strongly and reliably correlated from year to year - and you've checked and shown it! - then you can deduce partial information about one year's temperature from neighbouring years.

Mann didn't even check, and for temperature data it isn't true. The year-to-year variation in temperatures is many times larger than the signal you're looking for, and while correlated, the autocorrelation year-to-year in global temperatures is about 70% (IIRC) and so will fall off to about 15% in 5 years. The autocorrelation in temperature at an individual location will be even worse. And the relationship between temperatures and tree rings makes the connection even murkier. That's why Mann had to do all that processing of hundreds of data series to extract a signal in the first place.

"I don’t know what I am supposed to make of your new series. Apart from oscillating between -0.1 and 0.1 without ever hitting zero, the only thing that seems clear is that it contains two probably bad values at the end of series 3."

Hooray! That's exactly the right conclusion!

"If it is physically plausible that the signal rises nearly 2 orders of magnitude in 2 samples then one might estimate 10, 20, whatever."

And you put your finger precisely on the issue!

What method should we use to estimate the last value? Do we extrapolate linearly, (or quadratically? exponentially?), and report a big number? Do we assume a constant value and report the same number? Do we assume the two outlying values were an error, and revert to the 0.1s we see in the rest of the series? Without some understanding of how neighbouring values are related, and indeed, what mechanisms underlie the generation of the values, we can't say.

"Whatever is estimated has no effect on the earlier part of the dataset."

It has an effect on the early part of the dataset, in that it further dilutes it with more random noise.

Once you can understand that it's not simply a matter of changing 5 years, that adding those five years enables you to incorporate another five hundred values into your mixing pot, you need to know about the other part of the recipe, which is the weighting. You don't simply average all the series with equal weights, you give a higher rating - up to 390 times higher in the case of MBH98 - to those series that correlate with the post-1900 rise in global temperature. So you give series 3 up to 390 times the weight than series 1 or 2. The result is now almost entirely series 3, with all the other data zeroed out.

So now do you see why it makes a difference?

"To be pedantic, it is not random noise, but, I think, high persistence (0.9) red noise like Wegman/McIntyre used."

To be pedantic, red noise is still "random". What it's not is "independent". That's mathematically a different property.

" What that means, if anything, I don’t know, but it seems to have nothing obvious to do with temperature. And notice that the shaft of this curve is flat - and it is the shaft that is contentious as everyone knows 20th C temps have risen."

Give the man a prize! Yes! Yes! Yes!

I've taken a bunch of data series that have nothing to do with temperature, fed them through a complicated-looking algorithm, and come up with something that matches observed temperature during the calibration period where I forced it to match and has no relationship whatsoever to temperature anywhere else, which is the bit we're interested in. So if I claim that the flat bit at the front *is* the temperature, proving the rise at the end is "unprecedented", why do you believe me?

This is how paleoclimate reconstructions work. That's what we're complaining about.

"The ramp you generated is arbitrary or are you suggesting that Mann has a similar ramp in his calculations?"

Mann used the observed temperatures post-1901. (And used a comparison of the result with the observed values 1854-1900 to check that it was working.) I only used a ramp because I couldn't be bothered to type them all in, and this demonstrates the effect more cleanly and simply. Feel free to type the real temperatures in yourself. The effect will be the same.

Raff you read at least 10 pages and ythat was the only error you found? And it was not really an error, but rather a misstatement because Montford did actually write about the reconstruction starting from 1400. That was the only "error" you found (except I think you borrowed from the singularly moronic Frank o'Dwyer, but you are not honest enough to admit it). It is an error that you can get through within a page of writing. Unblike the gross mathematical errors of Mann. Do you possess a sense of scale, Raff? What is bad...what is horrible?

the singular tense....what does that term mean?

still dancing on your high heels, Raff? I think you have no idea about argumentation.

NiV, I don't know why you find it plausible that Climate Science would pick an unknown to do a groundbreaking study with the explicit aim of getting rid of the MWP and allow him to go only to 1400. It is daft. And of the fact that the curve is flat, have you considered the possibility that globally temperatures were indeed fairly flat back to 1400 (or 1000)?

You seem at least to accept that infilling can be a valid procedure and is not an automatic error. It is still not clear to me why you think that infilling those 25% with half a dozen values at the end makes a difference. You didn't answer what you thought would happen if the last 6 values of 25% of your 500 simulated data were to be replaced by the 7th last value. If you think doing so is such an error, you must think something would happen to the result.

Your example with two false values in series 3 is rather unrepresentative and difficult to take seriously. Clearly you are just using a toy example to explain something more complicated but it seems to have no relation to what you'd like to explain. The real data is mostly noise with little clear signal whereas your example is the opposite. If these end effects make such a difference it should be possible to chop off the last 10 years and get a totally different result. That this apparently obvious step wasn't publicly done by sceptics is suggestive to me that it makes no difference. It is stated in the HS Delusion that Mann did this and got the same result.

I'm having trouble with your correlation and the implication that this is what Mann did. Your ramp is artificial, added for the purpose of generating an HS, or more accurately a zigzag. There's no artificial data in MBH98 so any correlation is from within the dataset and hence is perhaps (or perhaps not) a real signal.

> There's no artificial data in MBH98 ...

Well clearly there are infilled values but I discussed those already.

he means a singular noun, but he is just deluding himself because NIV put him straight. I wonder if the fool will reappear. With another bizarre concocted excuse why he has no data to support his position.

I recall that Raff was talking abouyt how "sceptics" continually change the point of debate...obviously without giving any evidence. And now he is circling around the fact that his sources of information are inadequate.Like all the other warmists we meet here, he depends on the SS site and it always fails to stand scrutiny.

diogenes, raff is part of the wider SKS community. There is some variation in the tone and style of his responses, indicating input from a variety of SKS retained 'experts'.

HSI is now under sustained attack. Whether there is a link to long awaited US court appearances, and possible lines of legal argument, I wouldn't know.

Diogenes, Golf Caddy perhaps you can explain why Montford's conspiracy theory is plausible - namely that Climate Science would pick an unknown to do a groundbreaking study with the explicit aim of getting rid of the MWP and allow him to go only to 1400. As conspiracies goes, isn't that daft?

Raff, name calling again? Gosh, ooh-err. Petulance suits you.

Don't you need to explain and justify Mann's conspiracy, why so many people provided assistance, and why it bothers you all so much now? You could always try reading Hockey Stick Illusion yourself. There is a clue in the title.

Weren't you insisting that climate science has moved on from Mann's failure?

Anyone would have thought there was a sense of alarm in the Global Warming Alarmist camp.

"NiV, I don't know why you find it plausible that Climate Science would pick an unknown to do a groundbreaking study with the explicit aim of getting rid of the MWP and allow him to go only to 1400."

Conspiracy theory? Tut. I'd imagine he chose himself.

Bradley and Hughes weren't unknowns. That they get students to do all the work while they get the glory is SOP in academia.

"And of the fact that the curve is flat, have you considered the possibility that globally temperatures were indeed fairly flat back to 1400 (or 1000)?"

Yes. Have you considered the possibility it's not?

Either way, bogus methods can't tell us the answer.

"You seem at least to accept that infilling can be a valid procedure and is not an automatic error. It is still not clear to me why you think that infilling those 25% with half a dozen values at the end makes a difference."

Because it allows the inclusion of corrupted data series like Gaspe, their end spikes causing them to be over-weighted.

"You didn't answer what you thought would happen if the last 6 values of 25% of your 500 simulated data were to be replaced by the 7th last value. If you think doing so is such an error, you must think something would happen to the result."

Yes. It unflattens the hockeystick handle. I already told you that.

"Your example with two false values in series 3 is rather unrepresentative and difficult to take seriously."

Montford takes half a book to explain all the complexities. You want me to explain it all in a couple of blog comments? To someone being deliberately obtuse? I'm flattered at your faith in my powers of explanation!

"Clearly you are just using a toy example to explain something more complicated but it seems to have no relation to what you'd like to explain."

Yes. So read the book.

" If these end effects make such a difference it should be possible to chop off the last 10 years and get a totally different result. That this apparently obvious step wasn't publicly done by sceptics is suggestive to me that it makes no difference. It is stated in the HS Delusion that Mann did this and got the same result."

I already told you. Mann did the experiment in his "BACKTO_1400-CENSORED" directory. The hockeystick disappeared. And by your own dubious logic, the fact that he didn't publish this is suggestive that it does make a difference.

I find this particular comment fascinating. What gave you the idea that sceptics haven't done this? Have you even looked?

"I'm having trouble with your correlation and the implication that this is what Mann did."

Why? What do you think he did?

McIntyre gives the maths to prove it, if you're interested.

Raff

I'd stopped commenting on this thread, partly because it was your thread and largely a discussion between you and NiV and I felt that I was gatecrashing; and partly because in view of your stubborn refusal to read the HSI while continuing to criticise it, I felt that the point on which I was commenting was one I could take no further.

Since then, however, I have been thinking. I'm sure we'll all agree that the debate between sceptics and alarmists is hampered by the fact that we've arrived at our conclusions, erected the barricades, and seek to defend our respective positions, seeing things exclusively from our point of view (I could argue that sceptics are more open to free thinking, but won't as it derails the point the point I wish to make).

My point is this. You have read maybe 5 or 10% (at best) of the HSI online, and have concluded that you have seen enough to decide that it's a load of rubbish, you don't need to read on, and can criticise it in its entirety without troubling to read it.

You might note that I have never (so far as I can recall) criticised MBH 98/99 directly, although I have read (and found persuasive) others' criticisms of MBH 98/99. The reasons I do not engage in a critique myself are because I am neither a scientist nor a statistician, and because I haven't read them. Imagine how you would react if I started posting comments saying that I had read the first small section of MBH 98/99 online, as a result had concluded that it was a load of rubbish such that I didn't need read further, but nevertheless going on to launch a determined criticism of it. How would you respond to me if I did that? I imagine you would go for my jugular, and with full justification too. Yet you seem to see nothing wrong in the analogous position you have taken on the HSI.

As I have said before, I respect the courtesy you normally show, I respect your intelligence, and I welcome the contribution you make on this site. However, I am completely baffled by your blind spot and double standards on this issue.

For those wondering what this is all about, 'The Hockey Stick Illusion' by AW Montford, is available, see links on this blog.

It is an excellent read, and very informative about the devious nature of climate science. I don't think 'Raff' would have forecast or projected giving it so much publicity, but predictions in climate science have never been reliable.

NiV, the conspiracy is Montford’s invention:

Climate science wanted big funding and big political action and that was going to require definitive evidence. In order to strengthen the arguments for the current warming being unprecedented, there was going to have to be a major study, presenting unimpeachable evidence that the Medieval Warm Period was a chimera.
Enter the Hockey Stick.

And the stick went back to 1400, after the end of the MWP. It is daft but it is the apparent theme of the HSI. You and everyone here seem to buy into that basic delusion. In fact buying into that delusion seems to underly a lot of such “skepticism”. Why do you find this conspiracy theory plausible? I expect neither you nor Mark will answer that directly because although it is clearly implausible, rejecting it means reexamining everything else that relies on it.

Yes. Have you considered the possibility it's not?

Of course. There is certain to have been variation. Everyone knows that. Paleoclimatology is all about studying such variations. The Earth might have been as warm as it is now 1000 years ago. But if there is good evidence of that, I haven’t seen it. Have you?

I still don’t understand what is wrong with discarding the last values to align all data to the same length. You say the stick disappears, but if we do this with your 4-line example, which I presume you think is valid, keep the same simulated data and just restrict to 494 points instead of 500:


> x = replicate(100,arima.sim(list(ar=0.9),500)) # Line 1!
> temps = seq(0,0.99,0.01) # Line 2!
> correl = apply(x,2,function(s){cor(s[401:500],temps)}) # Line 3!
> plot(as.vector(correl %*% t(x)),type="l") # Line 4!
>
> y <- x[c(1:494),]
> correly = apply(y,2,function(s){cor(s[401:494],temps[1:94])})
> lines(as.vector(correly %*% t(y)),col="green")


And there is no significant difference in the plot. Yet you suggest that is effectively what Mann did.

My point is this. You have read maybe 5 or 10% (at best) of the HSI online, and have concluded that you have seen enough to decide that it's a load of rubbish, you don't need to read on, and can criticise it in its entirety without troubling to read it.

I haven’t said it is a load of rubbish or criticized its entirity. But it has that underlying nutty theme of conspiracy. Do you accept that theme is sensible as discussed above. As I stated it above, it is clearly not sensible, yet you are prepared to accept/ignore that and assume that the rest of the book is free from such distortion, despite your (and my) not being in a position to understand where the distortion lies. Maybe it is like other fiction, you just need to suspend disbelief and enjoy the story. But most people reading fiction know all the same that it is not true.

Raff, you mean fiction like Mann's Holy Hockey Stick that you and thousands of others do 'believe', and dare not criticise, even though it is trash? (as aTTP would put it)

The interesting thing is that all subsequent temperature reconstructions, which are said by the cult faithful to verify the hockey stick, all show much greater variability than the hockey stick, which seems to destroy Raff's suggestion that temperatures might have been fairly flat in the mediaeval period. However, of course the begs the question of whether these reconstructions are actually producing temperature results rather than just some vague proxy of climatic conditions.

But I note that he has now changed his point of attack to a conspiracy theory one in which I have no interest. I would suggest he read the climategate emails but that would probably be unethical for a cult follower.