On request, this is a continuation of the conversation between NiV and me that we so rudely imposed on the "Theory, law or fact" thread. Unless NiV has run out of steam :-) Others are welcome too, of course.

This is a copy of my last comment on the other thread:
------------

NiV, as nothing becomes automatically true just by being published, and everyone knows this, there is no danger in articles not being the subject of comments. They just won’t get cited and will fade away.

I remember reading that McI’s comment to Nature had the stats wrong in some way, so that is grounds for rejection. How could a criticism be uninteresting? Well complaining about a 4 year extrapolation in a 600 year record would qualify, just as an example.

Still citing Mann years after? If McI had done a proper reconstruction, as you said he explicitly chose not to, people might have been citing him instead.

I have no idea whether you are competent to check the MBH98 analysis. I am certainly not. But even if you can and did, you have no way of knowing who did or said what, to whom, when or why, which is what a lot of debate revolves around. That is why it is 2nd hand.

Why do I believe what climate scientists say? Do I? I was talking about the HS controversy (“I have read around enough…”) and I don’t know who is right. It wouldn’t surprise me at all that the paper had its flaws. Neither am I convinced that tree rings faithfully tell the story of past temperatures (although if the trees are at the tree line, there is likely to be some information there). But I also have no reason to mistrust MB&H, to think they were dishonest or incompetent or whatever is thrown at them (Mann mainly it seems).

”… in particular the Lebesgue norms. He showed that L1, L2, and L4 gave completely different apparent behaviour, some increasing and others decreasing when applied to the same temperature distribution. The radiative cooling can't be calculated that way,…”

BS! They say, “R4 would appear in connection with black body radiation.” Then later they say:

“For the radiation example we essentially measure {Ti^4}, ignoring constants of the classical black body law. To get temperature we must transform the measurements by taking the 1/4 power.”

And their figure 1 R4 curve starts at about 28 ºC which, quite by coincidence I suppose, is the 4th root of (33^4 + 2^4)/2 where 33 and 2 are the temperatures of their coffee and ice water. They used ºC.

Nice try but now you have me wondering about your objectivity.

But when the output turned out to be uncorrelated with the thing it was supposed to be reconstructing, the attempt ought to have been recognised as unsuccessful.

You say the output was uncorrelated with an unknown? That is a clever trick.

That they [error bars] should be several times bigger?

Where are McI’s error bars (or yours, as you repeated the analysis) for comparison?

As far as retaining PC4, I’ve read that in a standard analysis, which McI claims to want, it should be retained. I’ve read before (maybe you said it) that only PC1/2 were retained because that is what Mann did. Now you want to say that there was some other justification. It seems like special pleading just to avoid retaining a HS.

From earlier:
…feeding random data (trendless red noise series) into such an algorithm almost always produces hockeysticks

If it generated a HS “almost always” McI could have plotted 100 random curves. Instead he needed to filter out only the 100 of 10000 that looked like a hockey stick. Again, I call BS.

And what if something bad was going on? Have you closed your mind so thoroughly to the possibility that you can't even bear to read someone talking about it?

It is more likely that nothing bad was going on. My impression was that the HSI didn’t consider that (it would hardly sell many copies if it assumed and reached that conclusion).

"On request, this is a continuation of the conversation between NiV and me that we so rudely imposed on the "Theory, law or fact" thread."

Good idea!

"NiV, as nothing becomes automatically true just by being published, and everyone knows this, there is no danger in articles not being the subject of comments. They just won’t get cited and will fade away."

Why would it not get cited? Hardly any of the journal's readers would know of the refutation. They'd see the published paper, see no disagreement, and assume that's because there wasn't any. The longer the paper existed with no counter-arguments, the more confidence they would have in its correctness. The entire process of publishing scientific papers relies on the assumption that any refutations will get published.

And by the way, I've come cross a lot of people who don't know that peer-reviewed publication isn't proof of truth - judging by the number who use it as an argument from authority. Scientists know, but journalists and members of the public don't.

"I remember reading that McI’s comment to Nature had the stats wrong in some way, so that is grounds for rejection."

Mann and his associates said so, but then they would, wouldn't they?

Nature's rejection letter (which McIntyre published) said it was because there was a 500-word one-figure limit, and M&M's detailed critique couldn't be condensed into such a space. Maybe they were just being polite?

"How could a criticism be uninteresting? Well complaining about a 4 year extrapolation in a 600 year record would qualify, just as an example."

The extrapolations are significant because it allowed Mann to include particular series in steps that they didn't meet the criteria for, not having complete data. This materially affected the results. It also made it impossible to reproduce the published results, since downloading the original source datasets and following the instructions as published would have resulted in them being excluded, giving different results to Mann's.

You might not find replicability or its lack to be interesting, but scientists do.

"If McI had done a proper reconstruction, as you said he explicitly chose not to, people might have been citing him instead."

His position is that it's not possible. So how could he comply?

"I have no idea whether you are competent to check the MBH98 analysis. I am certainly not."

I'm not asking you to trust me. I'm against that on principle. (Hence my pseudonym.) As a scientist, I checked the results for the reason Tom Wigley described so eloquently. Everyone has to find their own solution.

"But even if you can and did, you have no way of knowing who did or said what, to whom, when or why, which is what a lot of debate revolves around. That is why it is 2nd hand."

Sure. It could all be an elaborate hoax, with *both* sides of the dispute authored by the same pranksters. Does Michael Mann even exist? I only have other people's word for it. Should we believe what the newspapers or TV say? Should we believe what the newspapers say climate scientists say? Where do you stop?

"and I don’t know who is right. It wouldn’t surprise me at all that the paper had its flaws. Neither am I convinced that tree rings faithfully tell the story of past temperatures"

That's good enough for me.

"BS! They say, “R4 would appear in connection with black body radiation.”"

And so it would - as I already said.

You're confusing two separate issues - one is the mathematical property of the Lebesgue norms that they don't put sets of numbers in the same order, a point that can be demonstrated with any set of numbers, in any units. They're just talking about "averages" here, not any physics - as is made clear by the appearance of R{2}, R{1}, and R{-1} in the same diagram. The other is a potential application of the R{4} norm to black body radiation - something they don't actually do or use in any of their calculations. It's mentioned as an aside, to illustrate how other choices of average than the familiar L1 might be physically appropriate.

"Then later they say: "For the radiation example we essentially measure...""

This refers to the previous paragraph: "In examples such as kinetic speed or radiation energy as raw values, an average over derived kinetic energy or derived temperature would be expressed in terms of a power law." It doesn't refer to the coffee cup example you're talking about.

"And their figure 1 R4 curve starts at about 28 ºC which, quite by coincidence I suppose, is the 4th root of (33^4 + 2^4)/2 where 33 and 2 are the temperatures of their coffee and ice water. They used ºC."

They did indeed, but they were just demonstrating averages, not calculating radiation.

(BTW, the SI unit is just C, not ºC. But that's just minor pedantry.)

"Nice try but now you have me wondering about your objectivity."

That's OK. I'd expect you to. It's my task to argue well enough that it doesn't matter.

"You say the output was uncorrelated with an unknown? That is a clever trick."

That's the problem with you not reading HSI! If you had, you'd know how it was done.

If you'd read p67-68 of the book, you'd know that Mann used the instrumental temperature data from 1901 onwards to calibrate the ring width to temperature conversion, and reserved the instrumental temperatures from 1854 to 1901 for validation. The problem with these sorts of methods is that a fit to temperature during the calibration period is automatic, whether the results are real or not, because the algorithm is deliberately matching those bits of the ring width histories to the actual temperatures. The test is to see if it still matches for data that the algorithm hasn't seen.

Anyway, as described on p216 of HSI, the correlations found were numbers like 0.018, 0.010, 0.006, 0.004, 0.00003, 0.013, 0.156, 0.050, 0.122, 0.154, 0.189, for the different stages of the reconstruction from earliest to latest. Mann mentioned the last of these in the paper itself, so we know he calculated them. But the critical one is the earliest one - the one supposedly showing that the Medieval Warm Period hadn't happened, and it's correlation with temperatures from 1854-1901 was less than 2%. In other words, whatever it's reconstructing, it's not the global temperature.

"Where are McI’s error bars (or yours, as you repeated the analysis) for comparison?"

The parable of the Emperor of China's nose is applicable here.

A statistician, so the story goes, wanted to know the length of the Emperor of China's nose - but the Emperor lived in the Forbidden Palace and nobody outside had ever seen him, or any portraits or pictures. So what he did was to ask a hundred million Chinese peasants to estimate the length of the Emperor's nose - just have a guess. Then he reasoned thus: the standard deviation of the estimates was a couple of centimetres (all noses are pretty much the same), and everyone knows that the accuracy of an estimate improves in proportion to the square root of the sample size. Hence, he could average the 100,000,000 numbers, and draw error bars on the result of about 0.00002 centimetres. That's pretty accurate! Especially given that none of the people asked had ever seen the Emperor!

If you can figure out what the statistician did wrong, then you'll be able to see for yourself why the error bars should be floor-to-ceiling on the hockeystick graph.

"As far as retaining PC4, I’ve read that in a standard analysis, which McI claims to want, it should be retained."

No doubt written by Mann and his associates again. However, as I've already explained, it shouldn't because the data in it was already known to be corrupted and unrelated to temperature. How can it possibly be correct to retain corrupted data?

"I’ve read before (maybe you said it) that only PC1/2 were retained because that is what Mann did."

Yes, I said that earlier. The point is there are two separate errors here. One is including the bristlecones in the first place, when they're known not to be temperature proxies. The other was the short-centering error in Mann's implementation of the PCA algorithm that incorrectly promoted a minor PC4 pattern to the dominant PC1. McIntyre showed the effect on the PC1, to illustrate the impact of the latter error. But even correcting the latter error, it's still wrong because of the first error.

Go read HSI for the detailed explanation.

"If it generated a HS “almost always” McI could have plotted 100 random curves. Instead he needed to filter out only the 100 of 10000 that looked like a hockey stick. Again, I call BS."

99% of the series generated were hockeysticks, so yes, he could have picked 100 at random. He happened to pick the top 100, but didn't have to.

"It is more likely that nothing bad was going on. My impression was that the HSI didn’t consider that (it would hardly sell many copies if it assumed and reached that conclusion)."

It was already known that something bad was going on. All the arguments had been gone through in gory mathematical detail on the climateaudit site, and the case proven conclusively. But the argument was scattered over hundreds of extremely technical posts, that average citizens quickly got lost in, so Montford provided a layman's historical summary and explanation of the jargon and technicalities so that non-mathematicians could hope to understand.

It would be a bit like complaining that a historical non-technical recounting of the Enron collapse of 2001 "assumed that something bad was going on" just to boost book sales, and was probably all perfectly innocent.

Assuming you want to keep an open mind, just think of it as the case for the prosecution, and then go help build the case for the defence. It's no use complaining that the prosecutor is biased against your client, trying to make out they're guilty. That's their job. It's your job, if you can, to refute the argument. Not reading it and then saying you're not aware of any reasons for suspecting misbehaviour just doesn't cut it.

Hmm. Typo. Should be 0.0002 centimetres. Apologies.

In the Humanities, I remember reading many fascinating debates of original articles, both in reply articles of equivalent length to the original, or long letter debates. Why do the natural sciences impose censorship to the extent that seems apparent from the Mcintyre experience?

"Is it fair to say that this is the illusion, or self-delusion, that "precision" and "accuracy" are synonymous?"

That sort of thing, yes.

Tangentially related, I spotted this article today.
https://www.newscientist.com/article/2077380-in-science-is-honesty-really-always-the-best-policy/

They're getting pretty brazen about it.

Why would it not get cited?

Nobody thinks or thought it was the last word in reconstructions. It would have been just the first of many reconstructions if it hadn’t become a cause célèbre.

Scientists know, but journalists and members of the public don't [know that peer-reviewed publication isn't proof of truth]

But journey and the public don’t read such journals anyway, by and large.

The extrapolations are significant …

I never saw the beef there. They could just have started the recon from 4 years later if it really made much difference (which it seems not to). Who would care?

His position is that it's not possible.

It would be wouldn’t it.But that doesn’t make it true.

They did indeed, but they were just demonstrating averages, not calculating radiation.

If you believe that, where did the R4 curve starting value of 28C come from if not from the calculation I just showed?

If you'd read p67-68 of the book,…

The RE values say different. I’m sure you/McI reject those.

The parable of the Emperor of China's nose is applicable here.

That seems like an elaborate way to say that he hasn’t calculated them.

No doubt written by Mann and his associates again. However, as I've already explained, it shouldn't because the data in it was already known to be corrupted and unrelated to temperature. How can it possibly be correct to retain corrupted data?

If it is so obvious that the data is corrupted and so on, why all the fuss about centering in the first place? You remind me of Willard’s Never Ending Audit - complain about one thing, when that complaint fails, complain about another, repeat ad infinitum.

99% of the series generated were hockeysticks…

Can you prove that? It is the first time I have heard it and if true would make the code that selects the 100 from 10000 unnecessary.

It was already known that something bad was going on.

Are there any balancing arguments in the book? By that I mean that we can have a discussion on this and there are always alternative views of any of the “facts” on offer. My impression from skimming some of the book is that this balance is missing. But maybe I just missed it.

Not reading it and then saying you're not aware of any reasons for suspecting misbehaviour just doesn't cut it.

I’m aware of lots of purported reasons to suspect misbehavior, I just don’t find them convincing. I don’t believe Mann, Jones or any of the others are fundamentally bad or conniving or mischievous people, as seems to be the stock view here. Sorry, I just don’t.

"Nobody thinks or thought it was the last word in reconstructions. It would have been just the first of many reconstructions if it hadn’t become a cause célèbre."

It *was* the first of many reconstructions, many of which cited or even *used* Mann's reconstruction as an input.

"But journ[alists] and the public don’t read such journals anyway, by and large."

No. But they told me various things must be true because they were in such-and-such a peer-reviewed journal. They don't have to be able to read it, just cite it.

"I never saw the beef there. They could just have started the recon from 4 years later if it really made much difference (which it seems not to). Who would care?"

It's probably not the most important issue, but it's still an error, and it still has an effect.

For an example, consider series 56 which is from "Twisted Tree, Heartrot Hill". Mann used an obsolete version of the data which only went up to 1975. Since he needed data up to 1980 to get the algorithm to work, he simply made up numbers for the last five entries. However, the latest version of that series had data up to 1992, in which the final rapid rise Mann was matching up to the temperature record reversed and *dropped* equally dramatically after about 1970 to the end of the series. The numbers he made up were level, when the real data was dropping rapidly.

Because the rise up to 1970 looks a bit like the rising global temperature, it gets weighted heavily in the reconstruction as a "temperature sensitive" site. Had the full series been used, the reversal would have failed to match and it would have had much less effect. Had Mann not made data up, the series would have been excluded, and the spurious non-temperature-related pre-1970 rise would not have had the influence it did. By infilling 5 years, 600 years of data gets included. It has more effect than just those 5 years.

It's an interesting argument, and I wonder what would have happened if I'd used it at school. "Sorry, I didn't have enough data, so I just made up the values I needed and carried on. It's only 5 years. What's the problem?"

At my school, I'm pretty sure I'd not have got away with it. Perhaps your school was different?

"It would be wouldn’t it.But that doesn’t make it true."

It doesn't make it untrue, either.

"If you believe that, where did the R4 curve starting value of 28C come from if not from the calculation I just showed?"

I just *said* "They did indeed". Why are you acting as if I said the opposite?

They also give a curve for R{-1} which starts at a different value, again calculated using temperature in C. What bit of physics do you imagine uses an R{-1} power law?

It's not meant to represent any physical effect here - it's just an illustration of "averaging".

"The RE values say different. I’m sure you/McI reject those."

McIntyre did not reject those. He found that the RE values did *not* say different - Mann had miscalculated the significance threshold by assuming the sequences input to the RE test were AR(1). McIntyre showed that if you put AR(1) series through Mann's short-centering PCA first, as Mann had done with the real data, the significance threshold was pushed much higher - to 0.59 instead of 0. (See MM05 http://www.climateaudit.info/pdf/mcintyre.mckitrick.2005.grl.pdf).

"That seems like an elaborate way to say that he hasn’t calculated them."

That seems like an elaborate way to say you don't understand what's wrong with averaging a hundred million uninformed guesses and drawing tiny error bars on the result.

"If it is so obvious that the data is corrupted and so on, why all the fuss about centering in the first place? You remind me of Willard’s Never Ending Audit - complain about one thing, when that complaint fails, complain about another, repeat ad infinitum."

The difference is that in this case the first complaint didn't fail. There are dozens of things wrong with the paper, all at once. The reason for finding and mentioning all of them is because of people like you, who dismiss an individual mistake as a minor matter that you can ignore.

"Can you prove that? It is the first time I have heard it and if true would make the code that selects the 100 from 10000 unnecessary."

Figure 2 of the MM05 GRL paper shows the full distribution. Code is available at http://onlinelibrary.wiley.com/doi/10.1029/2004GL021750/abstract - click on "Supporting Information".

"Are there any balancing arguments in the book? By that I mean that we can have a discussion on this and there are always alternative views of any of the “facts” on offer."

It routinely describes most of the objections, arguments, claims, and so on made by McIntyre's critics. You have to, in order to refute them.

But on your general point - can you point to where I can find the balancing arguments of sceptics set out in the mainstream global warming literature, such as the IPCC reports? If this equal treatment of opponents is required before we take any presentation seriously, then why should we accept the mainstream story? If climate scientists are not required to present sceptic arguments in their works, why do you require it of Montford?

It seems to me that you demand all the space in your own works and at least half the space in ours. All the money and authority is on the side of the mainstream - you can find their version of the story anywhere. There is limited space to set out the sceptic case, so of course sceptics will concentrate on it.

By your comments, it's clear you've already *seen* the case against McIntyre, and with your ignorance of the counter-arguments, it's clear that *they* didn't set out the other side of the story. It seems a bit hypocritical for you to now require Montford to do so before you'll even read it.

"But maybe I just missed it."

Maybe you missed quite a lot? :-)

"I’m aware of lots of purported reasons to suspect misbehavior, I just don’t find them convincing."

Fair enough. I'm a supporter of freedom of belief.

In judging any dispute, you need to find out the best arguments on both sides, and make your own mind up. All I'm offering here is the best arguments I know of from my side. If I think you didn't understand something, I'll keep on explaining until you do, or we give up. If you do fully understand but are not convinced, so be it. That's how debate and opinion-forming work.

Without people who believe differently to us, and who are motivated to find fault with our beliefs, we'd have nobody to check our work. That would be bad, for the same reason that living on an island free of predators was bad for the dodo. We need each other.

"I don’t believe Mann, Jones or any of the others are fundamentally bad or conniving or mischievous people"

Probably not. Popular theories are that it was initial over-confidence, "noble cause corruption" (bending the rules for a good cause), sloppy standards, limited time, funding, training, and knowledge that caused the initial problems. Then when caught out, especially after entering the global limelight at the centre of the political arena, it was the usual human instinct to try to cover up mistakes instead of admitting and correcting them. You explained earlier how it was only human nature not to want to admit to anything knowing that critics are going to use it to attack the credibility of the entire field, and by extension, their jobs, careers, and reputations. I agreed with you.

That doesn't mean the technical criticisms aren't true, or that refusing to acknowledge them isn't a huge problem for the science. (Or the global economic policy, for that matter.)

But bear in mind that by the same reasoning, climate sceptics aren't fundamentally bad, conniving, or mischievous people either. They're as human, fallible, and imperfect as everyone else. And after having being kicked for years by the mainstream into the same bin as the moon-landing conspiracy nuts and holocaust deniers, it's only human nature to want to put the boot in when the opportunity arises. And most climate sceptics aren't professional scientists with standards to uphold, but more the "bloke down the pub" sort of people. If you want to see more sympathy and human understanding for climate scientists, would you be willing to demand the same consideration and tolerance for climate sceptics?

Thank you Raff and NiV. I'm enjoying the debate, and learning as you both go along. It's too far above my pay grade for me to comment on any of the science (as I'm not a scientist) but I do struggle to understand why Raff feels able to debate it so intensely without having read the HSI, and in the process being quite glib in his reasons for not doing so.

I consider myself to be an environmentalist (a real one, not the modern fake kind who claim that CO2 is a poison or a pollutant, and are happy to destroy our visual environment with useless wind turbines (or subsidy generators, as Salopian and I have just agreed to call them on another thread). Back in the '80s I almost joined the Green Party. I believed the AGW hype, and moved on from just being keen on re-cycling and walking and cycling where I could, to become quite excited about reducing CO2 emissions. At some point I started to read up on the subject, and eventually I came across, and read, the Hockey Stick Illusion. It was one of the (several) things which made me move away from the "greens" (I always put the word in inverted commas now since they're only really green these days in the sense of being naive and gullible) and towards scepticism.

Raff, please do read the HSI. It might change your mind (or if you read it and disagree with it, at least you'll be better informed). ;-)

Mark, I’m not more a scientist than you, so I’m wondering, when you read the HSI (a book that seems to have influenced you greatly), how did you evaluate the text? If you were to read, for example, The Montford Delusion (http://www.realclimate.org/index.php/archives/2010/07/the-montford-delusion/), how would you evaluate that and decide between the two?

NiV, without the notoriety MBH might be cited less - that is unknowable. Alternatively it might be cited just as much precisely because it was the first and introduced various themes and techniques but not necessarily because it was considered authoritative or even largely correct.

It's probably not the most important issue, but it's still an error, and it still has an effect.

For an example, consider series 56 which is from "Twisted Tree, Heartrot Hill”.

There you go again. The PC saga is inconclusive and the 4 year padding is just silly so you move on to a 5 year padding and question the quality of my schooling. Never ending audit indeed. What’s next on the list?

I just read (in the link just above, if I understood correctly) that all this PC1 and north American stuff boils down in the maths to just 1 of 22 proxy series (presumably that applies to all the verification and correlation stats (REs and R-squared etc) as well - all about condensing down the overweighted N.America series?) So there’s 21 other series for which all of that argument is irrelevant. If that is true, I can only say, Wow!

Mark Hodgson, you don't need to apologise here, or anywhere else, for not being 'a scientist'. Nor even does Raff. (He just needs to start reading and thinking a bit more before doing things like attempting to dismiss the work of scientists on another thread because a co-author is an economist.)

Clear logical thinking is essential in most human endeavours. If a portfolio manager tells you your investment will, say, go up 10% per annum over a decade and it goes down 10% per annum, then you don't need a qualification in finance to ask that person some very hard questions. They may come back with some highly technical reasons why the under-performance is not their fault and their estimate was reasonable. Or that you misunderstood what they really said (it's your fault). Or they may choose to re-define how it is calculated. The list can be endless, but they are still the people who sold you something and may still be trying to sell you something.

I hold that you can define a real scientist by the quality of the questions a person asks, and the questions they appear to have ignored when they seem slightly too confident in their abilities.

Trust yourself.

"There you go again. The PC saga is inconclusive and the 4 year padding is just silly so you move on to a 5 year padding and question the quality of my schooling. Never ending audit indeed. What’s next on the list?"

I apologise for questioning the quality of your schooling. I just can't get used to the idea of a version of maths/science where errors in a calculation are considered acceptable.

The 5 year padding was meant to be an explanation of what was wrong with the 4 year padding. Many of the same issues apply, it was just an easier example for me to find the details for. There are lots of them. The Vagonov, Texas-Mexico, and ITRDB groups transcribed values from neighbouring columns - that's 13 series affected. Then series 6, 45, 46, 52, 54, 56, 58, 93, 94, 95, 96, 97, 98, 99 are subject to the infilling we were discussing. Series 50 had all the values from 1962 to 1982 copied across from series 49. That's 20 years!

Then there's series 10 and 11 which weren't tree ring records, but marked as the central England temperature record, a thermometer record going back to 1659, and the central Europe temperature index, dating back to 1525. Except it turns out the figures don't match the CETR, they're actually just the average of the summer figures. And also Mann truncated them so his version starts in 1730. That's 70 years erased. Inconveniently cold years at the height of the 'Little Ice Age', it has to be said, followed by one of the longest, largest, and most consistent warming's in the record - big enough that it is very hard to distinguish (if you hide the dates) from the post 1900 period of warming. Coincidence, I'm sure. Similarly, 25 years got chopped off the central European temperaure index, which by another coincidence was the warmest part of the record, at the time of the 'Medieval Warm Period'.

As far as I'm concerned, doing it once for one year is still wrong. It doesn't matter if it has any effect on the answer - an error is still an error. And making up data is simply not acceptable. To mess up 30 times in one calculation has to set some sort of record! That's about a quarter of the series!

It would still be wrong even if it had no effect. But in the case you cited, it *did* have a major effect on the result. The series that got infilled 4 years at the start was the Gaspe cedars, that essentially flatten the 15th century all on their own. Their inclusion in this step enabled by the infilling (and use of an out-of-date version for which the first few decades contained only one or two individual trees) almost single-handedly "gets rid of the MWP". If you leave Gaspe and the American bristlecones out, the 'straight handle' of the hockeystick disappears as the MWP returns (see figure 1 of MM05 EE).

"I just read (in the link just above, if I understood correctly) that all this PC1 and north American stuff boils down in the maths to just 1 of 22 proxy series (presumably that applies to all the verification and correlation stats (REs and R-squared etc) as well - all about condensing down the overweighted N.America series?) So there’s 21 other series for which all of that argument is irrelevant. If that is true, I can only say, Wow!"

Yes. If you look at figure 3 right hand panel in MM05 EE (https://climateaudit.files.wordpress.com/2009/12/mcintyre-ee-2005.pdf) you will see that of the 22 series extending back to the 1400 step, 20 of them have a negative contribution to the hockeystick shape, and two of them - Gaspe and the North American bristlecones - contribute about 160% of the result! I can only say, Wow! ;-)

--

Like Mark and Paul above, it mystifies me why anyone on your side would want to defend the hockeystick. As we discussed earlier, it's not that important in a technical sense - it's one duff paper by one duff researcher, and we all know that anyone can make a mistake. Acknowledge it, bin it, bin everything that depends on it or makes the same errors, fix the review system that missed it, and then we can all move on.

The true significance of the hockeystick issue (besides the pure political symbology of the IPCC's headline graph), is the way nobody in climate science spotted the glaring errors, and then the way most of the people in climate science tried to defend it once the errors had been spotted. It's not the technical incorrectness of MBH98/99 that is the major problem for climate science, it's the way you guys keep on trying to claim that it's actually correct!

This pair of papers are some of the most comprehensively dissected and thoroughly debunked papers of all time. They are completely indefensible. We have every gap and hole in the defences thoroughly mapped. Entire books have been written setting out every point, with 19 pages of references. To even cite it nowadays in the debate is simply asking for a kicking, especially when there are so many better arguments and battlefields.

You was doing much better earlier, with your line that climate scientists understandably did not like admitting to errors knowing that their critics are going to use it to attack them. That's fair enough. I'm sure they don't, I'm sure none of us would, and it argues for us accepting climate science's *tacit* acknowledgement that the results were wrong. I doubt anyone on our side will be persuaded, but it's a decent argument.

But to try to defend it technically seems like a misguided approach from your point of view, unless you're genuinely interested in learning what the flaws actually are (or are claimed to be), in which case reading HSI would be a far more efficient way of achieving that than listening to me ramble on. Are you trying to learn more, perhaps to see if HSI is worth reading, or are you genuinely trying to claim McIntyre's criticisms are without foundation or weight? I'm happy to carry on talking, either way, but it might help me to help you if I know what you're trying to do.

----

"If a portfolio manager tells you your investment will, say, go up 10% per annum over a decade and it goes down 10% per annum, then you don't need a qualification in finance to ask that person some very hard questions."

Portfolio managers get audited, since they're playing with large amounts of other people's money, so they're usually more careful about what they say than that. (The prospect of prosecution concentrates the mind on maintaining quality standards wonderfully.) They might say "It's gone up 10% pa over the last 10 years, but future investments can go up or down." or "I expect 10% +/- 30% (2-sigma) over the next 10 years," or something like that. Models and predictions are all about error bars and quantifying uncertainty. Always ask what the error bars are - it's one of those distinguishing questions scientists ask.

Raff

I evaluated the text by being open-minded when I read it (remember it was only one of a number of events that moved me from being in the alarmist camp to being a climate alarmism sceptic), and by being a reasonably (though not super-) intelligent individual who found that the book set out a (for me) complicated subject in layman's language. I read it with the sceptical attitude I bring to everything, and I found it quite compelling. I then went off to read what I could find about McKitrick, McIntyre and about Michael Mann. Nothing that I then read changed the conclusions I was starting to reach.

However, I think that nothing is more important than continuing to learn, and part of that process is retaining an open mind. I'm a bit busy at the moment, but will do my best to make the time to read the Montford Delusion, to which you refer. Will you read the HSI? ;-)

Thank you Michael Hart for your kind words.

Raff - I have now read the Montford Delusion (sorry, I mistakenly assumed it was book length, and it didn't in fact take long to read, being just 5 pages long).

It's an interesting critique, but it didn't convince me, as NiV on this thread has dealt in much more detail with some of the critical issues than does the article you referred me to. It's a valiant attempt to provide a readable (and necessarily short) summary of the counter-argument to Montford and the 2 Mcs without being so long as to lose people's interest. The problem I have with it is that although it's 5 pages long, probably only 3 of those pages deal with the meat of the dispute, which isn't nearly enough to deal with the substance of the issues involved. The first page is mostly introduction, the last page talks about the Climategate emails, and much of the intervening 3 pages is devoted to questioning (rather snidely, I might add) the motives of Montford and the 2 Mcs.

You won't be surprised to learn that the Climategate emails were another of the issues that drove me more into the camp of those who are sceptical about the CAGW scare. For me, the article you referred me to rather undermined its credibility by blaming Montford for raising them at the end of the book, and claiming he was quote mining. They were hot off the press at the time, and were a huge mine of information supportive of Montford's case. Why wouldn't he bring them into the discussion? As I have mentioned on another thread, I am a retired solicitor, which means that I am unqualified to talk about the scientific issues, but I am (if I say so myself) rather good at assessing the credibility of witnesses, and knowing when a quote has been used out of context, or when it really does say (and was meant to say) what it appears to say. For me, anyone defending the Climategate emails needs to take a good hard look at themselves. They are close to the nadir of modern science. I'm with NiV in not assigning bad motives to people, but I do think that the climate science community needs to accept valid criticisms, especially of unacceptable behaviour, rather than trying to defend the undefendable, not least by criticising the motives of its critics.

Thank you for the link, anyway - an interesting and thought-provoking read. Now I need to try to find the time to re-read the HSI...

I just can't get used to the idea of a version of maths/science where errors in a calculation are considered acceptable.

That is not what you said with reference to schooling, which was more on the lines of making up data. And the made-up data was almost all at one or other end of a long series or filling in a few missing values. Clearly it is better if data sets are all complete and of the same length. But If they are not (the real life situation), you have to make do with what you have. The supplementary info says that about 25% of series stopped a few years short of 1980. It might have been better if MBH had stopped their analysis at 1972. I find it hard to believe that it would have made much difference to the shape of the curve in the 15th-19th centuries (which is the part of interest) if the last 8 years from the 20th C part were either removed entirely or interpolated.

But there it is again, 4 year and 5 year padding are not enough so we go on to all the other padding. And then to temperature series. CETR was suspiciously cut off, despite Manley, who collected it, saying 60 of those 70 years were unreliable (the fahrenheit scale wasn’t invented until 1724 so interpreting anything before that might be a challenge). And a plot of the yearly data (extracted from Manley 74, which used Manley 59 data) http://snag.gy/VxiPj.jpg doesn’t look so interesting - certainly if you were not looking for a LIA there it doesn’t jump out at you (the plot is of whole year data).

Inconveniently cold years at the height of the 'Little Ice Age', it has to be said, followed by one of the longest, largest, and most consistent warming's in the record - big enough that it is very hard to distinguish (if you hide the dates) from the post 1900 period of warming.

Is that what the summer temps show - do you have a plot of them?

Similarly, 25 years got chopped off the central European temperaure index, which by another coincidence was the warmest part of the record, at the time of the 'Medieval Warm Period’.

It is a moveable feast, the MWP. You put 1525 as part of it yet when I read about it I generally see much earlier dates and even the famous Lamb graph quoted in the HSI shows it well and truly over by 1400.

You really seem to be scraping the bottom of the barrel with these observations. You call these things ‘errors’ but they are just choices made in processing imperfect data. They may be done ‘deliberately’ to get rid of the LIA (the MWP was well and truly over by 1400 according to the Lamb graph and other sources, so forget that). But more likely they seemed reasonable choices at the time.

Like Mark and Paul above, it mystifies me why anyone on your side would want to defend the hockeystick.

I’m not defending the hockey stick, I’m just interested in the skeptic obsession with it and with Mann. Although you keep referring to a litany of errors, apart from the upside down series, I don’t think you have identified any errors (maybe I’ve forgotten them, the parts of the story have been a long time in coming out). The PCA stuff seems debatable and the infilling is deliberate, not an error. You might say they are deliberate attempts to falsify something but that is just your or McIntyre’s opinion, not fact.

Mark, if you are re-reading the HSI, look out for mention of the Medieval Warm Period and bear in mind that the period had ended by 1400, the earliest date in MBH98. Consider why it has any relevance to the discussion at all.

Raff

In all seriousness, I will struggle right now to make time to re-read the HSI, much though I would like to do so. From memory (which I admit is fading, as it is quite a few years since I read it) I believe its critique of the Hockey Stick to be about much more than the MWP, but as and when I do get around to re-reading it, I will bear in mind what you say.

Will you now read it, please? I continue to struggle to understand why you are devoting so much time to criticising something you haven't read, or why you would even want to do so. Has climate science slipped so far that proponents of the current climate change mantra feel it is appropriate to criticise things they haven't read?

"That is not what you said with reference to schooling, which was more on the lines of making up data."

Making up data is an error. It's the wrong data.

"Clearly it is better if data sets are all complete and of the same length. But If they are not (the real life situation), you have to make do with what you have."

Yes. You *don't* make up data you *don't* have.

The correct approach is to exclude the series, since it doesn't meet the published criteria for inclusion.

It's bad practice to change the criteria after you've looked at the data, to try to achieve a desired outcome. (It's called 'data snooping', and introduces bias into the results, usually invalidating the statistical tests.) But if you do, then you *have* to change the published criteria to match what you actually did. You need to explain exactly what you did and why, so that other scientists reading the paper can replicate the calculation.

" I find it hard to believe that it would have made much difference to the shape of the curve in the 15th-19th centuries (which is the part of interest) if the last 8 years from the 20th C part were either removed entirely or interpolated."

Why? Do you not understand how the calculation works?

I'll give you a simle example. We have three data series as follows:
Series 1: 0.1, -0.1, 0.1, -0.1, 0.1
Series 2: -0.1, 0.1, 0.1, -0.1, 0.1
Series 3: ????, 0.1, 1.3, 6.4, ????

The third series has missing values, so we can't determine the true average of all the series there. We make do with what we've got.
Average of series 1 and 2: 0.0, 0.0, 0.1, -0.1, 0.1

But what gets published is:
0.033, 0.033, 0.500, 2.066, 2.200

Bit of a difference, right? And if you don't know which series have been infilled and which haven't, there's little chance of being able to figure out where the numbers came from. The numbers plugged in are the wrong numbers for those years, and given the peculiar nature of that series, it's hard to avoid the suspicion that the infilling was done specifically to ensure it was included and would have the effect it has.

"Is that what the summer temps show - do you have a plot of them?"

There's a plot of three segments from HadCET here:
http://www.countingcats.com/wp-content/uploads/2009/10/hadcet.gif

One of them is the modern post 1950 rise, another is from 1680, and the third from 1850. That is to say, one is said to be clear evidence of anthropogenic global warming, and the other two merely natural variation. Can you spot the difference?
The scale across the bottom is in years. The scale up the side is the annual average temperature in C.

"You really seem to be scraping the bottom of the barrel with these observations. You call these things ‘errors’ but they are just choices made in processing imperfect data."

OK. Explain to me why it's a "reasonable choice" to include data in a temperature reconstruction already known not to be caused by temperature? Or to publish a "temperature reconstruction" known not to be correlated to measured temperature? (But not mention the fact.)

Or for that matter, why you would "choose" to locate North American rainfall data in Paris? Hearing your explanation for this one should be good!

"I’m not defending the hockey stick, I’m just interested in the skeptic obsession with it and with Mann."

I already explained that. First, that it's evidence that climate science isn't checking the results they publish, and second that they're willing to defend even obvious errors to protect the reputation of climate science.

It's the perfect bait - believers feel constrained to flock to its banner and defend it, which enables us to knock them down easily. That makes us look good and makes the believers look like gullible idiots or liars, which is of course politically convenient. :-)

Can you explain, why the obsession with defending it? Given my earlier explanation of why defending it was a really bad idea, I'm not sure why you would still be doing it.

"I don’t think you have identified any errors (maybe I’ve forgotten them, the parts of the story have been a long time in coming out)."

It must be useful to have such a selective memory!

"The PCA stuff seems debatable and the infilling is deliberate, not an error."

The PCA stuff is definitely wrong. (I'll grant that if you're not a mathematician you might have to take somebody else's word for it.)
And if you would have it so, then the infilling was a deliberate error.

If you ask me what the average temperature in 1980 was, and I put in the 1975 temperature instead because I don't have all the measurements from 1980, it's not the average temperature in 1980. It's the wrong number; at variance with reality; not the truth; contrary to the facts; wrong.

If you told a tax inspector that you didn't have the profit figures for this year so you put in the numbers from five years earlier and didn't bother to mention it, and told him "That's not an error, it's a deliberate choice", I don't think you'd get much sympathy.

But I'm guessing that there's nothing that you *would* consider an error, or would remember if you did. If you don't have the data you need, you can just make it up. It's just a "choice", right?

---

"Will you now read it, please? I continue to struggle to understand why you are devoting so much time to criticising something you haven't read, or why you would even want to do so."

He won't read it. The more you push him to do so, the less likely it is he will.

Even if he read it, he wouldn't believe it. It's not about the science, it's about the identity politics.

NiV:
Making up data is an error. It's the wrong data.

That is just “skeptic” dogma and does you no credit. Interpolation of missing values is a common procedure. Search online if you disagree and tell everyone it is wrong. There is even an R forecast package to help fill in missing time series data.

I'll give you a simle example….

The data in question is 600 years long. As I said it is difficult to believe that it makes a difference to much of the data in those 600 years if the last half dozen values of 25% of the series are estimates. If you think it does, give an example with several hundred values; an example with 5 is just nuts.

"Is that what the summer temps show - do you have a plot of them?"

There's a plot of three segments from HadCET here:

That is not the Manley data used by Mann. What relevance does it have? The yearly plot of Manley data I showed has no such obvious trends and just eyeballing the summer months I don’t see trends there either. If you say there are, show me the plot of that data (you must have one to be so sure).

Explain to me why it's a "reasonable choice" to include data in a temperature reconstruction already known not to be caused by temperature?

Not sure what you mean there. Tree growth is clearly not 100% correlated to temperature, but there is likely to be a signal in there (mixed up with other influences).

Or for that matter, why you would "choose" to locate North American rainfall data in Paris?

So that sounds like a real error. That makes two, with the upside down series. That is a long way from the 30 you claim.

Can you explain, why the obsession with defending it? Given my earlier explanation of why defending it was a really bad idea, I'm not sure why you would still be doing it.

I don’t suppose you appreciate the irony of you accusing me of obsession, when you go to the trouble of reproducing some of the analysis and are so fluent in the supposed “errors” in MBH98. And that you do it on the website of an author who wrote a whole book about someone else’s blog about the HS. And that many of that website’s inhabitants are so unbalanced on the subject of Mann that they treat him almost as the devil incarnate. You have no self awareness if you see obsession in others but not in yourselves. I on the other hand have to go and look up this stuff you talk about and I find that what you claim is not on the face of it true. But hearing people like you claiming big significance from extending one of 100 or more series from 596 to 600 years (your second argument I think) is worth the effort for the laughs :-)

——

Mark, of course the MWP is discussed. In a way you could say the book is about the MWP and the dastardly conspiracy to get rid of it. Fig 1.1 in the intro section (available to read in the Kindle version online at Amazon) shows the Lamb graph and the MWP terminating at about 1400, then a section discusses how “The Medieval Warm Period becomes less warm” and the paper that suggested that it wasn’t global; then there is “The Deming affair” and allegations that climate science needed to get rid of the MWP. Then you get this priceless piece of conspiracy thinking:

Climate science wanted big funding and big political action and that was going to require definitive evidence. In order to strengthen the arguments for the current warming being unprecedented, there was going to have to be a major study, presenting unimpeachable evidence that the Medieval Warm Period was a chimera.
Enter the Hockey Stick.

So there you have it, in Montford’s eye, MBH98 was written with the aim of providing evidence against the MWP and it would do it by reconstructing temperatures back to….1400: after the MWP had ended! Okay, you or maybe others here probably do think climate scientists are incompetent enough to plan the defrocking of the MWP using a study with a start date after the MWP ended, but to anyone normal this is hilarious. Do I really need to read any further? If the author of the HSI can misrepresent MBH98 so laughably within a few pages of the beginning, what hope is there that anything else he writes will be believable?

Raff, is that the best the SKS Hockey Stick Repair Team Collective can come up with after all these years?

Are 'you' (however many you are) being called as an "Expert Witness" by Mann's Lawyers? Presumably Mann's Lawyers will pay more than you get for posting here.

Raff

"Do I really need to read any further?"

Yes, you do. You cannot criticise (in depth) something you haven't read, and expect to be taken seriously. I will make a real effort (despite being short of time) to re-read it, in order to try to respond more fully, but it will be weeks, rather than days, before I can do so, in view of everything I have going on at the moment. I suspect, however, that you still won't read it.

Unlike some of the aggressive and unpleasant tripe written by some climate alarmists on this and other sites, I enjoy reading what you say, because you try to keep it civilised (which I always value), and because you challenge my assumptions (we have all - on both sides - made assumptions in this debate, whether deliberately or unthinkingly, I would say the difference is that sceptics are better at challenging their own and others' assumptions. Many climate alarmists just unthinkingly accept what they're told and look no further. I give you credit for not being one of them).

However, on this occasion, I am rather running out of patience. I think NiV has you bang to rights, but you won't accept defeat. You should try re-reading your last contribution to the thread (indeed the last exchange) and try to do so, if you can, as a disinterested 3rd party, and really ask yourself if you come out of it well. And I repeat, I still can't take seriously someone who criticises so confidently a book he hasn't read (and who does so on the website - and with the evident tolerant permission of - the author of said book) and then says "I don’t suppose you appreciate the irony of you accusing me of obsession".

So Raff, please do keep contributing on this site. Sometimes I learn from you, sometimes I am baffled by you, sometimes I wonder if you're on the same planet as me (not meant as a criticism of you - maybe it says as much about me). But you're going to have to read the book if you want to criticise it and expect to be taken seriously, and you're going to have to do much better if you want to try to defend Mann's hockey stick, as you're starting to lose (and, to be honest, lose face too) hands down now.

"The data in question is 600 years long. As I said it is difficult to believe that it makes a difference to much of the data in those 600 years if the last half dozen values of 25% of the series are estimates. If you think it does, give an example with several hundred values; an example with 5 is just nuts."

I picked the example with 5 entries just to make it easy for you to understand. Once you understand the principle, it's obvious how to extend it to sequences of any length.

But just so you can't avoid the obvious, here you go...

Series 1: 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, ...

Series 2: -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1

Series 3: ????, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1, 0.1, 1.3, 6.4, ????

Do you see now?

---

And just for fun, here's how to construct a hockeystick out of garbage in four lines of code!

(You'll need to get R to run it. https://www.r-project.org/ )

# Generate 100 trendless red noise series of length 500 each
x = replicate(100,arima.sim(list(ar=0.9),500)) # Line 1!

# plot some of them
plot(x[,1],type="l")
plot(x[,2],type="l")

# Plot the average - note there's no pattern. The values average out to zero, with no hockeystick
plot(apply(x,1,sum)/100,type="l")

# create a representative vector of modern temperature observations ...
temps = seq(0,0.99,0.01) # Line 2!

# ... and find the correlation between each series and observed temperatures
# to identify which source series are "temperature sensitive"
correl = apply(x,2,function(s){cor(s[401:500],temps)}) # Line 3!

# Plot the correlation-weighted sum of the red noise series
# Note the strong "hockeystick" shape. We've reconstructed the observations
# from unrelated and trendless random numbers - tantamount to haruspicy
plot(as.vector(correl %*% t(x)),type="l") # Line 4!

---

Now, I've "reconstructed" temperatures from random noise - and it looks pretty good!

So, are you really willing to tell me that I've just made "choices" processing "imperfect" data and this is all perfectly legitimate and scientific, or is it actually an error to claim this is a valid method for reconstructing global temperatures back in 1500?

I'm interested to see if you hold me to higher standards than you do Mann and the IPCC consensus.