Wednesday
Mar212012
by 
Bishop Hill
Bishop Hill Climatologists go open
Mar 21, 2012 Climate: Surface FOI John Graham-Cumming notes that CRU and the Met Office have come over all open, as these excerpts from the CRUTEM4 paper make clear.
we are able to make the station data for all the series in the CRUTEM4 network freely available, together with software to produce the gridded data (http://www.cru.uea.ac.uk/cru/data/temperature/ and http://www.metoffice.gov.uk/hadobs/).
..and...
The code required to produce the CRUTEM4 fields and timeseries will soon be made available. Note that code previously released for CRUTEM3 cannot be used to exactly reproduce CRUTEM4 temperature series due to changes in processing methodology.
Reader Comments (48)
Beware of Greeks bearing gifts.
bewear of geeks bearing gifs
Could Richard Betts clarify whether we now have the digital originally recorded raw station data and the 'value added' adjusted data with the station selection and adjustment justifications. I thought some or all of the history on the station adjustments was not kept and the metoffice were to do a rework of the whole thing?
Currently, all the individual station data and the CRUTEM4 gridded product is available at the Met Office Hadobs website, also both of the recent papers, on CRUTEM4 and HADCRUT4. The code itself does not seem to be there yet but I expect it will be soon.
JGC says that he finds the statement
"our preference is that the underlying station data, and software to produce the gridded data, be made openly available" slightly surprising.
I wonder what could possibly have brought about this remarkable change of attitude? Perhaps Jonathan Jones or Don Keiller might have some insights into this question :)
Hurray! If anyone inthe future asks what good came of climategate then this is one simple answer.
Ironically though if they'd done this five years ago there probably wouldn't have been a climategate because the driver for that leak appears to have been the continual stonewalling of FOIA efforts to get precisely this information.
"changes in processing methodology"
Settled science.
Andrew
Looking at http://www.cru.uea.ac.uk/cru/data/temperature/ there are two sets of input station data, one for CRUTEM3 and one for CRUTEM4. That seems strange to me, shouldn't there be a consistent single set of input data, regardless of the algorithm used to process it?
mm...
I downloaded some of the station data a few days ago, just out of idle curiousity. It's not representative I know, but opening up a few of the files, I was horified but just how much of data was -99C (i.e blank).
Presumably they've used the best records available in the various regions - makes you wonder though, how much reliance you can put on the end result.
May be worth going to this - Gleick the forger in Oxford.
http://www.bsg.ox.ac.uk/events/peter-gleick-oxford-amnesty-lectures
I'm with Paul Matthews in giving praise to Jonathan and Don for this surprising and positive turn of phrase.
May be worth going to this - Gleick the forger in Oxford.
http://www.bsg.ox.ac.uk/events/peter-gleick-oxford-amnesty-lectures
"due to changes in processing methodology": well, of course. Otherwise comparisons would be straightforward.
And wow, Fred: Gleick hosted by the Oxford Amnesty Lectures:
I haven't really been tracking the subversion of human rights groups like Amnesty by our New Unhappy Lords (GK Chesterton's phrase). Index on Censorship's take on climate openness at the Data debate Josh and I attended in December and in publishing Fred Pearce's forthright views was a great deal more healthy than one might have expected. Amnesty I have less idea about. Presumably Gleick was signed up for this before admitting his deception and identity fraud. What should be done to persuade the organisers that the invitation can't now be right? Do Oxford or Amnesty people have ideas on that?
"May be worth going to this - Gleick the forger in Oxford."
thanks, Frederick, I have a message for Oxford University from Leo Hickman:
"Beware of Gleicks bearing gifts."
Re: climatebeagle
CRUTEM3 uses a single file with all the different stations in it.
CRUTEM4 uses a separate file for each station.
Frederick Bloggsworth
Good to see Gleick's career is forging ahead through forging new contacts at Oxford.
Since it's just up the road from my Hampshire Riviera I've half a mind to squander £8 on a ticket and £2 on popcorn.
Alan, I don't think popcorn is adequate for this one. What about someone investing in face masks with Gleick's face printed on them? If a few people wore those throughout the lecture and had some photocopies of an well-worded explanation to give to people I think that could have an impact. But as I've said in an earlier post trapped in moderation:
Links left out this time. But to get an idea of the history of the lectures try
http://www.oxford-amnesty-lectures.org/index.php?p=Archives
I think this is a fine opportunity but it needs to be done really well. Best to persuade the organisers to dis-invite the guy. The face mask approach or some other stunt might be handy if they refuse to listen. But I'd be open to anything.
At least one member of the faculty hosting Gleick talks sense .............. I wonder if they will meet?
http://www.dieterhelm.co.uk/sites/default/files/Huhne_hype_060212.pdf
A further thought about Amnesty and Index on Censorship. They naturally support dissidents. Why was nobody from a sceptical viewpoint invited to the Data Debate? Has anyone from a sceptical viewpoint - economics, politics or science - been included in the seven luminaries this time?
http://www.oxford-amnesty-lectures.org/index.php?p=Lectures
Best guess would be that Olivier de Schutter is nearest:
http://news.bbc.co.uk/1/hi/7381392.stm
That links back to one of the most important Bishop Hill posts of all time, The Entrepreneur:
http://www.bishop-hill.net/blog/2012/2/20/the-entrepreneur.html
But look at the dreadful title he's been given or accepted. That's just two days after Gleick.
I think the dissident question is really worth asking the organisers. But there are masses of issues raised. We should organise and do some real good :)
FB
"May be worth going to this - Gleick the forger in Oxford."
Oxford coma I suspect.
@ pesadia
Beware of Greeks bearing gifts.
This is supposed to be a sceptic blog, not a cynical one. If the announcement is sincere, and we might as well take it at face value because if it is not sincere it will do more damage to the reputation of the Met Office than simply maintaining the status quo would, then the Biblical quotation below is far more appropriate.
There will be more joy in heaven over one sinner who repents than over ninety-nine righteous persons who need no repentance.” Luke 15:1-7.
Roy:
Very well put. We're going to have to up our game if we have a go at Amnesty and their ilk. Amnesty - what the prodigal son got. I need to think about that. Luke 15 rules.
I downloaded all the new station data for Hadcrut4 and then compared the results with the old Hadcrut3 data. There are 5549 stations in the set compared to 5097 in Hadcrut3. 738 new stations have been added while 286 stations have been discarded. Those added are all mainly in northern Russia. Quite a lot of stations from North America have been discarded. There are none added in the southern hemisphere but others have been lost, despite sparse coverage. The changes to the global anomalies are small and statistically insignificant. However they do psychologically change the impression of "warming" over the last 15 years - moving 2010 and 2005 up a bit and 1998 down a bit. Also the 19th century data has got just a tiny bit cooler. You can read more about this analysis at my blog
Clive
Are there objective criteria for station selection?
This approach is welcome.
It will be interesting to review the scale of, and the reasons behind adjustments to raw data.
As I understand matters, they have adjusted sea temperatures to take account differences between bucket measurements and engine room measurements. I must read their papers which presumably explain some of the genral approaches.
I think that it is often overlooked that ships are in effect measuring sea temperature at a depth in the region of about 10m below the surface (this is a typical depth of the water intake) and are not actually measuring surface temperature.
No need to waste cynicism on Gleick:
"Believing that people are motivated by self-interest; distrustful of human sincerity or integrity".
He only did it because he honestly believed he would never get caught.
If I wanted to accurately plot the temperature history of your car as a function of time with sensors attached to the the car's surface, that I periodically moved around every couple of years, and claimed I could 'munge' all the measurements into a single temperature using an averaging program....and every few years I reported that you car was actually colder in the past than I originally claimed...would you:
a) assume I was a statistical genius and award me a large grant? or
b) treat everything I said with suspicion?
Your choice!
@ Mike Jackson
Its not a typo
giff (hard g) stands for graphics interchange format
Read the small print on the notice advertising the Gleick event in Oxford: "Please note, the lectures are not part of Amnesty International and they are not University events".
Paul, CRU have not suddenly turned over a new leaf.
They are still fighting a rearguard action, through their solicitors, to prevent the release of requested emails.
I will provide more details when (legally) possible.
Clive, do you not think it suspicious that some stations in the southern hemisphere have been lost, despite the already sparse coverage?
Could this be because the SH is not "on message" temperature wise?
I can't imagine how they can justify dropping stations from the Southern Hemisphere. It certainly is true that SH temperatures (not anomalies) show no warming at all since about 1950. The arctic is warming and most new stations are in the far north. A couple of Arctic stations have even been dropped for example this one which appears not to be warming much. Perhaps CRU have a perfectly rational explanation - I don't know. Antarctica is certainly not warming and there are no new stations added there.
In reality the changes to the global anomaly are statistically insignificant. However the front page headline for "policy makers" is that 2010 is the hottest year. This is a meaningless statement since statistically there has been no observed warming for nearly 15 years, and this remains true.
Peter G ained
L ittle
E vidence
I llegally
C aught
K lutzy
I can find no explanation by CRU or by Hadley as to their station selection criteria. They really need to be open and transparent about this.
However their software requires the following:
Anomalies are defined for each station by subtracting the monthly values for a particular month from these normal values. Stations without normals for 1961-1990 or where any anomaly is > 5 standard deviations are excluded. The standard deviation is calculated from the 1961-190 monthly normals.
The large majority of the dropped stations pass the above criteria !
Something went wrong there.Hope u can work it out.
Anoneumouse
I'm well aware what a gif is. I'm less sure about "bewear", however. (|:-|)
Mike Post:
Thanks for this. They raise money for Amnesty - if anyone buys the resulting book - from what I can tell and they use the name. They've attracted a stellar list of speakers over the years (to a Guardianista more than to me I guess) and there's this:
Time to contact that Law Professor at All Souls I met at a family do in 2008? Could be. I don't really think either organisation can credibly distance themselves from this one. Gleick should be stood down.
believe it when I see it , I will and watch out for that where they hide the thimble .
But you have to say given there us already a lack of data the reasons for drooping any station must be very good indeed , Anyone got any idea why these stations were drooped or are we back to the 'trust us' approach to science?
I'm an Amnesty member and I don't remember seeing much if any mention of CAGW or sustainability in their recent campaigns.
Although I don't by any means pretend to represent the membership as a whole, I see AI aligning most closely with social libertarianism, which doesn't sit well with many of the ideas and "solutions" touted by those of a more alarmist nature.
"our preference is that the underlying station data, and software to produce the gridded data, be made openly available"
I certainly welcome that. But it does not necessarily mean that they have done so, or are going to do so. Wishing is not having, and the statement leaves wiggle room for later saying "So and so wouldn't let us do that" or "we've lost it". I note, without prejudice, that the Met Office makes no mention of the two biggest countries in the world, Russia and Canada, at
http://www.metoffice.gov.uk/hadobs/crutem4/data/download.html
It all sounds too much like fudge fudge, wink wink.
Richard said;
"I think that it is often overlooked that ships are in effect measuring sea temperature at a depth in the region of about 10m below the surface (this is a typical depth of the water intake) and are not actually measuring surface temperature."
More accurately, I believe;
At a variable depth depending on the size of the ship and its lading on that voyage but typically in the range 2 to 8 metres.
So a tanker, departing in ballast from the Gulf of Mexico (for example), will produce completely different data from that gained on its inbound voyage, since it is now measuring temperature at a completely different depth.
Clive says station 710820 (Alert, Canada) has been dropped.
In fact it is there in the new CRUTEM4 dataset, but with a new number, 713550.
Just a quick look so far but the "raw data" seems to be much the same in Western Australia.
E.G. The Kalgoorlie P.O. is ignored prior to 1942 and the Grid is filled with the last 4 years of the 1800's (1896-1889) fromSouthern Cross which global warming caused to cool ~ 1.2 degrees C between 1990 and 1999..
The Geraldton Town is ignored before 1942 and the grid is filled with the last 4 years of the 1800's from Hamelin Pool.
I shall check the rest in detail when I get some free time as the big oil cheque is well overdue.
Mar 22, 2012 at 7:58 AM | NicL
///////////////////////////////////////////
Fully agree that it is dependent upon the design, size, configuration of the vessel and whether the vessel is sailing fully laden or not.
In general, since vessels' earnings are usually made on laden voyages, voyage planning seeks to minimize ballast legs (ie., non revenue earning legs). Some trades do not permit this, and they are essentially round trip voyages having a laden out and ballast back leg. But all of that said, trade, as a world wide whole, will be biased towards the laden ship. As such shup's are not recording SST.
Richard Drake,
"Best to persuade the organisers to dis-invite the guy."
Isn't that the sort of thing that Gleick's chums do? I think it's an awful idea in about seventeen ways. For a start, giving Gleick a platform now can surely only be a good idea. Let the web get more tangled, I say.
The mystery deepens. CRU have changed the numbering of 110 stations between CRUTEM3 and CRUTEM4. Another 175 stations have been discarded.
This was one of the stations which has changed numbering. 710820 -> 710823. This is where the mystery starts because in CRUTEM3 the start date was 1950 whereas in CRUTEM4 the start date is now 1987 ! This causes the station to be excluded from the anomaly averages because it has no normals. So in reality it too has been discarded. You can see this by comparing these graphs:
ALERT (crutem3) and this new one ALERT (crutem4)
Now there are no anomalies defined for this station and so it is discarded anyway from the global average.
However I just noticed that Paul is also right - because it then appears a 2rd time in CRUTEM4 as 713550.
I hope there are not too many other duplicates like this !