Richard,

Thank you for your prompt reply.

So if my understanding is correct, in order to calculate the Level of Scientific Understanding, the body of evidence, which I would understand to be peer reviewed papers, needs to be assessed in order to see what the consensus of opinion is.

If for example, forgive my use of examples I find it makes things a lot easier for me to comprehend, I had an argument that had say 950 peer reviewed papers supporting it and 50 peer reviewed papers not supporting it, then I would be in a position of having a large body of evidence which has strong consensus. Therefore attracting a high level of scientific understanding.

Interesting, because if there was a subsequent increase in the number of papers arguing against my position that would then reduce my position, because even though there is still a large body of evidence, the consensus is starting to change.

But as long as there was the same number of papers published, perhaps within days, supporting my case, I would still maintain a high level of scientific understanding, as the body of evidence is still increasing but the consensus remains the same.

Obviously all of these papers would have to have been published and peer reviewed so there wouldn’t be much chance of flooding the process in order to maintain my high level of scientific understanding due to of the length of time it takes papers to be written, reviewed and published.

I will have a good look through the document that you linked to though, and see if it satisfies my concerns and obviously if my interpretation of what you have explained is incorrect then please do let me know.

Sep 30, 2011 at 12:58 PM | Richard Betts

Thank you for embiggen my vocabulary with your perfectly cromulent words.

From the Ecclesiastical Uncle, an old retired bureaucrat in a field only remotely related to climate, with minimal qualifications and only half a mind.

Giving way to optimism, it occurs to me that the extent by which the skeptics’ cause is allowed to be represented in the final version of the next assessment report may be an advance indicator of the extent by which change of policy on CAGW may be occurring in the corridors of power

Just a thought..

Sep 30, 2011 at 2:25 PM | Brownedoff

You're welcome - frabjous!

Sep 30, 2011 at 1:25 PM | Lord Beaverbrook

Well, it's not really as simple as counting papers - in your example, most of the 950 papers could have been written by the same few people using the same models or datasets, while the 50 could all be independent from each other.

So unfortunately, since there are limits to how far it can be clearly quantified, there is still a degree of "expert judgment" involved - but this should nevertheless be backed up with evidence. This is the kind of thing the reviewers need to check out.

Richard,

Thanks, I was hoping that the direction was leading towards Quality Control and perhaps a description of how the evidence which must be vast is determined and used. I suppose my example of quantity over quality didn't provide the trigger. In effect how are the following recommendations going to be enacted to convince me that the evidence is good and all views have been considered.

The IPCC should strengthen and enforce its procedure for the use of unpublished and nonpeer-reviewed literature, including providing more specific guidance on how to evaluate such
information, adding guidelines on what types of literature are unacceptable, and ensuring
that unpublished and non-peer-reviewed literature is appropriately flagged in the report.

Lead Authors should explicitly document that a range of scientific viewpoints has been
considered, and Coordinating Lead Authors and Review Editors should satisfy themselves
that due consideration was given to properly documented alternative views.

What is the more specific guidance, is this documented?
Is this documentation by Lead Authors on the range of viewpoints going to be publicly available?

Richard,

I have decided to withdraw from the field (again), but this time, on this thread, permanently.

I see that you have an on-topic exchange in progress with LB, and it should not be disrupted.

Your Grace,

Further to my 2:49pm today, would you please remove my "contributions" on this thread at 11:56am and 12:40pm today.

Thank you.

Oct 1, 2011 at 8:05 AM | Lord Beaverbrook

Sorry I got the wrong end of the stick - thanks for clarifying.

I think Annex A of the IPCC Principles Paper may help you here. I'll highlight a few key sections below, but I'd also encourage you to read the whole thing.

I'll address the "range of views" issue first. From the IPCC principles paper:

4.2.2 Selection of Lead Authors

...

The composition of the group of Coordinating Lead Authors and Lead Authors for a section or chapter of a Report shall reflect the need to aim for a range of views, expertise and geographical representation

4.2.3 Preparation of Draft Report

...

Experts who wish to contribute material for consideration in the first draft should submit it directly to the Lead Authors.

...

In preparing the first draft, and at subsequent stages of revision after review, Lead Authors should clearly identify disparate views for which there is significant scientific or technical support, together with the relevant arguments.

4.2.4 Review

...

Review Editors should not be involved in the preparation or review of material for which they are an editor. [Note from RB - there is more than one RE per chapter - indeed some chapters have several REs]. In selecting Review Editors, the Bureaux should select from developed and developing countries and from countries with economies in transition, and should aim for a balanced representation of scientific, technical, and socio-economic views.

4.2.5 Preparation of Final Draft Report
...

If necessary, and timing and funding permitting, a wider meeting with principal Contributing Authors and expert and government reviewers is encouraged in order to pay special attention to particular points of assessment or areas of major differences. It is important that Reports describe different (possibly controversial) scientific, technical, and socio-economic views on a subject, particularly if they are relevant to the policy debate. The final draft should credit all Coordinating Lead Authors, Lead Authors, Contributing Authors, reviewers and Review Editors by name and affiliation (at the end of the Report).

ANNEX 1

TASKS AND RESPONSIBILITIES FOR LEAD AUTHORS, COORDINATING LEAD AUTHORS, CONTRIBUTING AUTHORS, EXPERT REVIEWERS AND REVIEW EDITORS OF IPCC REPORTS AND GOVERNMENT FOCAL POINTS

1. LEAD AUTHORS

...

Lead Authors are required to record in the Report views which cannot be reconciled with a consensus view but which are nonetheless scientifically or technically valid.

3. CONTRIBUTING AUTHORS

...

Input from a wide range of contributors is a key element in the success of IPCC assessments, and the names of all contributors will be acknowledged in the Reports. Contributions are sometimes solicited by Lead Authors but unprompted contributions are encouraged.

5. REVIEW EDITORS

...

Although responsibility for the final text remains with the Lead Authors, Review Editors will need to ensure that where significant differences of opinion on scientific issues remain, such differences are described in an annex to the Report.

So we are given a very clear steer that differing views should be included. The makeup of the author teams and review editors should itself help to provide differing views, but differing views will also come from reviewers. Also contributions from non-authors are encouraged - these do not have to be specifically invited, they can be unsolicited.

Oct 1, 2011 at 8:05 AM | Lord Beaverbrook

(Part 2)

Now the issue of non-peer reviewed literature. Here is Annex 2 of the principles paper in its entirety (but again see the original if you wish.

PROCEDURE FOR USING NON-PUBLISHED/NON-PEER-REVIEWED SOURCES IN IPCC REPORTS
Because it is increasingly apparent that materials relevant to IPCC Reports, in particular, information about the experience and practice of the private sector in mitigation and adaptation activities, are found in sources that have not been published or peer-reviewed (e.g., industry journals, internal organisational publications, non-peer reviewed reports or working papers of research institutions, proceedings of workshops etc) the following additional procedures are provided. These have been designed to make all references used in IPCC Reports easily accessible and to ensure that the IPCC process remains open and transparent.
1. Responsibilities of Coordinating, Lead and Contributing Authors
Authors who wish to include information from a non-published/non-peer-reviewed source are requested to:
a. Critically assess any source that they wish to include. This option may be used for instance to obtain case study materials from private sector sources for assessment of adaptation and mitigation options. Each chapter team should review the quality and validity of each source before incorporating results from the source into an IPCC Report.
b. Send the following materials to the Working Group/Task Force Bureau Co-Chairs who are coordinating the Report:
- One copy of each unpublished source to be used in the IPCC Report - The following information for each source:
- Title - Author(s) - Name of journal or other publication in which it appears, if applicable - Information on the availability of underlying data to the public - English-language executive summary or abstract, if the source is written in a non English
language - Names and contact information for 1-2 people who can be contacted for more information
about the source.
2. Responsibilities of the Review Editors
The Review Editors will ensure that these sources are selected and used in a consistent manner across the Report.
3. Responsibilities of the Working Group/Task Force Bureau Co-Chairs
The Working Group/Task Force Bureau Co-Chairs coordinating the Report will (a) collect and index the sources received from authors, as well as the accompanying information received about each source and (b) send copies of unpublished sources to reviewers who request them during the review process.
4. Responsibilities of the IPCC Secretariat
The IPCC Secretariat will (a) store the complete sets of indexed, non-published sources for each IPCC Report not prepared by a working group/the Task Force on National Greenhouse Gas Inventories (b) send copies of non-published sources to reviewers who request them.
5. Treatment in IPCC Reports
Non-peer-reviewed sources will be listed in the reference sections of IPCC Reports. These will be integrated with references for the peer-reviewed sources. These will be integrated with references to the peer reviewed sources stating how the material can be accessed, but will be followed by a statement that they are not published.

Actually, Working Group 1 tend to prefer to avoid non-peer reviewed literature as much as they possibly can - here is the Working Group 1 guidance paper on their approach. Clearly, for their work on the physical science basis, most (if not all) of the relevant material should be in established scientific literature.

Working Groups 2 and 3 have a different approach because their subject areas are more applied. Taking the example of WG2 (since I am part of it) there should still clearly be a strong basis in peer-reviewed literature when discussing issues which require scientific or technical analysis and understanding, but when moving into more practical issues the relevant material may not be in the peer-reviewed literature but in (for example) consultancy reports or government reports. So if the issue under consideration is adaptation planning, the most concrete examples of this in practice will in the reports commissioned by the bodies actually planning the adaptation.

A specific example of this would be the major piece of work carried out by the Environment Agency, Met Office and Centre for Ecology and Hydrology on informing forward-planning for the Thames Estuary flood defences. If you believed alarmist messages you'd be spending billions of pounds on a new Thames barrier immediately - but we were commissioned to do a proper scientific analysis in order to help estimate when upgrades to the existing defences should start to be considered. This was done essentially as a piece of consultancy, but it should still be assessed in IPCC as it's the kind of thing which is affecting policy and planning.

(Indeed, my view is that assessment in IPCC should be a good sense-check on the robustness of such consultancy reports - unfortunately there's an increasing number of bad examples floating around these days, as environmental consultancies get in on the act and start re-hashing reports from NGOs - or indeed re-hashing statements from AR4 which have since been found to be in error, or have been superceded by more recent literature!)

As the guidance note says, the IPCC authors have a clear responsibility to check the pedigree of non-peer reviewed literature to make sure it is of good quality. Moreover, all material must be made available to the reviewers, so the TSUs will have copies of any non-peer reviewed literature being cited.

Is that the kind of information you were after?

@Richard Betts Oct 2, 2011 at 12:23 AM

5. Treatment in IPCC Reports
Non-peer-reviewed sources will be listed in the reference sections of IPCC Reports. These will be integrated with references for the peer-reviewed sources. These will be integrated with references to the peer reviewed sources stating how the material can be accessed, but will be followed by a statement that they are not published.

I'm afraid you're reading from the old song-book, Richard :-)

Pls. see When task group says let’s “disappear” a rule, IPCC agrees

The IAC's recommendation:

The IPCC should strengthen and enforce its procedure for the use of unpublished and non-peer-reviewed literature, including providing more specific guidance on how to evaluate such information, adding guidelines on what types of literature are unacceptable, and ensuring that unpublished and non-peer-reviewed literature is appropriately flagged in the report. [emphasis added -hro]

However, the writers of the new, improved song-book had determined that the (longstanding, but rarely practiced) flagging rule was "too impractical". So it will be brushed out whenever they get around to publishing/posting the amendments to the text recommended by the Task Group - and accepted by the IPCC (notwithstanding its earlier endorsement of the IAC's recommendation), and confirmed by their Decisions Taken on procedures

And while I'm here ... since you had raised the question regarding participation by WWF Climate Witness SAP members in AR5, I thought you might want to know that Donna's latest post provides this information:

78 Names

In scrolling quickly through the list, I see that it includes (inter alia) at least 11 CLA's for AR5. It is, perhaps, also worth noting that until his untimely demise, the late, great climate science communicator, Stephen of Stanford, was included in their rollcall for AR5.

Richard

Absolutely, a lot to go through there.

Just getting the lamb shoulder and veg on the barbeque for a slow roasted Mediterranean style Sunday lunch with the family so I might get chance this evening to make a start. Making hay while the sun shines and all that.

BTW give the boys a pat on the back for this week’s forecast, the shops were well prepared for the weekend with charcoal next to the Halloween costumes and advent calendars. I hope that the Met office had a hand in advising for the winter preparations at Heathrow, looks as though common sense is resuming.

hro001

I'm glad that I'm not the only one taking this opportunity to scrutinise the process, hopefully others will find this just as interesting.

Lord Beaverbrook

Thanks for the complimentary feedback, I'll pass that on. I hope you had a wonderful day in the sunshine. I even had a swim in the sea!

Sadly it looks like it will be back to cold, wet & windy for many of us in the UK from the middle of the week...!

We have indeed been talking to various key organisations about the importance of not confusing the projected long-term (many decades) trend with the very large natural variability, ie: even if we expect cold winters to be less common by the end of this century, this certainly doesn't mean they are not worth being prepared for now or in the next few decades. The simplistic message that "everything's just getting warmer" has a lot to answer for....!

hro001

Thanks - interesting - I'd not noticed that!

But what are the actual implications of this? I wonder whether it might actually make the AR5 appear to cite less peer-reviewed literature than it actually does? (ie: leave it more open to easy criticism). I say this because it will be easy for someone to go through the reference lists and simply see which references are from academic journals, and this might be assumed to be the indicator of which references were peer-reviewed - however, non-journal publications may also be peer-reviewed (I know that my own consultancy reports often get independently reviewed even if they are not for a journal).

NB this is not an excuse, just an observation! I'd be interested to see what you think.

BTW It has been made very clear to us as LAs that we have responsibility for checking that non-journal sources check out OK. The Himalayan Glaciers thing still smarts....

Yes, I'd seen Donna's post on the 78 AR5 folks on the WWF Climate Witness SAP list - Richard Tol (another AR5 CLA) tweeted it!

I should point out that my team have done a couple of projects funded by WWF (although we're not involved in the Climate Witness thing). However we absolutely do not let them dictate our science - indeed one of our projects was, unfortunately, delivered late because we needed more time to get the science right. Also of course Robert Napier is chairman of the Met Office board, but he's never tried to influence our science.

We also do projects funded by the oil, coal and airline industries. Hopefully this means that my potential conflicts of interest might balance out.... :-)

@Richard Betts Oct 2, 2011 at 11:03 PM

But what are the actual implications of this? I wonder whether it might actually make the AR5 appear to cite less peer-reviewed literature than it actually does? (ie: leave it more open to easy criticism). I say this because it will be easy for someone to go through the reference lists and simply see which references are from academic journals, and this might be assumed to be the indicator of which references were peer-reviewed - however, non-journal publications may also be peer-reviewed (I know that my own consultancy reports often get independently reviewed even if they are not for a journal).

From where I'm sitting, the (continued, but now "legitimate") failure to flag certainly leaves the IPCC more open to criticism! The IAC had handed the IPCC a "lifeline" which it clearly failed to grasp. Although I'm inclined to think that your concern that this might make AR5 "appear to cite less peer-reviewed literature than it actually does" is, well, highly unlikely :-)

Going through the lists of (18,000+ in AR4) references to see which ones actually were from academic journals is precisely what we had done in Donna's Citizen Audit - and is what precipitated the "birth" of AccessIPCC (where, as you know, we identified a number of other concerns).

For years we have been told that the IPCC's "gold standard" assessment reports are based on "all peer-reviewed" literature - when in fact this is clearly, and significantly, not the case. As we found with AR4 (and - as the IAC had noted - as was found by other researchers for AR3) approximately 1/3 of the references were not from academic journals.

This oft-pronounced "all peer-reviewed" mantra has been wielded as a sword [to exclude valid criticisms and contrary views] and as a shield [to protect the "integrity" of the science]. Particularly by Pachauri - although since the Citizen Audit, he has changed his tune somewhat, as I had documented over a year ago ... But I'm not sure what his position might be today!

What is also not acknowledged is the huge elephant in the peer-review room: i.e. peer review does not include any audit or examination of the data and methodolgy underlying the results. And, unless I'm mistaken, the IPCC assessment process does not include such audit or examination either.

FWIW, my reading of the new, improved (but apparently as yet uncirculated to AR5 authors) guidelines is that blogposts and newspaper articles are not deemed to be appropriate source material. However, my recollection is that the new guidelines are silent on the appropriateness of citing material produced by NGOs such as WWF and Greenpeace.

Yes, I'd seen Donna's post on the 78 AR5 folks on the WWF Climate Witness SAP list - Richard Tol (another AR5 CLA) tweeted it!

Just for the record - and in the interest of truth in posting - Donna was not claiming that all 78 from the WWF Climate Witness SAP list are involved in AR5, but rather that some of the 78 were involved in both AR4 and AR5 and others in one or t'other :-)

We also do projects funded by the oil, coal and airline industries. Hopefully this means that my potential conflicts of interest might balance out.... :-)

LOL ... Now you're beginning to sound a bit like Pachauri ;-)

Richard

Lots to digest from your previous post but just to ensure that I am up to date with IPCC current practice the IPCC principals paper was last amended 'Twenty-Ninth Session (Geneva, 31 August – 4 September 2008)'.

The IAC recommendations we were discussing are from October 2010, is there an amendment in the pipeline or a meeting planned to update the current practice, as indicated by hro001?

hro001

If "non-peer reviewed" status is indicated simply by not being in an academic journal, then isn't this clear simply by looking at the reference list? ie: if a statement is made in the chapter and a citation is given, readers can simply look at the reference and see whether it's in a journal or not.

Mind you, I guess that relies on people actually reading the main volume as opposed to the Summary for Policymakers or Technical Summary - most people don't! So it's probably a fair point that great care should be taken to avoid headline statements in the SPM and/or TS that rely heavily on "grey literature". As I said, we've been clearly instructed to be careful with this kind of material.

I think part of the problem with grand statements being made about the IPCC being "all based on peer-reviewed papers" is that WG1 make a big deal of this (or at least, it being mostly based on peer-reviewed papers!!) and this gets interpreted as being representative of IPCC as a whole - even by people who should know that WG2 and WG3 have always had to use much more "grey literature" for the reasons I described earlier.

Unfortunately there is has always been a bit of a tendency for the IPCC community (and indeed the wider climate science community) to be somewhat tribal - physical scientists being somewhat disconnected from biological scientists and social scientists etc.

Good point about peer-review not being perfect anyway - I meant to make that point myself in an earlier post. We can only do our best under the circumstances of the current setup! I guess the main point is that the whole thing should not be over-sold as some kind of authoritative final answer - just the latest picture in an ever-evolving an uncertain field.....

FWIW I am engaging with authors of some of the key papers I'm looking at for AR5, testing some of the statements they made and asking for further info. So far everyone's been very helpful!

Thanks for the clarification on the smaller numbers for the SAP / AR5 overlap!

hro001

Oh, I meant to say, yes the unsuitability of blogs and newspaper articles as information sources has been clearly communicated...!

Hi Lord Beaverbrook

The info on updating the procedures etc is on the IPCC home page - see the boxes on the right-hand side which give information arising from the IPCC plenary meetings. The most recent was May 2011, and the next is in November. I'm not part of that process (thankfully!) - this is for the govt representatives, not the science report authors.

Richard

Also:
http://judithcurry.com/2011/10/04/climatic-change-special-issue-on-uncertainty-guidance-for-the-ipcc/

This can turn into a full time occupation just keeping up with the recommendations, I now understand your previous comment.

Statistician and sceptical blogger William M Briggs has signed up as an IPCC reviewer. Authors of WG1 Chapters 9, 11 and 12 will need to make sure their statistics are correct!

Lord Beaverbrook: Indeed!

PaulM: Good! Hopefully he'll sign up to review WG2 as well, when the time comes.

He's going to have his work cut out to beat Vincent Gray's reviewing of AR4. I am still convinced he's one of the few people on the planet to have read the whole WG1 volume cover-to-cover, line-by-line!

Another post by Dr Curry, one of the comments is telling:

Richard S.J. Tol | October 6, 2011 at 3:30 pm | Reply

As a CLA for WG2, I see little appetite for change in the IPCC. The 0th order draft of AR5 makes many of the same mistake as AR4: Selective citation, gray literature, a tendency alarmism. The conflict of interest policy is a farce. Although there are forces in the IPCC that want to change this, there are other forces that do not.

The IPCC may have lost the trust of some or even many, it seems to continue to enjoy the confidence of the majority of its clients. Many of the IPCC member states are autocracies, who could not care less about what you write on a UK or US blog. Other member states are democracies, but the IPCC delegates are faceless bureaucrats, with a limited understanding of the new media.

At the end of the day, the IPCC is a UN bureaucracy serving the needs of national bureaucracies. Mutterings on the internet will not affect those needs and how they are served.

Oct 7, 2011 at 1:46 PM | Lord Beaverbrook

Thanks for that. Hopefully Richard Tol will have made the most of the opportunity he had to comment on the Zero Order Drafts of the WG2 chapters (I know I did).

Hans von Storch will also have had access to the ZOD - I hope he made the most of this opportunity too.

Hopefully when the WG2 First Order Draft goes out for review next year we'll have lots of people signing up to review it, so if the things he describes are the case in the FOD they can be challenged by more people.

We need to get folks like Roger Pielke Snr involved again - sadly he gave up on IPCC years ago and didn't review AR4, which was a great shame.

It appears that I owe Richard Betts a pint! (See page 1 of thread).

I applied to be a reviewer of AR5 WG1, which requires declaring oneself to be an expert.
On Friday I got an email from the IPCC inviting me to be a reviewer and after clicking a box agreeing not to distribute the draft, was given access to the 'First Order Draft' and full instructions.
See this post where Anthony Watts describes his surprise at being accepted as a reviewer and several others say that they have signed up too. I'm not aware of anyone who has signed up and been refused. There are now over 1000 reviewers.