Discussion > What does "robust" mean?
Impervious to evidence.
geronimo
Martin A
This came from discussion of the Lewis method.
Unfortunately that does meet the scientific definition of robustness, being very sensitive to the size and start/ finish dates of the temperature record used.
For example, consider this post at AndThenThere'sphyaics.
http://andthentheresphysics.wordpress.com/2014/02/01/understanding-lewis-2013/
The writer points out that Lewis adds six years to Foster's existing data and gets a much lower sensitivity. Hardly robust if the output varies so much with small variations in input data.
Entropic man
1). It took me ages to torture the data into producing this result and I don't want any discussion of how flaky it is.
or 2) It's essentially punctuation, boilerplate, padding. It has no meaning.
(Perhaps I am being unfair. To punctuation.)
rhoda
In the chemical scientific literature "robust" is a term commonly used to describe a procedure or preparative method which is reproducible (Not necessarily useful. That is something else.).
Thus a person with ordinary skill in the art should be able to follow the same method and get the same results. In fact, even a slightly cack-handed operator or lackadaisical approach might be expected to produce approximately the same results by using a "robust" method.
It goes without saying that IPCC model-ensembles of climate simulations do not meet this standard.
michael hart
Michael hart
Your definition of robustness works well on the laboratory bench, under controlled conditions with pure reagents. Under such conditions the link between theory and experiment is easily made.
Unfortunately life is a little harder when the scientist has to take a further step from the laboratory into the real world, where observation is possible, but replicates and controlled experiments are not. There is also much more wear and tear on the experiment and the experimenter.
Just for fun, try repeating your standard robust laboratory procedure, using the same apparatus and reagents, outside in a thunderstorm. :-)
In an earlier conversation with Martin A we discussed the difference between measuring the conductivity of silver under laboratory and field conditions. Studies of climate face similar problems.
If you insist on applying laboratory standards to all science, then you limit science to the laboratory and deny the possibility that any science can be done beyond its walls.
Entropic man
Thanks for the advice Entropic Man.
I've done quite a few experiments outside. I've also done them inside living organisms, which is pretty complex too. Students of 'climate-change' seem to have the problems you describe, not me. They need to escape from inside their computers every now and again. They also have a problem with the English language, at least as far as the word robust is concerned.
michael hart
Apr 3, 2014 at 10:09 AM Entropic man
EM - this thread is a discussion on what "robust" means in climate science/propaganda contexts - not on whether some result obtained by Nic Lewis is robust or not. Perhaps he'd respond to what you have to say if you posted it on a thread related to his paper.
Do you actually have a reference to "the scientific definition of robustness"?
Martin A
Great subject for a thread and excellent shooting down of a diversionary move from EM, Martin. I've also learned about a perfectly reasonable use of the word in chemistry from Michael Hart. But I have to point out that you shot yourself in the foot with the example you initially gave:
The scientific basis for significant long-term climate risks remains robust, despite the points raised by the Mail.
Despite the points raised by the Daily Mail. Is there a clearer or more important use of the term than to be able to dismiss criticism from that source and not have to dignify it with proper consideration?
Richard Drake
Shot myself in the foot? Huh?
Please carefully re-read my original posting.
Did I fail to make it clear that the quote I gave was an example of how the word is used in CAGW propaganda to give the impression that something is impervious to being challenged?
Martin A
I know. I was just pointing out that in the demonology of those that push CAGW propaganda the Daily Mail has a special place. Your bullet through the foot was wholly ironic; theirs isn't.
Richard Drake
Michael hart
Then you'll understand the problem. Measurement of the existing climate is not the problem. Standardised measurement techniques generate temperature, humidity and radiation data to the limits of accuracy of the equipment.
That data and physics theory is then used to generate hypotheses. The test of a hypothesis is its ability to predict the result of experiments, or further observations.
Under lab conditions you can design experiments wit proper controls, to isolate the dependant and indepepent variables of interest from all the extraneous effects.
To achieve this for a planet is harder. There is only the one, so replicates using different rates of CO2 increase and a null control with stable CO2 are not available. The closest you can get is to use comparison with past conditions, or comparison with other planets. This is why the PETM is of interest, as the closest past parallel to the current situation.
The other approach is modelling, using physics and data to project conditions. I think you underestimate them.
Given accurate data they produce good results. The problem is that some of the data is not available. If your want to project future climate you must approximate unpredictable variables such as future vulcanism, aeroaol production, CO2 etc. This is impossible. To get around this you do repeated runs of your model with different values chosen on the basis of past experience. They are then summed into a range of outcomes and probabilities. The average of the model runs reflects faster warming than actually occurred because the average parameters were more forcing than actually occurred.
The real world temperatures were towards the lower bounds of the projected temperature range because the real world forcing variables were also close to the lower bounds of their expected range. Only those models incorporating real world variables projected real world temperatures, as you would expect.
Martin A
I like Michael Hart's definition of robustness, despite its limited applicability outside the laboratory. It does, however rely on the assumption that laboratory practice can be applied.
Since field and observational sciences such as ecology, astronomy and climatology cannot test hypotheses using laboratory techniques the teat of robustness would need to be based on probability.
How about this?
"If the value of a dependant climate variable projected from theory is within the 95% confidence limits of the observations of that variable, the projection is robust. If the underlying theory is sufficiently skilful to predict a number of projected dependant variables to the same level of confidence, the theory is robust."
Entropic man
So sometimes occasionally a model gives a value roughly consistent with observation. That really does not mean anything. If all the models all the time got it right, even that would not be robust unless they were validated in some other way. I have often asked for intermediate results from models which could be checked as they go along. By which I mean anything one could measure in the world which was independently predicted by a model. Like the tropospheric temp, for example. Or humidity. Or cloud. Never had a reply from a modeller on that, but never mind. Never had a clear response on the number of aborted runs. Nor on why the worst-performing models are not excluded from the ensemble. This does not smell of robustness. Nor does taking the very lowest of the range of predictions and claiming it meets with observation in any significant manner. Clutching at straws. Straws are not robust.
rhoda
Rhoda: +1. I've gone further and asked for all model results that are used in reports for policy makers to be reproducible by any of us on an affordable computer, so that we could study any intermediate results simply by stopping the program and stepping through it in a debugger. Likewise we could change starting conditions and see what happened. The affordable machine part would mean no doubt that current AOGCMs would have to be simplified. But given the apparent success of Guy Callendar's even simpler model, run on a bit of paper and pencil, I don't see this as a major problem.
Richard Drake
I like Michael Hart's definition of robustness, despite its limited applicability outside the laboratory. It does, however rely on the assumption that laboratory practice can be applied.
That's kind of you, EM, but I don't consider it limited to the laboratory. I think you almost got it right when you identified the ability (or lack, thereof) to ensure adequate control experiments. The good scientist is always aware of this, wherever the experiment takes place.
I think my original statement reflected this when I used the term "a person with ordinary skill in the art". I think the use of the term "robust" seems rather more forgiving in climatology, which says, I think, a lot about the state of their art.
[If you think climatologists have large, unavoidable, uncertainties (I dislike "confidence levels", but that discussion is for another day), then you should see Immunologists'. But Immunologists don't pretend otherwise.]
michael hart
...Immunologists don't generally pretend otherwise.
michael hart
" In fact, even a slightly cack-handed operator or lackadaisical approach might be expected to produce approximately the same results by using a "robust" method"
I like that (semi-) definition, since it calls to mind the sense used in a large variety of technological contexts.
During an oral argument a judge asked me once what I meant by "robust," and, although I believe my answer was fine, the episode stuck in my mind because the judge's having asked it showed me that I had been obscure.
In that case the context was a control system, but the answer applies to communications systems, user interfaces, and much more: even in situations that depart from the precise target situation, the system produces a result close to what was intended if it is robust. The control system is robust if the desired position is pretty much reached even in the presence of forces external to the control force; the communications system is robust if the recovered signal is correct even in the presence of noise; and the user interface is robust if an error in entering a command is relatively unlikely to trigger an unintended operation.
As several commenters have observed, CAGW proponents tend to use that term when findings are anything but robust.
Joe Born
Martin
Your objection to the use of 'robust' is just a re-run of your objections to 'consensus'. It means what you suggested it means, ie "not sensitive to the precise details of assumptions, parameter values, small measurement errors, and so on".
> The scientific basis for significant long-term
> climate risks remains robust, despite the
> points raised by the Mail.
That is a polite way of saying that the Mail doesn't know what it is talking about.
Chandra
Rhoda
When some of the variables included in a projection run are unknown at the time the model is run, how do you define the worst- performing models, except in hindsight ?
Richard Drake
Models come in various levels of complexities. The 1D model used for Hansen (1981) was a single vertical column of atmosphere and ocean with minimal detail. The latest models simulate energy flow through and between a planetful of 50km grid squares, using all the physical processes they know of. I suspect that simplifying a model to the point at which you could tinker at home would greatly reduce its effectiveness.
The simplest open source model is at Roy Spencer's website here it runs on Excel.
http://www.drroyspencer.com/2010/04/simple-climate-model-release-version-1-0/
NASA has a downloadable model on its website, it is a frozen CMIP3 mosel known as Model E1, available via this page. Other source code and data is also available.
http://www.giss.nasa.gov/tools/modelE/
The more complete physical models do take a while to run. If I remember correctly, three weeks on a high end PC is not unusual. To process the models used in weather forecasting needs something much bigger and faster, since they have to do multiple runs in faster than real time.
Michael Hart
The climate scientists are very aware of the potential errors. Read the literature and you will see considerable discussion thereof. The variability in climate sensitivity estimates derived by different methods is only one example.
Read AR5 WG1and you see the output expressed in terms of a range of probable outcomes from several different possible scenarios. The uncertainties are described and quantified.
The problems come when scientists have to interact with laymen, especially politicians. It is hard to discuss probabilities with those to whom any hint of uncertainty is seen as a weakness.
I' m interested in exploring your concept of robustness further. In studying climate, comparison and modelling have to substitute for replicated cotrolled trials. Might I ask what strategies you would use to maximise robustness under those conditions?
Entropic man
EM, if you are saying that models given incomplete information sometimes give good results and sometimes miss, and all the models have good and bad days, then none of therm are much good for this particular purpose. This is the main problem with modelling climate. Not the models but the use to which the results are put. At the stage we are now, the models are only good for learning how to make better models. A noble ambition. They are not good for predicting the future. Now I would contend that some are better than others. But 'they' never throw out the worst performers. I contend that this is for political reasons. They can't offend the national institution which produced the offending model. Haven't they heard of natural selection? We would get better models if they had to compete, not all contribute to some unconvincing fan of 'predictions'. And you wouldn't have to come here defending the indefensible for the honour of the climate gang who are hiding under their desks right now.
rhoda
That is a polite way of saying that the Mail doesn't know what it is talking about.
Apr 3, 2014 at 3:09 PM Chandra
Chandra,
It's better to take words on face value than imagine things about them and then discuss what you have imagined as if it were fact.
If you had said "I surmise that is a polite way of saying that the Mail doesn't know what it is talking about"
I could accept what you say.
But what the CCC said also be taken to mean (depending on your viewpoint):
- The Mail knows perfectly well what it is talking about but is lying.
- The Mail is perfectly correct but what it says is utterly irrelevant.
- The Mail is perfectly correct and we are embarrassed but we are going to bullshit our way out of this one.
Discussing things we have imagined is unlikely to lead anywhere, so best avoided. Does that make sense?
Martin A
"If the value of a dependant climate variable projected from theory is within the 95% confidence limits of the observations of that variable, the projection is robust. If the underlying theory is sufficiently skilful to predict a number of projected dependant variables to the same level of confidence, the theory is robust."
Apr 3, 2014 at 1:05 PM | Entropic man
EM - I assume that you made that one up and I see what you are getting at.
The fact (assumed) that you made it up indicates that a standard definition of the term does not exist. Otherwise someone as well-read in the climate science literature as you would have pointed to it immediately.
This confirms to me that it is currently used only as a bullshit word without any real meaning but with the intention of impressing the gullible or the unquestioning.
For example...
(...) Using Virtual Met Mast™ is a reliable and scientifically robust method of generating a long-term wind climatology.
(...) The UK enjoys a reputation for strong and robust science on the international stage. In the field of climate research the Met Office is widely acknowledged as world leading.
(...) As the Met Office report states ‘With a credible modelling system in place it should now be possible to perform scientifically robust assessments of changes in storminess, the degree to which they are related to natural variability and the degree to which there is a contribution from human-induced climate change.’
(from Met Office publications and communications)
Martin A
Martin, I'll go for 4) unyielding.
Rob Burton
Martin:
The fact (assumed) that you made it up indicates that a standard definition of the term does not exist. … This confirms to me that it is currently used only as a bullshit word without any real meaning but with the intention of impressing the gullible or the unquestioning.
Case proven, unless some standard definition can be found asap, as you say.
EM: Thank you for your thoughts on my "all model results runnable on an affordable machine" proposal. I don't think this is the place to delve into it further - or, at least, I don't have the time to do that this week. I accept that it will take a little while more for climate openness to mean this but I think in the end it will. And this will be a good influence on many other forms of software modelling.
Richard Drake
" Might I ask what strategies you would use to maximise robustness
under those conditions?"
You can't as the model is only a conjecture or a theory. Robustness would come from observing things or measuring things or conducting controlled experiments (which you obviously observe and measure too)
Rob Burton
The C.O.D. gives the following for "robust":
1 (of a person, animal, or thing) strong and sturdy, esp. in physique or construction.
2 (of exercise, discipline, etc.) vigorous, requiring strength.
3 (of intellect or mental attitude) straightforward, not given to nor confused by subtleties.
4 (of a statement, reply, etc.) bold, firm, unyielding.
5 (of wine etc.) full-bodied.
There is another colloquial usage in scientific discussion, generally meaning something like "not sensitive to the precise details of assumptions, parameter values, small measurement errors, and so on". I'm not aware of a precise definition in scientific use.
I have noticed that in things related to CAGW the word seems to have taken on a different meaning. It seems to be an add-on bullshit word to give the impression that something very questionable is not open to being questioned. For example:
Any more examples?
Any good definition of what "robust" really means, in the context of CAGW propaganda?