Monday
Apr162012
by 
Bishop Hill
Bishop Hill Quote of the day
Apr 16, 2012 Climate: Models One climate modeller we interviewed explained that the climate is a ‘heterogeneous system with many ways of moving energy around from system to system’ which makes the theoretical system being modelled ‘tolerant to the inclusion of bugs'.
From the Pipitone and Easterbrook paper on validating climate model software, currently the subject of a guest post at Judith Curry's.
Reader Comments (129)
To use the plane flight control software analogy....
This sort of shit just won't fly anymore...
Anyone with a knowledge of exponential error accumulation would know that.
2 weeks is all that's needed to turn any recursive prediction into a random number generator.
Which would indeed be a major problem if the GCMs were recursive.
Mounting errors in the GCMs appears as drift, which is not the same thing at all.
I did go through the paper some more, cut out all the unnecessary flannel and it reduces to a couple of pages of relevant information.
The question I always ask is why is Climate Science different? It always is. Put elements of it front of seasoned professionals and they go "that is not the way it is".
Microsoft pulled the same stunt many years ago. Steve Ballmer was threatened by Linux. Not only was the Linux world producing better quality software, it was doing it for nothing and with no marketing budget. It was the old Captain Kirk scene of getting the computer to form a paradox: "Cannot compute, cannot compute". Ballmer could not compute.
Starting to see the parallels?
He, and everyone else, knew that Windows did not deliver based on its marketing. Trouble was he could not buy Linux out - there was no one to give wads of money to. So Microsoft had some "independent" studies showing Windows was better quality.
The market did not buy it. The consumers stuck with Windows, but the mission critical server market was lost to Linux. Microsoft had to change and the result is that the current Windows are the first Windows that are actually any good. And the words "Open Source" do occur within the Microsoft Universe.
Strange parallels.
Apt definition:
Model - a small imitation of the real thing
People mention aeroplane software in relation to bugs... why do I immediately think Chinook FADEC? (A salutary tale)
Bugs can be nailed to a certain extent by auditing - but I see no mention of any independent third party inspection/test. The dismissive hand wave of ‘tolerant to the inclusion of bugs' without qualification smells very strongly of "we know the answer we want, and we can ignore anything we like that's contrary to the required result" - also commonly known as full blown stupidity.
Many of he climate modeller's assertions are self evidently wrong on so many levels that it's difficult to know where to start dismantling them - hopefully they'll collapse unaided - but let's not take the risk eh? Controlled demolition as a spectator sport.
I suspect that the reason they think they can 'get away' with bugs is that the software is highly tuned to the results, so in the end it doesn't matter if bugs are contributing or compounding errors, as long as the algorithm gives the matching curve at the end.
Of course this blows out the water the idea that the models are based on known physical laws, because chances are there are algorithmic twists introduced by poor coding, fudge factors, edge-case defects etc.
I have a guess that the code isn't written in a modern discrete way that makes it injectable, testable, etc.
What is clear to me from that paper, and the reason for its scratching around trying to define what "quality" is and miraculously saying the models are "quality" is that there is no Customer.
There is no hand over to, and quality assurance process by, the end Customer.
If there was an independent body (the Customer) set up to go through every aspect of the deliverable, and present the results, then we would soon find what quality is enshrined in the whole process. But as the models deliver pretty pictures that fill a narrative no one is interested in that.
Using CVS changes on a closed project as a basis for quality definition is not a valid metric.
"...there is no Customer."
I expect that the customer is the government, who pay for it with our taxes, and use the results to formulate public policy. Unfortunately, HMG's record with large software projects (NHS, HMRC, FADEC, etc.) is truly terrible - a fact which I think that they now admit. Literally billions have been wasted on projects that never saw the light of day. The usual gang of management consultants hired by HMG to oversee projects clearly failed as well.
In applications where "failure is not an option", such as the A380 flight control software, effective procedures of quality assurance have been found - but I understand that they are very expensive. Many years ago I was told that, line for line of code, safety critical software on an aircraft cost ten times more than any other software - this may still be the case.
I do not have a solution, other than to opine that it is a job not worth even attempting, as it is mathematically impossible to produce meaningful, long term forecasts using ill defined climate models, containing so many uncertainties.
Jiminy Cricket writes:
'What is clear to me from that paper, and the reason for its scratching around trying to define what "quality" is and miraculously saying the models are "quality" is that there is no Customer.'
Have you ever worked on a big project in which there is no customer? (This is not intended as a hostile point, it is a genuine query). Above, Jiminy makes the point is made that things in climate science are 'always different' to the way that professionals do them.
But in very large parts of science - from Isaac Newton through Watson and Crick to the Large Hadron Collider - there is no customer, at least in a direct sense that there would be for a commercial project. So is Jiminy saying that science without a customer cannot be relied upon, or that it is harder to get right, or impossible, or what? (You could say that climate science is different because direct comparison to experiment takes longer than in other branches of science. That's an interesting discussion, but I don't see what it has to do with there being no customer.)
What experience can software professionals bring of projects with no customer? Do you suggest setting up a 'surrogate customer' or is that just trying to force climate model development into the only box you have experience of?
Not if the code leaked as part of Climategate 1 is anything to go by, but those techniques are part of software engineering, and therefore not relevant. ;-)
There is always a customer, even if the customer is only me. This is, however, missing the point of unit and integration testing.
In an environment where the requirements are fluid, the unit and integration tests act as a de facto requirements specification. In other words, the tests express our understanding of how the software under test - usually a single method, in the case of unit tests - is supposed to work. If you want to know how the software works, the test suite is often a good starting point.
The tests should encapsulate everything from "known good paths", including parameters at the extremes of their acceptable values, and "known bad paths" including parameters that exceed those extremes. The latter tests are important as understanding the way a method breaks is as important as understanding how it works.
Tests also provide a pointed stick with respect to the way you have implemented your functionality. If writing a test is difficult, perhaps because the method in question is complex, then this is usually a good indicator that the problem in question should be broken down into smaller functional primitives that can be tested.
For Agilistas, tests provide reassurance, allowing us to code quicker. The change -> test cycle allows us to make a series of changes with the knowledge that we haven't inadvertently broken something else that relies on what we just changed.
Throg writes:
'There is always a customer, even if the customer is only me.'
So when Jiminy Cricket wrote:
'What is clear to me from that paper, and the reason for its scratching around trying to define what "quality" is and miraculously saying the models are "quality" is that there is no Customer.'
he was either wrong, or just meant something like 'they are not taking this seriously'? Re-reading his comment at 9:46 it looks to me as if he really intended to say that without an external customer, quality control in a large project is hard.
Sorry, I didn't answer the final point.
Software development is an engineering discipline. Software developers don't work in isolation - like all good engineers we learn from each other and share good practice. Indeed, we share our experiences of bad practice (anti-patterns) as well. It is therefore not a case of what any individual has experience of as such, as opposed to the wider community.
This particular sellsword has worked for numerous clients over the years. Every customer believes that they're a special case. Usually, they're wrong about that.
@JK
No-one is saying that quality control isn't hard. Surely this is true of every engineering discipline?
Thanks Throg. I agree with much of what you say about engineering and agility.
When I tried to paraphrase Jiminy Cricket as saying 'without an external customer, quality control in a large project is hard' I got that wrong (always better to quote, but then I thought I would land up quoting the whole post...). Of course quality control on a large project is always hard. Jiminy seems to be saying that an external customer is in some way necessary for it, but better to wait an see what he has to say for himself.
As to your points on agile methods, the Pipitone and Easterbrook paper highlight two problems which they say are commonly encountered in scientific computing (not just climate models):
They write:
"There are two fundamental problems that make impossible the traditional notion of testing by way of directly comparing a program’s output to an expected value. The first is what Hook terms the tolerance problem: it is impossible, or very difficult, to tell if errors in output are completely free of unacknowledged error since it may be difficult to bound acknowledged error, and even with a bound on acknowledged error it is impossible to detect unacknowledged errors that fall within those bounds....
The second problem is the oracle problem: "available oracles are problematically imprecise and limited". That is, for certain inputs there may not exist a source of precise expected outputs with which to compare a program’s output."
You can read that in a bit more detail on their pages 353-354. I think they make a reasonable case that these problems must indeed arise where computers are being used to extend scientific understanding of a system. This seems a bit different to the problem of changing specifications, which as I understand it, agile engineering is primarily designed to solve. What are the closest parallels to these problems in other areas of software engineering?
PS on the structure of climate model code, you don't need to fully rely on guesses based on leaks in emails. I found at least four GCM model codes available free online with a few minutes googling (NCAR, MIT and GFDL required registration, which they claim is a formality, but you can look at GISS model E without registering).
@jk
"Above, Jiminy makes the point is made that things in climate science are 'always different' to the way that professionals do them." that is putting words in my mouth. This is no implication that people involved are unprofessional as individuals. "Seasoned professionals" (my precise term) look at the results as a whole from the outside. And I have done enough Project Audits in my time to understand the difference between a project and an individual. And I certainly never said quality is not taken seriously - that still does not mean quality will result.
There is a boundary for a project. When something crosses that boundary it is a deliverable. To some entity. I used the word customer because I think that is the easiest word to use - someone always has to pay the bill, whether they are interested in reading the invoice or not. If it doesn't cross that boundary the deliverable does not exist. It stays within the black box.
The output of Climate Models should be delivered. They should cross the boundary. They are used to form policy costing billions of pounds.
I believe we actually have a negligent customer. An absent customer. There is actually no real delivery taking place. They are not reading the invoice. The bill is just being paid. The vendor is trusted.
You can quite rightly say anything staying within the box is of high quality. That is meaningless. To talk about quality without reference to results it is just academic masturbation.
And when they try to compare the quality of such a closed project to an OPEN Open Source, which has very short delivery time-scales to real customers? Based on CVS check-outs? I find that an unsupportable approach.
I have done every single QA role. From running an integration testing team for Barclays, through to running the acceptance test, on behalf of the Central Bank, for a one country's domestic payments system. For every step there was a boundary and I took delivery. For deliverables that were defined. Of course that can present a problem. Much is made that science cannot be written in a specification that gives expected results to be tested against: new frontiers. But equating something like CERN with Climate Modelling would be a little disingenuous. That is against the bedrock of known physics. Not chaotic system like the climate.
The point is not that the Climate Models produce interesting information. It is the confidence you place in those deliverables. I believe that actually the quality of those deliverables is poor, because the modelling confidence percentage is actually low. No one wants to hear that.
I wanted a Mini and I get Rolls Royce. The Engineer in me (yes I am one as well) say great quality. The customer in me says I am not paying the bill it is a piece of crap.
Perhaps it can be said that Science is a unique case. That is applies its own internal QA, through it's networking and mature communications. Perhaps, but when entering the political arena and running after huge budgets that quaint view of science can be difficult to trust in. I have never given open trust to a Vendor, and that is a good place to start with an any QA exercise.
Good points, Throg and Jiminy. To reiterate Jiminy's point, recall the anonymous reviewer's vital paragraph:
All the unit testing in the world (and I fully agree with Throg's emphasis as an agilista there) isn't going to cross this divide. Political leaders themselves are the customer, on behalf of all of us. They need to start to act like it.
What experience can software professionals bring of projects with no customer? Do you suggest setting up a 'surrogate customer' or is that just trying to force climate model development into the only box you have experience of?
Apr 17, 2012 at 1:10 PM | JK
You talk rubbish and have quite clearly never worked in a 'real' software development environment. CERN will almost certainly have a software audit unit, independent of the development team as well as a full version control and interface management team. Software quality would be assured through the quality management system which contains such gems as required software standards.
The strategic development objectives would have been in the funding document and this would have been used to develop both management tactics and unit objectives for each module of the software culminating in a test script for each module and an interface specification document for each team and each module.
Now, please show me where CGMs you mention on the web have hidden this documentation. It was made blatantly obvious by climategates 1 & 2 that database management in that particular community of modelists was non-existent as was any form of version control.
Having been both a customer (for an expensive but very modest economic model - which worked pretty well) and a project manager (for a customised interactive product between members of the public and head office) I am staggered by some of the assertions made by scientific model users and their cronies in the IT field.
The economic model I was the customer for was a synthetic model of income and expenditure across the national population, broken down in various ways among population sectors. It was required because the Bureau of Statistics national data was only fully updated every three years, and then required a lot of further work to break down into the information we needed a couple of years before it would be available. This model, which when checked against the real data a few years later, was remarkably accurate. But - it cost a lot of money, and was based on extrapolating from previous ABS data and adjusting for changes in demographics, wages and prices, income support policies etc. It took about 4 months to build, and another 5 months of testing and fiddling before it was signed off. And, no-one claimed it would be accurate beyond the forecast period of about 3 years.
It was rigorously tested against the highest QC standards before either the vendor or the customer signed off on it. The guys who built it had tremendous commitment to getting it right, and frankly I doubt if they made much money out of the process, except learning how to do it for future clients. There was no trace of the casual attitude to QC, testing and verification that infests climate science. The documentation (every step was logged) was massive, and accountable in every particular.
I won't enumerate my project management story (life is too short!) but one thing is for sure - the attitude that stuff like documentation, systematic testing and debugging, milestones, deliverables, clarity of objectives and parameters etc are for hacks in the commercial world is just so off the graph that I wouldn't trust these models to do what the chip in my $20 watch does so well.
From the smug experts here, it sounds like programmers have got software quality nailed - it's just those ignorant scientists who can't get it right! So it must just be gremlins causing my phone to freeze, my word processor to screw files, my PC to sulk when I open the lid and iTunes to lose my music.
In case you get too rosy a view of software QA, be aware that many companies have no QA standards, do not adhere to supposed 'coding standards', do not hold code reviews or do so just as a box-ticking exercise, don't know what validation is, do minimal testing, don't know how properly to use their very expensive (and most likely obsolete) revison control systems, use outdated versions of compilers and other tools, and on and on..... Shocking? Not really. They have deadlines and products must be shipped. Like many well known companies, they fix problems later when they arise. Even paragons of virtue like the Debian distribution of Linux don't imagine they will fix all bugs before a release (see http://bugs.debian.org/release-critical/).
Are the climate models perfect? No of course not. But if they were written by scientists and post-grads etc they cost a fraction of the price you would pay if professional programmers got anywhere near them. And my guess is they would not work much better in the latter case. And if you think the pro's would document their models any better you can forget it. There is only one thing pro's like less than testing and that is documentation.
By all means be skeptical of models. But scientists are not programmers so don't judge them by the same standards.
If the issues is, scientists aren't programmers, then rather than leave coding to amateurs we should get scientists hire programmers to do the coding. After all nobody would let amateurs play with dangerous chemicals in labs, and that's why there exist lab technicians.
But there's a cheaper option, and that is to publish the code so others will do the QA and for free.
Judith discusses further:
From Systematic Evaluation of Complex Simulation Systems with Uncertainty Quantification
"publish the code so others will do the QA and for free"
Bitbucket, JK,
Surely, this is an offer that you can not refuse?
Bitbucket,
"In case you get too rosy a view of software QA, be aware that many companies have no QA standards, do not adhere to supposed 'coding standards', do not hold code reviews or do so just as a box-ticking exercise, don't know what validation is, do minimal testing, don't know how properly to use their very expensive (and most likely obsolete) revison control systems, use outdated versions of compilers and other tools, and on and on..... Shocking? Not really. They have deadlines and products must be shipped. Like many well known companies, they fix problems later when they arise."
It depends upon the application, and the cost of failure.
If johanna gets it wrong she gets fired. If the A380 software screws up aeroplanes crash and burn......
The cost of climate models getting it wrong about carbon dioxide is £100 billion in extra taxes.
Roger Longstaff: "The cost of climate models getting it wrong ... is £100 billion in extra taxes."
And the cost if they are right and we do nothing about it?
omnologos: "hire programmers to do the coding"
That is bound to work out well - just like it did for the new NHS system, right?
"And the cost if they are right and we do nothing about it?"
If there was any evidence that anthropogenic CO2 could harm humanity then you may have a point - but there isn't. The output of unverified and unvalidated computer models is not evidence.
Apr 17, 2012 at 9:46 AM Jiminy Cricket
JC has hit the nail on the head. That is the key thing.
Years back, I was asked to help sort out quality issues in a software department in a well known US company.
My first question was "What is quality?" Until they had understood what they were trying to achieve, they were not going to achieve it.
A deep question with one just correct answer. All other answers (bug counts per code line etc etc) just show a profound level of ignorance.
You can find the answer in "Zen and the Art of Motorcycle Maintenance". Quality is whatever the customer decides it to be, using whatever criteria the customer wishes.
I think, in this case, it is worse than saying the customer is asleep at the wheel. The customer's chief requirement has been that the models provide evidence of AGW/CAGW.
Hence the logic of the - at first sight - crazy statement
But as Roger Longstaff correctly says
But as Roger Longstaff correctly says "The output of unverified and unvalidated computer models is not evidence."
How well do models match historical temperature records?
I mean, if the models are set running from (say) 50 years ago, do they match the actual climate of the last 50 years? There must be a huge dataset available for such testing, and it should be clear whether the model cuts the mustard. Of course if the model does match history, there is no guarantee that it will continue to do so, but it would give a good degree of confidence.
@BitBucket... 50 years? Ignoring the fact that I can curve fit to anything... I do not need a super computer for that.
I always find it interesting that the time frames of Climate Science always relate to the span of a man's life, or even perhaps more interestingly the span of the transistor age. I do believe these two facts form part of its psyche.
We have others fields where the overall time frame is much shorter, where the data is more numerous, where the amount of human knowledge expended on analysis is much greater.
Yet we cannot predict the weather more than a few days in advance. We cannot create financial stability.
Let us create something called Knowledge Power.
Climate Science is like the 1 bar 1kw Electric fire I had in my seedy bedsit as a student, running since 1981. Compared to the all the furnaces of accumulated knowledge since the industrial age began.
BitBucket - your analogy makes no sense. I'm saying, just like there are technicians for highly professional scientific work in the lab, there should be technicians for highly professional scientific work on computers.
The NHS's and other giant projects have nothing to do with that.
Saying we should just accept shoddy code by scientists is like saying we should accept shoddily produced music where eg the Boss or Adele play every instrument having fired everybody else.
Not everybody, actually almost nobody is born a Lenny Kravitz. This doesn't mean Springsteen is no good, or worse than Kravitz.
The last thing we need is another 30 years of unprofessional climate-change-related computer code.
@Martin A
You pre-empted what I wanted to add. Follows on from your "Quality is whatever the customer decides it to be, using whatever criteria the customer wishes.".
It may come as surprise to those who practice the dark arts of programming (and I am a closet Perl programmer myself) is the Coding is a very small part of the Delivery Life Cycle. There are no "Masters of the Universe" coding geniuses. No spotty teenagers with a chin beard and skateboard in the employ of a Bond villain. No heroes hacking between the FBI, KGB and GCHQ networks at will.
Even with Agile style development (something I am great fan of, and was practicing long before it became a hip term often misused and misunderstood) the Programmer is still a tool - sometimes even in both senses of the word.
What about Open Source projects? Aren't they Programmer driven? Well they can appear to be. That though is often a function of a few things. The spreading of seeds (ideas) across a huge concrete market with most failing but some finding a crack and growing and sometimes cross breeding with others. And also that the people involved are multi-talented. You meet them and see a guy with a bad haircut, a MacAir in his Man Bag, holes in his T-shirt and riding a trendy Fixie bike. Your media driven preconceptions makes you see a Techie. Actually they have all the skills of entrepreneurs. Coding is only one part of their skillset.
So if a Project's quality is not about coding what is it about? As Martin A. said it is whatever you want it to be.
Projects/Companies often make the mistake of creating a QA Team/Department. And they will employ outsiders without a knowledge of the culture or the Products. These QA people will get out their Bible's and set forth. I have met many. my current company has two. They are so anally retentive. They are happy to have the authority of pissing everyone off. However, they have a panic attack when they fail to avoid dodging any responsibility.
There is a perfect example today in the news. The relaunch of the Shuttle in 2005 almost ended in disaster. This was despite going through one of the most rigorous quality regimes possible. They had missed this:
It turns out that the thermal cycles associated with filling the tank could crack the foam, especially in areas where there were two or more layers of foam.
As someone whose 200lt gas water boiler has an equivalent problem, and for any engineer, would be so blindingly obvious, how could this be missed? In an organisation that after losing two ships, clearly had to publicly show that quality was nailed.
Real Quality is simply about Risk Management. Risk Management is the interpretation and balancing of any number of factors. There is no manual, no bible, no ISO qualification. It is whatever you want it to be.
Risk Management has to be an integral part of the Project, its delivery and acceptance. However more crucially there has to be independent Risk Management. "Excuse me my water boiler had the same problem.".
And how can that that be summed up? Real Quality? Effective Risk Management? What they all should do is ADD VALUE. That is it: ADD VALUE. And you would be surprised how that is forgotten in overly complicated world of QA.
And how is that related to Climate Science?
The thoughts expressed here and in the paper are about why a few coders (Scientists or Programmers) can produce Quality results. What everyone in involved in this Life Cycle should have above their monitor is the following:
That defines the effective quality, not a few bug counts against number of lines of code.
Apr 19, 2012 at 9:17 AM Jiminy Cricket
Agree completely.
If a software organisation considers that "bug counts" have any meaning, that implies that it is a chaotic software development environment, probably where a bunch of programmers are told "ok we need a new system. Get cracking. Have you started coding yet? If not, why not?" .
The emphasis on counting bugs in the Pipitone and Easterbrook paper makes me feel that climate modelling software is probably produced in a pretty chaotic manner - maybe not quite as bad as revealed in HarryReadMe.txt but chaotic all the same.
In systematic business software development, errors can occur at different levels. Each level will be tested, verified and signed off before dependent work at the next lower level is started.
- At the highest level, errors in understanding the customer's business requirements can be costly and time consuming to correct. Such errors often arise because the customer's business processes are ill defined. Delivering quality requires a strong project manager who will tell the customer "I am halting work until this aspect has been analysed, documented and agreed".
The counterpart in climate modelling would be: "What is the purpose of this model?" (eg To predict next week's weather? To test our understanding of climate physics? To produce scary results to feed the CAGW propaganda machine?)
- High Level design of the system. I guess errors in database modelling would count here.
The counterpart in climate modelling might be using an incorrect physical model. You might say that lack of knowledge of some physical aspects of climate would automatically preclude the production of a quality product - depending on what the customer's real requirements were. Lack of understanding of feedback effects might be an example.
- Detailed design of the system. A counterpart in climate modelling might be an error in mathematical formulas.
- Coding errors. My observation is that when modules are produced by professional coders, these are uncommon and are usually revealed in module testing, long before the software gets anywhere near a customer.
But in a chaotic software environment, all errors tend to be counted as "bugs" irrespective of the level at which they were made.
omnologos: "your analogy makes no sense..." - respectfully, I think it does. Throwing 'professional programmers' at the job is no guarantee of success. I hate to break it to you but, speaking as a one, programmers are in general not the sharpest tools in the box. Your average scientist is likely to be significantly more intelligent. Sure he doesn't have a programming background, but many programmers have little formal training either. There are no 'professional programmers'; it is not a real profession.
@BitBucket, that is an interesting viewpoint. I would guess that over 29 years, 90% of the programmers I have dealt with have had University degrees.
Maybe I was just lucky.
"Your average scientist is likely to be significantly more intelligent", now that is just unfounded snobbery. .
No Professional Programmers... hmmm... methinks you are either winding people up or your world has been very limited.
Though I am beginning to see part of the problem. Scientist thinks programming is a trivial exercise. They are on a lower storey of the ivory tower.
JC: what does curve fitting have to do with climate modelling? Please don't
confuse the issue. The only way I can see to verify a model is to run data
through it for which the outcome is known. For example, run historic data
through the model and see if it predicts what actually happened. Are there
other methods?
@BitBucket, with the greatest respect some people are worth the effort of dialogue.
Whether it is your fortune or not, you do not fall into that category.
If you want to take that as an admission of defeat then please do so. My ego can take it.
JC: sorry, I posted before seeing your latest post. Maybe we are in different fields (electronics/embedded systems). Sure they may be graduates but not necessarily in programming (eg arts).
Am I a snob? Whatever floats your boat. But note that I included myself :-)
No professionals: please quote your title. Mine is BitBucket BSc (Hons). That is it. No membership of professional bodies. And yet I can call myself a programmer. I don't mean that I am not 'professional' in my attitude etc. Just that anyone can claim to be a programmer.
BB - Many scientists are obviously sharper than the lab technicians they work with. Many singers and band leaders are obviously sharper than the musicians they play with. This has still nothing to do with those lab technicians and those musicians being able to provide the experience, professionalism, abilities that the scientists and the singers and band leaders lack outside of their profession.
Take the best genetist in the world, take the best lab technician in genetics in the world, and I bet I know which one of the two will get the best DNA data.
We have started this discussion saying scientists can't be expected to write professional-quality computer code. If you believe nobody can, then professional-quality computer code does not exist in your world view and there is nothing to discuss.
Otherwise...back to square one: scientists, and in particular climate scientists, are not know for quality coding behind the papers they publish. It is simply mostly of low quality, written by obvious amateurs.
Question is, how do we improve the situation? And the answer cannot be, "let's give up and hope for the best".
BTW...I have a MSc in Electronic Engineering and I call myself a scientist too. That's a distracting argument you're making as well. The trouble is not what people call themselves, but what level of quality their work is when it is presented to the world.
ps stop moving goalposts
Apr 19, 2012 at 3:16 PM BitBucket
The Met Office states that this approach validates their models. But it's a fallacy. It's the same error that researchers trying to build pattern recognition systems made: "testing on the training data".
Data used to "parameterise" the models comes from observed historic data. If the models could not even reproduce the history used to construct them, then their programmers would have failed at the first hurdle.
But even if they predict what happened, when "what happened" was used to tune them, this does not confirm the correctness of the physical models used to construct the progams.
I think this is what JC probably means - you can fit a curve to historical data and the "model" defined by a few coefficients will reproduce the historical data well. But because it is not based on any sort of representation of the physics, it is useless for predicting future outcomes.
I do not believe there are other methods. Can you guess anything about my view of the usefulness of climate models?omnologos: did I give the impression that we should just give up? I have been nicknamed Marvin in the past so perhaps that is possible. What I meant was that just piling in the pro's was not a guarantee of a better result. And man is it going to cost!
Whether there is any point in doing so is moot. My guess is that deniers would reject any result that did not fit their ideology and warmists likewise. We'd be back where we started.
Martin A: I take your point. I said something along those lines above (Apr 19, 2012 at 12:16 AM):
"Of course if the model does match history, there is no guarantee that it will continue to do so..."
However, I was under the impression that the models do indeed try to capture the behaviour of greenhouse gasses like CO2 and other climate-related physics. Is that not so?
Oh boy...small-minded statements about programming followed by a gratuitous use of "deniers". BB is just another pathetic troll.
Apr 19, 2012 at 7:23 PM BitBucket
I am sure that the makers of climate modelling software do their very best to model the physical effects as well as possible. But there are lots of effects that are simply not as well understood as they need to be. It's simply a different theatre from modelling, say, the dynamics of a satellite in orbit, where the physical models give results that agree with reality to multiple decimal places.
For example, there is a belief (no more than that) that small amounts of warming due to CO2 will result in much larger amounts of warming, due to increased atmospheric water vapour. These positive feedback effects have to be guessed at.
The dynamics of the concentration of CO2 in the atmosphere assumed in climate modelling is based on the "Bern model". This is based on assumed dynamics of CO2 exchange between atmosphere and other reservoirs but is inherently incapapable of validation. This model also seems to give results differing (by orders of magnitude) from what anyone can readily compute for themselves from the history atmospheric C14 levels since the end of atmospheric nuclear weapon testing.
By the way, the majority here consider the term "denier" extremely distasteful - equivalent to saying someone who does not believe what is published by the IPCC is on a moral par with someone who denies that the genocide of the jews took place during WW2. Did you know that?
Martin A: "Did you know that?". No, sorry. I had no idea of that and apologise if it was taken that way. I thought it was the accepted antonym for 'warmist'. Is 'skeptic' preferred?
BB
"Is 'skeptic' preferred?"
Smart arse troll.
The antonym for 'warmist' is 'skeptic', not 'denier'. 'Denier' is antonym for 'doomsayer'.
Happy with that?
Now go spread the word to the doomsdayosphere, where you clearly belong to.
I remember reading Robert Laughlin saying that he used to force all his students to write their own code as he felt strongly about it. In my line of work, scientists and especially trainees brute-force every assignment that is handed to them, because, 'the results are what that matter.' To this end, I would say, science trainees are being driven to mediocrity rather than creativity
Martin A: I was interested in your Bern model and 14C references. After searching around I found some stuff on http://www.john-daly.com by a Peter Dietze. I wondered if this is the source of your comments or whether there are other more recent discussions.
Such errors often arise because the customer's business processes are ill defined
This quote from Martin A above holds the secret of the NHS computing fiasco. Bugger all to do with programming simply a failure to get the high level questions answered properly before coding (and implementation!) began.