Martin A. I wouldn't dream of discussing computer software development. I have already acknowledged my utter, utter lack of understanding. My only interest is in situations of more general interest. During my short study of this subject I was reminded of a TV programme covering the inquiry into the shuttle disaster, where the problem was identified and solved by Feynman who brought his unique gifts to play.

Now you tell me that solving software problems are different because of the degree of complexity. But the o-ring problem was also fiendishly difficult. I also don't understand why starting again is a solution. Why don't the same problems arise again or variants of the same problems appear? Don't you learn anything from failure?

To be perfectly honest I'm not exactly in the market to be re-educated as a software engineer.

Martin A. You beat me to it. I took the dog for a walk and thought about what you had written. I was willing to concede that complex programming could be orders of magnitude greater than I previously have had to consider and thus a "white knight" would be impossible. Before I could convey this insight, I read your last post. So you did get your point across, it was just a little belatedly absorbed by me. You must be patient with me, for me these are all new concepts.

I still do not understand why nothing can be gained from previous experience. You mentioned that really complex problems have to be solved by compartmentalization, with links between compartments established from the outset. Surely if success is achieved in some compartments, that success is not thrown away, or am I grossly misunderstanding the problem yet again?

I recall a discussion about climate models in BH where I suggested that modellers must learn from previous errors, to which you responded that the correct approach would be to start again from scratch. But are climate models that complex? I suppose an obvious reply would be that a climate model capable of doing anything worthwhile would need to be hellishly complex.

I get the feeling from you that the most complex programming is testing human abilities to organize and manage problems. Is this, in part the reason for AI development?

You can't compare the software development problem with the O-ring problem without seeing that they are far different animals. Basically the O-ring thing is a puzzle. Feymann was not an expert in solid rocket boosters but he was a skilled problem solver and loved a puzzle of any kind. He had to find out how it worked while he was solving the physical problem and more importantly the problems with NASA management which had allowed the disaster to happen.

Software development requires skill, management and organisation of a different kind. And routine grinding painstaking work. I've shared an open-plan office with a software team and seen how it happens, but when they invited me to learn the language (PL/1) and join in I declined. I'm a puzzle solver too. I like a problem which will be finished when I find the cause. I'm good at it. Find out what is happening, how it's supposed to work, find the problem anf give back to the programmer to fix. Sometimes in a language I didn't know (this was back in the day when you could read code). If like me that is your skill set, you can be a fixer but not a creator.

Our lead programmer used to go down the pub at lunchtime and sleep through the afternoon. Nobody minded because of his productivity in the morning. Another person I've worked with caused a colleague to comment ' When Charlie's in it's like having three men off sick.'

rhoda. You may well be correct - in fact I'm sure you are. I also see myself as a puzzler (I don't always solve my puzzles) and perhaps also as an ideas generator. The last usually because I'm essentially lazy and want to take short cuts. My last few research papers were on puzzles I identified and failed to solve. Most papers in my field believe they have solved the task they set out to do. So I submitted my efforts very hesitantly (one took me more than ten years to send off) and was surprised and gratified at the receptions they gained.

I am not that interested in computer programming as such, but have been intrigued by the application of these (to me) new concepts to complex problems but of a lesser degree of complexity. Thus the tar pit analogy does I think apply also to very difficult problems where the original team set to solve them are floundering in the pit. Adding more of the same manpower won't solve these problems but the addition of a "tractor driver" might.

I think Feynmans brilliance was also due to his abilities to talk to, and listen to others of varied background. After all he listened to a female astronaut as well as engineers who were convinced the o ring problem was not due to excess vibration. He also possesses great communication skills. He belongs to the group my better half and I call obituary people - people with so many positives that you don't know all of them until you read their obituaries.

Its good to see this discussion on program development. Over the decades many methodologies have been dreamed up to ease complex program developments.

Anything that you dream up can be implemented in a program (a big statement I know) on the basis that you know how the system you want, works.

Back in never never land, our favourite climate modellers, do not yet fully understand how the climate works, how each variable interacts with each other variable, and even do we know what all of the variables are.

So in the climate world, currently, it matters not how their complex programs are designed and implemented, and will not, until we know all of the underlying processes.

"AI was a big thing in the 70's and 80's but was by and large essentially a giant wank"

Artificial Intelligence is not a cure for Human Stupidity - despite what the wankerati may tell you.

I am a software developer and I work as a contractor with my own ltd co. Almost every contract I get is to fill a knowledge gap or meet a looming deadline.
The solutions to a problem are extremely variable, there is always more than one way to skin a cat. Quite often the tune might be the same, but the words are different.

EO
In the days when I worked on a variety of developments throwing people at a late running project late in the day was guaranteed to delay the work further. Each additional person took one of the team off development and into training. This was true no matter what level of skill and experience was being drafted in. As Martin A says about other issues the late recognition of problems is probably a symptom of poor management.

A poorly performing large and complex system is beyond saving and needs a complete redevelopment from requirements specification upwards. Climate models fall into this category.

Aye Sandy.
I would never pick up a half finished job, I would just plain refuse. My standard line is 'I can finish that off in nine months or I can start from scratch and do it in three'

The problem with crummy half finished projects that the 'Owner' wants you to finish for them is that they are basically asking you to implement their solution for them. The three month option is me saying that I will implement my own solution.

I don't see the failure of climate models as an IT implementation problem. They would have still been heaps o shoite if they had been worked out with a pencil and paper