Wednesday, 29 May 2013

Great Software Disasters - Do They Never Learn?

We can all get very legitimately steamed up about the Beeb's epic Digital Media Intiative fail - £100 million of licence-payers' money down the pan with nothing to show for it.  BBC Must Learn Lessons, thunders the Telegraph.  Yes, and re-coup the money from their salary budget, if there was any justice.

But I suppose we must settle for 'learning lessons' - without much optimism that it will do any good.  The public sector is notoriously prone to great systems disasters, with the NHS well to the fore.

Actually, it is my strong belief that the private sector is just as bad - but finds it easier to cover up.  In one of the more chequered phases of my career I got an excellent insider's perspective on this; and since then have watched many a software project disaster played out in energy companies. Many, perhaps most of these have been truly avoidable - proven solutions being installed in conventional circumstances, and still cocked-up to buggery, at enormous expense.   Why does it all go so wrong, so often ?  Here are my Top 5 Lessons Learned (from a very long list indeed). 
  • keep Systems Integrators on a very short leash.  Or dispense with the bastards altogether, their added value is less than their fee by an order of magnitude - when it's not actually negative.  (I am being very restrained here, in the wake of Sally Bercow's little difficulty)
  • after a careful selection process, get the vendor to install their standard package with as few customisations as possible, ideally none whatsoever.  It's much easier for you to adapt to a proven solution (which, almost certainly, actually works) than the other way around - and to hold the vendor to their performance obligations
  • never, ever be a beta site for any mission-critical or enterprise-wide system.  Or ever, really
  • demand to meet - and vet - all the vendor's team who'll be installing the product.  They will resist this like hell.  Ask yourself why ...
  • dispense with System Integrators.  I may have mentioned this already ...
 I am guessing that many C@W readers have seen much of the same.  What are you diagnoses and prescriptions ?

ND

21 comments:

Raedwald said...

I'd add

1. Bandwidth and storage requirements will extend to the maximum extent to which applications can be misused

(Yes, when I retire I too will email a 150MB one page 'word' invite with flashing letters and natty graphics to around 300 people.)

Anonymous said...

Yes, in private companies the projects just sink without trace and are never mentioned again.

Ask CU what's the crack with EMED, btw. Is it just the delay, or is there something unpleasant in the woodwork? It's lost 50% since the last update a week or so ago. Fill boots time, or 'thank God I didn't buy' ?

Kemi said...

From my personal experience as a project manager:

1) Always start with a fully functional pilot before getting the real thing.

2) The sample of pilot users must be large and varied enough to get a good understanding of how much the users like the system and how well it works, but small enough not to have any serious impact if it doesn't work well.

3) If the pilot comes with only negative feedback DO NOT continue anyway because of sunk costs. You are throwing good money after bad. All problems must be fixed to the satisfaction of your harsher critics BEFORE you pay for the full application.

4) Make sure you have a plan in place to roll back if things don't work properly.

5) Do not be scared of sacking contractors.

6)From Day 1, temporary contractors must work WITH your permanent employees. It is not enough for them to report to one of your senior managers.
Do not allow your contractors to be the repository of knowledge and undermine your internal staff. The contractors will leave and you will need to hire them back at a higher cost to keep things running, further demoralizing your staff.

7) Ignore RAG and project risk status reports sent to management. They are reverse engineered and will never tell you how well or badly everything is going. Instead, have someone junior working on the project listen to the gossip and grumbling of other staff and pass on concerns.

8) Double the amount of time and money estimated for project completion, if you're still happy with this then go ahead and be sure to manage expectations.

9) Have a very vigorous and transparent competitive tendering process. Have all bids available to all staff for viewing and offer prizes or bonuses to any employees who can pick holes after going through them with a fine tooth comb.

That's the first few off the top of my head. I'd have to charge you for anything more than that! They may seem obvious, but every project I have seen fail, did so because of a failure to observe a combination of some or all of the above.

oobee doo said...

Make sure all business units actually know what they want and go through a proper discovery process.
Requirements by commitee or internal politiocs never really works.

Also stop Senior management bringing forward release dates, to prove how macho they are - throwing more developers at a schedule that is mid -development does NOT increase the speed the project is finished.

Watch out for the quality, off shoring does not alwas cheap software delivery make...

So being a software vendor I place most blame at the door of f*ckwit client who dont wven know what they really want and then complain.

CityUnslicker said...

Anon - EMED - Must be shenanigans unfortunately. Experience of Aim suggest to em a placing at 4 or 5p hence the big fall as the insti's off load to replace their stock with cheaper shares.

IT stuff - i wholly agree with ND, except about integrators. the biggest issue is customisation by far. it's just not needed.

Oh and also complex systems, as soon as people start selling you middlewhere and ETL's then you should move on - if you can't buy something off the shelf that works for you it is becuase it has not been invented yet. You have to be one brave company to decide that you should be the company to gain commercial advantage from having such a syste; the risks are huge.

Elby the Beserk said...

Working in IT for 25 years for a company that did a lot of work with the public sector (libraries, and hence, local councils), I was constantly amazed at the incompetence of council IT departments.

Many years ago we were installing a new library system, with one of our new machines, at Somerset CC HQ in Taunton. We came in one morning to find that there had been a power cut.

We had our machine up and running, with a roll-back recovery taking place automatically, ensuring that the library database (huge for big public authorities) was up to date, and integrity checked.

When we left around 7pm that evening, the council guys were still struggling to get their ICL mainframe working. No one knew who had ultimate responsibility for getting it running again, it had "happened before and we didn't have any problems", and ... frankly, none them, nice guys tho' they were, would have lasted 3 months in the private sector.

I'd add to that, that having spent some time looking at the CRU code that was released as a result of ClimateGate I, none of CRU's software people would have lasted 24 hours in the private sector. Of course, it is this software that has been used to extract millions of pounds from the taxpayer to fund the CAGW farrago. On which matter, the Met Office were finally pinned to the wall and made to confess the other day that there has been (excuse me shouting but this makes me angry) NO SIGNIFICANT WARMING SINCE THE LATE 1800s.

http://bishophill.squarespace.com/blog/2013/5/27/met-insignificance.html

Billions of pounds, pissed away, in the name of an ideology masquerading as science. When do the prosecutions begin?

Elby the Beserk said...

"We had our machine up and running, with a roll-back recovery taking place automatically, ensuring that the library database (huge for big public authorities) was up to date, and integrity checked." ... within an hour.

Blue Eyes said...

So much and yet so little to add.

I would say the most dangerous person in IT (especially procurement) is the guy who thinks he is quite good on computers.

And never, ever, ever, let this person do automatic updates on a Linux server.

ivan said...

Before all the above kick in the first and foremost requirement is to define what the project is to do. Once that is defined it should be set in concrete - anyone that tries to change it should be shot!

Hand in hand with this should be penalty clauses in the contracts of suppliers and contractors and those clauses should be adhered to without any excuses!

Those two concentrate the minds of those proposing the project and those implementing it.

In my 50 years experience of project of all types and sizes, the ones that go wrong are those that have significant changes made as the project progresses, usually because some manager wants something one of his buddies has or has heard about.

I agree that Systems Integrators should be shot when first sighted - I have yet to find one that has out any value into a project

phil5 said...

"the biggest issue is customisation by far. it's just not needed."

Absolutely! And ignore people who say "Oh but we do things differently here at XYZ plc, so we need special modifications". Even if you do (unlikely), it's better to change your procedures to fit in with the standard computer solution that others in your sector have bought.

Of course, accepting the standard package build won't work if the vendor doesn't use source control and doesn't actually have one! (No names, no pack drill I'm afraid.)

Sebastian Weetabix said...

In a previous life we introduced a -ahem- comprehensive soup-to-nuts MRP system from a large German company. It rapidly became known to the troops as 'systems against people' or 'stop all production'.

In fairness it reliably did exactly what it was specified to do. The problem originated with the morons who did the spec. from our side. The moral of the story being: be careful what you ask for; you might just get it.

Anonymous said...

Agree with Kemi. Contractors should really work alongside staff and train them or knowledge transfer. Trouble is, what if the guy you're with isn't very switched on?


CU - EMED - "Experience of Aim suggest to em (me?) a placing at 4 or 5p hence the big fall as the insti's off load to replace their stock with cheaper shares."

Does someone know something that isn't out there in the market - quite a few people given the drop?

Or have analysts looked at the receding dates, looked at the spend and said 'they'll need more money' ?

Taff said...

No experience of IT, but working for a major contractor, first onto site was the portacabin. Then the kettle. The the computer to record the variances i.e. to capture the profit that was forgone in bidding lowest. I would imagine it could be the same in IT.

Taff

CityUnslicker said...

Anon - sadly, to the tune of 10k or so for me, it seems obvious there is a need fro cash and I am guessing that will be filled via a dilution at around 5 or 6p.

No it should not happen that the insti's know and front run, but yes it does, that is AIM!

Timbo614 said...

My singular experience of getting involved in a major IT installation was back in 1999. The supplier was eventually labelled as "Quite Shitty Programs" - They went bust in 2001, during the contract. Two examples of their mentality:

1) At the start of the project they flew 6 guys out to "foreign country" for six weeks(at the clients expense), we being on the client side, kept asking when progress would be made? When we would start seeing some actual "installation" or preparation for it. Turns out they spent those first six weeks fine-tooth combing the contract to work out what they didn't have to do!

2) When we eventually got rolling they sent a guy out to "foreign country" to do the initial server test install. I wrote him off when I discovered that he had flown 1500 miles with: A ticket Home, 25 GBP in his pocket(not even in local currency), and no credit cards! Of course it went tits up so he had to stay over... he was Diabetic and also did not bring enough jabs with him :(

Otherwise I can concur with lots of points above, but would add what is always needed is one or two guys/gals with an overall TECHNICAL picture of the project (not 20 specialists). Run a pilot. Use real stuff for it as much as possible. Keep the team a small as possible.

Anonymous said...

Ha ha ha,
Never generalise.... Sorry Kemi, never trust an IT pm, they are generally full of shit and make it up as they go on. IT systems can be complicated, hmmm, never generalise. ITIL generally a good framework, in practice ruined by prince practitioners.
Keep up the good work. IT projects will fail, Dilbert says so.

Graeme said...

the vital thing is to ignore the folks who say that the computer workflow has to follow your human processes...it sounds so good but it entails endless customisations, which have to be maintained and updated constantly. I think Berite Woposter would nagree that buying off the shelf is a lotm less hassle.

Weekend Yachtsman said...

Yes, let me be the third or fourth person to say DO NOT CUSTOMISE.

Change your business to fit the package, not the other way around.

Nobody is unique, whatever they may think, and it saves money, hassle, and probably complete failure down the track.

Lurid said...

If one systems analyst is saying that a proposal is doomed to fail due to a fundamental limit and another is saying they can do it for a reasonable fee, which do you believe?

Anonymous said...

In addition to all these (excellent points), please do think about the poor buggers who will have to use what you spec and pay for -> Training!

...and don't cut that when the FD comes to you saying we need to be more efficient with our IT costs. If they can't use the thing, then you ain't gonna realise all the projected benefits which is why you bought it in the first place.

Agence communication said...

this is very good and very interesting ""
keep Systems Integrators on a very short leash. Or dispense with the bastards altogether, their added value is less than their fee by an order of magnitude - when it's not actually negative. (I am being very restrained here, in the wake of Sally Bercow's little difficulty)
after a careful selection process, get the vendor to install their standard package with as few customisations as possible, ideally none whatsoever. It's much easier for you to adapt to a proven solution (which, almost certainly, actually works) than the other way around - and to hold the vendor to their performance obligations
never, ever be a beta site for any mission-critical or enterprise-wide system. Or ever, really
demand to meet - and vet - all the vendor's team who'll be installing the product. They will resist this like hell. Ask yourself why ...
dispense with System Integrators. I may have mentioned this already ...
""