24 February 2011

We spend £16 billion a year on Government IT but our approach is deeply flawed. When faced with high levels of uncertainty about specifications and technologies we need to adopt a more agile approach.

Cycling in London requires agility. By that, I don’t mean being limber (although that does help). You need to be prepared to react quickly to changing circumstances and to revise your planned route as circumstances require.

Why flexibility is key

My commute takes me from Bethnal Green to Westminster. As any London cyclist will tell you there are plenty of alternative routes you could take depending on the time of day, weather, preference for quiet roads and whether you are on the inbound or outbound leg.

For example, cycling down The Strand, east to west during the morning rush-hour is a waste of time. The traffic backs up from Trafalgar Square to Aldwych. In the evening, The Strand is fine in the reverse direction except on days where there is a protest taking place in which case you might as well use Victoria Embankment. You get the picture.

Imagine if, before setting out, I decided the exact route I would take regardless of any of the conditions I came across along the way. That would be ridiculous and would be likely to make my journey unnecessarily long and maybe even dangerous. When dealing with unpredictability, being flexible is the best strategy.

The risk fallacy

For too long, our approach to IT in government has been similar to cycling along without any regard to road conditions or other vehicles. The punchline being that this is supposed to reduce the risk associated with the £16 billion spent on government IT each year.

Massively complex, multi-year projects are described in extreme detail upfront in an attempt to lock down specifications. The procurement process has become so complex that it takes an average of 77 weeks just to get a supplier signed up. Once up and running, projects are delivered against these specifications with little regard to changing requirements or technologies. At the back end, assuming the project is not terminated mid-way, the inevitable change requests end up costing taxpayers a premium.

Why do this? Sadly the incentives in the system encourage such behaviour. Those tasked with drawing up the contracts in the first place are encouraged to put as much detail on paper as possible for fear that this is their one and only opportunity to describe the functionality they need. Gold-plating becomes inevitable.

Large IT suppliers are similarly incentivised to agree to massive contracts with the promise that if requirements change, they will get paid anyway. The size of the contracts also limits competition. There are relatively few players who are willing to spend the millions of pounds necessary to participate in the procurement process. Small and medium sized enterprises are priced out.

An agile solution

Next week we will be launching a report that outlines a new vision for government IT. Our research has focused on ‘agile development’ - a set of techniques originally used for creating software but now deployed widely in many leading companies and government agencies around the world.

Agile turns the traditional logic on its head. Rather than specifying detailed requirements in advance you let specifications evolve over time. Functionality is built in from day one, with users involved continuously rather than during a ‘test’ phase at the end. Flexibility is valued over following a predetermined plan.

It’s the IT equivalent of cycling with your eyes wide open, and being fully prepared to change direction when necessary. Wise advice for cyclists and chief information officers alike.


It is also worth mentioning that digital application construction is a very painstaking field of engineering. It requires perfection in a way that does not parallel the real-world. Analogies with traditional building and building planning really don't help with software projects. For instance, a very well constructed software 'window' can instantly be duplicated and made available throughout a system, in various sizes and shapes thru object orientation. You can't get this kind of weirdness on a construction site.

It is also very important to get software architecture right - it is not just a case of agility to make sure the solution profile fits in a bottom-up way (top down NEVER does) but also a case of ensuring that there is encapsulation, primarily and heavy use of established software framework. For this reason open source is the best technology to embrace. Battle-hardened, modified by the community to solve lots of common use-cases. The best way to ensure reliability is to have lots of components pre-made and tested,. so that a large project is more about assembly than painstaking construction. A divide and conquer approach to building is crucial. I would not want to embark on any project that didn't feature 100% open source in this day and age. After all, open source is an internet phenomenon. And 90% of large software systems these days are going to be internet-related communication ventures.

I've been looking at large scale IT systems for 20 years now, including the development and operations stages of the lifetimes of systems.

While there are certainly some aspects of Agile development that could help govt. IT spending (and more particularly Agile operations/devops, management of 'technical debt' and continuous delivery), I think that you're falling into the usual trap of looking for a silver bullet.

Whatever is actually used to deliver IT systems and then run them its name is not a very good indication of how well the processes and technologies are actually implemented.

I've looked at many govt. IT systems, and, like most large systems, the issues are mostly to do with people and misaligned objectives/skills, both organisational and personal. A simple wargame ahead of the procurement would help flush out some of the more stupid contractual models.

Also, no agile approach is going to help with some of the standardisation that does need to be imposed to get around the groundswell of discontent about govt. systems 'not talking to each other'.

There's just too much Technology in IT: the emphasis should be on the Information.

I have successfully applied agile methodology to implement 4 public sector IT projects for the Pensions Regulator. All were delivered on time, fully functional, and under-budget. I was amazed when I was told that this was a first for government IT projects, since throughout my 25 year career in the private sector, successfull delivery was the norm, not the exception.

Completely agree - the problem isn't the technology. The focus should be on who needs what information, when for what purpose and getting that to them in the most flexible/secure way possible. Achieving this in a way which balances risk calls for really trusted/insightful relationships between suppliers using robust platforms and customers taking a level of ownership of the data. The MP example provided is one of a number of case studies in government which is improving performance and saving a lot of money.

Spot on Tim Coote IMO. The interface and interaction between the organisation and the techies is the key to most issues I have faced when implementing SCRUM processes within large organisations. I have found that if all project participants understand and follow the rules, the liklihood of successful delivery increases dramatically. The key barriers to successful delivery are nearly always people and process related, not technology related. Changing the culture within the organisation is the real challenge.

Having been intimately involved with IT System development for 30 years, reading the blog and listening to Iam McGhee on the today program I was struck by a sense of deja vu. Agile methodolgy would appear to be a a rediscovery of RAD development of the 1990's , which requires intimate contact between users and developers, prototyping and frequent iterations. The development cycle creates the specificaction, and assumes that requirements will change before completion.

There are a couple of problems with this methodology. Firstly it cuts out a layer of managment on both sides of the contract, which exist to shuffle paperwork between users and developers, ensuring compliance and justifying cost overruns against the agreed specification. This cohort will not readily accept redundancy. Secondly, is the prolem of deriving a suitable form of words for the contract between customer and contractor ensuring that both parties are fairly treated.

The fisrt one can be side-stepped with sufficient courage from the top, but the second is more problematic. We were able to do so, in the private sector, by fixing prices for objectives rather than specifics. There was never a dispute over whether and objective had been met.

Like David Revill I remember the RAD approaches of the 1990s, if fact I was delivering RAD based projects in Government back then. But the approach had it's problem and met with resistance from management cut out of the process.

While it can work Agile doesn't fit well with a management culture of outsourcing that needs to know costs up front. Bring IT back in house and it would be more agile friendly.

Agile says nothing about visualisation of complex solution designs. I find this very difficult to accept. It is a bit like approaching a large shipbuilder and saying don’t worry about the design; all you need is an agile approach! Software development is all about designs that: satisfy the user, the developer, and the poor devil at the end of the chain responsible for modification / repair.
Business requirements analysis and solution design need a major overhaul, because the status quo is just not working as well as it should, and the evidence is there for anyone who cares to look. Until both Business and IT stakeholders take this fully onboard, all of us: buyers, providers, consumers and taxpayers will pay much more than is necessary or justifiable.

Having worked in government IT between 1973 and 2007, latterly holding the position of e-business manager for a large govt department, I would say that government IT failures arise for these reasons

1. Project too large - small is beautiful.
2. Too much intergration - avoid complex interfaces between projects and departments wherever possible.
3. Stakeholders failing to understand that IT cannot solve organisational dysfunction. Users must bite the bullet of sorting out their own system problems before embarking on complex IT projects
4. And most important of all - the failure to lock down the specification of work that's underway. I have seldom worked on any project which has not been subject to political or top management pressure to introduce major changes during the development process.

Add new comment

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.