Why So Many Software Projects Fail
DURING MY 30 YEARS IN THE IT INDUSTRY I'VE WATCHED IT GO FROM BEING FRINGE, NAIVE AND UNSTRUCTURED, TO BECOMING AN INTEGRAL PART OF EVERYDAY LIFE, FOR BOTH OUR BUSINESS AND PERSONAL LIVES (I'M TALKING ABOUT NON-MAINFRAME IT SYSTEMS SUCH AS DESKTOP, WEB AND MOBILE PLATFORMS.) AS PART OF THAT GROWTH, MANY ORGANISATIONS HAVE INVESTED HEAVILY IN IT, WITH VARYING OUTCOMES. FOR SOME, THERE'S BEEN A GREAT RETURN ON INVESTMENT AND FOR OTHERS THERE'S BEEN HUNDREDS OF THOUSANDS, IF NOT MILLIONS, OF DOLLARS WASTED.
Understandably, Finance and Procurement Departments are playing an increasing role in the overall procurement process. In many situations, this has helped, but it has also hindered in some. The reason for the hindrance, generally speaking, is that people from a finance background like to benchmark what’s being purchased, or put another way, they like to compare “like-for-like”. This method is very useful if one is comparing products that currently exist, however it tends to fall down when it comes to procuring things that are yet to be built and/or fully implemented. The fact of the matter is, that building custom or bespoke software is part art, part science; there are a lot of moving parts to contend with. It’s a creative process, not just a technical one. This means that using a like-for-like approach isn’t very suitable. Having said all of that, I believe appropriate financial governance is an absolute necessity for every commercial software development project; without it, software projects could easily kill the sponsoring business.
As much as people have been trying for decades, innovation in software development can’t be commoditized. It’s based on layers and layers of code that is outside the control of any one developer (for example, the operating system, connections to outside systems and 3rd party libraries) blended with the technical skills and experience of the developer(s) on the job. Added to that is the methodology in play and the collaboration/communications capability of the team involved. All of this is founded on commercial arrangements, which more times than not, aren’t optimal for the project. On top of all of this is the client’s known requirements, the business’s drivers and strategic goals and the most poignant component, the Unknown Unknowns (thank you Donald Rumsfeld!)
And I think the thing that most people battle with is who is responsible for the Unknown Unknowns. Clients often say “well you should have known that” and developers often say “you didn’t mention anything about that during the requirements gathering stages”. When you apply this kind of friction to a waterfall procurement process, you have a recipe for disaster. Finance wants to know that the “milestones” have been reached before issuing payment and the vendor wants payment for efforts expended. What often is the case, is that the finance people do not want to hand over the money until they see “tangible” results against a list of milestones, but developers have often worked more hours than planned to get to that payment point, so naturally resent payment being held up. And often, the unknown unknowns have diverted efforts away from the pre-defined milestone.
So how do we solve this dilemma? In 2001, the Agile Software Methodology was born; a methodology that allows for the unknown unknowns because much of the specifics of what will be built is left for discovery during the development process itself. So, instead of leaving large periods during the development between measurement points, the work is broken down into small units, often known as “sprints”. At the beginning of a sprint there are planning sessions and at the end there are reviews and retrospectives; this allows teams made up of both delivery people (which may include software engineers, testers, BAs, SCRUM masters and project managers) and SMEs (Subject Matter Experts) to jointly take ownership of what has happened and the decisions being made about what will happen. So together they steer the direction of the development.
Well, “this sounds good”, I hear you say…ah ha, but there’s a little problem…true Agile doesn’t work to a budget….the EXACT OPPOSITE to what finance people demand. So if true Waterfall doesn’t work and true Agile doesn’t work then what is the answer? The answer lies in the grey area of tight collaboration. In our company we advocate what we call Cost Target, as part of our Practical Agile Methodology (“PAM”). Cost Target means we do some upfront discovery work, documenting requirements at a higher level, such as User Stories. We then estimate against those User Stories and create a backlog. And that is our starting point. The project team then go away and raise approval against that estimate which is now the Cost Target.
In the business world, software is usually a capital expense and usually created to generate a return on that capital. As such, a very significant measurement of success of a software development project (at least in the business world) is the level of return on the spend. As such, the less the spend and the higher the return, the greater the success. At this point I’d like to introduce the concept of the Minimal Viable Produce (“MVP”) as described in Eric Reiss’s book “The Lean Start-up”. If we apply the concept of building the smallest amount of software that is viable, we increase the chance of showing a return (or success in some way) and we reduce the chance of wastage. So, in the Cost Target model, as unknown unknowns appear, the joint project team work together to de-prioritize scope of lesser value to allow for high value discovered scope to take its place. Throughout my 30 years in the industry, it is this method of high collaboration and transparency that steers a software development project to success and a win-win for all parties.
Written by Kareem Tawansi