According to a 2012 report by McKinsey, 66% of software development projects run over budget and 33% run over time.

So it’s understandable that many companies continue to use fixed price contracts to ringfence costs and timings to not only try to understand how much things are going to cost before they get started, but also to try to avoid budget blowouts. The problem with this approach is that these fixed prices are based on estimates, and an estimate is, by definition, an “approximate calculation or judgement of the size of a thing”. At best, it is an imprecise measure and, at worst, usually wrong (it’s really just the degree of incorrectness that’s up for debate). Worse, this inaccuracy is multiplied for large projects where hundreds of individual smaller estimates are combined into a single, larger whole, and especially when this happens very early in the process, like during the RFP stage, which is when most fixed price contracts are defined.

Better defining the scope of work and requirements is an important first step in ensuring estimates (and therefore fixed price contracts) are as accurate as possible. The first two articles of this series, “Project planning: Customising solutions and weighing up off-the-shelf options” and “Efficiencies at scale and the risk of cutting corners”, provide some reflections to help with these stages.

But looking beyond that, can developer and client teams actually collaborate to improve the costing process, rather than relying on antiquated processes like upfront costings and legal sticks that have been proven, time and time again, to fail anyway? Let’s have a look into that.

Bugs and budgets

When learning to be a software developer, there are several truisms that you eventually make peace with over time.

The first is that there are always bugs. In fact, the purpose of software testing is not to eliminate bugs but to identify as many as possible before the software goes live. Testing is important not because it eliminates bugs, but because once a bug makes it to production, it becomes exponentially harder and more expensive to fix. Software quality processes therefore focus on a concept called “shifting left”: identifying bugs as early as possible in the software development process, because the later a bug is found, the more expensive it is to fix.

If we understand that bugs are not necessarily a sign of poor quality work (though admittedly this is probably a separate, longer subject to debate), and that they are just part and parcel of humans writing complex and intricate custom logic in a subjective and time-poor environment, then the ultimate upshot of this is that there will always be bugs. Eliminating bugs will therefore always take time out of the project schedule, and so estimating to allow for this is a critical part of getting overall timelines and budgets as accurate as possible.

The second truism is even more important, and that is that estimates are never correct. The reality is that estimates are simply too subjective and depend on too many factors to ever be more than a guide: the complexity of the work, the clarity and detail of available information, the number of variables and risks involved, even the skill level of the professionals estimating and executing the work; these are just a few of the factors that conspire to hamstring any effort to lock down a guaranteed price.

Estimates can be made more accurate by attempting to address these and many other factors, but they will never be perfectly correct.

What’s worse is that usually, when estimates are wrong, they are almost always too small and rarely, if ever, too large. People are usually optimistic when asked how long it will take to build something, and psychology also plays a role: developers don’t want to disappoint by giving an estimate larger than what is “expected”, so they choose something smaller and invariably struggle to deliver. With no margin for error, when the first bugs arrive (which they always do), then things are only ever going to take longer.

Multiplied over hundreds of individual estimates, the flow-on effects compound. Try to base a multi-million dollar budget and contract on them, and you have the perfect recipe for a budget blowout.

Is the agile model the answer?

The agile development process was designed at least partly to acknowledge and compensate for inaccuracies in early estimation. The theory goes that the most accurate estimates are obtained just before the work is completed, since this minimises unknowns and variables and ensures that the people doing the estimation are almost certainly the ones who have context of, and who are actively doing, the work.

However, while most businesses say they want to “do agile”, their procurement processes don’t allow it. Funding a team to deliver something without agreeing to and locking in what that something is at the beginning of the project is a procurement hurdle that is nearly impossible to overcome in many organisations. Add legal teams’ concerns around contract wording into the mix, and you won’t be surprised to know that most businesses who want to “do agile” still try to operate on fixed-price contracts and demand adherence to scope as defined in an RFP.

The result ends up being what the industry refers to as “agency agile”, “wagile”, or even (pejoratively) “fragile”. This hybrid process involves defining an upfront cost to satisfy procurement and an up-front discovery and design process that attempts to understand enough to estimate and document what will be required to deliver the project. It also ignores the fact that most of this will be wrong, or will change, within a few weeks of starting. These steps are then followed by an iterative release cycle that loosely refers to “sprints”, and employs a stakeholder-expectation-management dance predicated on “reasonableness” and good faith give-and-take as a substitute for blanket “change requests”.

This isn’t a bad process, but it’s not agile. And if either party doesn’t fully understand the ways of working, it can become its more pejorative namesake: fragile.

For both parties, willingness to compromise in this type of model is essential, as is keeping an eye on the big picture. When done well, there is a mutual understanding that, sometimes (often), estimates change. Some will go up, some will go down. There needs to be a willingness to reasonably allow for the unforeseen by balancing efforts based on agreed priorities. A productive relationship means a client can say, “We didn’t think about this. Can we do this instead?” and the feature is included for no additional charge. It also means a developer can say, “This is taking longer than expected. We’ll have to reduce time elsewhere to compensate”. In such circumstances, this, too, will happen without complaint.

When it’s not done well, things go pear-shaped very fast. Demands about delivering the scope as per the documentation and estimates start, change requests proliferate, and “reasonableness” takes a back seat. Most of the time, this is the reality, and in this type of relationship not only is everyone is a loser, the budget and timeline almost invariably blowout as well.

To achieve your expectations, you must ensure your expectations are first set in the right place, and you must allow for the unexpected, because in software development, something unexpected will always come up. It’s nobody’s fault: it just is. Estimates are never foolproof, nor are the people who are doing the work, no matter how experienced they might be.

Counterintuitively, higher levels of accuracy are achieved by allowing for unknowns, both in terms of individual estimates as well as broader program and delivery challenges. It’s a reality of what takes place on the ground, no matter how good the people doing the work might be, and assuming otherwise on your project is the same as putting your head in the sand.

But it’s also understandable that many organisations struggle to cater to this level of uncertainty. Being open and honest with your partners, and demanding that same level of honesty in return, as well as working with as much flexibility and “reasonableness” as your process allows, is the key to good relationships, and therefore good outcomes.

Christian Brenner has spent over two decades building enterprise websites for businesses and organisations. He is currently the National Head of Technology for Merkle Australia, the country’s leading customer transformation partner, empowering brands to create distinctly better experiences.

This article is the third in a five-part series. Follow Merkle Australia on LinkedIn for more.