Measuring Agile Success

As soon as you talk to anyone senior in an organisation about agility, one of the first questions that comes up is "How do we measure the benefit?" It's a perfectly valid question to ask. We are (usually) asking them for a bunch of money to implement change and quite often the benefits we describe are fairly poorly defined - better staff engagement, the ability to respond faster, faster time to market. That sort of thing. They all sound great but what really matters to an executive sponsor is the bottom line impact. Will spending this money generate additional profit? 

Unfortunately, the question of measuring agile benefit is a hard one. Other than time to market (which is actually a hard one to pull off for a bunch of reasons), all the traditional agile benefits are pretty soft. It's hard to come up with a set of hard, bottom line benefits we can easily measure and show the difference agile is making. This tends to make a lot of agile sales pitches a bit like the underpants gnomes in South Park - Step 1: Implement agile. Step 2:Ummmm. Step 3: Profit! While the ultimate metric is the profit, for the reasons I mentioned above, that's actually quite a hard metric to impact on the short term. What we need are a set of good metrics that can show progress towards increased profitability that we can measure and track in the short term. Don Reinerstein is an advocate of using an economic model to show benefits and while this is undoubtedly the right way to go, most organisations simply don't have the maturity to even consider things like cost of delay accounting until they are quite a long way into their agile/lean journey. What we need are things that any company can measure right from day one that we can use as leading indicators for increased profitability. Over the years I have come up with a set of metrics that seem to do the trick.

We really want to measure progress in three areas - the ability of the organisation's delivery process to deliver stuff; the impact those deliveries have in the marketplace; and the impact the delivery system has on the organisation and its employees. In all these areas we want to use hard financial measures where possible and steer right away from bogus "number of" metrics. These sort of "number of" metrics - things like number of people trained in agile,  are useless and misleading but are easy to measure and tend to be easy to make look good, so they crop up all the time. The Lean Start up folks would recognise these are good example of vanity metrics.

We don't want too many measures. That gets confusing and takes a lot of effort to maintain so I'm going to be a little bit contrived here and suggest that we have a nice symmetrical 3 measures in each of the 3 areas for a total of 9 in all. A nice symmetric set which makes dressing it up nicely on a fancy chart really easy (which is no coincidence).

Let's look at the delivery measures first. What we want to show is that agility is improving the organisation's ability to get stuff in front of customers quickly. As I said before, that's a hard thing to actually do but we can show progress towards that goal. The measures I would propose here are

• Total Lifecycle Cost

• New Dev : Maintenance ratio

• Feature Lead/Cycle Time (to prod)

Feature lead/cycle time is of course the classic measure here. What usually happens is that we see no change in lead time but a reduction in cycle time - the delivery teams are getting stuff done but the rest of the organisation is queuing that up because it can't handle that amount of change and sticks with quarterly releases or whatever they are used to. That's not a bad story to be able to tell. It's a much better metric than the other standard which is frequency of releases because that will only show that releases are still every quarter.

New dev : Maintenance ratio is the relative proportion of time the teams spend doing bug fixing and other maintenance versus new feature work. You should be able to show an increase in the proportion of new feature work as agile practices like focusing on a definition of done, early testing and the other agile tech practices take hold. This is a good one because it's the new feature development that makes all the money so they are getting more money making work done for the same cost.

Total Lifecycle Cost is similar and shows the total cost of a feature from first concept right through maintenance to its eventual decommissioning. You would want to show a reduction in total Lifecycle Cost as code quality and tech practices improve. You may not be able to measure right through to decommissioning as that as take years, so a Lifecycle cost to production plus 3-6 months is a good compromise.

Now for the marketplace metrics. What we are trying to show is that the organisation is getting better at producing the right things - things that make money because customers love them. Again, the real measure is increased profitability but that can take a while to occur so here are three that should show progress towards that goal

• Customer NPS

• Revenue/Employee

• Benefit realised : budget burned ratio

Customer NPS is pretty obvious (if you don't know what an NPS score is look here), you want to show the NPS going up as customers respond positively to new features. This is easy to measure through surveys and smart companies usually have a pretty close eye on their NPS scores already.

Revenue/Employee shows that the organisation is getting better at making more money for less cost. Anyone can make more raw revenue by throwing people at a problem but an agile company should be able to produce exactly the right things to delight their customers without needing to throw money around.

Benefit realised : budget burned ratio is trying to show that the organisation is getting better at delivering benefits early. It could fit in the delivery metrics just as easily as here but then we would have an asymmetric number of metrics and our pretty executive chart would look less pretty. What we are trying to show is the delivery of incremental benefit. In a traditional project all the benefit comes at the end when the budget is all spent so the ratio is pretty bad. If the project delivers its benefits incrementally then the ratio looks much better.

Last up are the organisational metrics. The impact Agility is having on the organisation. These tend to become terrible "number of", metrics but I have managed to come up with some that aren't.

• Team stability (% change in team structure over time)

• Employee NPS

• Management leverage (manager : doer ratio)

Team stability is measuring whether the organisation runs project teams or long lasting feature teams. Ideally we would want to see more team stability as they move from a project to a product or portfolio view.

Management leverage is measuring the ratio of managers to people who do actual work. As the organisation moves more to self organising teams you want want to see a much better manager:doer ratio.

Lastly, the employee NPS should measure how much the people like working there. That should improve as self organisation and all those other great agile things take hold.

So there you are. 9 fairly solid measures of agile progress. All of them should be fairly easy to measure. None of them require complex reporting or high level maths. No statistics required. Let me know what you think. Are there any I should add to the set? Any I should remove?