Measure What Matters
So far we have looked at two of the four key elements for real business agility - distributed decision making and execution efficiency. It's time now to look at the third element - measuring what matters. Organisations tend to collect a lot of data They measure a lot of stuff. The problem with many of those measurements is that they are often data that is easy to collect rather than data that is important.
What's the problem with that? Data is data. If it's easy to measure, why not measure it? Having more data has to be better than less. Not necessarily. There is something important about making a measurement that makes it vitally important to measure the right things, rather than just measuring stuff just because you can. The important thing about making a measurement is that measuring drives behaviour. As soon as you measure something, people will naturally try to optimise that measurement and if you're measuring the wrong things, that can drive some very bad behaviour.
Let me give you an example - back in the dim and distant past, I was working as a software engineer. As an engineer, the primary measure of engineer productivity used by the company I worked for was a thing called the KLOC. The thousand lines of code. How many KLOC did you produce each week? They used that measure because it's easy to collect, just run a counter across the source repository each week, and surely, more lines of code are better than fewer. They used to publish weekly tables of top KLOC writers and people would strive to hit the top of the table. Your bonus was based on being a prolific producer of KLOC.
OK, so anyone who has ever written code will realise that the number of lines is a pretty poor indicator of the value being produced. But surely there is no real harm in having a dumb measure? Well, put yourself in the position of having your bonus impacted by your KLOC score and now imagine that you are writing a new piece of code. There are two ways to approach that. One is to write simple, elegant, concise, maintainable code. The other is to maximise your KLOC count. Which one do you think you would pick? Now multiply that by 200 engineers and imagine what our code looked like. Our management used to brag to customers that "we have 24 million lines of code in our product". I can tell you for an absolute fact that 20 million of those lines didn't have to be there. They were there solely to pad people's KLOC counts.
By measuring the wrong thing, they were driving the wrong behaviour. We've all seen these situations, like testers who have a KPI on number of bugs raised. I worked with one test manager who bought a cake for every 100 bugs a tester raised. The result? The bug tracking system was swamped with dozens of insignificant, duplicate or just plain un-reproducible "bugs". If an issue affects multiple versions of a product, don't raise one bug and tag it against multiple versions. Raise one bug per version. Each bug gets you closer to the yummy cake. All the while making it harder to actually find and identify real problems.
Or what about the sales team whose KPI is number of new sign ups. Sounds fine, but the easiest way to get people to sign up is to give them something in return. 60 days free! 90 days free! Bonus this! Extra that! I have worked with teams that were absolutely nailing their sales KPI, people were signing up by the dozen, but leaving as soon as the free stuff ran out, meaning that each one of those customers was actually costing the company more to acquire that they made in revenue. Higher sales led to lower profits due to the excessive giveaways. I've seen marketing teams whose incessant spamming of their customer base in the search for more conversions (to meet their conversion KPI) leads to massive numbers of customers leaving.
Eric Reis in his book Lean Startup warns again the use of shallow "vanity metrics". Many of the metrics I have used as examples above are perfect examples of vanity metrics. Ones that are easy to collect and make things look great but tell us nothing at all about the underlying state of the business.
Measurement drives behaviour. As W. Edwards Deming said
People with targets and jobs dependent upon meeting them will probably meet the targets - even if they have to destroy the enterprise to do it.
Measuring something has real consequences. Much of what we measure in organisations is measured because it's easy rather than because it's important and measuring those things drives (in many cases) very poor behaviours from a system perspective.
The big problem is that we tend to come at decision making from the wrong direction. We start with "what data do we have available? " and then look at "how does this support the decision I need to make?", although in most cases it's more "how does this justify the decision I have already made?". We measure what is easy rather than what is important.
We need to flip this around. The sequence needs to be:
What decision do I need to make?
What information do I need in order to make that decision?
How do I collect that information?
Focus on the decision first, then look at the information support required. In many cases the information required to make the decision may not be readily available. It may be hard to collect. You may need to put a measurement system in place in order to gather the data. In an enterprise that has relied on easy measurements for a long time, getting the measures that really matter may be a difficult process.
You will probably discover that to really support the decision making, you need more than one measurement. Be very careful if you only come up with one measurement to support a decision. You are probably missing something. Most decisions are trade-offs and a single measurement will often only show one side of the trade-off — measuring the cost of projects rather than cost and value leads to a drive to lower costs often at the expense of delivering value. You need both measurements to make a good decision.
I would also add a fourth step to the decision making process above:
What behaviours will measuring this drive? Are they the behaviours we want?
It is not enough to have the information we need, we have to collect it in a way that doesn't drive the wrong behaviours. You may find that you can't collect the information you need without driving behaviours that you don't want. In that case, you may well have a deeper problem. Does that decision need to be made at all? Is our strategy or direction at odds with our values?
That would be an ideal time to stop, and inspect and adapt. Inspect and adapt is my fourth fundamental element and I'll be looking at that next time.