The Measurement Fallacy
As soon as someone starts looking at the topic of metrics, the measurement fallacy pricks up its ears (I always imagine it looking somewhat rodent-like with mangy fur, evil eyes and sharp teeth) and gets ready to emerge from its hole behind the database server. When people start discussing what should be measured in order to keep track of a process, it gets ready to strike. Most people have fallen prey to it at one point or other. Mostly without ever knowing they have been bitten. The bite is painless. The only symptom is that the bitten suddenly assumes that because we can measure something, it must be important. More serious cases assume that the easier something is to measure, the more important it must be. This dreadful scourge is responsible for making Lines Of Code the primary measure of developer output for years.
It's a typical case of a severe bite - we can measure lines of code. Therefore it must be an important measurement. It's really easy to measure so it must be a really important measurement. Therefore we must measure it and use it to drive developer behaviour. Once it sets in, it's hard to shift. Despite the fact that the behaviour it drove - writing masses of wordy code to inflate your LOC counts and never, ever remove code - was completely counterproductive, the LOC (or KLOC) still hangs around to this day.
The measurement fallacy is the belief that because something can be measured, we should measure it. This doesn't seem too bad at first glance. If it's easy to measure, why not measure it? What's the harm?
The problem is that our brains relate to numbers in a particular way. We look for patterns automatically. We assume that the number means something and we will give it meaning if no meaning exists. This is how measurements like LOC came about. It's super easy to measure so let's measure it. Now that we have it, it must mean something... obviously a developer who is working really hard will write more code than one who isn't, so let's use it to measure our developers.
Of course, lines of code bear little or no relationship to how hard a developer is working. They may be working really hard refactoring code to remove lines. What does a negative LOC count say about them?
Other measurements based on the same fallacy crop up all the time. I spoke to one company that told me that "we are a very mature agile organisation... 75% of our staff have attended an agile training course". Really? On a closer look, there were no stable teams, no backlogs, no standups, no VMBs, no agile practices of any kind. There is no real relationship between the number of staff trained and the agile maturity of an organisation. In this case 75% of their staff knew that the organisation wasn't the slightest bit agile. They had a high level of agile awareness in the organisation (awareness that they weren't agile) but a low level of maturity. Someone decided that the number was easy to measure and that it had meaning where it didn't. Classic measurement fallacy.
Whenever you look at metrics, it is really important to be aware of the measurement fallacy. Just because you can measure something, it doesn't mean you should. It may well be the case that a lot of the information you need, you may have no way to measure. The first step may be to put systems in place to enable you to measure what you need.
It's important to start with a clear idea of what your goals are for the metrics you are designing. What behaviours are you trying to drive? Don't fall into the other measurement trap of thinking that you can measure something without driving behaviour. As soon as you start measuring something, people will automatically change their behaviour to optimise the number. It's instinctive. So, have a clear goal. Know what you are doing. This is the "why" stage - Why do we want to do this?
Once you have a clear goal, do some research. Do some experiments. Find out what you need to measure. Find out what measurements actually measure what you want to measure. Don't assume that you have the data already. You may not. Don't assume that it will be easy to measure. It may not. Don't assume that what you already measure is important. It may not be. This is the "what" stage - What do we need to measure in order to do this?
The next stage, of course, is "how" - How will we make the measurements? You may have a system in place for it already; if so great. You may not, though, and you probably don't. In that case you will need to work out how to measure it. You may need to put new systems or processes in place in order to be able to measure it. Don't assume it will be easy. It probably won't be. You may find that the amount of work required to get the measurement outweighs the benefit you expect to get by measuring it. Don't be afraid to pull the pin if it's too hard, but don't fall into the trap of substituting an easier (but meaningless) measurement instead.
Metrics aren't something you can just jump in and start doing. They require a lot of thought. Corporate information systems tend to measure what is easy rather than what is useful. Don't fall prey to the measurement fallacy by assuming that the measurements you have are in any way useful to the problem you are trying to solve. You may need to do a lot of work to get the measurements you need.