Posts tagged measurement
Measure What Matters

So far we have looked at two of the four key elements for real business agility - distributed decision making and execution efficiency. It's time now to look at the third element - measuring what matters. Organisations tend to collect a lot of data They measure a lot of stuff. The problem with many of those measurements is that they are often data that is easy to collect rather than data that is important. 

What's the problem with that? Data is data. If it's easy to measure, why not measure it? Having more data has to be better than less. Not necessarily. There is something important about making a measurement that makes it vitally important to measure the right things, rather than just measuring stuff just because you can. The important thing about making a measurement is that measuring drives behaviour. As soon as you measure something, people will naturally try to optimise that measurement and if you're measuring the wrong things, that can drive some very bad behaviour.

Read More
Replace The Burn Down With Confidence Indicators

Ok. So I have a team and I'm looking at their board. On that board, if they are like 90% of the teams out there, they will have a burn down chart somewhere. Let's take a closer look at the burn down shall we? Does it, as so many do, show a flat line until the day they all update the tool at the end of the sprint? Does it show a nice progression of tasks done over the sprint but at the end of the sprint are there no stories completed because they worked on everything at once? 

The burn down is one of the classic tools for an agile team. Everyone does burn down charts. Mostly they do them badly. Over the years I have come to dislike the burn down chart for a number of reasons. Chief among those reasons is that it really doesn't do what it is supposed to - give an indication of whether the team is going to achieve its sprint goal. On the surface it looks like it should - there is a list of stuff to be done and a rate at which it needs to be completed to meet the date but in reality, the burn down doesn't give a very good indication of where things really are. The other big problem is that while they are supposed to be easy, most teams find keeping a burn down chart up to date a massive headache. So let's take a look at the burn down and some of the things that cause teams pain. We'll also look at a really good way to replace it completely.

Read More
Measuring Agile Success

As soon as you talk to anyone senior in an organisation about agility, one of the first questions that comes up is "How do we measure the benefit?" It's a perfectly valid question to ask. We are (usually) asking them for a bunch of money to implement change and quite often the benefits we describe are fairly poorly defined - better staff engagement, the ability to respond faster, faster time to market. That sort of thing. They all sound great but what really matters to an executive sponsor is the bottom line impact. Will spending this money generate additional profit? 

Unfortunately, the question of measuring agile benefit is a hard one. Other than time to market (which is actually a hard one to pull off for a bunch of reasons), all the traditional agile benefits are pretty soft. It's hard to come up with a set of hard, bottom line benefits we can easily measure and show the difference agile is making. This tends to make a lot of agile sales pitches a bit like the underpants gnomes in South Park - Step 1: Implement agile. Step 2:Ummmm. Step 3: Profit! While the ultimate metric is the profit, for the reasons I mentioned above, that's actually quite a hard metric to impact on the short term. What we need are a set of good metrics that can show progress towards increased profitability that we can measure and track in the short term. Don Reinerstein is an advocate of using an economic model to show benefits and while this is undoubtedly the right way to go, most organisations simply don't have the maturity to even consider things like cost of delay accounting until they are quite a long way into their agile/lean journey. What we need are things that any company can measure right from day one that we can use as leading indicators for increased profitability. Over the years I have come up with a set of metrics that seem to do the trick.

Read More
Before You Start Changing, Measure Where You Are

What's the first thing you do when you look at a map? Find your destination? Maybe. Start planning a route? Sounds logical. But there is something missing. One fundamental step that renders the other two useless. That first step is locating where you are. Obvious really, but essential. Unless you can position yourself accurately on the map, no amount of accuracy in destination identification, or time spent in route planning, will get you where you want to go.

That's obvious when looking at a map. Very few of us (my mother excluded) will locate our destination then confidently set off without working out where we are now. My mother, on the other hand, will locate her destination, see that it is on the left hand side of the map and confidently set out towards the left. Consequently her excursions often end up in interesting places. Trouble is, the same principle applies to organisational change and in that context, very few of us perform the first step. We jump straight into desired state, plan a few actions and off we go. We don't spend much time, if any, on step one. We don't measure where we are first. The result is exactly the same as looking at a map without locating youself on it. You will start off confidently in a random direction and end up... somewhere. If it's at your intended destination that will be by good luck (or the help of someone you asked for directions) rather than good map reading.

Read More
Release Predictability. Not Speed.

When talking to stakeholders about why they want their project to go agile, the most common reason they give is speed. Faster time to market. Faster delivery. Fast, fast fast. If you dig a little deeper though, and ask what they mean by fast, they don't say things like "before our competitors", they will say things like "when you promised it". A lot of the time, speed is a kind of code for "no delays". Make us a commitment and stick to it. Don't jerk us around. What they are really looking for a lot of the time is not more speed in delivery but more predictability.

Predictability is really important to the wider business. They may have trade shows booked, advertising campaigns locked in, shareholder briefings prepared. If commitments aren't met, it can have big impacts on the rest of the organisation. I know of one organisation who sent out letters to a million customers advising of a change on a particular date, only for that date to slip by 6 months. Not only is that not a good look, it's expensive too, as a million other letters needed to be sent out advising that the change would not, in fact, be happening, then another million when the new date was announced. Fortunately, an agile approach is ideally suited to giving the business the certainty it needs. How can this be when the agile approach doesn't try to lock down everything in advance? How can you have predictability without certainty?

Read More
The Measurement Fallacy

As soon as someone starts looking at the topic of metrics, the measurement fallacy pricks up its ears (I always imagine it looking somewhat rodent-like with mangy fur, evil eyes and sharp teeth) and gets ready to emerge from its hole behind the database server. When people start discussing what should be measured in order to keep track of a process, it gets ready to strike. Most people have fallen prey to it at one point or other. Mostly without ever knowing they have been bitten. The bite is painless. The only symptom is that the bitten suddenly assumes that because we can measure something, it must be important. More serious cases assume that the easier something is to measure, the more important it must be. This dreadful scourge is responsible for making Lines Of Code the primary measure of developer output for years.

It's a typical case of a severe bite - we can measure lines of code. Therefore it must be an important measurement. It's really easy to measure so it must be a really important measurement. Therefore we must measure it and use it to drive developer behaviour. Once it sets in, it's hard to shift. Despite the fact that the behaviour it drove - writing masses of wordy code to inflate your LOC counts and never, ever remove code - was completely counterproductive, the LOC (or KLOC) still hangs around to this day.

Read More
Control Charts

A couple of posts ago I promised you a post on Control Charts. Here it is. For those of you who have never come across these before, they are come from the field of Statistical Process Control (no... really... don't go...stay with me here... it's worth it, I promise). They provide a means of charting process data in a way that answers the single most important question you should be thinking of when looking at a chart of process data. No it's "not when can I go home?", or even "I wonder whether stabbing myself in the eye with this pencil will be more interesting?". It's – "what's normal?". When does the chart show normal variation and when does it show something I should be concerned about? Is this spike in the data something I need to investigate, or is it normal?

There are about a dozen different types of control chart for different types of data and you can use the various types to build a chart for just about any metric you choose, but for an agile project the most useful ones to chart are Velocity and Lead/Cycle Time. Even better, these two metrics use the simplest possible type of control chart – the IMR chart or Individual & Moving Range chart.

Read More
Agile Metrics - Lead And Cycle Time

Last time we looked at the most common question we get from management -

What metrics will I get?

And saw that our usual answer - velocity - really isn't very good at answering that question. It's really only designed to be an internal measure for a single team. You can't use it to compare teams. I mentioned that Cycle Time is a much better metric to use for this sort of question, so let's take a look at Cycle Time. We will also look at its cousin - Lead Time.

Read More
The Perils And Pitfalls Of Velocity

When implementing agile in an organisation, one of the first questions management will ask is “what metrics will I get”. This is not an unreasonable question. They are financially responsible for their part of the business and are used to dealing with a particular set of metrics that let them assess the performance of their teams and take action if action is needed. RAG (Red Amber Green) status on projects, schedule variance, earned value, actuals vs estimates, risk profile, and so on.

When agile comes on the scene, those measures go away and management feels like they have lost control. So a question about agile metrics is perfectly reasonable. How do I, as a manager, see that my process is working well? How do I tell when things will be delivered? How do I tell how much stuff will cost? When answering this sort of question, the answer we most frequently turn to is velocity. Most agile systems have some concept of velocity – how much a team can produce in a given time. The problem is that velocity is a very poor way to measure those things at any scale beyond a single team. Velocity just wasn’t designed to answer this sort of thing. Why? Well to see why velocity isn’t a good measure for these questions, we need to look at what velocity is designed to measure.

Read More