When implementing agile in an organisation, one of the first questions management will ask is “what metrics will I get”. This is not an unreasonable question. They are financially responsible for their part of the business and are used to dealing with a particular set of metrics that let them assess the performance of their teams and take action if action is needed. RAG (Red Amber Green) status on projects, schedule variance, earned value, actuals vs estimates, risk profile, and so on.

When agile comes on the scene, those measures go away and management feels like they have lost control. So a question about agile metrics is perfectly reasonable. How do I, as a manager, see that my process is working well? How do I tell when things will be delivered? How do I tell how much stuff will cost? When answering this sort of question, the answer we most frequently turn to is velocity. Most agile systems have some concept of velocity – how much a team can produce in a given time. The problem is that velocity is a very poor way to measure those things at any scale beyond a single team. Velocity just wasn’t designed to answer this sort of thing. Why? Well to see why velocity isn’t a good measure for these questions, we need to look at what velocity is designed to measure.

 So what is this thing called velocity? Simply put, it’s a measure of team capacity – the number of story points (or just stories if you count them) the team can produce in a given time period. For teams running scrum, the time period in question is the sprint length. It allows a team to do some basic forecasting – "We did 30 points last sprint, how much should we take into our next sprint?”. Well, 30 points would be a good starting point. We do 30 points per  sprint and we have 10 sprints in the release. How much should we plan for the release? 300 points would be about right. All very handy. So where’s the problem? Well, the key word in all that is team.

Velocity is a team measure. It was designed as an internal measure for a single team to guide its own planning. It was never designed to be released into the wild for anyone other than the team to see. Using it for anything other than that will land you in a whole world of pain. Let's look at a few common misuses –

Comparing teams – As soon as teams make their velocities visible outside the team, the inevitable comparisons start.

That team does 60 point in a sprint, why does this one only do 30?

The obvious answer is that story points are not comparable across teams so velocities are likewise not comparable. But that doesn’t stop them from being used for comparison purposes. We instinctively see the larger number as better and start pointing fingers at the team with the lower velocity. This is made even worse when teams try to extend the velocity in some way by relating it back to some real figure like time or even worse, dollars. Each point, they will proudly announce is equal to a developer working for 4.5 hours. Or each point costs $1500. Easy enough calculation – 2 weeks team time costs us X, we deliver Y points, therefore cost/point = X/Y. Easy.

The problem is that it's meaningless outside that one team. If another team of the same size (so same cost) estimates on a different scale, their cost/point will be different. They might deliver 60 points/sprint so their cost/point is half as much. We (agile folks) all know that the same amount of actual stuff is being delivered by both teams, the only difference is that one team’s totally arbitrary story points are twice the size as the other's, so one team will estimate a piece of work at 2 points, the other will estimate exactly the same piece of work at 4. But to management, one team now looks twice as expensive as the other. I know which team I would rather work for when it's layoff time. This sort of team comparison by velocity drives the teams to inflate their estimates – more points = more velocity = good. The same amount of stuff is getting delivered but the numbers are bigger.

Driving teams –

You do 30 points per sprint. We need you to pick up the pace to get everything done. We need a  velocity of 60.

Well done. You just got the team to double their estimates. Again, same amount of stuff being delivered, bigger numbers.

Normalising story points – OK. So if the problem is that story points are totally arbitrary, let's normalise them across teams so that each team will estimate a story at exactly the same size.

Welcome to estimation hell. Now you need to make sure that all your teams, who may be doing completely different types of work, in completely different environments, use and maintain a common estimation baseline. You will need regular cross team estimation base-lining meetings and some cross-checking to make sure teams are still aligned and teams will need to make sure their estimations are correct and… welcome to hell.

Your teams will end up spending most of their time estimating and not delivering. You can get some level of consistency if you count stories instead of estimating them, but even then there are much better metrics available to you.

So, when used to compare teams, velocity is a really bad measure. It drives some very poor outcomes and really doesn’t measure anything real. So what do we say to management? They are still waiting for an answer to their question. Fortunately, there is an answer we can give them. There are much better metrics available than velocity. What they really need to measure is cycle time, and we’ll take a look at that next time.