There has been a lot of talk at work recently about agile maturity checklists. There are dozens of them out there in the wild - the Nokia test, the unofficial scrum checklist, spotify checklists. Every organisation, at some point in their agile journey, seems to become consumed by a burning desire to measure just how agile they are, and sets about creating some sort of ultimate agility checklist mashed together from ones found online plus whatever else they can think to throw in.

There are a lot of reasons organisations want to measure agility. Some of them are good reasons, like looking for capability gaps so coaches and leaders know where more guidance is needed. There are also a whole lot of really bad reasons for measuring it, like comparing teams against each other at bonus time, or looking for conformance to a corporate agility standard. The biggest problem though with most agility checklists is that they measure the wrong things. They tend to focus on specific practices rather than focusing on desired outcomes. They are doing agile checklists not being agile checklists.

What do I mean by that? Let's take a set of questions that crop up all over the place in agile checklists. They will usually go something like this -

Daily Standups

• Is the daily standup at the same time and place each day?

• Is the standup under 15 minutes?

• Are updates directed to the team not to a manager?

• Does the team answer the 3 standup questions?

• Are solution discussions moved offline?

So what's wrong with that? That's standups 101 straight out of the scrum guide. Well, for a start, that's a lot of questions about just standups. Multiply that by everything else and I have seen these checklists grow to hundreds of questions. They can really become a chore for the teams to do, which means they tend not to get done unless they are forced to, and when they are forced to, they tend to rush through them and not really think about the answers.

The other problem is that this is scrum standups 101, straight out of the scrum guide. What about a team that has found that what works best for them is to have a longer standup with some solution discussion? I have seen this quite often, especially with distributed teams. It's a good opportunity for the whole team to come together and discuss stuff. One team turned their standup into an hour long daily design jam session. They inspected and adapted their process (just like we coaches tell them to) to optimise the way they work. Are they being agile? I'd say yes. Unfortunately for them, the corporate checklist said no so they had to add a useless (to them) 15 minute traditional standup to the beginning of their design jam to keep the agile auditors happy.

Another team abandoned the standup altogether. They all sat together within arm's distance of their board and collaborated all day. They didn't need a standup. Again, the checklist said they did, so they had to reintroduce it.

It's the same with other practices. Is the team holding a backlog refinement session each sprint? Plenty of teams say no. For them backlog refinement has become an activity that just happens. There is no backlog refinement meeting, it's something that happens in small chunks all the time.

This is the big problem with checklists. On one hand, we tell teams to inspect and adapt, and on the other hand we measure them by compliance with specific practices. Practices that they may no longer need at all, or may have altered to make the practice more useful to them.

Measuring practices is the wrong approach. What we need to do is look beyond the practices at the desired outcomes. At the "why" behind the practices. Why do we do standups? Why do we do backlog refinement? Why do we do retrospectives?

If we take standups as an example again, I would say that we do standups to help the team communicate effectively about the work that is being done. Standups are one way (one very good way) to teach a team to communicate. If the team finds another way, but still meets the objective, are they agile? Outcomes based checklist says yes, practices based checklist says no.

What about teams that do a standup by the book, but when you observe it, you realise that no real information was exchanged. It's just a group of people standing around a board mechanically repeating "worked on my story yesterday, working on my story again today, no impediments". Are they being agile? Practices based checklist says yes. Are they fulfilling the desired outcome though?

Which of those two teams would you rather have working for you? The one that communicates effectively and actively improves their process, or the one that mechanically repeats the practices with no real insight into why they are doing it?

A practices based measurement rewards the wrong sort of behaviour.

We can still include the 101 stuff in checklists, but as guidance for new teams. Are you communicating effectively? If not, here are some suggested practices you can try to improve communications... 

Checklists should be outcomes based. By measuring outcomes rather than specific practices (outputs) we enable the teams to innovate and still have a way to measure how far along the agile journey the teams are.

Outcomes based checklists tend to be much shorter - we just replaced a whole bunch of questions with a single outcome, and so are more likely to get done willingly. Also, by asking questions like "are we communicating effectively", rather than "is the standup at the same time and place", we encourage the team to think more deeply about the way they work together rather than just ticking off a list of actions. An outcomes based checklist can become a tool for the team's own self reflection (they make a great input to retrospectives) rather than a chore they are being forced to do.