Scaling Up To Scale Down

Last time we looked at the trend towards massive scale in the agile community and some of the problems scale leads to. We looked at an alternative approach. A scaled down approach -

"Imagine, instead of a huge program, we have small groups of teams, say 2-5 teams in a group. Each group manages its own stakeholders, environments, dependencies and the like. Each group is directly aligned to a set of business stakeholders with a common set of outcomes, is funded through an investment pool aligned to business outcomes, not specific project deliverables, and delivers value end to end for the stakeholder group."

This approach would allow the organisation to deliver value efficiently without the need for massive scaled structures and the complexity and inefficiencies that go along with them. The only problem of course is that such a structure is impossible in most organisations because they are built around large programs and large platforms and simply don't have the ability, architecture or processes to handle a scaled down operation.

So where to from here? Can we move beyond scaled approaches to a scaled down approach? I think we can and the first step in that journey is to scale up.

There are generally three impediments within an organisation to scaling down -

• Organisational structure

• Processes

• Technical architecture

Of these, technical architecture is the crucial one. It is both the hardest of the three to change and the easiest of the three to change. Hardest because it is a concrete thing. It is real, implemented, in production and fixed. To change it requires a massive investment in time and money. Easiest, because compared to the other two, it requires the smallest change in thinking to achieve. The other two not only require investment in time and money to change the physical artifacts, they also require a much more massive change in thinking.

The other thing that makes technical architecture crucial is that it is the physical embodiment of the other two: 

"Organisations that produce systems are constrained to produce designs that are copies of the communication structures of these organisation"

Conway's Law

The technical architecture locks in and makes concrete the other two. The technical architecture develops as a result of the organisational structures but then the organisational structures strengthen around the technical architecture to optimise the maintenance of that architecture so the architecture essentially ends up locking the organisation in place.

If we could somehow make the technical architecture more flexible, it would allow the rest of the organisation to become more flexible. Fortunately, there is a way to do this, and it involves leveraging the incredible delivery power that our scaled agile approaches give us.

Most of the time, when we set up a release train, or other scaled structure, we leverage its enormous delivery power to deliver more new features. But what if we turned part of that delivery capability in on itself? Used it to deliver, not new features, but a new, more flexible architecture? But how should we do that?

What if we carved out a chunk of our architecture, right through the stack, from top to bottom. A chunk that is about the right size for a set of maybe 2 or 3 teams to work on independently. We make that chunk as independent as we possibly can from the rest of the architecture using microservices or some other form of encapsulation so that we can develop, build and release independently. 

We also select our chunk so that it delivers value for a small set of business stakeholders. This is important as it gives us a small set of stakeholders with a direct stake in what we are doing.

Now we separate our small chunk from the rest of the train and set it free to run on its own. So we now have a small, independent set of teams, able to deliver value quickly to a small group of stakeholders independent of the rest of the organisation. Let's call this small unit a clan (because the Spotify model has already snapped up most of the cool sounding names and I don't want to confuse things by re-using names).

This clan and its stakeholders now have a direct incentive to develop processes that optimise delivery of value. So by changing architecture, we have started to change business processes.

Now we do it again. We carve out another chunk and spin up another clan. We keep doing this until we have a collection of clans, each with its stakeholder group and processes. Of course this doesn't fit within the current organisational structure but if it is delivering value, there will be pressure to change the structure. Conway's law can work the other way as well.

Now of course all this will cost a bunch of money, both in direct costs and in opportunity costs, as the enormous delivery power of the scaled teams are put to use refactoring rather than delivering new functionality. There will be enormous pressure to prioritise new functionality and relegate refactoring to a "when we have time" activity. This must be resisted. "When we have time" means "never" because new functionality will quickly fill up the entire delivery capacity and the chance to use that capacity to refactoring will be lost.

This can't happen as an undercover project run on the side by some keen agilists within the organisation. This has to come about as a deliberate and considered move by the organisation as a whole. It must walk into this with its eyes wide open. It must make a full commitment to this. It must carry through with that commitment when the going gets tough. This requires leadership.

The role of leadership in this is crucial. They must not only know about this change, they must actively sponsor and lead the change. They must want this. They must see this long term investment as crucial to their long term success. They must commit to change not only the organisation but themselves. A new organisational structure with new properties requires new management. If they want to be that management, they must change with the organisation.

I mentioned three impediments to change before. There are actually four and the fourth is leadership mindset. While technical architecture is crucial for implementing change, leadership mindset is absolutely crucial for enabling the change to even start. Without that in place, not only will you never succeed, you won't even get started.