One of the challenges with agile delivery is the idea that planning and making delivery commitments is somehow “non-agile”. While delivery teams work hard to complete their commitments within a single sprint, they are uncomfortable making commitments further down the road. However, our organization requires a commitment. They need to understand when something will be delivered.
This is one of the major friction points between agile delivery teams and the rest of the organization. From a funding or management support perspective, having an idea of what will be delivered by when is the starting point to gaining buy-in and support for the work. If a business is going to finance the work, they expect something in return. A plan. A deadline. A commitment of what they will get by when. And then teams need to deliver against that plan or manage expectations as they head towards an agreed objective.
I live in Vancouver, on the North Shore, which is the home of the Grouse Grind, a 2.5-kilometre trail up the face of Grouse Mountain, commonly referred to as “Mother Nature’s Stair Master". It’s one of many hikes in the Vancouver mountains that are both rewarding and challenging. Every time we head out into the mountains for a hike such as the Grind, experienced hikers register a plan. Where they are going, what time they will be done, and contact details. This plan - a commitment of what we will do by when - allows our friends to know when to worry. If they don’t hear from us by a certain time, they know where to look.
Committing to a long-term objective is similar to the plan posted at the beginning of a hike in the North Shore mountains. It does not necessarily describe the actual plan, but it gives us an indication of what the plan is so that if things go wrong, decisions can be made.
The similarity does not end there. If we are in the hiking party, tracking our progress can be done in a number of ways. Most of us now have a pedometer, some wearable, that is tracking the number of steps and therefore the number of kilometers we have hiked. While this is a measure of how much work we have done - activity if you like - it is not the best way of tracking our progress, especially if we head off in the wrong direction or zig zag backwards and forwards. We really want to identify view points or markers along our route that show progress along an expected path. As we hike, we can track our progress against these markers, we can track our time to each marker, and from that, re-assess how long it will take us to achieve our goal.
On the Grouse Grind, there are large markers at each quarter distance of the hike. At each quarter marker, as hikers on the Grind, we can re-assess our progress and see whether or not we are on track. The first quarter mark on the Grind is famously motivating (or demotivating), offering the last opportunity to turn back, and reminding hikers just how much more work is needed to get to the top.
Agile delivery can use a similar approach to mapping out a major deliverable. Exactly how does this metaphor help our Product Owners and agile delivery teams manage delivery over the long term?
- First, we need a plan. An idea of where we are going and what the markers are along that journey (our target deliverable and the delivery increments along the way).
- Second, we need to continually track progress towards our goal by tracking our teams reaching each marker (outcome or increment), not the activity they have completed. The steps we have taken don’t mean as much as the markers we have reached.
- Third, we use this information to make adjustments. If we are off track or not making the progress we expected, we have the opportunity to reset expectations. Turn back, take fewer detours, or manage stakeholder expectations.
Many delivery teams do one or two of these, but all three are required to answer the CIO’s question above. What is your plan? How are we doing against that plan? And what can we do to adjust if we are off track? Sounds a lot like project management, doesn’t it? There is one major difference, however - planning in uncertainty.
Planning in Uncertainty
Agile planning differs from traditional project planning in one key aspect: it assumes high levels of uncertainty the further ahead we look.
In this case, agile planning is more like weather prediction. Our understanding of the physical models that control our weather and the computer systems that calculate what our weather will be are incredibly accurate and powerful. However, they are also complex, with sensitivity to many variables that determine what the weather might be. This translates into very accurate near term predictions of the weather - we all believe our TV weather prediction over the next few days - but poor long term forecasts that differ widely between forecasters. This is not because the forecasters or their models are bad. It is because weather systems are complex - they are sensitive to many variables which make long term predictions inherently uncertain.
Traditional project planning assumes that the estimates we make today for the work we have to do in the distant future are as accurate as the estimates we make for work today (this week or this sprint). The entire plan can be estimated and mapped out at the outset with a high level of certainty. This is also why a common ask of management is for more ‘accurate’ estimates - which is a little like asking for more accurate long term weather forecasts. Our experience tells us that the quality of the traditional plan is determined by the quality of the estimates.
Agile planning assumes that those estimates are uncertain. More specifically, near term estimates are pretty certain but distant estimates become less and less certain. Knowing this, there is little point in creating a plan based on a complete set of estimates because they will change significantly between when we make the plan and when we deliver the work. But we have to provide something. So, much like weather forecasters, we can make long term predictions of when we will deliver the expected work. But, again like weather forecasters, we will continually update those predictions as our understanding changes.
Planning Matters, Plans Don’t
Projects require planning. Even the Agile Manifesto talks about responding to change over following a plan, with the implication that we still need a plan - a baseline from which we can respond to change. In agile planning, this baseline is a working document, continually updated and refined as we learn more about the sensitivities of the work to different variables.
In the past, software delivery has been generally predictable with little sensitivity to project variables; today this is not the case. Coupling between systems, fickle customer needs, and a continually changing delivery environment make planning highly sensitive to many different variables, and therefore inherently complex and uncertain.
From a business perspective, it's reasonable that the business understands what will be delivered and by when in order to govern how money is spent and whether it's spent fruitfully. At a basic level we need to be able to look at a problem and roughly determine the cost. We may not know the precise dollars-and-cents, but we need to provide a guideline to our business stakeholders; roughly how much it will cost and how long it will take.
Coming back to our hiking metaphor, we need to be able to say where we are going, what markers we expect to see along the way, and estimate roughly how long it will take. This rough plan allows us to make decisions and provides the baseline plan against which we can refine our expectations as we make progress towards the end goal.
In agile delivery, this is made easier because we have dedicated teams that frequently deliver shippable product increments. Therefore, we define increments of delivery (the markers) and estimate how many sprints we need to deliver an increment (the time between markers). In real terms, we describe these increments in terms of epics - large user stories that teams can estimate roughly how long they will take to deliver. Since the teams are dedicated, and therefore the cost of the team per sprint is fixed, we can create an estimate of what will be delivered by when. We can turn a timeline and sequential list of delivery increments into a roadmap with a cost per increment. First step complete. We can make a commitment to our business as to what will be delivered by when.
Track Increments Not Activity
We are trained in many organizations to tell people how busy we are, and not what we have achieved in our plan. In our hiking scenario, if we just track activity - how many steps we have taken, or elevation, or miles completed - we can provide implied progress without describing real progress. We can cover lots of ground without making progress towards our objective.
Instead, the markers along the route allow us to break up our plan into significant measurable outcomes. In our hiking example, we know much more about where we are when our hikers tell us they are at the quarter distance marker on the Grind than they do when they tell us how many steps they have taken. While we can make similar predictions - if the Grind consists of 2800 steps, and our hikers tell us they have completed 700 steps, we may think they are at quarter distance. We can even look at the time taken to complete 700 steps and estimate when they will complete the full distance. But there are a lot of unknowns. They may have taken the wrong route. Or not even left the parking lot yet, but just been pacing backwards and forwards finding coffee, washrooms, various hiking party members.
When we get notice that they have reached the quarter distance marker, we have much less uncertainty about where they are. We can make more accurate predictions about when they will get to the next marker, but now we have certainty about where they are starting (from the quarter distance marker). We effectively collapse the various unknowns back to a new baseline and can recalibrate the plan. This is like looking at the last few days of weather, and using this to recalibrate our long distance weather forecast. We have removed a whole bunch of possible scenarios and can be more certain about where we are and what is coming next.
Activity updates don’t give enough context for us to know that we are heading in the right direction and how that has impacted our ability to get where we are going to next. The value of tracking increment delivery is not simply a measure of progress, but the ability to collapse uncertainties down to make a more accurate prediction of the future. As the long term weather forecast gets closer, our forecast is continually recalibrated to become more certain.
Leaning on the hiking metaphor a little more, as we hit each marker we learn to make more reasonable predictions of when we will get to the next marker. Anyone who has hiked over distance knows how difficult it is to make up time. You can rarely just step on the accelerator and speed up. Capturing the details matters for the team's ability to learn about how they are making progress and what their pace of delivery is.
This is our second step - making our progress real rather than hypothetical by tracking the progress of small delivery increments that are markers on the way towards our long term objective.
The final step falls to the Product Owner. They are responsible for continually updating our delivery plan to provide insights into when the work we agreed on will actually be delivered. This means continually updating the plan with new information. If our teams are delivering more slowly than anticipated, what impact does that have on the timeline? What about expected costs? Clearly, this only really becomes an issue when our original plan looks to be more expensive or is taking longer than anticipated (and when does it not?).
Epic budgeting is a way for Product Owners to continually refine their timeline or delivery roadmap. It starts by establishing a process for writing epics, prioritizing, sizing, and then calibrating delivery timelines. This is the process by which we create an initial baseline - with approximate sizes or costs for each delivery increment.
In the traditional project planning, we hold on to a plan for as long as possible and fight to keep deviations to the plan to a minimum, holding scope constant as much as possible. In agile planning, we recognize that deviations from the plan are inevitable and instead continually adjust the scope with the end goal in mind, instead of protecting the scope that was originally agreed upon.
Let’s take personal budgeting as an example. A common practice for personal budgeting is to allocate a certain budget to each primary cost category, a budget to food, rent, utilities, sport, entertainment, and so on. In fact, a common practice (in the days of the cash economy) was to have an envelope for each category and keep the right amount of cash in each envelope.
This works well until it’s ‘birthday month’, when several of your friends have birthdays in the same month. This throws the entertainment budget out, with multiple birthdays eating up more budget than planned. So how do we handle it? Well, you probably end up doing one of two things: either don’t go to somebody's birthday and run the risk of losing your friend or steal from another budget category - eat less, spend less on coffee, whatever it might be. Anything to make the available cash reach the end of the month.
Let’s get back to our agile planning. We have defined delivery increments which consist of multiple epics. As with the personal budget example above, the same goes for epics. As we move epics into refinement and begin to understand the true cost of each epic, we will find out how good our original cost estimates were. Having broken an epic down into user stories and estimated them with the team, we will discover whether the total amount of effort required to deliver the epic (epic estimate) is about the same as, less than, or more than the total number of story points required to deliver all the constituent stories (see figure).
As we do this for each epic, we can quickly recalibrate our timeline with more informative estimates. The Product Owner is then responsible for managing the backlog of refined stories - where the story breakdown sizing is the same or less than the original epic estimate, there is little for the Product Owner to do. However, when the epic estimate proves to be an underestimate - that is the estimates of the user stories that make up the epic add up to more than the original epic estimate - the Product Owner has to get to work.
They can either remove stories that are not essential to the epic or they can steal capacity from another epic, reducing the work done for a future less essential epic. This is the equivalent of taking from one budget category to boost another budget category. Consequently, we know that the epic we are stealing from is now going to be smaller and skinnier and less impactful than it was before.
As Product Owners, we continually think about elements essential right now and which can be built given the finite capacity we have available to us. This decision is based on knowing the end goal or outcome, so trade-offs that deliver the original objective can be made.
Agile planning and the coordination of work to meet deadlines is similar to the work of the project manager. We are used to the idea that as we learn more, we need to adjust the plan and manage expectations. However, in an agile world, this management of the plan is more dynamic. It falls to the Product Owner to continually inform their stakeholders on progress and to continually make decisions to achieve the desired end goals. Here are three activities every Product Owner should be familiar with to make long term commitments when working with agile teams:
- Build a Product Roadmap. Prior to funding approval and the go/no go investment decision, a high level description of the product roadmap, including sized delivery increments, is necessary to agree scope, schedule and cost.
- Track Progress to Increments. Governance is based on completed delivery increments rather than completed stories, updated at the end of every sprint.
- Epic Budgeting. The roadmap is continuously refined based on empirical information gained at the end of each sprint.
If you want to manage your agile delivery towards a distant deadline, applying epic budgeting is the key. Agile planning gives you and your teams the tools, culture, and practices to plan long term with confidence while steering your programs towards delivery.
Agile planning is fundamentally different from a traditional planning mindset, based on embracing uncertainty and working with it to achieve future commitments. Don’t go it alone. At IncrementOne, we offer tailored consulting and training that takes the challenge out of your agile planning. We propel you toward a product delivery organization that is resilient, effective, innovative, and capable of lightning-fast change.