Why Prioritizing Your Product Backlog for ROI Doesn’t Work (Part 1 of 3)
Saturday, July 26, 2008 at 02:40PM There is a problem in the Agile community. Pundits tell us to prioritize our backlogs to generate the best possible ROI. But no agile teams that I've ever worked for or with do this. So the pundits must be wrong, because Agile continues to provide stellar results for a lot of product teams. This post is the first of three posts to explain both why the pundits are wrong and how to fix it, and is a precursor to my talk Converting Business Value into Actual Money at the Agile-08 conference on Tue Aug 5th. In this post, I'll cover the basics of backlog prioritization and the problems with attempting to prioritize based on financial metrics.
Backlog Prioritization 101
In its simplest terms, your backlog is the discrete set of deliverables that your team must create. Regardless of your particular flavor of Agile, all successful Agile projects (and, truthfully, all successful projects, Agile or otherwise) rely on the effective prioritization of work. To prioritize effectively, product managers must identify set of attributes for backlog items that will be used for prioritization, assign them a value, and then sort the list based on the values of these attributes.
The pundits say that you should value your backlog based on ROI, NPV, net cash, time for payback, or any number of other financial metrics that boil down to comparing revenue vs costs for individual backlog items. In all of the years I've been doing Agile, I've never seen a product backlog where each backlog item slated for the release has an associated ROI analysis. Heck -- I've never seen a backlog where more than a very few items have an ROI.(The previous statement does not hold true for portfolio backlogs, which are often prioritized for ROI based on sensible market analysis).
Product backlog items that do have revenue associated with them tend to be directly associated with contractual obligations, often referred to as Non-Recurring Engineering (NRE). This kind of ROI is simple: customer so-and-so is willing to pay $x for this feature, it will cost $y to implement it, and since $x is >= $y we'll do it. Except that in quite a number of cases product teams will do it even is $x <= $y, because customer so-and-so is either a really big customer, or they rationalize that if customer so-and-so wants it other customers will want it, or that they need the revenue, or, because only these items have ROI calculations, they bubble up to the top because they have more data, regardless of whether they are the best choice.
When the performance gap between espoused theory and actual practice is large you need to look harder at both to understand what's causing the gap. Sometimes, practioners are simply "street smarter" than their more academically motivated counterparts, and the espoused theory has to change to better match the realities of practice. At other times, the practice is just perpetuating old-thinking, and the theorists have a better approach. In this case, I think that street smart Agile practitioners are smarter than those Agile pundits who spend most of their time teaching theories instead of learning from practice.
The Problems With Prioritizing Your Backlog Based on ROI
There are several reasons why prioritizing your product backlog based on ROI is problematic. Here are just a few. Feel free to add your own.
- To establish a realistic ROI, you need to engage in appropriate market research. Unfortunately, most market research that can establish realistic ROI, such as a conjoint analysis or a willingness-to-pay study, is usually based on relatively large chunks of functionality that may or may not correlate to what is on the backlog. The unfortunate side-effect is that the release planning process, in which the development team negotiates the actual features with product management, can result in changes to the tested features that are so large that they original research is invalidated.
- In attempting to solve this problem, many Agile teams are adopting the Incremental Funding Methodology (IFM) popularized in the book Software By Numbers by Denne and Cleland-Haung. Simplifying quite a bit, IFM is based on using market research to establish Minimum Marketable Features (MMF) that can be used to maximize NPV. While I love this approach in theory, because it celebrates direct customer input, I haven't found any examples of this being used in practice. I suspect that one reason is that the method, while relatively easy to explain, has a significant amount of overhead, and defensible NPV needs to be based on defensible revenue data.
- The IFM also suffers from the same problem as much traditional market research in Agile projects: the backlog items tested are created before release planning, but the backlog items implemented are changed during release planning. This isn't to say that the concept of market research in an agile project is fatally flawed. It isn't. We need more market research in the Agile community, not less. We just need market research tuned to Agile projects, something I'll cover in Part 3 of this series.
- A related challenge is that Agilists prefer to manage relatively small user stories, but customers tend to value groups of related and often interdependent stories that together solve complex problems (something Jeff Patton refers to as "task" focus). And while any good Agile Product Manager hates interdependent backlog items, the reality is that sometimes backlog items really are interdependent. To illustrate, one Enthiosys client was building regulatory compliance software in the trucking industry. Many of their customers had large operations in the US and smaller operations in Canada and Mexico. The NPV for the MMF "provide compliance software in all geographic markets" would be positive, but the NPV for the MMF for each individual backlog item (one for each market) would be negative.
- Creating, executing, and analyzing a statistically significant market research project can take longer than a release cycle for a good agile team. In one client project, Enthiosys participated in a comprehensive conjoint analysis for a rather large system. In this case, it was the right choice for the client, but the overall project took about 7 months -- far longer than the release cycle of most Agile teams. And the expense ($100K's) means that it is highly unlikely that any company would fund more than one of these projects per year.
- Statistically significant market research may be antagonistic to Agile. Consider that the co-creation of value, which involves that active participation of companies and their customers, is evolving to the point where many market segments don't expect to be asked to respond to traditional surveys. Instead, they expect to be allowed to collaborate with the company -- and ther customers -- to jointly develop specifications and solutions to complex problems, something our client SAP (among others) is pioneering through their Collaboration Workspace.
- Agilists celebrate our ability to respond to change rather than following a rigorous plan. Which sounds great until you're asking a product manager who just invested a lot of time and money in her market research to reshuffle the backlog based on new information. They'll be torn. The market research says that they should stick to the plan. The new information says that it should change. The greater the investment in the market research, the more challenging it is to make the change.
I suspect that even with all of these problems, pundits continue to recommend this approach because it sounds easy, it creates long-term, and therefore, expensive consulting projects, and it makes Agile easier to sell to MBA-trained senior executives ("See? We're prioritizing by business value, in terms that you've been trained to understand -- NPV. Just don't ask me to show you how I got that number, OK?). And, since I don't see much emphasis on rigorously testing if the projected NPV matches the actual NPV, and what to do if they don't, I'm admittedly skeptical of just how well ROI based prioritization methods perform over time (which is, if you'll notice, the core aspect of NPV; money sooner is better).
In summary, consistently prioritizing your backlog for ROI is a fool's errand. In part two of this series, I'll discuss how to create sets of attributes for prioritization that every Agile Product Manager can leverage. In part three of this series, I'll show how to gather the data for these attributes and how to use them to prioritize your backlog for sustained success.
{see the second and third posts in this series}
Reader Comments (13)
Hi Luke,
Interesting article, and although all three blog posts in the series are out, I am commenting on this one before reading the other two. A few thoughts:
1) Who are the "pundits" that say prioritize by ROI? I see a lot of agile folk say prioritize by business value, which is really a way for development teams to abdicate responsibility for making intelligent decisions about the order of delivery for backlog items to the product owner, without providing any guidance on how to do that. But honestly, I haven't seen too many folks say prioritize the backlog by ROI directly. Granted, some of the few that bother to try and help Product Owners prioritize use that as a method...
2) I completely agree that things like ROI, NPV, etc. are completely inappropriate for prioritizing the product backlog. They are much better suited for making decisions about projects, for many of the reasons you site above. Especially the fact that a lot of product backlog items tend to be interdependent when it comes to truly delivering value.
3) I also agree that the full blown MMF model is probably too cumbersome to be done in practice, but it is a nice way to group product backlog items or features together to address the issue described in #2.
So if you don't use financial tools like ROI, NPV, etc. to prioritize, what do you use? I suppose it depends. I find that relative weighting which is determined by factoring in the Projects Purpose, Considerations (such as risks, assumptions, and constraints) and some understanding of the impact to cost and benefit can work, when done in a barely sufficient manner.
Nice post. I wanted to attend your session before reading the blog post. It will certainly be an option worth considering during that time slot.
Kent -
Re: (1), we agree that those claiming to prioritize by "business value" without "showing their work" are indeed abdicating their responsibility for true business value. As for the pundits, there are too many to mention, but a simple search on web turns up a lot of people who promote prioritiation by ROI without explaining the pitfalls of the approach.
Re: (2) & (3), we're in agreement. As for the rest of your comments, I hope and trust I answered them parts 2 and 3 of this series.
Hi Luke, I couldn't agree with you more! I also bumped into lots of people saying "prioritize by ROI?" at agile 2008.
On the teams I have seen try this, it consumed vast amounts of energy and people just gamed the ROI numbers to move their stories up the stack.
I think it's important to understand the business rationale behind the funding of the project. I like to get the business to explain the model behind their thinking. I then use this as a litmus test for the backlog items.
[...] Several speakers at Agile 2008 talked about business value “Story Points”. Joe Little proposed assigning Business Value points to each item in a backlog. My experience is that this consumes a vast amount of energy and the business people start making up large values to game the prioritization process (Luke Hohmann has a nice posting on why prioritizing the backlog by ROI like this does not work). [...]
[...] fantastic way to do your planning, but their were some flaws. Luke Hohman stated that "Prioritizing a Product Backlog for ROI is a fool’s errand".The whole discussion between Scott and Luke is very educating, I suggest you all read it and [...]
[...] [1] Luke Hohmann, Why Prioritizing Your Product Backlog for ROI Doesn´t Work, [...]
[...] [1] Luke Hohmann, Why Prioritizing Your Product Backlog for ROI Doesn´t Work, [...]
[...] is a review of Luke Hohmann’s excellent blog series on Product Backlog Prioritization. As usual, I have captured what I believe to be the salient points in a visual note. The main [...]
[...] is a review of Luke Hohmann’s excellent blog series on Product Backlog Prioritization. As usual, I have captured what I believe to be the salient points in a visual note. The main [...]
[...] And they should do this, as I describe in my three-part blog series Prioritizing for Profit: 1, 2, [...]
[...] is a very useful set of articles on the Enthiosys website that go into quite some detail on a new approach to looking attributes that allow prioritisation of [...]
[...] Why Prioritizing Your Product Backlog for ROI Doesn’t Work (Part 1 of 3) [...]
Hi, Luke!
I agree with you that prioritizing a backlog by ROI is very difficult. We can, however, include other metrics that helps us prioritize well. I like the approach used by Tom and Kai Gilb (gilb.com). There is a great article about their method here:
http://tiagomjorge.wordpress.com/2011/06/15/measurable-value-with-agile-the-impact-estimation-technique/
What do you think about it?
Best regards!
Tiago.