Sunday, February 18, 2007

Rockets, Cars and Gardens: Visualizing waterfall, agile and stage gate




The further I dig into new product development practices the more I crave a simple way of helping folks new to the concepts visualize them quickly. In that spirit, I've assembled a little pictorial journey through the intriguing landscapes of waterfall, agile, portfolio management and stage gate. For fun, there is also a description of how you can apply portfolio management techniques to individual agile project as a technique for additionally reducing the design risk.

The concepts illustrated here are readily applicable to most software projects including games, websites and of course full desktop applications.


Software development as a learning exercise
All the diagrams start with two premises:
  • Knowledge is critical to success: A large number of software projects fail because we build software that has little real world value. If you dig into the reasons, even teams with great production skills claim they focused on the wrong issue or they really didn’t understand what the customer actually wanted. When you build the wrong software, you end up with useless code, loss of political support and eternal development cycles. If only the team had reliable knowledge of what to build, they could deliver value early.
  • Most product teams start with very little practical knowledge: Knowledge is great, but we start projects with hardly any. We don’t know exactly what our customers want. We don’t know how our design will work when it runs smack into the complexities of the real world. We don’t know what new opportunities and constraints will emerge in the future. Even when we are experts from a related domain, the best we typically have are theorems, guesses and opinions.
Product development is about rapidly producing targeted solutions that generate maximum customer value. In order to do this efficiently, our teams need to be learning teams that are constantly gathering concrete domain knowledge about customer needs. A team that learns the quirks of its customers, code, and business rapidly will often out perform teams operating without this knowledge. With this concept as a filter, let’s take a look at some typical development scenarios and how they are affect by team learning.

Typical product development of a single product
In most traditional waterfall models, teams gather requirements, develop the product and then test it in order to see if they implemented the spec correctly. Only after they release do they gain any insight into what the customer actually desired. The metaphor that is often used is that of launching rocket towards a destination. You get to aim it once and then you fire it off and pray that you hit the target. All well and good, except A) this is a rocket that burns money for fuel and B) when it misses its target, entire teams (and sometimes entire companies) get the ax.


Now, not all is lost. If your company has enough money to burn, they can try again. There is a good chance the team learned quite a bit about what their customers actually desired by failing in the marketplace, so the next moonshot has a better chance of landing closer to some actual customer needs.



The old joke is “If you want to build a good piece of software, rewrite it four or five times.” Eventually, if you keep trying long enough, you’ll hit a target market that wants your product. Some very big companies have made a staggering amount of money following this dogged pursuit of success. They fail again and again and again, but their few successes fund the further pursuit of new growth businesses. The learning is quite expensive but very valuable.



Room for improvement
This pattern of eventual success through multiple failures has been well established in all sorts of new industries. A few smart companies have identified two basic strategies for speeding up the time it takes to identify a new successful market.
  • Standard portfolio model: Fail in parallel. By trying lots of different options all at once, a few will succeed.
  • Iterative model: Fail sooner. Try something simple, test it to see it works and repeat.
Standard Portfolio Model
If the waterfall process is like firing a rocket at a target, the portfolio model is like firing a swarm of rockets and hoping one hits. The company greenlights a large number of projects, funds them fully and hopes that one of them will blossom into a success. In return for spending more money, you rarely have to wait until version 3.0 to observe success.



This model tends to be highly useful when the target market is poorly understood or ephemeral in nature. For example, in the music industry, when user tastes change every month, it doesn’t make sense to spend three years attempting to evolve an individual album to fit the market. It also tends to be useful when the cost of development is low and potential payoff is high. The incremental cost of green lighting one more project can be easily offset by the incredible amount of money a successful record produces.

The big benefit of the portfolio model is that it allows you to try completely different options simultaneously. Companies are often put in situations where more than one idea has potential and they lack the information to choose which ones to fund. If they were forced to choose only one, they would often miss out on creating multiple product lines. Instead of putting all your eggs in one basket, a portfolio model spreads the risk and increased the chance of multiple innovative breakthroughs.

The Achilles heel of the traditional portfolio model is cost. Smaller companies have difficulty funding one project, never mind ten or twenty.

The iterative model
Smaller teams have learned to maximize their learning opportunities by building lots of opportunity for rapid feedback into their process. The agile software development movement is the poster child here, but many of the same lessons are found throughout lean manufacturing, kaizen and other successful practices used by product development companies across a broad spectrum of industries.



The common metaphor is that of driving a car as opposed to launching a rocket. At every possible opportunity, you are looking ahead and adjusting the team’s trajectory in order to steer towards. Each change may seem subtle, but due to all the rapid cumulative adjustments, the team hone in on their targets quite efficiently.



It ends up being okay that the team makes little mistakes. If they veer off a little bit to the left, that's fine. They rapidly learn it was a bad idea and correct their efforts. The short feedback cycles ensures the big mistakes happen far less often.



Instead of taking 12 to 18 months to create and evaluate a new concept, they build and put new version in front of users every 2 to 4 weeks. They also work in high bandwidth environments where all the team members are close together and close to the customer. Team members converge on and build concensus around good ideas over a period of hours, not months. Teams become experts through intense hands-on problem solving and testing. This ends up building products much more likely to serve real needs than those imagined in Platonic specs by ivory tower experts.

Agile development is favored by small start up teams because the techniques greatly reduce the risk of an individual team failing. If you only have room for one shot at the target, you might as well steer your way to success using lots of rich information instead of launching blindly into the unknown. Long term, agile processes delivering more value sooner, with lower overall risk.

An agile project is intensely focused. In the rush of completing only high priority features, many alternative concepts never get the chance to be explored. Customers, a rather vague and argumentative bunch at best, are required to speak with one voice in the name of reducing noise and thrash for the team. For many teams struggling just to get software out the door, these traits are a godsend. The downside is that there is little concept of strategic portfolio management.



Agile projects can be a bit like a hill climbing algorithm. They will steer towards the closest customer success story, but may ignore a bigger opportunity further away.

Stage Gate: Cross breeding portolio models and iterative development
We can do better than either pure agile or pure portfolio development. The stage gate process borrows a little bit from both iterative and portfolio techniques. You are still launching multiple products, but then you use iterative development techniques to guide each project towards success. You kill projects that fail to meet your goals.



Imagine that we are gardeners. We seed a diverse portfolio of projects, some high risk, so low. As we add team members and customer feedback, the projects begin to flourish, winding their way up towards the sunlight of launch. The core team development practices can be agile since we want to encourage short cycles of rapid feedback. The best ones start to blossom and ripen with obvious customer value. However, some projects are sickly and seem unlikely to produce much of anything.

As gardeners, it is also our job to groom the garden. The weak projects are trimmed back and turned into mulch that helps nourish the most successful projects as well as create a bed for seeding even more promising ideas. The best projects are harvested and sent off to customers, rich with ripe features, bursting with value. We are constantly seeding new projects to keep our portfolio balanced.

We’ve added three items to the traditional agile model.
  • Diverse seed teams: The first is the concept of multiple teams heading in different directions with different goals. There isn’t a single customer, but multiple customers driving multiple potential success strategies.
  • Kill Gates: The second item is the concept of gates every few iterations that kill poorly performing projects. It is better to return those resources back to existing projects than to continue investing. You can think of the success criteria that drives the gates as a set of reasonable constraints. The team is allowed to do whatever it desires as long as it meets the outlined constraints that are measured at each gate.
  • Concept banks: The third is the concept bank where old ideas are stored for possible recycling. You never know when the remnants of an old idea will nourish a strong new project.




You may start out with dozens of projects, but in the end you’ll only launch a few into the market. The combination of trying lots of directions with very small low cost agile teams dramatically increases your opportunities to learn compared to either pure iterative or pure portfolio techniques.

You spend dramatically less than you would in a pure portfolio model since you don’t have the heavy cost of finishing and deploying the bulk of unsuccessful market explorations. All that waste can go back into either exploring new options or building out projects that show success.

You have a higher success rate than a purely iterative model since you are exploring multiple hills instead of climbing a single hill and hoping that it is the highest one around. This recognizes that not all successes are created equal and it can be easy to get stuck in rut serving a niche. By keeping your options open, you can always shift resources around if a truly great opportunity emerges.

One downside is that cost is a little higher than a pure iterative model. You need to devote a portion of your overall efforts to generating and exploring new opportunities. This is typically more than offset by the earlier access to successful products. Another downside is that some long term projects (such as say…a fusion reactor) are culled before they show their promise. This can be handled through careful tailoring of gates so that break through innovations are properly weighted.

The Harvest Process: Applying a stage-gate model to a single product
A single product can benefit greatly from the ideas in the stage gate model. A single product is composed of a variety of separate features, scenarios and audiences. We can treat each chunk of customer value as a separate project and bundle of high value projects as a product. The basic model is the same as the stage gate process outlined above. However instead of spinning off multiple products, your individual teams tend to be focused on critical workflows or user scenarios for a single product.



Returning to the garden metaphor, as you nurture these areas of value, you should always have projects that are ripe for release. Publishing can be as simple as creating a public release that turns off all the experimental projects, leaving only your best features to show the public. I think of this as harvesting a ripe crop for your favorite customers.

As time passes, the little ideas that you seeded earlier will mature and produce a new crop of high value features. The process just keeps repeating, season after season, release after release.

The cliché in our business is that ‘ideas are cheap’. The shame of the matter is that some of the best ideas are left to rot because they don’t come from the right people and the production schedule has no room for creative thinking. The Harvest process builds innovation in at the lowest level possible. Everyone is encourage to come up with ideas and there is a clear and official process by which those ideas can become reality.

Making it work
In order to pull this off you need some basics in place. There is a substantial expense to running multiple projects and constantly integrating their efforts. With the right people, the right process and the right values in place, it is a very achievable goal.
  • Multiple, small agile teams: A single monolithic team has difficulty guiding a multitude of projects. Smaller teams can champion their area more easily.
  • Highly refactored and regression tested modular code: You need to be able to change one element without breaking the rest of the projects.
  • Innovation at the edges: Teams need to be able to innovate in interesting directions without constantly referring back to the ‘master planning committee’
  • Kill gates: If you don’t understand success and kill projects that don’t meet the bar, you’ll end up with a cancerous mass of bizarre, unhealthy features that have grown in random directions. Quality kill gates that actually kill projects are critical to success.
  • Shared design: You still need to present a unified face to the customer. Each team is under additional constraints to share visual styling and minimize conceptual complexity for the product as a whole. These constraints are built into the kill gates.
  • Customer feedback: At every stage, you need to be testing your products with real customers. You should be able to quickly deploy a working product to your customers every day. None of these ideas can succeed without people using working versions of your software and the team being wiling and able to learn from the customers’ reactions.

Conclusion
In the end, any of the techniques outlined above have a chance of working. You need a driven group of people with the right circumstances in place. With the right amount of random luck and persistence, even a waterfall project can change the world. For some fat cats working in bloated large companies, this is enough. However, if failure has a personal cost, you’ll want to look into the alternative techniques such as agile, stage gate, or harvest. Over the long term you’ll notice less overall portfolio risk and your customers will start enjoying your superior, well informed product designs.

If you take away one thought today, it should be that learning is fundamental to software development. We do not live in a Newtonian world where huge mechanical brains can predict the future. Instead, we operate in a constantly shifting organic universe of cultural trends, customer urges, fluctuating cash flow and skeptical sponsors. In order to deliver great products on time, we need to learn from our environment and build a response rapidly and precisely. Portfolio technique increase learning by trying more options. Agile techniques increase learning by shortening the feedback cycle. Build both of these key concepts into your teams and you’ll find that they make smarter decisions and deliver more value to your customers.

Take care
Danc.


References and resources

Original pretty pictures
I was inspired by these image of agile development when I began making diagrams.
http://www.gamesfromwithin.com/articles/0608/000110.html

Small initial teams? Try 20% of one person.
With Harvest and Stage Gate, initial teams can be very small. For example, you could devote 20% of one person’s time for the initial concept stage. That doesn’t sound like much, but if you have every single person in the company devoting 20% of their time to new ideas, you’ll be seeding the equivalent of hundreds of new projects every year for no additional headcount. Google is the golden child of this concept right now, but 3M has been doing it successfully for decades.

Put those hundreds of ideas through a simple checklist that looks for potential customer value. Spin off the best ideas into tiny teams of a couple people and give them an iteration or two to come up with something cool (and testable.) Repeat the process several times and you rapidly gain some surprisingly innovative products .

Iterative techniques
Agile Overview: http://www.agilealliance.org/
Scrum: http://www.controlchaos.com/about/
Kaizen: http://en.wikipedia.org/wiki/Kaizen

Stage gate
Winning at New Products: Accelerating the process from idea to launch, 3rd edition Robert G. Cooper, copyright 2001

Project Horseshoe Report on building games that sell
Many of the ideas in this essay were heavily influenced by discussion inside the Play Dough group at Project Horseshoe 2006.
http://www.projecthorseshoe.com/ph06/ph06r5.htm

14 comments:

  1. Would you like to speak at our Agile user group. Been following your blog for some time and think it would be great. We are in Denver, if you are ever in the area. See http://www.agiledenver.org and there is contact info there.

    ReplyDelete
  2. Your diagrams are so much nicer than mine. :)

    The trouble with the Harvest process is this: Game development tasks are almost never orthogonal. Unless your technology is 100% set in stone when you start, you're going to be making changes to things like the file formats and code libraries the whole way through development. You make those changes because of feature A or B, but they often affect sections of the game outside of those specific features. Arranging these features into silos that don't interact with each other is a recipe for disaster.

    Of course if you smudge the circles together a little and include the failed-at-the-last-stage feature in the finished product you have a pretty accurate picture of the development process of an average game.

    ReplyDelete
  3. I think if I worked at Microsoft, I would not let the phrase "Kill Gates" appear so many times in one blog entry. ;)

    ReplyDelete
  4. Applying harvest to Games
    Joe, applying something like the Harvest process to games is an interesting challenge. However, it is the same basic problem that you run into with any multi-team agile effort. I won't say this is solved problem, but there are companies doing it right now with reasonable success.
    - Integrate frequently: Everyone is integrating their code into a shared code base. Some throwaway prototypes might not make it in, but building together and running everyone's regression tests makes it more difficult for things to completely diverge.
    - Individual teams still talk to other teams. Most of their high bandwidth communication will occur between one another, but you still want meetings periodically between different teams and different disciplines. Programmers should meet every couple weeks and chat. They'll enjoy and good things will come out of the conversation.
    - Shared Library team: One Scrum technique is to have another team that comes in after teams have build something and figure out how to refactor that code into libraries that the other teams can use. This still allows innovation on the edges, but also helps a) prevent unnecessary rework and b) create a shared technology platform that accelerates the progress of each team.

    There is a nice article on applying Scrum to game development in the February 2007 issue of Gamasutra ("Scrum Rising, Clinton Keith, pg 21). He covers the basics like developing in vertical slices and the overall process. He also runs a blog that I enjoy linking to called http://www.agilegamedevelopment.com/blog.html

    Your last comment about including the failed-at-the-last-stage features is one of the sad things about modern game development. Many of those features could have been killed earlier and a lot of unnecessary wasted effort could be saved. Strong Kill Gates can be politically hard to implement, but are immensely useful once you follow them.

    It isn't your job to build bullet points
    It is better to release less than it is to release more of poor quality. Players are motivated by the quality of the experience, not a check list of features. If you need a checklist for a product, that is marketoid task. Any MBA worth their salt can take the simplest and easiest to use product on the market and generate an impressive feature list.

    See this video about the iPod:
    http://video.google.com/videoplay?docid=36099539665548298

    Cabal
    I also recommend that folks check out any articles they can find on the Cabal process used at Valve. The terminology is slightly different, but it shares many of the same attributes. They rely heavily on customer testing to ensure the good stuff gets additional love and the bad stuff gets killed.
    http://www.gamasutra.com/features/19991210/birdwell_01.htm

    take care
    Danc.

    ReplyDelete
  5. Thanks for the fascinating post!

    As a NPD consultant working in industries outside of game/software development in addition to a geek/programmer when I first read this post I thought it was about general software product development.

    After a second read and looking at the very interesting project horseshoe discussion I realized that this is mostly focused on games. But I think that it still applies to other software product development practices, maybe more so!

    I know very little about the game industry. Coming from a NPD perspective I think that it has much more in common with the movie industry than it does with the industries that have implemented Cooper style stage gate processes.

    One thing that may be confusing to many is that the first few stages of this process don't involve development work.

    Assume the example stage gate: Idea > Scope > Business Case > Development > Launch.

    We usually advise our clients that the end of the third stage (business case) they build a prototype. It's hard to say that you've made your business case if you can't overcome the technical hurdles. In consumer products they need this for the first steps of Development - mainly focus groups. In software development its one of the primary things that agile methodologies have taught us.

    Most of the upfront work (fuzzy-front-end) has to do with identifying and exploring unmet needs. This is also where the majority of our clients need help. However when the primary function of the product is entertainment how do you answer questions like:

    How is it done now?
    Whats wrong with it?
    Whats the value of the improvement?

    If I'm talking about P&G's Swiffer these questions are easy.

    If we are talking about a game engine the functional requirements are much easier to nail down because your primary customers are the people who have to build the game on top of it. I realize that their needs are in turn driven by the end user but I hope I make the point.

    The need is never for the product - its for what the product can do.
    What does your product do? It engages and entertains me. Isn't this the same function a movie fulfills? Perhaps a better model than stage-gate would be the hollywood model. Although their success rate isn't much better they are able to spread the risk around

    This applies to the overall success of the game and the decision about whether to make it at all. However, if we are talking about specific features I think an adapted stage gate process might be useful for separating the wheat from the chaff.

    Anyway, thanks for the interesting topic. I am very curious to see where it leads and how you apply these NPD and Stage Gate concepts.

    ReplyDelete
  6. Doesn't Hollywood basically use a stage-gate system for TV production?

    The typical model is this:

    * Pitches for shows are submitted.
    * The survivors are filtered and pilot scripts and production estimates are submitted.
    * The survivors of that stage are then commissioned as pilots.
    * The surviving pilots are then commissioned with episode runs
    * The shows that are performing poorly by mid-season are then canned.

    So, every year they end up receiving thousands of show ideas which turn into maybe 3 good shows. It's a very expensive process.

    ReplyDelete
  7. Re: Using Movies as a model

    This is in fact what the industry has been doing for the past decade or so. It tends to work poorly. By bringing the language of NPD into the picture, the hope is to give developers and publisher new tools that help improve success rates.

    A couple of differences between movies and games give me hope that this tactic might work:
    - Games can be tested incrementally. The core 15 minutes of gameplay is generally indicative of the broader experience. You can test games with your audience earlier and need to rely less on expert opinion.

    - There are still numerous market niches that are not being tapped. Nintendogs, Sims, Bejeweled, Various Serious Games, Brain Training, Sing Star all tapped into huge markets that most game developers didn't even realized existed. Just yesterday I was using a pedometer mixed with a virtual pet mixed with a DS game and it was amazingly useful. That is a new to the world product if I've ever seen one. A process that actively seeks out these niches is something worth striving for.

    I'm fascinated that games are utilitarian software as much as they are fluffy content. We have the ability to dramatically change their functional capabilities so that games fit into different customer environment. The environment. With game, you can serve entertainment while riding a train vs entertainment while walking vs. entertainment while at home vs entertainment while hanging out with friends. Each of these need different entertainment applications with distinctly different functionality. This is less the case for static content with a fixed viewing platform.

    People have a wide variety of entertainment needs. Games, as software, have the ability to morph and meet those needs in ways quite unlike any other media. This flexibility presents a vast landscape of new-to-the-world entertainment opportunities ill served by committees of inward looking cultural judges.

    We need a process that looks outward and can ask the housewife when she has the need and opportunity for entertainment. Is it in the 15 minutes that she picks up the kids before soccer practice? Who are the other people around her? Other moms? What are their shared interests? Do they need something to break the ice? If we come up with a solution, how can we judge the market validity of the idea? These are the sort of questions that good NPD practices excel at asking, answering and converting into blockbuster products.

    In movies, you have the script writer, the director, the suits making decisions. It is only at very, very end of the process that customers are involved. This is not a process that fills unserved needs. It is a process that serves minor variations of content to a well established audience with a well established need. With games, we are still discovering new needs and new markets. We need a process that helps, not hinders our progress.

    take care
    Danc.

    ReplyDelete
  8. Have you read Mary and Tom Poppendieck's Lean Software Development books? I would assume so, since you mention both lean manufacturing and agile here, and those books explain how applying lean ideas (lean product development, actually, rather than manufacturing) to software gives you agile ideas, but some of their recommendations that aren't part of traditional agile seem closely aligned to what you mention.

    In particular, their discussion of funding models is closely aligned to your multi-project scenario (if I'm remembering them correctly - my copies of their books are at work), and their set-based design looks a lot like your single-project version of this. Concept banks show up in there, too.

    ReplyDelete
  9. Great article, as usual. Thanks for this. We're heavy into Scrum at Relic and one of the things I'm specifically working on is our new products strategy. I've been reading Cooper's book and I had read your Project Horseshoe stuff, so by now I'm pretty convinced we should be using a Stage Gate system for new product development.

    Love the diagrams too. :)

    Adrian

    ReplyDelete
  10. David: Yep, I've read one of the Lean Software Development books. It is a great read and has some wonderful concepts. Recommended. It is good to see these useful concepts from other industries making their way over to the software world.

    Adrian: Very cool! Go Relic! Stage gate techniques are a pretty rational way of bringing a touch of sanity to a rather messy and creative process. The real trick is in the design of the gates and how they are applied. It is a great opportunity for the company to ask "What do we really value?" :-)

    In the best of cases, the process gives a clear and ongoing opportunity for new-to-the-world projects to thrive. That is pretty darn cool (in addition to helping reduce risk)

    take care
    Danc.

    ReplyDelete
  11. Has anyone come accross a good article on Agile development methods applied to hardware-based products? Is it possible? What are the trade-offs?

    ReplyDelete
  12. Great article, thanks for sharing.

    Our team uses a hybrid prototype+agile method.

    We fire off multiple rapid prototypes to test opportunity areas then use agile to get there. This is a good approach for small teams.

    Here's a diagram - http://twitpic.com/otkdc

    Cheers,
    twitter.com/aussie_ian

    ReplyDelete
  13. Hello Danc,

    Your post is great! Many thanks! I've translated it into french :
    http://www.fabrice-aimetti.fr/dotclear/index.php?post/2011/01/03/Fusees-Voitures-et-Jardins-%3A-visualiser-le-modele-en-cascade-le-processus-agile-et-la-methode-du-passage-de-portes
    Regards,
    Fabrice

    ReplyDelete
  14. As a potential plant breeder is resembles what I am being trained to do but I am not convinced that it will work in this case.
    The usefulness of this method rests on several assumptions
    1 that you have limited resources to devote to your teams (nearly always true)
    2 that you can successfully tell the difference between a good game and a bad game (not usually a problem)
    3 that early quality is highly correlated with later quality (this one is the cause of most problems)

    I don't believe that you can currently judge the finished project reliably by the demo.

    There is a pretty simple way to test how good you are at predicting. You run the whole project like normal but this time instead of actually culling teams at each stage you just write down on paper which teams you would cull. When the projects are finished you compare the teams that you would have culled with the teams that passed every sage and see how different their success rates are.

    ReplyDelete