Click here to read the report. Now for a bit of commentary on the topic of innovation and business.
We are bad at designing profitable games
If you've read this site for some time, you'll notice that I keep returning to the themes of measuring success and rapid feedback cycles so that you can make necessary changes early. Unfortunately, it became very clear that most players in the game industry do not apply these lessons to the business of designing games. It isn't purely a publisher issue, nor is it purely a developer issue, but the overall issues are rather clear:
- We don't know what makes a successful game.
- We don't measure our games regularly from the customer perspective to see if we are going down a successful path.
- We don't act on information that our projects will be market failures.
A Solution
You might expect a good deal of venting at one of these designer retreats, but everyone was far more interested in fixing the problem. The game industry isn't the first to run into these issues and the solutions we found used in other industries are surprising palatable. Our group ended up exploring the adaptation of the Stage Gate new product development process to game develpment.
Stage Gate is a well known options-based portfolio management technique that focuses on improving product launch success rates. It is particular appealing to the game industry because it allows companies to try out lots of different ideas ranging from highly innovative project to product line extensions and then guide the most successful ones toward a market launch. Hundreds of companies in other industries use this model and it has been proven time and time again to dramatically improve success rates, reduce design and execution risk and also reduce time to market. These are all good things. :-)
Low hanging fruit
My basic belief is that innovative products are a highly desirable and profitable component of any publisher's portfolio. However, in order to make this obvious to all the egos involved, you need a process that provides hard data and a way to mitigate risk/fear. The Stage Gate process does this with surprising efficiency.
It is also obvious that a competitive market eventually punishes stupidity. Ignoring new markets while releasing products with a poor chance of success only works until someone figures out that there are better ways. The first party story has some obvious suspects and I know at least one forward thinking 3rd party publisher has already started rolling a version of the stage gate process out across their teams. It will be fascinating to watch the techniques we talk about in the paper inevitably take hold over the next decade.
If you work in any organization that deals with multiple retail game titles, you are likely to save millions of dollars by taking the lessons of this report to heart. Perhaps that is a big claim, but one that is well justified by the historical results from other industries. The opportunity to become the savior hero in your organization is immense.
Kudos go out to everyone in the PlayDough group. We had the perspectives of independents, studio presidents, and publishers represented. Everyone is looking for solutions and everyone who worked on the report feels quite hopeful about the future. It is a wonderful thing to be able to point to business best practices that hold the promise of dramatically improving the industry as a whole.
take care
Danc.
- Read the PlayDough report online here: http://www.projecthorseshoe.com/ph06/ph06r5.htm
- Download the PDF of the report:
Project Horseshoe Report - Building Innovative Games That Sell.pdf - Read all the Project Horseshoe 2006 reports: http://www.projecthorseshoe.com/reports.htm
I feel like I maybe already posted this in another comment so excuse me if I'm repeating myself.
ReplyDeleteThis seems related so I thought I'd post it.
According to friends that work for Nintendo of Japan, whether or not they use a Stage Gate, they have something called the Mario Club. You'll notice "Special thanks to the Mario Club" in lots of games.
Well, according to friends this Mario Club is a game design/feature playtesting group. Unlike any other developer or publisher I know of, supposedly Nintendo has this always available, always fresh group of testers who's sole point is to do DAILY TESTING STARTING IN THE PROTOTYPE PHASE.
Most games are only put in front of people outside the team 2 or 3 times during an entire project. It's a big deal. There's usuall a crunch for each one "we've gotta get the game ready for testing by March 18th!" type of BS. Because it's such a big deal to get a game tested it's rarely done.
Nintendo instead has their system setup so it is EASY TO ALWAYS TEST YOUR GAME. Every day you can get feedback on your latest feature. You tweeked the jump so your character jumps 1 meter further so you pass that version on to the Mario Club and find out if the game played better, worse, etc. You put in your first puzzle. The Mario Club will tell you if people get the puzzle, how many found it too hard or too easy or too obscure.
This isn't design by committee and it isn't desiging by asking your customers what they want. Instead it's verifing that your design decisions are actually meeting your goals. You want people to be able to both swing the sword, block with the shield and move the camera at the same time. Create 2 or 3 different control schemes and send it over the fence to the Mario Club and get feedback within a day about what's working, what's not and what might work. It's instant feedback, not just once a quarter but daily if you need it.
The Mario Club rotates it's members so no person has more than 2 months experience.
Because it's available it's used. Every other developer I know of quick testing is not easily available so it basically does not happen.
It seems to me, more stuff like the Mario Club at each developer or publisher would go a long way to helping more games be a market success. I know that it's not the only factor but certainly Nintendo arguably has the single highest track record of hit titles and a very high track record of critically acclaimed titles as well. I'm sure the Mario Club is a big reason for that and would go a long way to helping more titles get through Stage Gates in good shape to be market successes.
Wow, that's really cool about the playtesting group.
ReplyDeleteIt makes me wonder if there could be an Internet community group like that available for all of us indie developers.
I just posted my thoughts on the Project Horseshoe report. I don't think stage gates are really the answer, mostly because , as you write above, game development is an iterative process. Stage gates are not an iterative model: They have a series of fixed stages that a product goes through from idea to release with fixed exit criteria for each stage.
ReplyDeleteI think something like the Mario Club in the first comment plus an iterative process at the high level would be much more useful.
Anyway, there's the link if you're interested. :)
Appreciate the comments.
ReplyDeleteRe: Mario Club. This sort of user testing points out one of the big 'success criteria' that the market leaders have known for quite some time. Games that play test well tend to be pretty good games. :-) It's a rather obvious observation, yet many companies still struggle with not having testable games until very late in their development cycle.
The best testing setups tend to involve watching users directly, but there is still a lot to be gained from remotely recorded play sessions. Indie developers with access to existing customers (say from a previous game) have some useful options with beta tests, forums and remote logging.
re: Joe's essay
Appreciate the link. I cross posted this comment on your site as well, but thought it fit here also.
You've made a leap here and fundamentally mischaracterized a stage gate process as just another name for waterfall development methodologies. One has absolutely nothing to do with the other. You can use stage gate with waterfall. You can use it with agile. I personally feel it fits agile development much better and you'll see that more evolved stage gate process have a striking similarity to many agile technique even though you find them in non-software industries.
As you may know from my other essays, I'm a huge fan of various agile process and put a big emphasis on both prototyping and rapid iteration. The stage gate process maps quite nicely onto practices like Scrum or XP and brings a lot of common lessons to the table
Start small: Do the simplest thing possible that gives you critical information. You may learn something.
Don't put all your eggs in one basket: Try out multiple ideas early on. Since you can't know which ideas will pan out, it makes a lot of sense to try several and then invest further in the promising ones.
Test regularly: The whole point of the stage gate process is to have formal testing and validation *throughout* the development process, not at the end.
Understand what success looks like: Even the most rapid iterative process is useless if you are making random changes that do no improve the product. By understanding what success looks like, you can iteratively guide your products towards that that goal.
Invest late, not early: Product development is a learning process. You might as well invest money as you discover success mechanics, settings, etc instead of dumping a wad on hand waving specs early on.
It is also worth noting that just because there are big formal stages that are used for large scale portfolio management, there is nothing in the process that rejects the hundreds or thousands of iterative loops within each development stage. In fact, I'd recommend it since it helps teams meet the gate criteria more successfully.
Stage gate has some serious potential for helping us create innovative product. It brings portfolio management out of the darkness of Skull&Bones green lighting committees and encourages that smart companies set aside money to build new-to-the world, high payoff products.
It is worth taking another look at the concept and imagining how it might fit into the iterative work flow you so rightly target as critical to success in our industry. I'm guessing you'll be pleasantly surprised at how the two compliment each other.
Appreciate the article and the opportunity to start a conversation. :-)
take care
Danc.
All the games released from the game industry could be successful if they'd just produce about 80% fewer of them. :)
ReplyDelete--Ray
I think you are confused with Mario Club. Mario Club isn't a playtest group, but the entire testing dept. NOA has the Mario Club which does the daily testing of project, and then there is the Tree House which goes into more detail and design. NCL might have a different system.
ReplyDeleteThis looks strikingly like the good ol' spiral methodology of prototyping.
ReplyDelete