Wednesday, April 5, 2006

Managing game design risk: Part II - Data Driven Development

Read part Managing game design risk: Part I here. It provides an overview of different classes of production processes and their relationship to various forms of risk. The next two essays will talk about common techniques for reducing risk.

The first technique, data driven development, involves lowering execution risk by investing in lower risk product features and processes. This low execution risk strategy is the predominant technique used by game developers to reduce risk and is currently considered state of the art by most development houses. Compared to naïve product development there are considerable advantages. However, there are also some unexpected side effects worth taking into account.

If we think about data driven development in terms of our process complexity spectrum, we want to drive processes out of the complex and chaotic zones down into the simple zone. Simpler development processes are easy to manage, easier to scale to larger teams, and far less likely to fail catastrophically.

Conceptually, data driven development relies on two techniques
  • Investing in existing low risk activities
  • Converting moderate risk activities into low risk activities.

Investing in existing low risk activities
The first step is to identify low execution risk items and invest in them. If we were to rank some common game elements according to risk, they would look something like this:
  1. New core game mechanics: High risk
  2. New to the world technologies such as better AI: Moderate-high risk.
  3. New Setting / IP / Brands: Moderate risk
  4. Level design: Moderate risk
  5. More of the same technologies such as improved rendering: Moderate-low risk
  6. Graphics / Cut scenes: Low risk
If you look at the resources spent on modern games, you’ll find that game funding priorities are roughly the inverse of their execution risk. Core game mechanics have the fewest number of people working on it during the length of the project, graphics of various types have the most, level design tends to sit someplace in between and so forth. It is a rough rule of thumb, but it works reasonably well.

Converting moderate risk activities into low risk activities
The next step is to simplify complex or complicated tasks in the hope of turning them into simple production processes. Game development in the past was saddled with some truly unpleasant risks.
  • Crazy custom code written under tight deadlines for hardware platforms that are still in flux.
  • Artist fumbling their way around new technology often introduced unacceptable schedule risk.
  • Assembling the game elements at the end of the production cycle often meant that level design and minute-by-minute game play wasn’t testable until very late in the development cycle.
Wouldn’t it be nice to simplify these areas dramatically?

Historically, the simplification of complex processes has occurred during the early stages of any new mass media life cycle. Let’s consider animation as an example of the typical evolution. Once upon a time, each artist drew in a unique style. If you wanted a Norman Rockwell illustration, you pretty much had to go to Norman Rockwell. In the case of early animation, you typically required one animator to create most of the frames of the animation. Animations were short, labor intensive affairs

Walt Disney (and others) came along with production techniques for the highly repetitive task of animation. They reduced the risk of producing a series images with the same visual characteristics. Suddenly, instead of having one artist working in a particular style, you could have hundreds, all laboriously following style sheet, all drawing Mickey Mouse. By defining specialized roles and standardizing the production process, early animation houses reduced risk and enabled the task to scale in a linear fashion.

The same process of risk reduction is occurring in game development.
  • Artist fumbling? This issue is solved by using standardized tools like Maya and Max and Photoshop. When they are tightly integrated in a unified tool chain, they increase the speed of creating complex graphics while reducing schedule risk.
  • Crazy custom code? Replaced by 3rd party rendering technologies like Unreal or Renderware which reduce the risk of rolling your own technologies.
  • Assembling game elements at the end of the production cycle? I sat through a lovely talk on God of War in which 7 programmers created a highly tuned tool chain that allowed designers to act as highly reproducible, lower skill production cogs. The entire issue of game mechanics was reduced to a ‘push button and play animation’ sort of affair. You could also play test levels early and often.
Data driven development
The result of these process simplification efforts is a production model known broadly as ‘data driven’ development. By focusing on low risk development elements, tasks like creating graphics, models and other static content becomes the primary cost center for the title. The following practices are common.
  • The company invests in a centralized engine maintained by a relatively small number of programmers. Tool development takes on a hitherto unprecedented role.
  • Game mechanics are usually borrowed from an existing genre.
  • The team streamlines the flow of data into the game engine using a well defined content pipeline. This allows people without programming skills to build much of the using artist friendly tools. The data is rapidly translated into the game engine where it can be viewed running in the actual game.
  • The team then ramps up the number of artist / designers to fill the game with content.
  • The focus of the title is on providing the most exciting, highly polished static content to wrap the core game mechanics.
Data driven development optimizes many but not all aspects of game development. Some gameplay elements such as new game mechanics have not yet been reduced to simple production processes. These remain stubbornly in the land of complex or even chaotic process. Companies try to reduce these risks as much as possible by selecting well explored game mechanics.

God of War is a great example of this development process. They gained impressive production efficiencies by building a strong content pipeline around the proven gameplay. The core game play differed only mildly from beat ‘em ups from ages past. However, they make the experience feel fresh by skinning the mechanics with flashy content. Every time you pressed a button to open a door, you saw a new animation. Every time you killed a monster, you saw another new animation. It looks great and it doesn’t play all that badly either.

Benefits of data driven development
Data driven development is clearly superior to naïve game development processes of the past. It results in dramatic reductions in all forms of execution risk:
  • Technical risk: Because most of the game rewards come from content instead of complex systems, the need for taking on technically risk work decreases. There is no need to put in a complex physics engine when an animator can produce the same results with less risk to the overall project. The overall code size is much smaller and you need fewer programmers. The God of War executable was only 1.5 MB in size and this was produced by a mere 7 programmers.
  • Quality risk: By create a tight content pipeline that is regularly exercised early in the development process, errors are found early and fixed early. Because you are dealing with data instead of code, code errors decrease dramatically.
  • Schedule risk: Because game content is produced in a manner that scales linearly at low risk, you can easily throw more people at the problem or cut back scope if there is a threat of schedule risk.
  • People risk: By having content readily playable and instituting style guides, Disney-style directorial control can be imposed on areas like level design. This lets companies use relatively low skill designers as ‘production cogs’. They are replaceable, easily trained, and scalable to large work teams.
This is an impressive feat. If you’ve ever lived through a poorly run game development process, the God of War presentation at GDC sounded like heaven. We all want to create a high quality game that lets us spend the last day of development on the beach. If you don’t understand the techniques and processes involved, your skills are behind the curve and your games will suffer.

Opportunities for deploying data driven development
There are two classes of game that benefit strongly from implementing a data driven development process
  • Genre king brands competing in mature genres
  • ‘Story games’ that rely primarily on quality static content
The obvious win are products that are at the end of the genre life cycle. Such teams are serving a well-defined market populated primarily by late and laggard adopters. Since the hardcore ‘genre addict’ customers they server are highly risk adverse, it is okay for the product to sport low risk features. Does anyone who buys Halo 2 really want a radical shift in game mechanics like you find in Animal Crossing? Not likely.

There are also very specific genres of game that benefit greatly from data driven development. Japanese RPG titles, graphic adventure games, and other games whose primary value comes from giving the player a strong linear narrative can use a robust content pipeline to cut their execution risk.

These ‘story games’ compete head-to-head with the tales in other linear narrative formats such as movies or books. The gap between the experience of a movie like Advent Children and a ‘game’ like FFXII diminishes with each new sequel. Such titles have forsaken interactivity as the primary method of providing players with value. The more high quality static content, the better.

Problems with data driven development
Data driven development is the future for a large portion of the industry. It is simply so much more effective at managing risk than older methods. However, there are issues that the smart development team should take into account.

  • Market issues: Low product differentiation
  • Cost issues: High production costs
  • Game play issues: High player burnout
Market issues
How do new games compete? This is a big question. I humbly submit that the unique value propositions in games typically are centered on innovation in interactivity, not the static fluff wrapping the interactivity. For example, in Nintendogs, the value comes from interacting with a dog, not from the resolution of the wallpaper in the room. Even in God of War, the core value comes from the combat, not the pretty animations.

The low risk activities that data driven development relies upon also happen to be low value activities. You can easily add considerable graphics, level design and animation to a product and only add incremental value to the customer. When companies compete using fluff elements such as graphics and level design, you end up with a slew of undifferentiated products.

In the early days of middleware, Epic started licensing their Unreal engine. Licensees replaced the graphic and the levels, but failed to innovate in terms of new game mechanics. The licensees believed they were offering new and exciting content. Players on the other hand, saw this flood of titles as inferior FPS with tire game mechanics. They ran smack into markets dominated by genre kings Quake and Unreal. These titles failed. Ah, the irony. Differentiated content is not enough to compete.

Cost issues
The other issue is that the strategy of risk reduction is not a cost reduction technique. By ensuring that production activities scale in a linear fashion at low risk, all that happens is that development teams are encouraged to do more of the same in order to compete. Most companies blindly invest in the most obvious and lowest risk competitive option.

This is creates a vicious cycle:
  • The competition releases a title with ‘The best graphics ever’
  • Your team chooses to compete by putting money into the lowest risk production element. By simply spending more money, they can keep their risk of execution failure low and still mount a competitive product.
  • When another competitor comes out with ‘Even better graphics’, then your producer doesn’t mind at all hiring a baker’s dozen of new art cogs to up the ante. It won’t delay the product and could give the game an edge. This is an easy management decision.
  • The product releases and sells a very reasonable amount. Unfortunately, due to the extra production costs, it posts a loss.
  • Even though your title is a financial failure, it provokes other competing companies to spend more on the quality of their titles.
The cycle slowly drives profitability out of the marketplace. The results are low profitability games that launch in intensely competitive markets. A small percentage remain profitable by capturing the genre king crown, but most lose money.

Gameplay issues
If we look at data driven games from our action / reward game play model, we can also predict some key characteristics of data driven games.

Data driven games encourage player addiction by relying on two classic game design elements:
  • Visceral rewards: Visceral rewards are ones that attempt to trick the brain into reacting as if it were put into a real world, high intensity situation. You see a strong focus on emotional content and realism. Final Fantasy does this quite well by providing highly emotional drama in almost every cut scene.
  • Unique rewards: Unique rewards are rewards that are different each time the player successfully completes an action. The classic quote from David Jaffe is his desire for "Lots and lots of special cases. I don't want any two doors to open the same way” You are still pressing the same button, but the results appear unique.

Visceral rewards create very intense rewards. This can be very delightful to jaded players. Unique rewards, in the form of new animations, ensure that the rewards give the player a solid buzz each time they are encountered.

Visceral rewards have a high burnout rate, especially if they are reused. When you kill the Minotaur for the first time, it is quite exciting. After dealing with several of the beasties, the thrill of the kill wanes.

Unique rewards might seem like a good game idea, but they too have a dark side. With repetition, players will eventually grok the deeper patterns behind the ‘unique’ rewards. Ultimately, they see the impressively tumbling statue in God of War as yet another animation that opens a door. When the gameplay patterns become visible, players can prematurely burn on the title far sooner than the designer predicted.

This can lead to situations like Quake IV. Here the franchise was damaged because players focused on how the mechanics are ‘more of the same.’ They dismissed the new expensive graphics engine and the plethora of content that was intended to differentiate the product in the market place.

From a game design perspective, data driven games are a quick jolt of brain candy to be consumed and forgotten. They are low touch, consumable entertainment product that fails to build a deep and loyal customer relationship. When your ability to connect with your customer at a meaningful level is threatened, you have surprisingly high franchise churn. Crash Bandicoot? Sonic the Hedghog? Lara Croft? These titles’ relationship with the customer was much less secure than they imagined.

Contrast this to Mario, a franchise that relies much more heavily on constantly updated game mechanics. By offering players a series of unique high value game play experiences, Nintendo creates a strong customer bond with the character that primes the channel for future titles. Nintendo generally avoids player burnout and accidental damage to their highly valuable franchise by focusing on gameplay, not merely lobbing undifferentiated content at their customers.

Closing thoughts on Data Driven game development
Data driven development represents one vision of the current state of the art in game development. It eases a large number of issues that have plagued game development for many years.

If you care at all about why you do the things you do, you need to understand the practice and repercussions of data driven development. It is the current future of the console industry and its driving philosophies will shape the work environment of most professional game developers. Many of us have been doing flavors of data driven development for years, but expect extremely polished pipelines like the ones found in God of War to become the default for the upcoming generation.

It is a potent technique if used correctly. Pure data driven development is a solid strategy for titles in mature genres that have a brand capable of capturing the first or second place in the market. Such genre king titles are the bread and butter of any publisher and reducing execution risk is of paramount importance. However, companies should be careful of producing more than two or three titles that use the same game mechanics or they risk devaluing the franchise.

For all others, data driven game development tends to be a very expensive way fail in the marketplace. The execution risk may be low, but design risk and financial risk are actually higher than if you did naïve game development. If you can’t capture the genre crown, exciting static content is rarely enough to carve out your own niche within the marketplace. You’ve invested a lot up front and you won’t get the sales on the backend.

Even a title like God of War is a huge risk for Sony. They’ve supported it with a large marketing budget and a decent sized development budget. It certainly has gotten them rave reviews, but sales have been disappointing for AAA title. First month US sales are reported at only 200k for the month of April and by June of 2005, they had reached 500k world wide. The mythical typical AAA title needs to sell anywhere from 900k to 1.2 million depending on how the books are kept.

With continued promotion, I’m hoping they’ll at least break even or turn a small profit. Admittedly, Sony is attempting to create a new next generation IP. They are willing to take a moderate loss on the first title. It is a simple strategy. Create an expensive splash, get the new brand in the public consciousness and then scoop up the preorders for GOW2.

I have to wonder though. If we’ve already gotten to the point where one of the world’s top five publisher has to release a very expensive title as a loss leader in order turn a future profit, where does this leave the smaller companies? Screwed and tattooed. At the very least, it suggests that launching original content-based IP that relies on older game mechanics is an expensive and risky strategy.

Data driven game development is one risk reduction strategy. If everyone invests in this ‘safe path’, then there will be some remarkable market failures. This is just another tool and should be used with care and wisdom.

What’s next?
In the last part of this series, I’ll talk about another take on the risk reduction game, a little technique called prototype driven development. This focuses on reducing design risk, instead of execution risk. It should provide a fun counterpoint to many of the issues found in data driven development.

Take care

God of war sales figures “disappointing”

The spin on God of War sales figures: “Grand Turismo + God of War sell over 2.5 million.” Right…now how many did just God of War sell?

Ub Iwerks leaves Disney in the dust

GDC: God of War: How the Left and Right Brain Learned to Love One Another


  1. It is sort of like the big movie studios making films on huge budgets, using standarized stories (hero fights crime boss, and ends up with woman) with few variations. Splashing on some FX and a fairly well known actor...shazaamm. People watch it. Then they release number 2, on a drastically lower budget...and make tons of money on it.

    Whilst the smaller studios create interesting films, that usually get little interest from the public, but sometimes they hit the nail and BAM; new "genre".

    I think its always gonna be like this as long as we are dependent on huge distributors to sell our products. Like record companies sell cds. Once we rid ourselves of that outdated form of distribution, I think innovation will rise to a new level. Purely because developers/musicians/movie makers aren't that dependent on a contract to secure distribution any more.

    2 cents from me.

  2. I think its always gonna be like this as long as we are dependent on huge distributors to sell our products.

    The absence of large distributors (and I agree that it would probably result in more innovative titles) will not free you from the risks of the development process. Even if the middlemen are somehow removed and the path from developer to player is shortened, development costs will still be up there in stratosphere. You'll still be playing the same game of chicken with your dwindling budget, in many cases still beholden to some sort of investor, still trying to manage risk. Throwing off the yoke of the publishers (as they exist now) would probably be a positive thing, but I don't think it's quite sufficient to break up the dynamic you described.

  3. Oh Danc! You godless capitalist... With all this corporate speak... you've become the very thing you despise. Twist the thumbscrews! Twist 'em!

    What happened to the little boy that created space pidgeons for a game dream called Star Traders?

    It is sad what a broken dream can do...


    PS> :) We need to get together more often... I'm starting to get bored and am ranting across your perfectly clean and intelligible essays... :(

  4. Great post, as always. It made me question my own notions on data driven development. At first, I'd considered games as a split between the engine and the content.. that evolved into engine, game-specific code, scripting, and content, with scripts bridging the gap, and giving the designers something safe to play with.

    I take a little issue with data driven development causing low innovation; I think this comes from saying it relies on two techniques.. doing low risk stuff as much as possible and making moderate risk things low risk. My experience is that a lot of it is actually reducing the risk of everything.. low->trivial, moderate->low, high->moderate. High->moderate could be things like tools which allow a designer to design an FSM for an AI, allowing them to visualize it and convert it to code or script
    (or whatever) that can be ran in the game.

    And that, the high risk things being made moderate risk, is the key to how data driven development can boost innovation rather then stifle it.

    (of course, the current drive is maximizing the low risk, not making high-risk more palatable)

  5. Nice essay and I agree with you that DDD leads to genre-lock and spiraling development costs all in a single movement. I also with Tomas' comment about publishing, though concede that it applies much better to book publishing and music than the far more expensive fields of game design and movie making. Afterall, someone's always got to put up the money!

    Looking forward to part 3. Expecting to hear good thoughts about rewarding innovative design ... we can always hope!

  6. Loved the article! Another great piece of writing Danc. But where's the promised 3rd part to the article series? Or did I just miss it?

  7. The third part is sitting on my hard drive in a state of disrepair.

    The basic thrust of the third part is that instead of treating games like a production excercise, you can treat them as a development exercise.

    You focus on create a agile team that solves problems quickly and learns rapidly. There's a strong focus on:
    - new gameplay
    - prototyping and play testing
    - reuse of static content (or replacement of static content with procedural content)
    - play metrics
    - Iterative development

    The idea is to reduce design risk by introducing players to highly addicting experiences that own a unique competitive position in the market.

    Production risk is also reduced, but not by minimizing variation (the technique used in production focused models) but by creating highly adaptive and flexible feedback systems.

    Instead of borrowing from outdated waterfall-style production models (that ironically were left behind in manufacturing about 30 years ago), it borrows from the philosophies of lean manufacturing and agile software development.

    take care

  8. *cough*, Renderware is dead. Perhaps you meant Gamebryo? :-)