Monday, July 7, 2008

Soul Bubbles: A classic game ill treated by expert reviewers


I wanted to turn your attention to a delightful little title called Soul Bubbles. I had a chance to play an early version of the game and was impressed by its lead designer, Olivier Lejade, careful attention to the difficulty level of the game.

When it finally launched, I was intrigued to see its aggregate reviewer score hovering at 77%. That is a middling score, but I expected better. Yet when I glanced at the user rating, it was pegged at an impressive 92%. From the user's perspective, we are talking about an instant classic, with a higher aggregate user ratings than either press favorites Halo 2 (91%) or Halo 3 (89%).

Why the disparity? When I looked closer, the professional reviews with lower scores almost all commented on the difficulty level, the one area I knew for a fact that the developers spent months polishing. Alex Sassoon Coby over at Gamespot intones ominously,"The shallow difficulty curve and lack of challenge in the main goals are the only things that let Soul Bubbles down."

Yet, users have the opposite reaction to the same exact features. One fellow gushes "It's very easy to get into thanks to the excellent tutorials, which introduce you effortlessly to the physics-based gameplay as you go along. The game's 40 levels will keep you busy for some time, but chances are you'll play the missions back to back only to be left craving for more!"

There's something odd happening here. Is Soul Bubbles a simplistic, middle of the road experience or is it a classic new game that deserves to be promoted as one of the more playable and innovative games of the year?

The answer tells us a lot about what it takes to make a great game and also happens to highlight one of the grand philosophical flaws in modern game criticism.


Games are about learning skills
First, a bit of background. Games, particularly one built around exploring innovative new game systems like Soul Bubbles, are all about learning new skills. There is a lot written on the topic, there are some articles at the end of this essay for you to peruse. The short of it is that learning new skills yields fun.

You can think of a game like Soul Bubbles as a bit sheet of bubble wrap. Each challenge is a little bubble of fun waiting to be popped. Most games are like this. However, once you've learned a particular challenge, doing it again is usually less exciting. By playing, you've been changed. You've learned the challenge and you'll never be able to revisit that challenge and relive the same emotion that you felt the first time through. You can't re-pop the bubble.

People who play a large number of games tend to rapidly morph into expert gamers. Reviewers, specifically, are almost by definition experts. In order to multiply their meager paychecks, they train themselves to quickly plow through dozens of games. They've crunched through so many levels, powerups, puzzles and collectibles that they are walking encyclopedias of game design techniques.

Since the act of learning is where a large amount of single player fun arises from, many expert gamers find it more and more difficult to derive pleasure from each new title. Games often reuse mechanics and the even an innovative game like Soul Bubbles starts feeling the same. It's like handing the reviewer a sheet of bubble wrap with all the bubbles already popped.

The result is that an initial game activity, such as a tutorial, that would delight a new user instead appear at rote obstacles that need to be skipped past as quickly as possible. Reviewers will use their impressive pre-existing mastery to zip past carefully constructed levels in the hope of find a challenge that will teach them something new. For most, this is subconscious behavior. They just know that they are looking for the thrill that they once experienced as child playing games for the first time. Due to the fact that they have changed, that they are now experts, only the most refined and challenging games still give them a hint of that sweet learning delight. Everything else is labeled 'crap.'

The Expertise Bias
This phenomenon is well understood in the game development community. Game developers also suffer from being experts. Not only do they have encyclopedic knowledge of exist game mechanics, they also have an intimate understanding of how their game is supposed to operate. Surely with such vast expertise, they would be the ideal critics.

Yet, because games are a learning activity, expert game developers often have surprising difficulty understanding how new users will react to their creation. Things they feel are incredibly important end up not mattering. Elements they dismiss as trivial annoyances end up stopping players dead in their tracks. The very fact that designer knows their game intimately makes them a poor critic.

Observation is the solution
The well documented work around for the expertise bias is to observe other people, who aren't experts, play the game. The best designers follow a simple process:
  • Observe target players
  • Take notes on potential issues.
  • Leverage their intimate knowledge of the game to come up with elegant solutions.
Valve does it, Nintendo does it, Microsoft does it. Admittedly, the process is time consuming and not always the easiest path. However, testing with real users is the only proven way to accurately ascertain a game's difficulty and balance.

By observing real people in your target audience learning for the first time, you can realign your heavily biased perception of the game to be more in sync with reality. It becomes readily apparent that 'obvious decisions' do in fact need improved tutorials. Entire systems that you thought were essential are often ignored as players telegraph their delight in simple things like picking up shiny coins.

Game developers have learned the hard way what happens when you ignore the practice of observation. Periodically, schedules become tight and the expensive act of observing real users ends up on the chopping block. Someone with more ego than wisdom stands up and proclaims that they can use their infinite expertise to balance the game using brain power alone. Inevitably their products suffer.

They are fighting the fundamental physics of our medium. Experts, in the absence of observation, make for heavily biased critics.

A tale of Soul Bubbles
Mekensleep, the developers of Soul Bubble, are enlightened developers. They spent months polishing and balancing the difficulty of their game. They held playtests, they observed real users playing for the first time and they fixed the problems that they ran into. They knew that that Soul Bubbles featured some very unique movement and herding mechanics, so they were under no assumptions that they could use their expertise to predict a user's initial reactions.

In the process they learned a lot about how their customers wanted to play Soul Bubbles. Their target player picks up a few games a year and plays in short burst for a long period of time. Many are not looking for intense competition or a massive challenge. Instead, they want a way to relax and explore a delightful world.

As a result Soul Bubbles targets exploratory and easy fun play styles. These feel very different than the traditional hard fun that is the mainstay of many titles played by the core. Yet they are equally enjoyable and often more profitable. Much of the game is about peacefully exploring with the levels designed so that around every corners there is something new to learn or play with.

Through a rigorous and iterative process that involved going to real users, they nailed the difficulty level. That is why the aggregate user ratings are up at 92%.

The flaw of expert reviewers
The reviewers of Soul Bubbles didn't observe real users. Instead, they reacted to the game as expert gamers. The tutorials were a bore, the game could be 'beat' in a short amount of time and the number of times they were forced restart were low. So reviewers told their audience that they should not buy the game on the assumption that the player would likely feel the same thing that the reviewer felt. This represents a basic philosophical approach to game criticism.

I read a short essay by Andrew Doull that sums up this philosophy with the gem of a quote "Fundamentally, the process of being a game critic is the same as being a game designer (is the same as being a game player). That is, it involves the exploration of a possible game space, and trying to validate whether that game space is interesting."

To me, this represents a fallacy of epic proportions that results in badly designed games and inaccurate reviews.

Due to the fact that games are learning systems, good game critique requires two elements.
  1. An expert understanding of the game: Playing the game, knowing mechanics, player psychology, design patterns gives the critic powerful tools for understanding and reacting to what they are witnessing.
  2. Observation of representative users: Expert knowledge biases the priorities of most players, so it is critical to see how real users react to a title in order to get actual target audience data. Having sat through hundreds of hours of observing users, you don't actually know how the virgin value of an inactive system until you see others use it.

Game reviewers only follow one half of this unified process. Since most reviews eschew observation of others (often for timeliness), there is nothing to counter balance their expert bias.

Games are not movies. Please repeat...again and again and again.
There are good historical reasons why experts fail to incorporate player observation into their game reviews. The concept of a review comes from reviewing movies, books and plays. These are what I think of as 'empathetic media'. The process of enjoying these works follows a clear psychological pathway.
  • The viewer observes a universal stimuli, such as a pretty girl in a movie,
  • They empathize with her situation based off their extensive memories of related situations
  • Finally they recall and synthesize an emotional response.
The best works of linear media deal with broadly identifiable stimuli, archetypes of human experiences. Most people have experience with loneliness or the boy winning the lovely girl. Empathetic media gains its mass appeal by dealing in universal truths.

When a reviewer watches a movie, they are asking themselves the question "Do I, as a passable representative of humanity, react strongly to the stimuli in this movie? If so, there is a great chance that others will as well." There is very little expertise bias involved in this exercise. It asks the reviewer to empathize with the stimuli like any other person would.

There is a big question if games work well as empathic media. Their stories are weak, characters flimsy and their exploration of universal truths are usually laughable. Instead, games tend to be strongest when they focus on learning, exploration and first time experiences. Games, more than any other media, are less about reacting and more about changing who we are.

Because the deep, underlying theme of games is change and learning, you need to take into account your level of mastery and the level of mastery of the target audience in your criticism. Otherwise, you end up, like in the case of Soul Bubbles, being the PhD student claiming that Physics 101 is a waste of time because you've 'been and done that' already.

Traditional reviewing techniques taken from the world of empathic media are ill suited for critiquing games. They lack the essential observational techniques that working game designers have found to be so important.

Looking into the future

So, yes, the current game reviewing system is broken. As is often the case with games, we've adopted wholesale the techniques of movies and literature without asking if they even make sense in the context of our brilliantly vibrant new media. I'm certainly not the first to say this.

Yet with single player games, the hack still almost works. Single player games have generally followed a linear path padded with cutscenes, where a reviewer will typically have a similar experience to that of most other players. As such, the expertise bias usually only throws off scores by 10 to 20%. Long term, this practice shrinks the gaming community and it has certainly caused a few developers to miss out on royalty bonuses, but overall it clunks along.

Yet, the world is changing once more. As you introduce multiplayer games into the mix, social dynamics take over and who you play with has as much impact on the experience as which quests you take on. The types of learning and the experiential paths that each player takes are exploding. One player's experience playing with his new girlfriend will be radically different than that of a old school guild settling into the game as a respite from World of Warcraft. An empathetic expert reviewer will not be able to speak for everyone in a convincing fashion.

It is again instructive to observe how developers are using an observation-based understanding of the game to create a proper practice of game criticism. Right now there are hundreds of teams building complex metrics and logging systems that track their player's experiences on a minute level of detail. The best have psychographic and business dashboards that tell them how people are reacting and where problems are emerging. In the future, developers will be observing, tracking and improving the experience of individual guilds and social groups. Practical game criticism, the sort performed by actual game design teams, will be even futher fueled by deep observation and timely intervention.

Unfortunately, these tools are not available to most reviewers. In the coming years, developers will have a vastly superior understanding of how customers are reacting to their game than reviewers will. This is already the case for many titles, such as Soul Bubbles, and the trend will only continue.

What is the future role of professional reviewers?
What role does the expert reviewer have in this situation? As the audience for games broaden, as the benefits of a single expert judging an entire game diminish as their opinions become even more divorced from the actual experiences of real players. The air of objectivity dissipates and the reviewer becomes no more than yet another guy with an intricately detailed, heavily biased opinion.

This represents an intriguing crisis in game criticism. There are many paths for the ex-reviewer to wander down:
  • The news announcements: The factual (though still flavorful) announcements of new games, events and updates. The goals is to let people know that something is happening.
  • The analysts: The elitist community that uses their expertise to deconstruct games according to various theoretical frameworks. The goal is a deeper philosophical understanding of games (and strutting rights within their small incestuous circle.) This is my world. :-)
  • The tourists: Every Man players who approach writing about a game like a travel journalist on a safari. The goal is to evoke the emotions that the individual reporter experienced, not to predict what everyone's experience might be. They succeed if they provide simple entertainment.
  • The opinion mavens: The high energy personality who crystalize the trends and fashions of their target culture. The goal is to pick hits in a heavily biased but entertaining fashion and enhance the maven's personal brand.
  • Ranking sites: Sites like gamerankings are still of questionable value, but over time sites that use a broader range of data will emerge. The goal is to provide a public thermometer that, with reasonable accuracy, states if the game is worth trying.
I already see this evolution occurring. I've given up on reading reviews and instead find myself frequenting gaming blogs, the news portals of our age. Many traditional reviewers are popping up in more experientially-focused sites like The Escapist or Rock, Paper, Shotgun. Even next generation ranking sites are appearing in the form of portals like Kongregate. And what is Zero Punctuation if not our very own flavorful equivalent of Oprah the Opinion Maven?

Closing thoughts
Go out and try Soul Bubbles. It is a great example of what happens when a developer balances their title for their target audience and not the expert reviewer. If you are an expert reviewer, play it with an eye towards seeing how a first time user might experience it. It is an interesting and remarkably difficult exercise. Then give the game to someone who isn't an expert gamer and watch them play it. I suspect that they'll highlight elements that you didn't even realize were important.

If you are serious about providing objective insight into a game, either a title you are building or one your are reviewing, your expertise is not enough. In fact, your vast mastery of game related skills is mostly likely causing a giant bias in your judgments. You need to fight this bias by observing other players over and over again. They will do things with the game that are a source of wondrous insight. Your expertise becomes a tool for making great changes based off these insights, not one for predicting a priori exactly how all users will react to the game.

As for the current review industry, it is built on the unstable foundation of expert opinion in the absence of actual player observation. As games evolve and become ever more about first time learning experiences, the traditional game review will become increasingly irrelevant. It is arguable that they've already stopped informing most buying decisions and now serve as little more than entertainment for the hardcore niche. As the value proposition of reviews falter, the vast, churning, capitalist forces of creative destruction will replace them with a much richer set of game criticism that offers real value to its readers.

It's a beautiful day outside, so I'm off to pop a bit more bubble wrap,
Danc.

References