Monday, July 7, 2008

Soul Bubbles: A classic game ill treated by expert reviewers

I wanted to turn your attention to a delightful little title called Soul Bubbles. I had a chance to play an early version of the game and was impressed by its lead designer, Olivier Lejade, careful attention to the difficulty level of the game.

When it finally launched, I was intrigued to see its aggregate reviewer score hovering at 77%. That is a middling score, but I expected better. Yet when I glanced at the user rating, it was pegged at an impressive 92%. From the user's perspective, we are talking about an instant classic, with a higher aggregate user ratings than either press favorites Halo 2 (91%) or Halo 3 (89%).

Why the disparity? When I looked closer, the professional reviews with lower scores almost all commented on the difficulty level, the one area I knew for a fact that the developers spent months polishing. Alex Sassoon Coby over at Gamespot intones ominously,"The shallow difficulty curve and lack of challenge in the main goals are the only things that let Soul Bubbles down."

Yet, users have the opposite reaction to the same exact features. One fellow gushes "It's very easy to get into thanks to the excellent tutorials, which introduce you effortlessly to the physics-based gameplay as you go along. The game's 40 levels will keep you busy for some time, but chances are you'll play the missions back to back only to be left craving for more!"

There's something odd happening here. Is Soul Bubbles a simplistic, middle of the road experience or is it a classic new game that deserves to be promoted as one of the more playable and innovative games of the year?

The answer tells us a lot about what it takes to make a great game and also happens to highlight one of the grand philosophical flaws in modern game criticism.

Games are about learning skills
First, a bit of background. Games, particularly one built around exploring innovative new game systems like Soul Bubbles, are all about learning new skills. There is a lot written on the topic, there are some articles at the end of this essay for you to peruse. The short of it is that learning new skills yields fun.

You can think of a game like Soul Bubbles as a bit sheet of bubble wrap. Each challenge is a little bubble of fun waiting to be popped. Most games are like this. However, once you've learned a particular challenge, doing it again is usually less exciting. By playing, you've been changed. You've learned the challenge and you'll never be able to revisit that challenge and relive the same emotion that you felt the first time through. You can't re-pop the bubble.

People who play a large number of games tend to rapidly morph into expert gamers. Reviewers, specifically, are almost by definition experts. In order to multiply their meager paychecks, they train themselves to quickly plow through dozens of games. They've crunched through so many levels, powerups, puzzles and collectibles that they are walking encyclopedias of game design techniques.

Since the act of learning is where a large amount of single player fun arises from, many expert gamers find it more and more difficult to derive pleasure from each new title. Games often reuse mechanics and the even an innovative game like Soul Bubbles starts feeling the same. It's like handing the reviewer a sheet of bubble wrap with all the bubbles already popped.

The result is that an initial game activity, such as a tutorial, that would delight a new user instead appear at rote obstacles that need to be skipped past as quickly as possible. Reviewers will use their impressive pre-existing mastery to zip past carefully constructed levels in the hope of find a challenge that will teach them something new. For most, this is subconscious behavior. They just know that they are looking for the thrill that they once experienced as child playing games for the first time. Due to the fact that they have changed, that they are now experts, only the most refined and challenging games still give them a hint of that sweet learning delight. Everything else is labeled 'crap.'

The Expertise Bias
This phenomenon is well understood in the game development community. Game developers also suffer from being experts. Not only do they have encyclopedic knowledge of exist game mechanics, they also have an intimate understanding of how their game is supposed to operate. Surely with such vast expertise, they would be the ideal critics.

Yet, because games are a learning activity, expert game developers often have surprising difficulty understanding how new users will react to their creation. Things they feel are incredibly important end up not mattering. Elements they dismiss as trivial annoyances end up stopping players dead in their tracks. The very fact that designer knows their game intimately makes them a poor critic.

Observation is the solution
The well documented work around for the expertise bias is to observe other people, who aren't experts, play the game. The best designers follow a simple process:
  • Observe target players
  • Take notes on potential issues.
  • Leverage their intimate knowledge of the game to come up with elegant solutions.
Valve does it, Nintendo does it, Microsoft does it. Admittedly, the process is time consuming and not always the easiest path. However, testing with real users is the only proven way to accurately ascertain a game's difficulty and balance.

By observing real people in your target audience learning for the first time, you can realign your heavily biased perception of the game to be more in sync with reality. It becomes readily apparent that 'obvious decisions' do in fact need improved tutorials. Entire systems that you thought were essential are often ignored as players telegraph their delight in simple things like picking up shiny coins.

Game developers have learned the hard way what happens when you ignore the practice of observation. Periodically, schedules become tight and the expensive act of observing real users ends up on the chopping block. Someone with more ego than wisdom stands up and proclaims that they can use their infinite expertise to balance the game using brain power alone. Inevitably their products suffer.

They are fighting the fundamental physics of our medium. Experts, in the absence of observation, make for heavily biased critics.

A tale of Soul Bubbles
Mekensleep, the developers of Soul Bubble, are enlightened developers. They spent months polishing and balancing the difficulty of their game. They held playtests, they observed real users playing for the first time and they fixed the problems that they ran into. They knew that that Soul Bubbles featured some very unique movement and herding mechanics, so they were under no assumptions that they could use their expertise to predict a user's initial reactions.

In the process they learned a lot about how their customers wanted to play Soul Bubbles. Their target player picks up a few games a year and plays in short burst for a long period of time. Many are not looking for intense competition or a massive challenge. Instead, they want a way to relax and explore a delightful world.

As a result Soul Bubbles targets exploratory and easy fun play styles. These feel very different than the traditional hard fun that is the mainstay of many titles played by the core. Yet they are equally enjoyable and often more profitable. Much of the game is about peacefully exploring with the levels designed so that around every corners there is something new to learn or play with.

Through a rigorous and iterative process that involved going to real users, they nailed the difficulty level. That is why the aggregate user ratings are up at 92%.

The flaw of expert reviewers
The reviewers of Soul Bubbles didn't observe real users. Instead, they reacted to the game as expert gamers. The tutorials were a bore, the game could be 'beat' in a short amount of time and the number of times they were forced restart were low. So reviewers told their audience that they should not buy the game on the assumption that the player would likely feel the same thing that the reviewer felt. This represents a basic philosophical approach to game criticism.

I read a short essay by Andrew Doull that sums up this philosophy with the gem of a quote "Fundamentally, the process of being a game critic is the same as being a game designer (is the same as being a game player). That is, it involves the exploration of a possible game space, and trying to validate whether that game space is interesting."

To me, this represents a fallacy of epic proportions that results in badly designed games and inaccurate reviews.

Due to the fact that games are learning systems, good game critique requires two elements.
  1. An expert understanding of the game: Playing the game, knowing mechanics, player psychology, design patterns gives the critic powerful tools for understanding and reacting to what they are witnessing.
  2. Observation of representative users: Expert knowledge biases the priorities of most players, so it is critical to see how real users react to a title in order to get actual target audience data. Having sat through hundreds of hours of observing users, you don't actually know how the virgin value of an inactive system until you see others use it.

Game reviewers only follow one half of this unified process. Since most reviews eschew observation of others (often for timeliness), there is nothing to counter balance their expert bias.

Games are not movies. Please repeat...again and again and again.
There are good historical reasons why experts fail to incorporate player observation into their game reviews. The concept of a review comes from reviewing movies, books and plays. These are what I think of as 'empathetic media'. The process of enjoying these works follows a clear psychological pathway.
  • The viewer observes a universal stimuli, such as a pretty girl in a movie,
  • They empathize with her situation based off their extensive memories of related situations
  • Finally they recall and synthesize an emotional response.
The best works of linear media deal with broadly identifiable stimuli, archetypes of human experiences. Most people have experience with loneliness or the boy winning the lovely girl. Empathetic media gains its mass appeal by dealing in universal truths.

When a reviewer watches a movie, they are asking themselves the question "Do I, as a passable representative of humanity, react strongly to the stimuli in this movie? If so, there is a great chance that others will as well." There is very little expertise bias involved in this exercise. It asks the reviewer to empathize with the stimuli like any other person would.

There is a big question if games work well as empathic media. Their stories are weak, characters flimsy and their exploration of universal truths are usually laughable. Instead, games tend to be strongest when they focus on learning, exploration and first time experiences. Games, more than any other media, are less about reacting and more about changing who we are.

Because the deep, underlying theme of games is change and learning, you need to take into account your level of mastery and the level of mastery of the target audience in your criticism. Otherwise, you end up, like in the case of Soul Bubbles, being the PhD student claiming that Physics 101 is a waste of time because you've 'been and done that' already.

Traditional reviewing techniques taken from the world of empathic media are ill suited for critiquing games. They lack the essential observational techniques that working game designers have found to be so important.

Looking into the future

So, yes, the current game reviewing system is broken. As is often the case with games, we've adopted wholesale the techniques of movies and literature without asking if they even make sense in the context of our brilliantly vibrant new media. I'm certainly not the first to say this.

Yet with single player games, the hack still almost works. Single player games have generally followed a linear path padded with cutscenes, where a reviewer will typically have a similar experience to that of most other players. As such, the expertise bias usually only throws off scores by 10 to 20%. Long term, this practice shrinks the gaming community and it has certainly caused a few developers to miss out on royalty bonuses, but overall it clunks along.

Yet, the world is changing once more. As you introduce multiplayer games into the mix, social dynamics take over and who you play with has as much impact on the experience as which quests you take on. The types of learning and the experiential paths that each player takes are exploding. One player's experience playing with his new girlfriend will be radically different than that of a old school guild settling into the game as a respite from World of Warcraft. An empathetic expert reviewer will not be able to speak for everyone in a convincing fashion.

It is again instructive to observe how developers are using an observation-based understanding of the game to create a proper practice of game criticism. Right now there are hundreds of teams building complex metrics and logging systems that track their player's experiences on a minute level of detail. The best have psychographic and business dashboards that tell them how people are reacting and where problems are emerging. In the future, developers will be observing, tracking and improving the experience of individual guilds and social groups. Practical game criticism, the sort performed by actual game design teams, will be even futher fueled by deep observation and timely intervention.

Unfortunately, these tools are not available to most reviewers. In the coming years, developers will have a vastly superior understanding of how customers are reacting to their game than reviewers will. This is already the case for many titles, such as Soul Bubbles, and the trend will only continue.

What is the future role of professional reviewers?
What role does the expert reviewer have in this situation? As the audience for games broaden, as the benefits of a single expert judging an entire game diminish as their opinions become even more divorced from the actual experiences of real players. The air of objectivity dissipates and the reviewer becomes no more than yet another guy with an intricately detailed, heavily biased opinion.

This represents an intriguing crisis in game criticism. There are many paths for the ex-reviewer to wander down:
  • The news announcements: The factual (though still flavorful) announcements of new games, events and updates. The goals is to let people know that something is happening.
  • The analysts: The elitist community that uses their expertise to deconstruct games according to various theoretical frameworks. The goal is a deeper philosophical understanding of games (and strutting rights within their small incestuous circle.) This is my world. :-)
  • The tourists: Every Man players who approach writing about a game like a travel journalist on a safari. The goal is to evoke the emotions that the individual reporter experienced, not to predict what everyone's experience might be. They succeed if they provide simple entertainment.
  • The opinion mavens: The high energy personality who crystalize the trends and fashions of their target culture. The goal is to pick hits in a heavily biased but entertaining fashion and enhance the maven's personal brand.
  • Ranking sites: Sites like gamerankings are still of questionable value, but over time sites that use a broader range of data will emerge. The goal is to provide a public thermometer that, with reasonable accuracy, states if the game is worth trying.
I already see this evolution occurring. I've given up on reading reviews and instead find myself frequenting gaming blogs, the news portals of our age. Many traditional reviewers are popping up in more experientially-focused sites like The Escapist or Rock, Paper, Shotgun. Even next generation ranking sites are appearing in the form of portals like Kongregate. And what is Zero Punctuation if not our very own flavorful equivalent of Oprah the Opinion Maven?

Closing thoughts
Go out and try Soul Bubbles. It is a great example of what happens when a developer balances their title for their target audience and not the expert reviewer. If you are an expert reviewer, play it with an eye towards seeing how a first time user might experience it. It is an interesting and remarkably difficult exercise. Then give the game to someone who isn't an expert gamer and watch them play it. I suspect that they'll highlight elements that you didn't even realize were important.

If you are serious about providing objective insight into a game, either a title you are building or one your are reviewing, your expertise is not enough. In fact, your vast mastery of game related skills is mostly likely causing a giant bias in your judgments. You need to fight this bias by observing other players over and over again. They will do things with the game that are a source of wondrous insight. Your expertise becomes a tool for making great changes based off these insights, not one for predicting a priori exactly how all users will react to the game.

As for the current review industry, it is built on the unstable foundation of expert opinion in the absence of actual player observation. As games evolve and become ever more about first time learning experiences, the traditional game review will become increasingly irrelevant. It is arguable that they've already stopped informing most buying decisions and now serve as little more than entertainment for the hardcore niche. As the value proposition of reviews falter, the vast, churning, capitalist forces of creative destruction will replace them with a much richer set of game criticism that offers real value to its readers.

It's a beautiful day outside, so I'm off to pop a bit more bubble wrap,



  1. I liked your analysis a lot. However, you also forgot about a very important detail.

    How good or bad you might find something is hard to predict by a rating made my someone else in the first place. Thus, ratings are never even meant to be objective in the first place.

    Who are those people reading gaming magazines and sites? Of those people, I expect very few to be new to the world of gaming. Over 90% of them know how to play games, they are expert gamers. So if there is a rating of 77% for Soul Bubbles, that means that expert gamers will not find it as fun as Halo, for example.

    I bet that in magazines and websites that focus more on the less expert gamers (I hate the word "casual gamers") out there, Soul Bubbles is not criticised for its difficulty level and might achieve a better rating.

    I'm wondering if with that background anybody actually wants more objective reviews that focus more on the audience that a game is made for, and try to somewhat perceive the game experience from that point of view.

    On a side note, the German actually advertises with completely subjective reviews and is quite popular and successful with that. I see that as proof that people are not really interested in "analyst" reviews.

  2. You should look at some reviews of flOw back when it came out.

  3. Re: "rating are never even meant to be objective in the first place"

    Rating have been traditionally used as a method of informing users whether or not the title is worth purchasing. In this context, a review needs to predict if the reader will enjoy the title.

    As it becomes more apparent that expert reviewers are poor at predicting the enjoyment level of others, some such as, have started to add the disclaimer that they are 'mere opinion'. Honesty is a great step in right direction. :-)

    Yet, we do not need to go down the path that 'everything is opinion.' In that way lies idiocy.

    There is an objective reality here since you can measure how many people play a game, how they rate their experience and when they stop playing. There will be average experiences and outliers. A game with higher average ratings in a specific target group of players is objectively 'better' than one with lower ratings. You can state that more people, on average, will enjoy it.

    In terms of sales and word of mouth recommendations, this is meaningful data. Like most statistical results, it doesn't predict the experience of everyone, but it is far more valid than the opinion of a single dude sitting amidst his pile of 300 games.

    take care,

  4. I agree with your general sentiment that Soul Bubbles is a great game and probably deserves better reviews.

    But, there are many things to nitpick here. The idea that expert bias doesn't happen as much in movies is clearly wrong. Most people comment on the fact that movie reviewers rarely like popular movies. Movies like ID4, Armaggedon, The live version of 101 Dalmations, Patch Adams all had horrid reviews by the expert reviewers yet were hugely popular movies that entertained people.

    An expert reviewer isn't just going in for pure entertainment, they want the story to make sense, they want the acting to be believable, and like games, they want something new. They've seen 5000+ movies and if this movie is telling the same basic story as something they saw before and is not as good they are going to rate it worse than if this was the first time they experienced that particilar plot point.

    Review IS opinion. The idea is supposed to be that you find a reviewer who shares your tastes. If they liked the game than chances are so will you. I don't really think it's fair to ask some reviewer to try to decide if some target audience will like it. They can only really know if they liked it.

    You appear to want enjoyment statistics instead of reviews. A agree with your reasons for wanting them but they would no longer be reviews.

    I suggest you start a magazine / website that just tests games and posts those stats 75% of female casual gamers age 16-21 liked game XYZ, of 21-25yr old expert gamers only 40% liked it.

  5. It is always good to read a note about game reviews. I consider myself a strange mixture between casual user and very hardcore one: I have habits like a casual user, buying a couple of games every three or four months, preferring small games like Picross DS than Contra 4, etc. But my favorite genre is the dungeon crawling, and take real pleasure in extremely difficult games like Etrian Odyssey and Shiren.

    At work we have a small testing group who checks our software builds for bugs. However, they don't test usability, that is left to our beta testers (clients who signed a NDA), because testers already expect certain things to work in a certain manner that is not always what the final client wants. For example, the tester may think a button would do something because it makes sense to do it at the current dialog screen. However, the client may think otherwise not because of the screen, but because of the situation within the screen.

    The main flaw of game reviewers nowadays is that they review every game as a gamer. A casual friendly game like Carnival Games is awful from a gamer perspective, but it is a million seller because the target public accepts it. The site where the review is published is targeted at gamers, true, and therefore their opinion may match that of the gamers. However, it does fail at analyzing the potential of the title. Just like the gaming industry before the casual-gaming craziness, the gaming media focuses in a single environment instead of opening to the world. As an example, EGM did not review Endless Ocean because it was not a game, you only explored, there was no danger, you could not die at all.

    Hardcore titles should be analyzed from a hardcore point of view. A game for girls should be analyzed from their perspective. A casual game should be analyzed from their perspective. The media, right now, analyzes everything as hardcore. And while the reader of the site is hardcore, some day daddy and mommy will try to reach them to know what they think about this or that game and the site won't be able to welcome them.

    Just as companies found their own divisions to build family-friendly games, so game sites should have their own divisions of casual gaming reviewers. They don't make an Xbox 360 reviewer analyze a Game Boy Advance title, so why they let a hardcore gamer review a casual game? That is the only way those titles will be reviewed correctly.

  6. I was lucky to also play an early version of Soul Bubbles and I wholeheartedly agree with Daniel.

    Soul Bubbles is a rare game. Not only does it teaches you new gameplay skills, as Daniel writes, but it also allows you to experience new feelings. The rhythm the game induces through the dragging motion that controls the winds is something truly new. The whole game is tailored around this experience. While I guess I fall into the "hardcore gamer" category, I couldn't help but be captivated by the simple ideas and twists the game offers. Very few games give you the feeling that they're built for exploration, that you should be rewarded for testing stuff and not only for finding the right answer. Playing Soul Bubbles brought up the same feelings in me that the first hours of Super Mario 64, Tomb Raider or Portal gave me: I had to explore all the nooks and crannies of every level.

    If you're interested in game design, you should definitely play this game.


  7. I disagree. 'Soul Bubbles' does stand out with it's beautiful art direction and a general 'feel' which isn't one of fear or constant over-stimulation, but of exploration and relaxation. As a gamer, I'm usually 'stressed' and end up speeding throught some levels to finish quicker. With 'Soul Bubbles', I truly felt like I could play at my own pace, on my terms, which is rare (and pretty nice).

    Still, 'Soul Bubbles', ultimately, is not this grand classic. The novelty factor wears thin way too quickly and in the end, the product feels more like a high-quality tech demo than anything else.

  8. I think that Danc has hit onto something here, and it's been mirrored in some of the comments left by others. Reviewers and critics are, by definition, 'experts' in a field, whether it's the culinary arts, literature, film, or games. And just like critics in other media, it becomes important to know how a particular reviewer/critic matches the likes/dislikes of the individual.
    For example, if you really dislike everything that Roger Ebert loves, you'll not give his 'thumbs up' much weight. On the other hand, you may have a local movie reviewer who has loved the same films you have. That person's opinion and analysis would have more weight. The trouble that I see is that the gaming industry doesn't have the maturity in critique or review that other media do. Coupled with the instability of the Internet in general as people come and go and everyone has a soapbox, no matter the size, it would seem impossible to find a game reviewer that you can consistently expect to look at games the same way you do.
    Finally, as Danc pointed out, the aggregate scoring used on some sites is a statistical fallacy. Amazon has spent millions trying to get their recommendations down to a science – and they miss more often than hit, I think, but the work is being done. And it can be done, but not generally by hobbyists, which are the people who start most of these review sites in the first place. That these sites may move into more corporate territory (IGN, for example) is no indication that they are applying the same business models and analysis that will help someone really understand whether they will like a game or not.
    And finally, the analyst group that Danc mentions comprises, in my mind, an extremely small set of individuals. In fact, in all of my years developing software, I’ve met one person who meets those criteria. And he, of course, is the author of this blog. Perhaps if we had more analysts, we could begin to better help the average user who Googles the title to see if it’s worth buying. As the ‘experts’ in the field, I think it’s incumbent upon us to do what we can to help them understand. Whether it requires a re-envisioning of how reviews/critiques are done or a more/better/expanded categorization model (or some other deconstruction) remains to be explored.

  9. And next time, I'll have only one 'finally' paragraph. :)

  10. Let's assume that indeed it was just a matter of "lack of challenge" that led the game to be underappreciated by the core gamer audience.
    Would it have been that hard to include a "challenge mode" that imposed, say, moderately hard time limits on level completion? You could have a prompt to choose between "adventure mode" and "challenge mode" at the beginning of the game, in order to funnel players into the experience best suited for them.
    It seems like they could have added at least part of the core gamer audience to their focus (and maybe 10 metacritic points to their score) with just a little extra effort.

  11. An off topic question for Danc

    Have you ever been to, or do you know anyone from Long Island, New York.

    -Daniel "Zeed" Joseph Cook

  12. Contact me if you have Traveled both Time and Space to be where you have been at

  13. While I think it's quite valid to say that using the same model for reviewing games and movies is a flawed concept, I think the flaw is a quantitative one, not a qualitative one. (I would characterize your description of the "learning" that games cause as fitting in with "reacting" that movies cause, but I think that's just a semantic distinction) Because games require more personal involvement from their audience, the flaw of the process of reviewing something (subjectivity) becomes more noticable. It's still there when someone reviews a movie, just not quite as noticable.

    You refer to the universality of the human experience as what makes movies (or any stories, really) satisfying, and I strongly agree with this, but stories about various things can resonate more or less with certain people. If I can understand that something would resonate with someone else, even if it doesn't impact me significantly, all that does is make me a halfway decent movie critic. I guess I'm just not clear on what inalienable quality makes this game reviewing process fundamentally different.

    Then again, I'm one of those elitist jerks who happens to think the only reason why games usually don't have anything meaningful to say about the human condition is because the job of writing is shoved onto some designer as a second thought. Show's what I know...

  14. As a reader of this blog as well as 'another guy with an intricately detailed, heavily biased opinion', I'm flattered that you've linked to my small essay on the state of game criticism. However, I'm a little perplexed that I'm expressing a fallacy of epic proportions.

    I will agree I've disregarded the place of the representative user in game development - at least for the purpose of that opinion piece. But the search for Citizen Kane was written to deride reviewers for merely trying to compare games to art and literature while ignoring the unique properties of games. I wrote: 'game criticism will come to resemble an ongoing dialog' with the intention that game criticism would involve more interaction between players and critics regardless of the skill levels of either. And equating game reviewers to game designers was to suggest that game design and design techniques had as much to contribute to the art of game reviewing as the converse - including (but again not stated) the iterative user testing methods of Valve and co.

    At least you and I both agree that games are not movies.

  15. There seems to be this notion that film reviews are better suited for making an informed buying decision.

    However, looking at metacritic, you'll see review film scores commonly diverging across the whole scale, while in the case of games, it's mainly more of the same. A ~70 game gets reviews from 60 to 80 etc.

    Wouldn't that indicate that the opposite is true, that films are a much *more* subjective experience, and therefore harder to review?

    In fact, I think it is clear that game reviews traditionally have attempted to be more of an objective test, like testing a piece of hardware. I have never been a fan of that.

    So I would actually like to see them take a slice from film, and *embrace* their subjectivity.

    Sure, we are probably all influenced one way or another by reviews in our buying decisions (at the very least indirectly). But does anyone actually think that it's guaranteed that he will not enjoy a game because it has a 50 score? I certainly don't. I don't put any pressure on a reviewer to be "right".

    Ultimately, it comes down to a very simple recommendation system for me. And there is a reason why personal recommendations (i.e. word of mouth) has always, and will always be much more effective in convincing people to go see a movie, or play a videogame.

  16. Elsdoerfer, I think you're quite right. The problem is that the subjective point of view expressed by Metacritic scores is taken to be the ultimate measure of quality of a title.
    We need to realize that videogames have matured to the point where one viewpoint no longer suits all.
    It is probably not an easy thing to do, as I myself am emotionally conflicted at the fact that videogames as a medium are no longer equivalent to the gamer culture I identify with.

  17. In some cases, people just need to read the full text of several reviews, look at some gameplay video on Youtube, troll forums about the game, and make a decision for themselves.

    This post makes me envision some strange future where designers write academic journal articles about the relative merits of Design Element X or Design Methodology Y and support their conclusions with data gathered from gamers crooning over video screens with electrodes strapped all over them. Hopefully in the near future we can at least form a common critical vocabulary that isn't entirely based on hazy recollections of past games like it seems to be today, and instead tries to decouple critique from the games being critiqued.

    Your point about how modern games are multiplying the 'possibility space' beyond the critical scope of linear media is interesting, and makes me wonder how, or if, reviewers can hope to really properly review a game which could contain a large deal of emergent "content" the reviewer has no hope of seeing.

  18. As someone who keeps the method of criticism in mind most of the time (and exercising it with varying degrees of success online), I found this article extremely interesting. It pours a slew of ideas into my Pot O' Discourse.

    It's what I stand in line here for, and dammit Danc you keep ladling it out!

  19. I'm afraid your premise here is flawed at the outset. One major disparity between expert and user scores in your example is simply proportional: when you posted this, only two users had reviewed Soul Bubbles, while a dozen or so expert reviewers had weighed in. Now that a total of three user scores have been posted, the user score is 9.6.

    A tiny sample of online user reviews is a poor measuring stick against, well, most anything, since a couple of anonymous fans mashing the 10.0 button is basically meaningless. Seems like a rickety foundation to base so many broad conclusions upon.

  20. I'd say one other factor in the low reviews is the approach to playing the game. I'm playing Soul Bubbles a level at a time, one or two levels a day, and really enjoying it. A critic, by sheer necessity, is going to play through the game in one or two sittings. I can easily see you get bored doing that; it just isn't how the game is meant to be played.
    Which is a shame, since it may well be the best game the DS has.

  21. @Steve

    There are currently 5 written user reviews, but 40 user votes. I'm assuming that the score of 9.1 is based off the ratings not the reviews. (otherwise the average would be 9.5).

    It is still an interesting comment. I didn't calculate the margin of error so it is quite possible that this is just a statistical fluke. A sample of 40 isn't bad, but my stats teacher would be horrified at the eyeballing. :-)

    There's also the chance of vote stacking. That's one of the problems with user rating systems.

    take care

  22. Certainly, there's a place for a Consumer Reports-like approach to game reviews; that's essentially the role that Gamespot, IGN, and friends are trying to fill. And when it comes to more casual, mass-market games, those reviewers' hardcore biases prevent them from doing a very good job, as you point out. If we're going to review a game as a product, then simply focus-testing it and assigning it a score based on "aggregate enjoyment" seems as good an approach as any -- but it's an utterly soulless one.

    But shouldn't we be encouraging real game criticism? Do we want to regard games as a medium and capable of being true cultural artifacts, or do we simply want them to be mindless escapism that we fill our free time with?

  23. Don't you think just like games have a target market that reviews also have a target market?

    I can't imagine that the people who like to go to sites like gamespot or 1up and read reviews, and look at screenshots and listen to podcasts are this alternative audience of non-expert gamers.

    The expert bias is only a bias if there are non-experts in the audience. I've never met anyone who wasn't a pretty avid gameplayer who had a subscription to EGM.

  24. Not to harp on this too long, but I assumed you were using Metacritic for your sample, which still shows only 5 user votes (now at 9.8) My apologies if you were pointing to a different site, as 40 user votes is more solid.

    Andrew above makes a good point: every "expert review" of Soul Bubbles comes from an "expert" (hardcore/fan) publication. It seems like non-expert reviews would find a home in publications with a broader audience, like Entertainment Weekly, regional newspapers and the like.

  25. Hi Danc,

    Just wondering if you'd had a chance to check out Johnathan Blow's long awaited Braid on Xbox Live Arcade.


  26. But shouldn't we be encouraging real game criticism? Do we want to regard games as a medium and capable of being true cultural artifacts, or do we simply want them to be mindless escapism that we fill our free time with?
    This is a false dichotomy, albeit a pervasive one. There's no reason that one precludes the other. And, for that matter, Bejeweled could certainly be considered an artifact of today's culture - naturally, it lacks literary merit and provides no thoughtful social commentary, but commonplace popular entertainment is as much a part of our culture as "true art" (whatever that means) is.
    Look at the progress of film as an artistic medium - you have profound, meaningful works of art, and you have light entertainment, and naturally everything in between. Neither precludes the other. To me, at least, games already seem to be headed toward a similar path.

    When a game takes itself seriously, it's appropriate that a reviewer would do so as well. But similarly, when a game sets out to be easy, fun entertainment, why should a reviewer hold it to some other standard? The hardcore gaming sites have rated the game in accordance with what hardcore gamers - not the target audience - will get out of it. I suspect the casual gamers don't spend time investigating game reviews in depth anyway - one of the key features of casual gaming is that it's low-investment.

  27. @nathan: You make a good point, and upon re-reading my post, I see too much hand-wringing "Games are Art!" self-righteousness in it. I was reacting negatively to the notion that the game reviewer's duty is solely to judge how a game's supposed target audience will receive the game - that is, to predict the aggregate enjoyment that the game will elicit. This seems to be the argument put forth by the article.

    Certainly, there are plenty of game reviewers out there who attempt to function in just this way. The Soul Bubbles article was written about them, and my response may be out of place. I just wish we saw more game *criticism* targeted at a mainstream audience; I wish games had a Roger Ebert or an Anthony Lane.

  28. The thing is regardless of what the average user experiences when they play a game for the first time, I've been playing games for 18 or 19 years. The reviewer's "expert bias" is exactly the same bias I'll be experiencing myself when I play the game. I couldn't care less how new players play the game and more importantly I don't think new players could care less about how new players play the game after 10 hours of play.

    While being fundamentally enticing and good on a level that appeals to everyone is a worthwhile design objective and an objective that creates markets and profits, all this article points out is that there are multiple audiences for games, a key case of stating the obvious. You don't review movies from the point of view of someone who's never seen a movie before so why should games be that different? There's already a big divide between mainstream reviews and "gamer" orientated reviews and unsurprisingly the mainstream will, on the whole, read mainstream reviews tailored to mainstream gaming experiences while "gamers" will read gamer-orientated reviews. I'll read reviews relevant to my circumstances and the last thing I want is for concrete reviews I rely upon to start pandering to the broadest possible spectrum, because it's simply not feasible to write a review for everyone. Furthermore, in my opinion, from an artistic point of view, there is little to no validity in evaluating something in it's most basic terms devoid from the environment from which it's born. If someone rants and raves about a new album it's virtually meaningless without reference to the artistic vista from which it has been born as well as comparison to it's peers and predecessors. Soul Bubbles seems like a fun and well put together experience but there's a reason that reviewers who play a lot of games have scored it as average and that's simply because out of all the games they've played it is nothing but an average experience, perhaps a well presented and fun average experience, but not much more than that. I'm curious though and will probably buy it, mainly on the grounds that swapping DS games around with friends is a major enough factor that a short play time isn't too big a deal.

    Ultimately there are some interesting questions of what a reviewer's purpose truly is, but the fact is that "expert bias", consciously recognised or not, does not invalidate the reviewer's experience in any way shape or form.

  29. Long and interesting post. Reminds me of "Blink" by Malcom Gladwell. Have you ever read it before?

    Ah, Tomb Raider! This has really been some kind of experience.