Tuesday, June 2, 2009

Engineering Emotions: More predictions come to pass

Back in 2007, I wrote about some hypothetical technologies like real time motion capture, voice recognition and biosensors that were on the horizon that would have  a dramatic impact on how we design emotional games.  Those technologies are now becoming mainstream with console accessories like Microsoft Natal and the Wii Vitality Sensor.  Others techniques like feeding player's input into internet API's like search and social networks are already easily implemented using basic capabilities available to even the most limited game devices on the market. 
In the essay Constructing Artificial Emotions, I described a 'crazy' futuristic game design called Bacchus: 
"Bacchus is a multiplayer dancing game with a religious theme. The selling point is its ability to evoke intense emotions.

Imagine if you will, a decrepit theater filled with writhing, dancing people. The lights flare and swoop in time and the people chant in unison. A massive screen shows a mirror image of the hall like some surrealistic portal into an alternate universe. Instead of blokes and lasses in street clothes, the on screen spirits are clad in ornate ritualistic garb. The movements on each side of screen are eerily synchronized. The pitch of the chant rises.

The screen zooms in on a girl in the center of the room. The crowd, as one, turns and watches her figure on the screen. She begins to dance. At first her movement is controlled and intricate. The screen pulsates and she yells to its beat. The room takes up her words and amplifies them, giving them god-like resonance. Bass mixed with reverb mixed with primal, guttural passion. Her dance becomes wild. The pace increases and she begins to confess.

The theater reacts. Each word she utters shimmers on screen, merging with ghostly photos from her past. In a beat, the entire room witnesses her sorrow over the death of her mother, her time alone in an empty apartment, and her first kiss. An inhumanly beautiful electronic chorus rises, matches and turns her words into a song. Her movements become a blur. Her glowing eyes are ecstatic. At the peak, her spirit on the large screen explodes in light and the girl collapses to the floor in fervent religious swoon.

The crowd goes wild. The screen zooms out and the next god dancer is chosen. 

Later, the girl writes to her online friends that the night she danced was the single most powerful spiritual and emotional experience in her entire life. It was the night she was touched by a higher power while playing a video game."

The original essay is an admittedly difficult read, but I recommend revisiting it.  In short, psychological experiments show that by intentionally mixing physical states of excitement with the appropriate context a designer can concoct emotional responses that are indistinguishable from naturally occurring emotions. The design techniques described within are no longer futuristic daydreaming.  A basic form of Bacchus could be made in the next few years. 
In the past games have been limited in the types of responses they can evoke in players because the range of human activities that we could model and reward were limited.  We've admittedly designed amazing experiences that only rely on the limited ability to press a button.  However, a cursory inventory of the human body and mind is surprisingly more comprehensive than a twitching thumb.  We can move our amazing and capable bodies, we can engage in complex social interactions, we can become excited or depressed.  All these basic elements of our humanity have been outside the realm of game design because we could not track them, build models around them or reward desired behaviors. 
Now we can. 
With these new tools and a mass market that embraces them, we have a vast laboratory of millions of players.  Early mini-games will act as experiments in the engineering of human emotions.  Initially, we'll focus on found fun since that is what our audience is currently trained to consume.  With time and enough experiments, we'll begin to notice that with the ability to manipulate body, mind, social context and excitement level, we gain the ability to evoke deeply meaningful emotions. Imagine visceral sorrow, lust, anger, happiness, cruelty, generosity, stress and contentedness.  All the emotions reproducibly evoked in psychology lab experiments become our creative palette. 
Every game becomes a reality television show starring the player. 
Every game designers becomes pragmatic engineers of the player's emotional experience, dissecting and reconstructing the ephemeral moments of human nature.  Our games turn into intricate systems of hardware and software that play players like a willful instrument. 
Hardware like Natal, MotionPlus, Sony's wands and the Vitality Sensor are really just the beginning. There is an entirely new class of middleware that tracks the torrent of new sensor information and teases out useful patterns of human behavior.  Fresh emotional game mechanics that are as new to the world as moving objects in Spacewar! must to be invented from whole cloth. There is great work to be done. 
Once again, I'm reminded what an exciting time it is to be a game developer. 
take care


  1. Is it an exciting time to be a game developer, or should I just feel overwhelmed? Obviously there is some awesome potential here, but there is a part of me that misses the more simple games of the past. The line between games and reality is becoming more and more foggy. Is it really a good thing when a game becomes the focal point of our lives?

    Of course, I'm probably over-reacting. There is plenty of opportunity here to do some pretty neat things and I suppose we just have to be sure that we take it and use it responsibly.

  2. Haha! Reading that excerpt from the article you quoted totally creeped me out!! :D It read like some kind of dark utopia, like Brave New World. It is apparent that technology is rapidly bringing the ability to closer tie into the human state, but I feel somewhat overwhelmed too... I think that even if technology allowed it, emotional mirroring in games might never go beyond niche use. I know that I tend to play games so that IT can effect my emotional state, not the other way around: that's what the arts are for!

    ...Hmm, now that I think about it, artistic expression is probably where this kind of technology could reign. Perhaps "games" are a slightly awkward fit, but for apps or what I call "game apps" that focus on artistic expression or social networking, this may well be the future.

    Your articles never fail to stimulate the intellect, Danc!

    Oh yeah, I think that any games that would mirror and/or amplify emotions, namely more aggressive emotions would need to have a rather physical aspect to gameplay to expend that energy healthily.

  3. I'm glad to hear I'm not the only one feeling overwhelmed...

    I thought learning games were new and exciting enough, and then I was introduced to the ideas of social emotion and social play as an entirely new area to design for. And now games that reach directly into our emotions? Wow, I have a lot to explore. I guess that's a good thing. :p

  4. To submit oneself to someone's idea completely, as to start playing a game as real as Bacchus would be, is by all means the greatest of experiences, coming from any direction - be it games, books, movies or even maybe real life, as we've become emotionally closed and hidden today. There might be a catch for freeing the 21st century man's mind in there; bringing him very close to reality in a simulated experience, showing him what and how he *can* really do, then letting him off the hook. Again, showing him what he can do can be less than rewarding if we don't pay attention to who (and how) designs the game. Emotions are a very sensitive deep pool, with electrical eels all around. Sure, swimming is nice, but...

    Nevertheless, I am open to the idea. I myself can't wait for the day when interaction between man and machine will be trivial on an organic level..

  5. It seems to me that the folks who wrote Caprica (the BSG prequel thing) might have been influenced (or just totally ripped off) this idea -- watch it and see if you don't agree. (I'm referring to the VR game the teenagers are all addicted to.)

  6. I think it is still way too early to say how ground-breaking this is. The sheer amount of psychological data points and programming needed to take into account a human being's emotional state from visual images takes a massive amount of processing. Robots and supercomputer AI specifically designed to handle visual recognition still make a lot of mistakes.

    Also there is the problem of people having a variety of body styles, facial expressions, and bio-mechanical quirks from person to person can easily mislead such a controller.

    There is also the "tangible" factor that is missing in something like Natal. The force-feedback of a controller is something that a lot of people like. It is one of the chief complaints against the touchscreen-only iPhone becoming a viable game platform.


  7. It is somehow scary although certain games had influence me from long time ago.

    Emotions is something complex. Behind common things about people we know, lie individuals with theirs darkest secrets, desires, traumas, monsters, and other things that most don't wish to see.

    I apologize if it might wrong, but i think even psychology hasn't really figure out everything about individual human, it mostly only know what globally acceptable, this include gestures. Will the AI recognize this and show what the gamer really wants or in the end world will be filled with reality-games that doesn't care about 'certain type of people'?

  8. Great stuff. I just found this blog, and I'm extremely excited that there is someone out there writing intelligently about games.

    Is this stuff scary? Overwhelming? Sure it is. Because our human nature is scary (not to say that it is all dark, just that it is deep and vast). But I for one am interested in discovering as much as possible about who I am, and who those around me are: that's why I read books and listen to music, watch movies, and go to art galleries.

    All we have here is a new medium to work with, and I am very excited, because not only is it new, but it is the most dimensional and maybe the most significant medium that we have yet encountered as a human race. Stuff like artificial, intelligent visual analysis etc. may seem "super computers away," but these days that isn't very far.

    Well done Danc.

  9. Great stuff. Visit www.theregalgamer.com for more gaming news, reviews, previews and cheats!

  10. @GameDevigner:

    That's not the least of it!

    I play games that mis-interpret my input all the time. Some games are BUILT around that theme. Street Fighter, for example, yields certain moves only when the player navigates a complex set of timing. Other games, like Uncharted, get confused when interpreting input within their "cover" and "climbing" systems, which are designed to be player-friendly and "intuitive."

    So, imagine taking all these new inputs, which are all analog, and mapping them not just to input, but to *actions in the game.* There's a subtle and important difference there.

    Most Wii games go little beyond "waggle" because interpreting player input is hard. Nest this problem inside the problem of capturing that input accurately. This is why we don't have these technologies yet.

    Now, imagine that some company creates a system that finally works, and works well: if that's patented and held secret, everyone else will have to invent it themselves, instead of benefiting and building on such an advance. I bring this up, because in such a case, we're not looking forward to these advances as a society, or as game designers, but as (for example) Microsoft customers.

  11. Hi! I am new on here and noticed your blog. I love the layout and how crisp it looks. The content is great too! Keep it up.

  12. Hello- Really enjoyed the drawings on your blog... I'd love to have a tattoo of a realistic dragonfly and white orchid... Let me know if you can create that drawing for me!! Keep up the creativity!

  13. I love this blog, such effective posting.

    I myself use MochiAds and it hasn't proved to be very much money since my website doesn't get very many hits.