Hi Christiaan,

I've been waiting for this article you mentioned! You had me at "Interactive Animation" :-)
I also very much enjoyed your input on the panel at our Game AI Conference '09 last week.

On the plane back from Paris, I actually saw an old Pluto episode, and just in the process of him "moving from A to B" -- I could count 4 different emotions in the animations showing the exact thought process that Pluto was going through.

Luckily, while animation technology is certainly not easy to get right, there's lots of low hanging fruit in this department.

Alex
AiGameDev.com

Christiaan,

Very interesting article and some great points about gesture.

You will be interested in the technology and products we are developing at Emotion AI (www.emotionai.com) using procedural animation techniques to "layer" personality, emotion and gesture over regular animation. Our goals are to save developers time and money by automating character animation production (or some parts of it) and to bring to games real time, emotional interactive characters while giving developers / artists control over that behavior.

Get in touch with me directly and I can let you know more details, or ask around, many have seen our technology in action and can give, I hope, rave reviews. Apologies for the "pitch" but I think readers will be interested to know that this technology is available now.

Best Regards

Ian Wilson
CEO
Emotion AI
tellme@emotionai.com
www.emotionai.com

Nice article. There is a really diappointing tendency in current game animation that no-matter how high-end the cutscene or ingame animation may be, facial animation is almost always close to zero in them. Seems like all the energy is spent on detailed assets, and motion capture.

Even simple/inexpensive, yet expressive elements like blinks are left out. See the new KOTOR MMO trailer by Blur. The jedi dies without even frown or grin. This lack of subtlety combined with realistc motion captured body animation and the rendering style just kills belivability and immersion. I don'really get how animation directors/producers on these project don't fight for some production time, spent on facial rigs and facial animation on par with the body rigs and animation.

I just don't get it!

To a leave you with a good example, that luckly exists, are the character showcase animations on the TEAM FORTRESS 2 site.

I'm always curious as to where people put their hands on themselves when they sleep and talk to others in situations, to balance themselves. One day I plan to make a 3d video that helps people do energy work (simular to jinshinjitsu) using the color and sound spectrum to harmonize..it will be for free tho'.

I can't help but think that part of the problem is that most developers are men. Don't get me wrong, please - but most men don't pay half the attention to expressions and body language that the average woman does.

And because so little research has been done in that area, most companies and developers don't see a good reason to develop things like that in a game.

Also, most games don't develop NPCs - they develop "spots where you get hints/quests disguised with a human avatar". There are few games that actually develop side characters - and of those few, even fewer expend some time and money to improve expressions and body language.

It's sad, actually. I hope that more developers and game designers and producers realize this necessity, so maybe we can see some games with more realistic expressions.

@Cindy: Yes we do, we're just looking for different things. *grunt un'gowa you'se gonna try smack me?*

On a more general note, game developers and animators who are looking for research on body language and how it's conveyed and read may want to look into research into Asperger's Syndrome (a handicap in which a person has true difficulty picking up others' body language and emotions... a particularly cartoonish parody of Asperger's is Milton from Office Space). This has been a topic of recent interest which has made its way into "pop psychology"... not necessarily good for understanding, but it does mean there's a lot of writeups on the subject. You just have to look in psychology circles.

One reason that Pixar movies communicate as well as they do is that Pixar's animators pay very close attention to subtleties in body language. They're also good poker players (at least the ones who've fleeced me, they say they can read me like a book...).

The other thing to remember is that body language is often closely tied to a particular culture. A posture or gesture in one culture vs. another can mean completely opposite things. Example... if I stand very close to you while talking (close enough so that you cannot see my hands and face at the same time), Americans will tend to interpret this as aggression (getting in your face). Mediterranean cultures expect this close proximity as the norm, and find the default American distance to be cold, aloof, or defensive. If body language is expected to be a part of storytelling, it also needs to be subject to cultural localization in the same way that idiomatic text is localized.

@Alex: yeah Disney really laid the groundwork for the art of character animation and much of their techniques and observations are as relevant now as they were then.

@Cindy: It may be true there is some difference between men and women in this regard, but I think it's fair to say there is a minimum of body language that we all pick up on and games have some way to go before they get even there. There's A LOT of research actually, if you look at the work of Desmond Morris as I mentioned in the article or Katherine Isbister's recent research... It's a whole field of science unto itself.

@Atilla: Apart from motion capture being grotesquely overrated (though that's another discussion) I agree that if you're going to have very detailed 'realistic' body motion you can't get away with blank stares. Eyes especially are hugely important, yet very few games pay attention to them.

Lance, you make a good point about cultural differences (Morris goes into this in some depth as well), though there are some things that are more or less universal. The idea of localization on animation is interesting (if rather labour-intensive) but games like any other art are inevitably a product of the culture where they are produced, and maybe that's okay...


Another game with good character development is "Vampire: The Masquerade - Bloodlines". Actually also made pretty good use of facial expression. The last screenshot in the article is from that game. The earlier screen is from "Outcast" (excellent open-world adventure).

@Ian: I shall have a look at it.

I hate to use the example, but I saw something just recently right up this alley in The Sims 3. Just by looking at a sequence of screenshots it was quite easy to see what emotion or actions were being portrayed.

Now for me, the real question is 'How can this be done effectively in an indie game with little or no budget?'. When you have lots of money and tools available, it really opens up the possibilities, but how can it be done without having access to those same sorts of money and tools?

Kudos to whomever is able to pull it off well in an indie game.

@Rohit: No I haven't, will try to play it though, have heard good things about it.

@Lance, @Christiaan: didn't want to say that men don't pay attention to body language, it's just that, on general, they tend to pay less attention to it than most women. And most big budget titles today ARE still aimed at young men. So, you show a prototype to a tester and he says "this isn't realistic enough" and you ask why and he will probably say, the physics aren't good enough, or there isn't enough blood, or the AI is stupid. They will hardly say something like "they have no body language".

For most of two reasons:
1) It's so rare to see characters expressing emotions through body language that the average gamer doesn't expect it.
2) Because the average gamer is so concentrated in the action, most of the time they don't care much about emotions.

Also, I didn't mean general research, I meant research that made it into games. While there are a lot of research about body language and the importance of expressing emotions in games, few mainstream games actually have it.

Two of my favorite games, Mass Effect and Fallout (1/2/3) have great stories, but I almost didn't see body language and expressions in those games. And these are games with big budgets that are pretty story-heavy.

Body language and expressing emotions should be a standard, not an exception.

Still, that so many developers nowadays seem to take notice on that and do research about it, makes me hopeful that some day characters that express emotions trough body language will me a standard, like realistic physics is today.

I should add that yes, there IS body language in Mass Effect, but it's mostly in cut scenes. I mean, they don't affect much my gameplay.

There have been some games that have successfully used body language in gameplay, The Sims among them as Joel mentioned.

I highlighted a few others in a recent blog post:

http://www.gamasutra.com/blogs/ChristiaanMoleman/20090405/1058/Good_practice__An
imation_As_Gameplay.php


As for doing it indie: that depends whether you're talking about 2D or 3D... If you can get your hands on something like the Source engine (as the Zeno Clash guys did) there's a lot you can do with animation blending and layers of different gestures, procedural Look At and such.

I have to admit I hadn't really pondered what shape this would take in 2D, but my guess is it would be difficult because having different overlapping states becomes very content-heavy unless you start breaking things up Flintstones-style.

Whether you're doing 2D or 3D though, the main thing is to put the effort where it counts: the eyes, gaze direction, overall posture (line of action) and timing... everything else is gravy.

Men don't...what? No, it has nothing to do with men or women. The game with the most expressive facial animation I've ever seen is Heavenly Sword, and the leading person behind that mocap work was Andy Serkis, who looks and sounds remarkably like a man.

Facial animation and body language exist where they're needed to tell a story. Kratos in God Of War expresses his violent nature in combat with body language, but subtlety isn't appropriate there and wouldn't read well.

@Bob: I've tried to explain my point of view about the fact that the majority of games are made for men might make some difference, if you choose to read as "ha, men can't create great games with emotions and body language" that's not my problem anymore.

@Christiaan That's a nice post, I'm reading it now. I might play Assassin's Creed again just to pay as closer attention to those guards... (even if I found the game to be somewhat boring).

Also, I agree that you have hit a very important spot here: "the main thing is to put the effort where it counts". And I wish more games put some effort into facial expression, it breaks my immersion to see close-ups of characters with NO expression, they just look so utterly fake...

@Bob: Oh, I see what the problem is, I wrote "because most developers are men" at the beginning, that was just stupid of me. What I wanted to say about men/women is better explained below, when I said that the usual gamer won't complain so much if they don't see good facial expressions and body language. I'd go as far as to say that The Sims is a great example of body language and that the fact that most The Sims players are women probably has a relation to that.

(while I can't say if body language was better developed because they had a broader audience in mind or if they hit a broader audience with the help of body language).

You know, I wish there was an edit button so I could write that in my first comment instead of that stupid thing I wrote. That's what happens when you write something withouth thinking much about it, will try to avoid that next time =p

Hey there! I am quite shocked about this article mainly because it is *EXACTLY* what I did my Master's Thesis on. I haven't placed it up online yet, but I will do so eventually and link it on this page. Great write-up, though -- I've been working on it and explaining this subject for about 6 months now. You've just scathed the surface...there's a lot to cover!

I think the lack of this in most games has do with a lack of time and resources. I'm sure a few big budget games will blow everyones minds at some point.

A easy technique is to use several static or lightly animated images for different emotions like in persona 3. Its a older style but it works well to display emotion. I assume its cheaper to do also.

I do get frustated working with creepy manikins in NWN2. I can do things with lighting, shadow, and sound but it still doesn't get things across that would be easy with real actors. I assumed that since half-life 2/Bloodlines did such a good job with faces that all games would be like that. It didn't happen.

It should probably be noted that for the purposes of this discussion cutscenes are entirely irrelevant, as they are not interactive.


I did a thesis (bachelors) on this myself as a student. This article is the result of me refining my thinking having spent a few years in the industry now.

With few exceptions animation is still very underused in games. It takes a real cross-discipline effort to do it well, but the results are clearly worth it (Shadow of the Colossus, anyone?)... If we want to advance this art-form, we need to start using animation properly.

Lack of resources is no excuse as we are more than happy to pour time and effort into hours upon hours of content that few players will ever see, whereas we can't improve character behaviour that will show up across the board?

My kingdom for an edit button!

@Ryan: Anyway, I'm curious to see what your take on it is. As you say, there's a great deal more to the subject.

A few ideas :

People Change:
For stances and body language a parametrical approach could be devised to indicate current mood and condition of the characters. This content layer can be more or less culture independent, since these sort of conditions and how they affect movement seem to be universal. Everyone moves slower when they are tired for a trivial example.

see this link to what I mean - http://www.biomotionlab.ca/Demos/BMLwalker.html

Stage it well:
For reactions and emotional responses a medium close-up (to include hand gestures) or a close up of the characters face is the way to go on a usual animation production.

Games using free roaming (1st, 3rd person) cameras could make use of overlayed close-up views/windows or even split screens to show these reactions, not having to break for a cutscene. This way you could retain the immertion yet make well staged story points in these overlays independent of the actual staging of the characters in the game world, or trying to restrain the camera.

One could separate body action to one instance of the charcter on screen, and gestures and facial animation onto another, and not try to squeese the two onto one.

Great article! Animation in games is something that has been overlooked for much too long. Most game characters feel like zombies with jerky mocap movements jiggling the brain in their half wit skulls. Characters just don't feel like they are thinking. They're just puppets triggering animations as they must. The bare possible minimum to keep the user in sync with what the hell is going on.

But enough ranting from me, you've already made a better case than I possibly could. :)

Excellent article! I can't help but believe that part of the problem is, ironically, the focus on 'realism' of characters in games. I think that with today's technology, credibilty -- as in making the player believe that characters are alive -- can be much better achieved with a more interesting and inspired character design than simply trying to copy actual human beings. Pixar is indeed king when it comes to bringing characters alive and emerging the viewer into their stories, yet I've never seen a 'realistic' character in any of their projects.

I've seen the attempt at giving expression to faces in Half-Life 2, but quite frankly I wasn't very impressed. Am I the only one who feels about it this way? Because apparently a lot of people are (or at least were at the time) pretty ecstatic about it.

I think that it is important not to stereotype any particular category when speaking of specific topics. For example, the question of verbal versus nonverbal communication is not a question of an individual's sexual identity (which is a spectrum, not a dichotomous choice of "male or female," which themselves are terms open for debate with respect to semantics). Rather, the question is one of individual psychology and communication patterns. This is also true for cultural variances; generalities are not good when applied to specific topics or products, especially when international communications creates an environment where differences are disappearing more and more. Or, to look at it another way, why stereotype a product's communications and alienate all the people who have a psychology that doesn't match the "average" (whatever that is claimed to be)? Such an approach is a very slippery, dangerous slope. For example, what about immigrant demographics or people who are living in a country for a few years for school or business? What about second, third, and onward generational descendants who embrace both their original culture and adopted one? Or what about people who simply do not have a psychological match with the culture they live in for whatever reason?

I think that Tytan points out one of the most important issues for modern Western developers: an unhealthy and self-defeating preoccupation with creating "realistic" environments and characters. East Asian artists tend to focus on "believability" and ignore "realistic" in favor of creating emotional empathy with the audience. This is a key feature of manga art education, and one of the primary differences in its artistic approach versus Western art education.

I'm afraid that I will have to strongly disagree with assertions that games are made by men for men, and that that's why they do not have presentations of body language. We must always remember that gaming and other media are global products, not restricted to only English-speaking markets. For one thing, there are plenty of women creating games and always have been, and there are plenty of women who enjoy media that is created by men just as there are plenty of men who enjoy works created by women. Consider the many female artists, doujin and professional, who create the bishoujo (and bishounen) characters in many Japanese works. This includes such famous women as Rumiko Takahashi, Naoko Takeuchi, Carnelian, CLAMP, Kazuko Tadano, Akemi Takada, Atsuko Ishida, and many, many more, as well as famous men such as Satoshi Urushihara who have created stories such as Chirality and characters for games such as the Langrisser and Growlanser franchises. These are artists that are enjoyed by both male and female consumers. In fact, as Tadashi Ozawa points out in his intro to How To Draw Anime & Game Characters Vol.5: Bishoujo Game Characters, most of the vast number of bishoujo characters are created by women, not men, and the fact that the characters have a wide variety of feminine traits helps explain why they appeal to both male and female fans. The same cannot be said for the various yaoi and shounen ai characters; relatively few male artists contribute to their production, so they do not generally have an appeal to male fans.

Just as importantly, you see the presentation of body language and facial expressions to convey emotion to the audience in any book that teaches basic art techniques, including the one just mentioned. In fact, even the names of the characters as well as their outfits, hairstyles, accessories, etc convey nonverbal meaning to the player. Japanese artists (and I think Korean, as well as other East Asian artists) would never think to skip this all-important aspect of a character, even for minor supporting roles. This is reflected in most any (Japanese) artbook/guide for any Japanese game. However, Western, English market guides never really show this background art development, concepts, etc.

You can see some of this in certain titles that have been released to the English market. Some good examples would be the Growlanser games (the couple that have made it here, that is) and the more recent Tales of Vesperia. Even 3D characters such as the ones in Star Ocean: The Last Hope or Silent Hill 3 show this type of attention to detail, albeit not to the extent that 2D, cel shaded works do.

Basically, it isn't true that such work is not done in gaming. On the contrary, there are vast numbers of games and entire markets where a focus on nonverbal communications, body language, facial expressions, etc are the norm rather than the exception. However, the English market may remain relatively unaware of such facts due to overlooking anything not in English, as well as certain examples that have been brought to the English market but are perhaps not as well-known as they should be. That's a bit different than such works not existing at all, though.

Western developers need to learn the East Asian focus on conveying emotional empathy through their characters. Western artists used to have this but have lost it in the past decade or two. After all, iconic Western characters such as Mickey Mouse, Pink Panther, Daffy Duck and Bugs Bunny gained their status due to empathic communication, not any sort of emphasis on "realism" (which they certainly are not). Perhaps the most iconic Western game characters that succeeded in conceying nonverbal meanings were Dirk the Daring and Princess Daphne from Dragon's Lair. Then again, that was largely due to Don Bluth's wonderful team, and it was done before the more recent focus on "realism is better." Most any other game character that had that type of success (Pac-Man, for example) was not Western in origin. Western artists and companies can do the work required, but they need to regain their focus on what is important (not "realism" but "believability").

Nice article Christiaan!

Nowadays, even when i play a FPS games, I get bored very quickly when the First Person Arms of the character aren't moving, just are static (Halo: Odst videos :S ).

The reason why I am waiting for Brutal Legend to come out because it actually focuses on facial and other expressions to convey its storytelling and humor. And since it has that simple look of a Pixar movie, it's going to be successful, at least that is what I predict. I'm not even a rock fan, yet the animation of the character alone quickly drew me into it.

@Cindy I think we're going to have to agree to disagree on your charges of gender bias' since you seem so headstrong in your opinion. It comes across to me that you may not be working in the games industry or you are very new to it. I believe you are ignorant (perhaps naive to use a lighter word) as to why things are the way they are in games. There are many factors, one of which being that Animators do not decide how a game is made. Second, games have but fairly recently been capable of dealing with this kind of stuff as each new generation pushes the boundary in some area (human expressiveness being at the bottom of priorities), we are a long way from Pac-Man but are still a ways away from Pixar quality in games. Also, until recently and still a limiting factor, any particular game engine utilizing skeletal animation cannot support several skeletons (on screen/in level) complete with articulated facial bone systems in each one. Third, it is deemed by those that be at the highest level, much to the chagrin of animators/creative artist types, that body language and facial expressions are not a major focus of the game and should not exhaust time and money developing such "insignificant" fluff. Forth, the Animation system is not robust enough to support such features and again it is decided to not spend time and money developing to provide such feature set. Fifth, schedule time dictates what gets done and what *can* get done and what will have to get flushed and/or cut. Sixth, animators can make animations, they cannot implement them, if the AI programmers do not have a mandate (again from higher up) it will not get implemented. Seventh, it is often extremely exhausting for artist/animators to work well with programmers to achieve the artistic quality level that artist types strive for and ultimately find it easier to get by with fewer frustrations and settle for lower expectations and still get a paycheck- this amounts to a degree of laziness, but really...ugh, sometimes a nightmare. Conversely, the animators on staff are inept at creating quality animation with nice body language/facial expression, etc. and the company has not corrected the problem and/or similarly, find it easier to get by with what they can get out of them with fewer frustrations and settle for lower expectations and still get the game out the door. I'd say my last point of factor being, which kind of sums all of this up, the level of competence (or incompetence) of the entire company/team/publisher/producer.
Video Games is still an infantile Industry and has yet to mature into a solid mold of production. It's already getting there, eventually the Art side of the Games Industry will completely embrace Movie Industry standards and practices and achieve it's epoch in conventional game crafting.

As an aside and worthy of a different topic of discussion, would not people agree that now Artist/Animators are absolutely equal to Programmers as far as being essential to game creation? Unless you're making another Tetris game, even still, you cannot make a game without competent artists just as much as the Artist cannot make the game without the Programmer. Why then, the discrepancy in pay scale? Because programmers "invented" video games? Have been there from the inception? Require a higher degree of "skill", education? I think it's by far overdue for leveling the playing field, imho, of course.

-shubes

>Animators do not decide how a game is made

Unless you're Fumito Ueda.

I think Team Ico and others have shown pretty clearly the value of putting an emphasis on strong animation and direct collaboration between programmers, designers and animators to achieve characters that are interactively alive.

I think it boils down to team vision, team structure, team cohesiveness, team competence (ability to do and how well in working with others), who leads it, what their priorities are, what they press for in excellence, communication and understanding, engine and system capabilities, time and money.