Gamasutra: The Art & Business of Making Gamesspacer
Persuasive Games: Gestures as Meaning
View All     RSS
October 24, 2014
arrowPress Releases
October 24, 2014
PR Newswire
View All

If you enjoy reading this site, you might also want to check out these UBM Tech sites:

Persuasive Games: Gestures as Meaning

June 30, 2009 Article Start Page 1 of 4 Next

[In his latest 'Persuasive Games' column, game designer and writer Ian Bogost examines both Natal-like gesture control devices and Brenda Brathwaite's experimental board game Train, suggesting: "Perhaps the souls of our games are not to be found in ever-better accelerometers and infrared sensors, but in the way they invite players to respond to them."]

Games have flaunted gestural interfaces for years now. The Nintendo Wii is the most familiar example, but such interfaces can be traced back decades: Sony's EyeToy; Bandai's Power Pad; Mattel's Power Glove; Amiga's Joyboard; the rideable cars and motorbikes of '80s - '90s arcades; indeed, even Nintendo's own progenitors of the Wii Remote, like Kirby Tilt 'n Tumble for Game Boy Color.

Recently, all three major console manufacturers announced new gestural interfaces. Nintendo introduced the Wii Balance Board last year, a device capable of detecting pressure and movement on the floor. This year, the company released Wii MotionPlus, a Wii Remote expansion device that allows the system to detect more complex and subtle movements.

At E3 2009, Sony demonstrated prototypes for the PS3 Wand, a handheld rod that uses both internal sensors and computer vision, via the PlayStation Eye camera, to track and interpret motion.

And Microsoft announced Project Natal, a sensor system that foregoes the controller entirely in favor of an interface array of cameras and microphones capable of performing motion, facial, and voice recognition.

Gestures As Actions

With few exceptions, designers and players understand gestural control as actions. Lean side to side on the Joyboard to ski in Mogul Maniac. Grasp and release the Power Glove to catch and throw in Super Glove Ball. Bat a hand in front of the EyeToy to strike a target in EyeToy: Play. Lean a plastic motorbike to steer in Hang On. Swing a Wii Remote to strike a tennis ball in Wii Sports.

Gestures of this sort also strive for realistic correspondence of the sort advocated by the direct manipulation human-computer interaction style. Input gestures, so the thinking goes, become more intuitive and enjoyable when they better resemble their corresponding real-world actions. And games become more gratifying when they respond to those gestures in more sophisticated and realistic ways.

Such values drove the design of all of the interface systems mentioned above: MotionPlus, Wand, and Natal all involve high-resolution technologies that hope to capture and understand movement in more detail.

Physical realism is the goal, a reduction of the gap between player action and in-game effect commensurate with advances in graphical realism. As one early review of MotionPlus put it, "It's like going from VHS straight to Blu-ray."

Gestures As Meaning

As much as physical realism might seem like a promising direction for gestural interfaces, it is a value that conceals an important truth: in ordinary experience, gestures not only perform actions, they also convey meaning.

Consider body language. According to an oft-summarized but infrequently cited study (psychologist Albert Mehrabian's 1971 book Silent Messages), half of human communication takes place through non-verbal actions.

Gestures like crossing one's arms, tilting one's head, and rubbing one's forehead telegraph important attitudes and beliefs. In these cases, gestures are intransitive; they do not perform actions.

Instead they signal ideas or sensations: impatience, disbelief, weariness, and so forth.

Other gestures take indirect objects. When we wave hello or flip someone the bird, we do not alter the physical environment in the same way a racquet does when striking a ball or a hand does when grazing a pool. We may, however, change the way the recipient of the gesture thinks or feels about us or the world in general.

(Lionhead's "Milo" demo for Natal offers one approach to such a goal, although it apparently uses vocal recognition over gestural recognition; hand gestures once again become actions, like reaching in to disturb the surface of a pond.)

But gestures, be they transitive or intransitive, direct or indirect, can also alter an actor's own thoughts or feelings about the world or himself. These sensations can be complex, and they can evolve. Flipping someone off may impress delight, then guilt, then shame. Reaching into a clogged drain may instill dread, then disgust, then relief.

Article Start Page 1 of 4 Next

Related Jobs

University of Central Florida, School of Visual Arts and Design
University of Central Florida, School of Visual Arts and Design — Orlando, Florida, United States

Assistant Professor in Digital Media (Game Design)
The College of New Jersey
The College of New Jersey — Ewing, New Jersey, United States

Assistant Professor - Interactive Multi Media - Tenure Track
Bohemia Interactive Simulations
Bohemia Interactive Simulations — Prague, Czech Republic

Game Designer
Next Games
Next Games — Helsinki, Finland

Senior Level Designer


Michael Silverman
profile image

When I hear about serious games like Train, it raises a lot of questions for me as a designer. Not so much about the holocaust but the motivation behind the serious games movement. I register many serious games as noble efforts, but there is a set of serious games that trouble me.

To the casual observer it may seem like many serious game designers are simply using tragedies to attract attention to the serious movement. IE: "If I make a game about the holocaust, it HAS to be important." I know isn't the spirit of the movement, but I'm having trouble figuring out the true purpose.

Is the idea to prove the power of games? If so, designers are evoking a very real tragedy to make what seems like a mundane point. As an industry, don't we all already believe in the power of games as an expressive medium?

The best motivator I can reverse engineer is the challenge of a "difficult subject?" If that is the case, I wonder if Train is really respecting the profound nature of this challenge? If that is the motivator, I'd like to read more about the process Brenda took on to address such a sensitive issue. This is perhaps the most profound design problem I can think of to the point where design failure could be considered a human travesty. A failed design here is a mockery of the holocaust. Do serious designers take responsibility for that?

With much respect,


Ian Bogost
profile image

I won't speak for Brenda's process, even though I do know something about it. She may or may not choose to do so here. But I also don't think she should have to speak for herself. The work is comprehensible, if ethically challenging, in clear ways. What I will say is this: I don't think this is a "serious game," nor that Train is "simply using tragedies to attract attention." I think it's an earnest work whose earnestness can be seen when it is played. How is that more worrisome than creating a zombie gorefest to attract attention?

I have the sense that you're invoking (without invoking it) the old, tired argument about ineffability, the unrepresentability of the Holocaust, the one so many misunderstand based on Adorno's old quotable. I think this is a retrograde idea. We must try, even if the risk is travesty.

Matthew Kaplan
profile image
"Is the idea to prove the power of games? If so, designers are evoking a very real tragedy to make what seems like a mundane point."

Michael makes an excellent point: that there is a fine line between a game interface that highlights the mechanism (in this case, the use of gestures as representational of agency) and one that highlights the rhetorical exigency (in the case of Train, the seeming complicity of an entire industrial complex in the deaths of millions). Take, for example, the aforementioned Manhunt 2, and let's compare it to God of War: Chains of Olympus. Is the act of moving one's hands in real life to execute a digital person any more a sign of agency and complicity in the imagined act than rapidly pressing the X button in GOW:COO to force Kratos away from his one love in life, his daughter? Unless Manhunt 2 gives the player a chance to make very real associations between the gesture and the act--to use the Wiimote to interact in a way that can ONLY be accomplished through gesture--the presence of the gesture can be taken as mere gimmickry that marginalizes the perversion inherent in the act.

To put it another way, in Trains, you are actively planning as those complicit have planned. There is no separation between semantic agent and semantic instrument. You, alone, are agent and you use the instrument (the set) to act. In Manhunt 2, as with GOW:COO, the instrument is still the agent: "It's not me executing that person, it's the controls, and through the controls, it's the character."

If there is a way around this on the Wii, I'm not sure what it might be. The Wii's controls always take center stage by the very nature of the paratext that surrounds every game ("Now with motion controls! Wowzers!"). Perhaps a game that invites others to act as witness to your supposedly heinous interaction with the game and then somehow report their opinions interactively?

The Eyetoy, Project Natal, and the PS3's forthcoming Eye-based motion control wand may be able to break that one dissonant step by crafting imagery in which the player is indeed making 1:1 connections between the interface and the act. By seeing oneself onscreen, acting upon others, the agency produced by gesture will not simply be a *facet* of the gameplay, but the focus.

E Zachary Knight
profile image
"To the casual observer it may seem like many serious game designers are simply using tragedies to attract attention to the serious movement. IE: "If I make a game about the holocaust, it HAS to be important." I know isn't the spirit of the movement, but I'm having trouble figuring out the true purpose.

Is the idea to prove the power of games? If so, designers are evoking a very real tragedy to make what seems like a mundane point. As an industry, don't we all already believe in the power of games as an expressive medium?"


There is a difference between "believ(ing) in the power of games as an expressive medium" and actually using them as such. Ask any game designer and most game players if they believe that games are an expressive medium and almost all of them will say "Yes." without question. But if you follow up by asking what did the last game they created/played (respectively) express and almost all of them will draw a blank.

There is no real "purpose" to the serious games movement beyond that of the game's designer. I know that doesn't really answer that question, but it is true. But there is a general consensus that the idea is not to just tell, or show but actually involve the audience in the learning process. Some people can learn by reading, some by watching, but there are many who learn better by doing. Yet there are somethings we cannot do such as the holocaust. We can read about it, watch movies about it, but when was the last time you took part in it? Train gives you an opportunity to take part on a certain level.

From personal experience, I can tell you that being an independent in the state of Oklahoma can be hard. Oklahoma is the hardest state to get an independent candidate on a Governor or Presidential ballot. We have the strictest ballot access laws in the country. I can tell you these things, show you stats give you articles to read, but in the end, you will not truly know how frustrating it is. Yet if I were able to hand you a simulation of the ballot access process an independent candidate has to go through to get on the ballot, you will come out with a better understanding.

Is the idea to prove the power of games? No. The idea is to USE the power of games. There is no need to prove the power of games. That has already been done and is recognized.

Alexander Jhin
profile image
Ian --

To reframe the thrust of your thesis: Imagine a computer chess program built with gesture and physics engines. Yes, you can play chess with it but you can also slam your queen on the board in victory, tip your king so it falls in sullen defeat or sweep the pieces off the board in disgust.

Physical, "reality based" games like 'Train' have what Douglas Hofstadter calls "spontaneous intrusions into a creative process." Or as Daniel C. Dennett explains, "All creative algorithms... it's very important that there be a lot of junk, a lot of noise.

However, traditional computer games are the antithesis of noisy, tending much more towards clean abstractions and explicit rules. And indeed, the problem is inherent in the technology of computers. To quote Dennett again: "In the real world if you make a hotel you have to go to a lot of trouble to insulate the rooms so people can't hear each other in the neighboring rooms. In a virtual hotel, it's just the other way around. In a virtual hotel you have to go to a lot of trouble so they can hear each other. They don't overhear each other for free, you have to add this."

Gesture based engines, physics engines, ray tracing engines -- all of these introduce noise and thus freedom and creativity. However, gesture based input is special because it places this freedom explicitly on the player, forcing her to take emotional responsibility for her freely chosen virtual actions.

Of course, this freedom must be somewhat limited or else the designer's message is lost. 'Train' without some rules is just a bunch of toys on broken glass. 'Train' with too many rules and not enough noise, is just a game.

Interestingly, this sounds like an argument against hyper-efficient interfaces for certain message games.

John Petersen
profile image
If i place my thumb down and twist and push with it, I can invoke the feeling of squashing something I don't like" or "The way I feel in the world- I've always got a thumb on me"

But what I really want in video games as far as gestureing goes is much broader. I want to feel what it's like to stick my hand through a stargate, Is it cold, hot, dry, wet, does it feel like liquid mercury, does it smell, how heavy is it? I want to ride a comet and feels the tears in my eyes from gettiting hit by the wind in my face.

I want the physical sensations, not to just act out a scenario. I want to smell the fish spawning.

Devin Monnens
profile image

Your article reminds me of a couple board games I worked on in my MFA portfolio. The act of gesture and the feel of the pieces as you place them on the board is something that may be easy to overlook in favor of rules and visuals. Gesture as well as the weight and feel of the piece can help give these tokens a sense of preciousness that rules and images alone cannot communicate. Isn't the feel of moving a chess piece or placing a stone on a Go board along with the accompanying sound an important part of the game?

There are some analogues to this in game studies through ergonomics, force feedback, unique control interfaces, and natural mappings (most of these articles should be on Gamasutra...). Digital games can allow a sense of gesture, and a feel of weight from acceleration of a digital object (such as the dual analog sticks in Katamari or acceleration in Super Mario Bros.), but not a sense of touch and weight beyond a piece of plastic - a feel that is universal to all games using the interface. For instance, you cannot replicate a pinball table in a digital game - though you might be able to come reasonably close with a very mechanical and well-designed control interface. That physicality and sense of weight is something that can only be SIMULATED in games, not enacted. Maybe this is an area for expression and emotion in games that is easy to overlook, but even has some potentials in the digital world...