Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
October 2, 2014
arrowPress Releases
October 2, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Can The Xbox One's Kinect Read Your Mind?
by Jamie Madigan on 07/07/13 07:30:00 pm   Expert Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

Well, no. Of course not. That's a silly question. Why would you even ask it? 

The updated supercamera on the Kinect 2.0 has been shown to be capable of some pretty amazing things, though. Microsoft demonstrated how it can tell where you're looking, estimate your heart rate from the color of your skin, and even infer your mood from your facial expressions. Finally, it has a sophisticated voice recognition system and the ability to see in the dark, which will come in handy when it wants to sneak into your bedroom at night and listen to your breathe for hours on end.

And though it hasn't been discussed, I wonder if the Kinect 's high definition camera could be programmed to measure one other important biometric: pupil dilation. This would be both awesome and worrisome, because while not exactly a mirror into our souls, the eyes can reveal a lot about what goes on in our minds. 

How Kinect sees you: a pulsing sack of meat and emotions.
 (Image from Wired's Kinect demonstration.)

Daniel Kahneman wrote  in his book, Thinking Fast and Slow, about how pupil dilation is a darn good proxy for mental effort. In a series of experiments he asked people to take a multidigit numer, then increment each digit in the number by one to form a new number. So 348 would become 459. The would then do the same to a new number, using a metronome to do one new number/sum every two seconds.

Try it yourself and you'll see that the task gets very difficult pretty quickly. If not, try adding 2, 3 or even 4 to each digit instead of 1, BRAINIAC. And if you had someone eyeball your eyeballs they would clearly notice that your pupils would steadily grow larger and larger as the mental machinery behind them started to work harder and harder --right up until the point where you gave up, when they would snap back to normal size.

In what sounds less like science and more like an exhibit you'd make a face at while visiting the museum of contemporary art, Kahneman and his colleagues would train a camera on subjects during these experiments and broadcast an enormous image of their eye onto a television in the hallway so that the pupils were about a foot wide and dilation was easy to measure. The results were pretty consistent: the more mentally taxed we are, the bigger our pupils get.

If the Kinect (or any camera) could detect pupil size, it would open up a whole new level of scaling game difficulty. A puzzle game could be made more and more difficult until you're taxed just the right amount to get you in the zone --something psychologists call "psychological flow." Imagine playing a rhythm game like Guitar Hero and having the game adjust the speed of the note highway until you're pushed just to the brink of your abilities based on how hard you're concentrating on the task. 

Xbone

Or what about knowing when to offer you a helping hand? If the Kinect can tell the point at which you've given up on a puzzle or sequence because your pupils shrink back to normal, it might offer you a hint. Possibly in a condescending tone.

Another, more unsettling implication would be that if the Kinect could tell when you are stressed and mentally taxed, it could use that opportunity to sell you stuff. Willpower is like a muscle that can be exhausted by any mental activity, and when it's depleted we're more likely to do dumb stuff like make impulse purchases or, one might imagine, place an impromptu Skype call to an ex-boyfrined when we really should know better.

Jamie Madigan writes about the psychology of video games. Read more at www.psychologyofgames.com and follow him on Twitter or follow him on Facebook.


Related Jobs

Vicarious Visions / Activision
Vicarious Visions / Activision — Albany, New York, United States
[10.02.14]

Character Artist-Vicarious Visions
Sucker Punch Productions
Sucker Punch Productions — Bellevue, Washington, United States
[10.01.14]

Concept Artist
Red 5 Studios
Red 5 Studios — Laguna Hills, California, United States
[10.01.14]

Web Developer/Web Architect
Halo
Halo — Kirkland, Washington, United States
[10.01.14]

Senior AI Engineer






Comments


Matt Marshall
profile image
I hate it how all of the awesome potential is now seemingly equally balanced with it's risk. So far we seem to have been pretty complacent about 'giving away our freedoms' for the betterment of our lifestyle such as our mobile phone revolution...although I guess many just assumed the government wouldn't do what it has done, in many cases. Not pointing any fingers or anything.

We seem to have come to the point where most of the next steps in entertainment and lifestyle have the potential to be used in a negative way. The question is, do they design these things with evil intent, hoping that we bite? Or do they truly believe this is the future and we need to accept them as is.

The problem, with most things we use, is that it comes down to intent. I feel that the problem now is that we, the users, are having less of a hand in that intent and are more on the receiving end to the point where we can't even opt out anymore. Microsoft's 24 hour check in plan was a hint of this. With all of it's benefits, to me it certainly pushed what I would call clean business practices. or at the least 'principles'. Which was my main issue with it, and I wouldn't buy it due to that.

Then there is simply the argument that if you don't want it, don't buy it. Which I also agree with. So I'm a bit of a hypocrite sometimes. The conversations in my head can be somewhat annoying I must admit :)

Either way, it comes down to the intent. Can we trust the people who are putting these gadgets in our homes that I admit have SO MUCH POTENTIAL to be used for nothing more than how to advertise to you and/or get money out of you in other dastardly ways.

In this case, if Microsoft's latest focus on making publishers and other 'moneymakers' people happy is anything to base it off, I would have to go with a planet crunching NO. Just looking at the ads on the CURRENT xbox live, the original DRM element of the 24 Hour check in, and a few other things they have mentioned that really seem to focus on anyone BUT the gamers.

In all this I can still say I respect Microsoft. They have done great things for the industry challenging Sony and Nintendo and bringing to market some interesting ideas...but unfortunately those resources are just not getting used to the full, or even right, use.

Matt Marshall
profile image
I must admit though, on reflection that MOST things we use CAN be used in a negative way, I feel that the difference now is we have lost control, and it's near expected to buy goods and services ANYWAY. Technology is changing so rapidly that it is easy for companies to do some pretty ridiculous things while we aren't looking. And even if people do see it, they buy it anyway because it has that new fantastic game or their friend Timmy has one.

I was reading the WB Terms and Conditions for Mortal Kombat the other day, something I doubt many people actually do, and the things we sign up for are pretty crazy...but I signed it anyway because I TRUST they will do right by me. But who's to say they will?

Freek Hoekstra
profile image
pupil dilation would be impossile, the resolution would not be adequate enough,
but body posture and overall facial expressions should defenitally be in the realm of possibility.

Ben Lewis-Evans
profile image
Pupil dilation can be used is a measure of mental workload, however, if the new Kinect can already detect heart rate then there is another measure that could be easier to access (i.e. wouldn't require the resolution that would be needed to detect pupil dilation, wouldn't need pupils to always be in view, and wouldn't have to deal with changes in lighting (say from changing action on a game screen in a somewhat darkened playing environment) and its impact on pupils).

Basically, assuming that the heart rate information is accurate and real time enough you could use it to calculate heart rate variability (basically the time interval between heart beats), which is a another well known, relatively sensitive, and established measure of mental workload (as mental workload increases heart rate variability decreases), especially in the high frequency bands.

A problem, of course, would be if the game being played was one with physical activity (jumping around in front of a Kinect) then the heart rate measure would be a combined physical activity/mental activity signal (That said, having someone jumping all over the place or moving a lot would also be a problem for accurately measuring pupil dilation). Furthermore, any measure of mental workload is essentially only a measure of arousal, which means in terms of trying to work out the emotional response of a player you are missing information on valence (i.e. if the arousal is positive or negative). That would be where recognising facial expressions could come in though (assuming people change their expression often when playing games, which they may not).

All and all though, with the addition of heart rate detection and facial recognition in the new Kinect there are may be lots of options that open up for biofeedback or biometrics in games. I mean, separate form the Kinect the guys at Valve, amongst others, have already been running experiments with stuff like having the AI director in Left4Dead react to changes in skin conductance, another arousal measure. Not to mention exciting opportunities to do things with the technology in research once the windows SDK is out.

Joshua Darlington
profile image
The new generation of kinect put all its resources into miniaturization rather than increased resolution. Some one would need to be really close to the cameras for that that sort of measurement. The idea is better suited for web cams or Google Glass.

These sort of biometric feedback ideas are part of machine social cognition. My opinion about subtle biometrics like pupil dilation is that they have secondary background value. Otherwise humans would actively use them and think about them. Instead, humans that have lived together for 20 years frequently ask each other how they are doing or what they are doing. I see that direct approach as more valuable. So I would recommend putting more effort into fun fluid machine conversational tools. or better yet - network in live humans to handle game balancing. Humans are an amazing resource for understanding and playing with other humans. If network capabilities are available why not plug in a game master?

If you think pupil dilation is valuable, go for it. I bet there are tons of people in mental health that would buy your tech.

PS regarding privacy - holding onto 20th century ideas about privacy is maladaptive. We are entering a new phase in the information age. It's better to work out new solutions that are suited in the emerging reality than try to live in the past. Luddite styles may be comfortable but set you up for information asymmetries and predation - see the red queen hypothesis.

Michael Hansen
profile image
one more reson not to by the console,, does microsoft share these data with the fbi and nsa
i think so ,, no clear statement on the subject


none
 
Comment: