Gamasutra: The Art & Business of Making Gamesspacer
arrowPress Releases
October 31, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Mind Reading
by Robin Arnott on 06/27/14 03:05:00 pm   Expert Blogs   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

I’ve observed two modes of playing SoundSelf. First there’s a playful mode: The player asks the system “what can you do, how do I play with you?” They dance with their voice, push SoundSelf around, explore its limits. They’re having *fun*. Players always begin in this space, but SoundSelf’s magic happens when they transition into a second mode that I’d describe as “surrender”: Their breathing slows down, their voice falls into repeating rhythms, and they stop thinking.

This is the “trance” I’m always talking about. SoundSelf’s interaction is designed to distract your “inner voice” for long enough that you temporarily fall out of the habit of listening to and identifying with it, thus leaving your sense of identity open to being hacked and expanded. However, not everyone makes it over the hump into the surrender phase of the experience, or they’ll surrender for a few minutes before the inner voice returns.

In zen meditation, when the mind inevitably wanders, you actively reign it in again by deliberately drawing your focus back to your breath. Unfortunately, such a mechanism is not directly available to me as a designer without introducing inelegant symbols and instruction into the experience, which  would change the nature of SoundSelf from play-partner to teacher, and a teacher is not what I’m interested in making.

So here’s the challenge: How can SoundSelf slowly seduce you into a deeper and deeper trance, but also catch you when you wander back into a playful frame of mind? I think it comes down to respecting the frame of mind the player is currently in – letting SoundSelf respond intuitively to your voice when you’re in the playful mode, but slowly and subtly leading you and moving with you once you’ve surrendered.

Ideally, SoundSelf would be monitoring the player’s brainwaves or heart beat variance and using that data to change the program. For better or worse this technology isn’t commonly available on commercial peripherals. But SoundSelf does have indirect access to a powerful biometric: your breath.

Imitone  (the pitch detection algorithm SoundSelf runs on, which is the pride and joy of our programmer Evan Balster) is very sensitive to tonal sounds like your voice, but it’s not designed for atonal sounds like wind and breath. This is a feature, as it effectively ignores background noise. So while SoundSelf can’t know when you’re breathing or how deep and long your breaths are, it can make an educated guess based on the length of your tones and the space between your tones. Combining a two minute rolling average of four elements…:

1. The length of time between your tones
2. The *consistency* of your rhythm between tone and inferred breath
3. The range of the tones you are expressing, from lowest to highest
4. The amount of different tones in your recent toning history

… and we get a pretty decent heuristic of how entranced you are, and thus how SoundSelf should behave.

It’s not perfect, and it’s quite sensitive to false positives (imagine if in the middle of a period of long low tones from the player, SoundSelf suddenly interprets a distant bicycle bell as a short high pitched tone), but smearing the measurement out over about two minutes gives me a pretty decent high-latency measurement of where your head’s at, and what SoundSelf should do to gently nudge you deeper.


Related Jobs

Activision Publishing
Activision Publishing — Santa Monica, California, United States
[10.31.14]

Tools Programmer-Central Team
Amazon
Amazon — Seattle, Washington, United States
[10.30.14]

Sr. Software Development Engineer - Game Publishing
Intel
Intel — Folsom, California, United States
[10.30.14]

Senior Graphics Software Engineer
Pocket Gems
Pocket Gems — San Francisco, California, United States
[10.30.14]

Software Engineer - Mobile, Backend & Tools






Comments


Kiel Gilleade
profile image
Interesting idea; though I suspect the act of playing SoundSelf (e.g. chanting) will have an overriding effect on any heartbeat rate (HR) based measures that could be used to infer the player's meditative state. You would probably be better suited to using brainwave activity instead (e.g. alpha wave activity -- high alpha being indicative of the player relaxing) which could be recorded using a consumer EEG such as the NeuroSky.

Have you tried manipulating SoundSelf using biosensors before? Having played SoundSelf myself, it be cool to see a time series of my HR overlapped with properties of the chant. Perhaps as a challenge the player could try to synch the sounds of their chant to the effect their chant is having on their HR (NB. during deep breathing exercises HR will synch with respiration -- player essentially tries to find a chant which has a similar effect).

Robin Arnott
profile image
I've not tried using biosensors, but it's a bit out of scope for the project. I wanted to make something playable with consumer hardware, and I'm already supporting one esoteric device (the Rift). Maybe as a performance piece at a later date or a follow-up project though :-). And there are some REALLY interesting projects out there using Heart Rate Variability (we have a whole subcuration of them at the Indiecade Night Games party this year!)

Robin Arnott
profile image
I've not tried using biosensors, but it's a bit out of scope for the project. I wanted to make something playable with consumer hardware, and I'm already supporting one esoteric device (the Rift). Maybe as a performance piece at a later date or a follow-up project though :-). And there are some REALLY interesting projects out there using Heart Rate Variability (we have a whole subcuration of them at the Indiecade Night Games party this year!)

Kiel Gilleade
profile image
Its a shame biosensors don't see more use in games. Coming from a research background I've worked on and seen many interesting applications of the technology (e.g. adaptive gameplay) but I can understand why commercially their a harder sell. Biosensors can be difficult to work with and don't have as readily applicable uses such as VR which is much easier to understand (a new viewing technology).

Not heard of the Indiecade event before, will keep my eye out, sounds very cool.


none
 
Comment: