ADAPTIVE GAMEPLAY AESTHETICS (PART 2): A Disruptive Game Design Framework
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.
Copyright © 2018 Chris Solarski, unless otherwise stated / www.solarskistudio.com
This essay can be used as a worksheet for educational and training purposes as long as the contents remains intact and the author, Chris Solarski, is clearly credited. Please consider donating via Gumroad if you wish to use it in a classroom context.
ADAPTIVE GAMEPLAY AESTHETICS (PART 2)
A Disruptive Game Design Framework
By Chris Solarski
TABLE OF CONTENTS
Please note: this blog post has been divided into 2 parts due the original essay's 80+ page length, to make it less overwhelming for the format. You can read Part 1 here, or download the complete essay in PDF format via Gumroad: https://gumroad.com/l/TtrHn
2. AESTHETICS AND MATH
2.1 The Origins of Shape Language
2.2 Secondary Shapes
2.3 Cultural Symbols
2.4 Duchamp, Objects and Meaning
2.5 Shape Language and Cultural Symbols in Composition
2.6 Overriding Cultural Symbols
2.7 Shape Language in Non-Western Art
2.8 Shape Language and Composition in Mainstream Culture
3. COMPOSITION FUNCTIONS
3.1 Visual Simplification
3.2 Visual Direction
3.3 Story structure
3.4 Holistic Design
3.5 Concealing the Composition
4. SHAPE LANGUAGE FOR GAME DESIGN
4.1 Shape Language for Game Art and Narrative
4.2 Shape Language in Support of Gameplay
5. THE ADAPTIVE COMPOSITION FRAMEWORK
5.1 The Magic Circle and Defining the User Experience
5.2 Shape Language and In-Game Music
5.3 Shape Language and Interactive Motion Lines
5.4 Interactive Motion Lines and Ludonarrative Dissonance
5.5 Formal Game Design: Single and Binary Control Schemes
5.6 Informal Game Design and Gameplay Vignettes
6. GAMEPLAY IS ART
The first iteration of this essay was published back in 2013 in the form of a Gamasutra article, after I’d completed a book tour around North America (kindly supported by swissnex, AlpICT and the Swiss Arts Council, Pro Helvetia) to promote my first book, Drawing Basics and Video Game Art (DBaVGA). I had the privilege of speaking at institutions and events that included The Smithsonian’s The Art of Video Games exhibition, USC Interactive Media Division, and NYU Game Center. Although DBaVGA is aimed at game artists (it features artworks by the likes of Leonardo da Vinci and Nintendo side-by-side), this string of speaking events gave me an opportunity to present the core concepts of traditional art techniques adapted for game art to varied audiences that consisted of the general public, academics, students and professionals working in the games industry. Through this string of speaking events, I gradually developed a new perspective of my work, which enabled me to broaden its application to the entire game development team. What emerged was a game design framework that elevated the aesthetics of gameplay without sacrificing game mechanics.
From left: English (Watson Guptill 2012), Japanese (Born Digital Inc. 2014) and Korean (Freelec 2015) editions of Drawing Basics and Video Game Art (Watson Guptill 2012), and Interactive Stories and Video Game Art (CRC Press 2017) by Chris Solarski.
My second book, Interactive Stories and Video Game Art (ISaVGA), was the result of that retrospection. I’m very proud that ISaVGA has received positive endorsements from distinguished creatives including Hollywood director, Marc Forster, and veteran game production consultant and Assassin’s Creed founding member, Stéphane Assadourian:
"The concepts presented in Interactive Stories and Video Game Art remind me of Joseph Campbell's powerful demonstration of a universal storytelling structure. Coming from a technical game design background yet being artistically inclined, I never considered that atomic elements such as shapes, lines of movement and transitions contain a vocabulary strong enough to sculpt our mood, whether consciously or unconsciously. This book creates that wonderful feeling of learning new letters, which soon prove useful to form new sentences for bringing new poetry to the world."
— Stéphane Assadourian
The 2-part essay that you’re now reading provides a disruptive game design framework based on Interactive Stories and Video Game Art, with original content and refinements that have arisen since the book was published. In Part 1, readers gain in-depth insights into shape language and composition, tracing the lineage of these long-standing techniques from antiquity to contemporary mainstream culture, where they are translated to adaptive composition for games in Part 2. Adaptive composition is a high-level game design framework for heightening physical empathy and the sensory experience of players with a focus on art and narrative-driven games and transmedia. The aim of the framework is to provide greater creative control of the development process, challenge game design conventions, and illustrate the impact of traditional art concepts on the entire development team of game and narrative designers, artists, writers, programmers, musicians, game user researchers, etc. Practical exercises throughout the essay additionally provide design tools for averting ludonarrative dissonance, demonstrate the shortcomings of the magic circle metaphor, and common misconceptions regarding Marcel Duchamp's teachings on objects and meaning. This essay is therefore sure to be of interest for those that are already familiar with Drawing Basics and Video Game Art, and Interactive Stories and Video Game Art.
Whether you’re a developer of video games, VR, AR or XR, a filmmaker, game user researcher (UX), a gamification and transmedia expert, game theorist, psychologist, or simply interested in experiencing resonant and meaningful interactive stories—you’ll find the following insights will awaken a new appreciation for art and craft, and provide you with creative tools that can be actioned immediately to influence the user experience.
3. COMPOSITION FUNCTIONS
If you haven't done so already, it is highly recommended that you read Part 1 before continuing. Part 1 is an essential primer for the entire development team, including those working in fields like game design, game programming, games user research (GUR) and user experience design. Many concepts in Part 2 may be confusing without their grounding in Part 1.
In Part 1 of this essay we learned that shape language and composition dates back to antiquity, and instances in contemporary mainstream culture are widespread and affect creative disciplines in all corners of the world including, film, comics, architecture, photography and interior design. These insights allowed us to safely conclude that these steadfast techniques are indeed a universal language and circles, squares and triangles have an important affect on the audience’s emotions—whether covertly interweaved with cultural symbols (as in The Lord of the Rings) or used more explicitly (as in Pixar’s Up). Shape language even has the ability to override the meaning of cultural symbols—counter to the teachings of artist-philosopher, Marcel Duchamp. We even found shape language linked to sound and language with phenomena like the “bouba/kiki” effect. Now that we’ve firmly established the natural basis and artistic scope of these principles, it’s finally time to take a look at the nuts and bolts of shape language composition before tackling question (2): what practical value does shape language and composition have for game design in the context of art and narrative-driven games?
Shape language and composition constitute visual storytelling techniques that have been refined for thousands of years dating back to antiquity. The artistic medium to which they are applied may be static (painting, comics and illustration), dynamic (film and animation) or involve audience agency (Buddhist sculpture and interior design) but their benefits remain consistent: to convey functionality, set the aesthetic tone of the subject and reduce visual complexity of the artwork. This sections explores these benefits in more depth, giving us a practical understanding of shape language in the context of composition that we can later apply to game development.
Massacre of the Innocents (c. 1611) by Peter Paul Rubens. (Image source: Wikipedia)
Take a moment to appreciate the complexity and details of the Rubens painting, above. Now imagine setting such a complex arrangement of visual elements in motion, as in a typical video game. We go from a semi-engaged “spectator mode” with time to casually explore and contemplate cultural symbols, to a fully-engaged “agent mode” where our attention becomes absorbed by interaction and the functional requirements of gameplay. Without an underlying composition framework for organization, visual information can quickly become overwhelming. The elegance of composition is that it enables artists and designers to reduce complex artworks to concise visual statements with the following benefits:
- Visual simplification
- Visual direction
- Story structure
- Holistic design
A composition based on a primary shape creates visual simplification.
3.1 Visual Simplification
To illustrate visual simplification, let’s remove everything from Rubens’ painting except the triangle at the heart of the composition. You’ll find other composition lines hidden within the painting—such as the vignettes at the corners to keep the viewer’s eye contained with the picture frame (a technique similarly used in Monument Valley (2014) using subtle shadowing instead of lines)—but our focus here is the central triangular concept, illustrated above. Stare at the middle of the triangle for a few seconds. Your eyes will be tempted to jump to the triangle’s three points because each point functions like a visual exclamation mark. However, keeping a fixed gaze on the triangle’s center is fairly easy. A circular composition would make it even easier to remain fixated upon because of its edge-free silhouette.
An image that doesn’t follow a structure can become visually chaotic.
Now try the same exercise with the above composition—the result of randomized brush strokes in Photoshop. You’ll find that keeping your gaze fixed on a single element is near impossible because of the increased number of points to take in. The visual chaos is just too tempting for your attention to remain in one place—illustrating one of the major benefits of shape language and composition.
Composition is essentially an aesthetic framework for averting visual chaos by giving order and balance to the numerous elements of an artwork. Visual simplification covertly increases audience immersion and prolongs engagement by reducing complex visual narratives into simple visual statements. There is so much going on in an average video game that such techniques help avert attention bottleneck (overloading players with too much information).
3.2 Visual Direction
The second function of composition is to serve as a visual guide that covertly influences viewers on a subliminal level. Elements that fall along composition lines (as illustrated below) will receive more attention than elements outside the framework, even if the audience is unaware of the embedded shape language.
Elements that fall along composition lines attract the viewer’s
attention more than elements outside the framework.
3.3 Story Structure
Conveying story or subtext is the third benefit of composition. Rubens’ Massacre of the Innocents is based on a skewed, unbalanced triangular composition that perfectly underscores the brutality of the painting’s theme. The shape language of the composition—which should ideally remain beyond the audience’s perception—can dramatically alter the tone of the cultural symbols being depicted.
3.4 Holistic Design
The final benefit of composition concerns the development process at large, by encouraging a holistic design framework that prioritizes the unity of the entire artwork over details. A great example of this approach is illustrated by the Hellboy comic examples explored in the previous Shape Language and Composition in Mainstream Culture section (see Part 1).
3.5 Concealing the Composition
In the following composition analysis of Raphael’s Niccolini-Cowper Madonna we can see the techniques that classical artists used to imply primary shapes through composition using the deliberate placement of figures, gestures and silhouettes, drapery, shadow edges, and environment props. Every element has a function within the composition and has been manipulated to support the larger shape concept. Extraneous details have either been made to align with the composition, been subdued or removed altogether. The resulting composition lines serve to covertly direct viewers to key storytelling elements, such as Mary’s gaze towards Jesus.
Classical artists manipulated every element of a painting to covertly imply a primary shape, including figure gestures and silhouettes, drapery, shadow edges, and environment props. Niccolini-Cowper Madonna by Raphael.
Note that the triangle embedded in Raphael’s painting is incomplete and “broken.” This was a deliberate technique to conceal the composition framework so that viewers would not be distracted by the craft. A lay-person exploring a classical painting (or comic) would therefore believe that they're absorbed solely in the cultural symbolism contained within the image, exploring the artwork at their own volition, while being covertly directed by the composition and the aesthetics of shape language. Using both affordances to affect viewer’s covertly on a primal level (universal shape language) and conscious level (subjective cultural symbols) is what leads a classically trained contemporary artist and teacher like Dorian Iten to say that ”shape language is as valuable as visual grammar."
Although we’ve been analyzing shape language and composition in paintings up until now, it’s also important to reference composition in classical sculpture because of its relevance to 3D video games. Sculpture is where we find painting composition techniques projected into 3D space so that viewers can freely explore shape language in the round (360 degrees). The sculpture below of David by Gian Lorenzo Bernini (1598-1680) is among the finest examples of a three-dimensional composition with a shape concept that can be perceived from all sides.
[The application of shape language in Greco-Roman and Renaissance art goes deeper to an anatomical level, which you can learn more about in books like Robert Beverly Hale’s, A Master Class in Figure Drawing (Watson-Guptill 1991)].
Three views of David (c. 1624) by Gian Lorenzo Bernini.
We inherit all the above benefits of shape language and composition—visual simplification, visual direction, story structure and holistic design—when we adapt the techniques to game design. Adaption is straightforward if we consider the player’s constantly shifting viewpoint and the dynamic properties of the medium. What is essential is that these techniques are appreciated by the entire development team if they’re to be implemented effectively. The following sections tackle the answer that we’ve been deferring for question (2), concerning the practical value of shape language for game design in art and narrative-driven games—demonstrating why such design principles are not just a concern for visual artists and storytellers.
4. SHAPE LANGUAGE FOR GAME DESIGN
Shape language for game design relies on the same principles that we explored in the chair example (see Part 1), and the previous section on the benefits of composition. Firstly, key symbolic elements of the gameplay experience are identified before an appropriate shape language and composition structure is considered that will evoke the desired aesthetic themes. Game artists regularly use these techniques to manipulate every visual aspect of a video game like character and environment designs however, the same techniques are also applied to game design to convey gameplay affordance. So let’s explore a few examples of shape language in game art and game design, in the context of Raph Koster’s definition of games as “systems built to help us learn patterns for survival.” We also wish to further challenge the notion that psych is not amenable to math.
From left: Mario in his original pixel form, and a contemporary,
high-resolution version. Artwork by Nintendo.
4.1 Shape Language for Game Art and Narrative
Let’s begin by taking the earlier chair and Tony Montana examples (see Part 1), and applying them to Mario from Nintendo’s Super Mario franchise. Mario’s base symbolic elements are his mustache, overalls, cap, protruding nose, and shoes. Color, of course, also plays an important part in identifying Mario but our focus here is on shape language. Even if the stylistic representation of Mario is altered, the character will remain identifiable as long as its base symbolic elements remain intact—illustrated by the blocky pixel version of Mario alongside the modern, high-resolution version, above.
Manipulating the shape language of Mario’s base symbolic elements results in characters with different temperaments. From left: Mario, Luigi, Waluigi and Wario by Nintendo.
Manipulating Mario’s shape language results in a character with different temperaments—echoing the example of glass objects (see Part 1). In the line-up below we see that Mario’s base symbolic elements have been aligned to the square (Luigi) and the triangular (Waluigi and Wario). Luigi’s forms remain largely the same as Mario’s but his overall shape has been stretched upward—visually aligning him to Mario but with a pillar-like aesthetic. Ignoring proportion differences, Waluigi and Wario is where we see the strongest shape language changes to Mario’s base symbolic elements—such as the ears, nose, mustache and shoes—which have been aligned to the triangle. The results of these shape shifts are aggressive, alter ego versions of Mario and Luigi, where the symbolic ingredients of one iconic character have been manipulated with shape language to convey different temperaments.
Every visual aspect of a game—the environment, props, user-interactive, etc.—can all have their shape language manipulated while keeping symbolic elements intact to elicit varying aesthetic themes associated with primary shapes and their derivatives. Shape tuning need not be so overt, with subtle manipulations of shape language possible by varying a character’s outfit to change its overall silhouette (see examples from Framed Ink in Part 1), complemented by drapery and the arrangement of accessories.
Exercise: #AngryKirby Challenge
Kirby by Nintendo.
Using a pencil and paper, first make a faint copy of Kirby, and then begin adjustments to the character’s facial features and overall shape. Can you give Kirby an aggressive or angry appearance without introducing diagonal lines or shapes? If you do succeed, please share your artwork via the #AngryKirby hashtag. The result will likely shift Kirby’s temperament closer to Meta Knight—Kirby’s archenemy.
A character’s poses and gestures are also responsive to shape language. Note that Mario, Luigi, Waluigi and Wario have each been assigned a pose that echoes their shape language to underscore their respective temperaments. Mario’s jump aligns to a larger circle; Luigi stands steady and upright; Waluigi’s pose echoes depictions of the Vajrapani in Buddhist art (see Part 1); and Wario’s arms create a thunderous zigzagging line across his form. We could imagine adapting the Zen teachings of Suzuki Shōsan, described earlier, inviting audiences to adopt the pose of each character, such as:
Luigi is Mario’s dependable, taller brother and sidekick. He resonates stability and confidence which you can absorb from him in times of uncertainty.
Character shapes and poses aligned to primary shapes help players distinguish characters more easily as well as prime them physically and emotionally for certain behavior. Additionally, shapes and poses that adapt dynamically to gameplay can imbue characters with a greater sense of life and emotional depth, and convey their intentions through nonverbal body language—reducing the necessity for a user interface and character stats.
Note that adaptive character poses appear in games like the Resident Evil series, where the protagonist becomes physically impaired when injured. However, this has more to do with communicating the character’s health stats (much like a user-interface icon), than an emotional change.
Shape language reference for character races in Destiny (2014) by Bungie. Created by Bungie’s former Art Director, now Game Director, Christopher Barrett.
In practice, shapes needn’t perfectly align to one of the primary shapes, as illustrated by the concepts above for character races in Destiny (2014). The overarching goal of this essay and Interactive Stories and Video Game Art is to foster shape sensitivity and an understanding of how shape language influences player movement and emotions—irrespective of whether the shapes and feelings evoked can be accurately defined. You need only consider whether something is round-ish, square-ish, or angular and be aware of the aesthetic and functional affordances that come with these visual references.
4.2 Shape Language in Support of Gameplay
A common criticism against shape language is that it leads to overly simple visual narratives and stylistic representations. However, classical paintings and comics like Hellboy (see Part 1)feature shape language on many levels—from the overarching composition to the smallest anatomical detail—and nobody would argue that they are visually simplistic or present subtext in an overt, obvious manner. Shape language encompasses all stylizations, and can be used to convey complex narratives. In the context of gameplay this also means that we can use shape language to organize the fast-paced flow of visual information, and make it easier for players to read a gameplay environment for functional and aesthetic affordances. This is where shape language has a clear advantage over cultural symbols.
Shape language is used in games with idealized as well as realistic style. From left: Super Mario Galaxy 2 (2010) by Nintendo, and Uncharted 4: A Thief's End (2016) by Naughty Dog.
For instance, the idealized style of Super Mario Galaxy 2 (2010), by Nintendo (above left), features characters and environment props that align to either the circle, square or triangle depending on their affordance: friendly non-player characters (NPCs) and bonus objects like warp pipes and mushrooms are generally round; jumping platforms are square; and enemies are triangular (albeit with softened tips to align with the family-friendly atmosphere of the series). The affordance of gameplay elements becomes easier to comprehend when pure shape language is used and players mustn’t second-guess whether, say, an NPC is friendly or will inflict damage.
Shape language illustration conveying gameplay affordance, by Emilia Schatz for 80.lv.
Games with realistic stylizations, like Uncharted 4: A Thief's End by Naughty Dog (above-right) use shape language for gameplay in much the same way as the Super Mario series, but their implementation is harder to discern at a glance. In an article for 80.lv titled, Defining Environment Language for Video Games, Emilia Schatz, lead level designer at Naughty Dog, summarizes the functional affordances of shape language for gameplay as follows:
“Round shapes evoke safety and well-being. They’re viewed as non-threatening due to their lack of points and edges. They blend in and disappear, lacking affordance. Rectangular shapes on the other hand are solid, stationary, and powerful. They evoke a feeling of strength, either as an obstacle to overcome, or something to be reliably used. Rectangular shapes are strong indicators of affordance. Finally, diagonal and pointy shapes evoke a sense of danger and imbalance.”
— Emilia Schatz
Shape language in Uncharted 4: A Thief’s End, by Naughty Dog. Illustration by Emilia Schatz for 80.lv.
In the above illustration you can see that Emilia’s overlays indicate how shape language was used in support of gameplay for Uncharted 4: A Thief’s End. By keeping the environment’s shape language as distinct as possible, Emilia ensured that the functionality and location of climbable objects remains easy to identify. Non-interactive objects—those that align to circles and triangles—are visually subdued or avoided. The visual distinction of the circle, square and triangle is why shape sorting puzzles for kids likewise feature the primary shapes, because they help in the development of visual perception.
MORF (2011) by SOLARSKI STUDIO.
Shape language can also be used as an illusion for narrative purposes, without necessarily affecting gameplay. MORF (2011) is a simple browser-based game developed to explore the aesthetic links between character and environment shapes, which you can play here. Players must guide a round character through two functionally identical environments that have differing aesthetic themes. I had the opportunity of presenting MORF at an art exhibition that included an audience of experienced players and individuals that had never played a game before. Players would bump their way through the first, circular level without incident, but when they reached the second, edgy level (pictured above) and accidently landed their character on triangles, they’d exclaim words like “ouch!”—words that we use when we hurt ourselves in reality. This suggests that interactions between playable characters and the environment need not always have gameplay value, when an aesthetic response is desired.
The shape spectrum below is a valuable comparative analysis tool to assess the aesthetic alignment of characters, and the gameplay affordances and emotional themes that players will likely attribute to their design. Characters that align to the circle tend to be friendly, caring, cheerful, dynamic and “jumpy”; square characters tend to be diplomats, wise, serious or heavy “tanks”; and triangular characters usually indicate a bad temperament, anguish or evil intentions however, triangles can also indicate a friendly character with a “zippy” personality. These aesthetic associations reference the tactile senses that we explored via the glass sphere (marble), cube (paperweight) and pyramid (spearhead) example in the opening section of this essay (see Part 1). The shape to which a
character aligns may be presented explicitly, in the character’s silhouette, or implied via discreet physical and costume details like the vertical stripes on Dr. Liara T'Soni’s uniform. Naturally, the shape spectrum can be used to assess a single playable character across a range of emotional states, a cast of characters and non-player characters.
The shape spectrum is used as a comparative analysis tool for assessing the perceived gameplay functionality and temperament of characters. Characters from left (incl. copyright owner): Sora (Square Enix), Elizabeth (Irrational Games), Isabelle (Nintendo), Dr. Liara T'Soni (BioWare), Mei (Blizzard), Adam Jensen (Eidos Montréal), Tracer (Blizzard), Bowser (Nintendo), Bubble Head Nurse (Konami).
The shape spectrum can be applied to every element of a game system—as we’ll see when we begin exploring the adaptive composition framework—which is why the high-level tool has important implications for game designers and game programmers who can use these concepts to implement game systems that aesthetically respond to gameplay and narrative contexts on the fly, in overt or subtle ways.
Players can, of course, be conditioned to understand that a triangular character is calm and friendly within the confines of a game’s system. Although such conditioning must be treated with care, since the game may feel unfair or suffer from usability issues if interaction outcomes don’t match with player expectations. In the words of Emilia Schatz:
“If you feel your visual language is not reading as clearly as you had hoped, adjusting your primitive [primary] shapes may be one way to help.”
— Emilia Schatz
Shape language for gameplay requires careful balancing, which is a task that is further complicated when cultural symbols must be considered. The shape of the playable character in Journey (2012), often referred to as the “traveler,” supports gameplay because it references wind kites—a symbol that is familiar to most cultures around the world. Familiarity with kites and how they function, means that the traveler’s visual references make the flying and gliding aspects of Journey’s gameplay easier for players to understand. While the traveler has a pointy shape, its gentle temperament is predominantly conveyed through its movements, which we’ll explore shortly in Shape Language for Interactive Motion lines. If Journey featured a larger cast of characters, then the development team may have encountered instances of players second-guessing whether the traveler is a friend or foe. In which case, the traveler could have been given rounder, Mario-esque forms while maintaining the kite reference but that would have had knock-on effects to all other visual elements of the game—not to mention, aligning the character’s visual theme with round forms, which would likely be at odds with the team's artistic vision. These considerations reference the design techniques as we explored with the Trinity, and shape language and cultural symbols in Christian paintings (see Part 1).
The traveler in Journey (2012), by thatgamecompany (left), uses shape language and the cultural symbol of a kite (right) to convey character and gameplay functionality.
Our exploration of shape language started with the Ancient Greek philosophers and their tactile associations with our three classic elements: Water, Earth and Fire (see Part 1). We found that shape language is a universal concept because it pervades Nature and is the result of physical laws. It’s no consequence, therefore, that we find shape language used in everything from engineering to art and design, from East to West, including typography, vehicle design, figurative art, architecture, interior design, furniture, comics, film and animation. The examples in this section demonstrated that shape language is equally important for video games to convey the aesthetic and functional affordances of gameplay elements like characters and environment objects (see shape spectrum earlier), for stylizations that range from from abstract and idealized (Super Mario Galaxy series) to realistic (Uncharted series). We can therefore conclude that shape language does indeed support Raph Koster’s definition of games as “systems built to help us learn patterns for survival.” Perhaps, more importantly, the math that defines primary shapes has aesthetic and functional value for the user experience with significant implications on game design and game programming.
Now that we have some answers to question (2) with a preparatory understanding of shape language in support of gameplay, we can go ahead and explore composition in a wider gameplay context—considering the users experience in its entirety via a disruptive game design framework inspired by traditional composition techniques.
5. THE ADAPTIVE COMPOSITION FRAMEWORK
Composition is nothing more than the act of combining parts or elements to form a whole. In the context of art, modern day master, Nathan Fowkes, describes composition as the “thoughtful arrangement of pictorial elements to best portray subject matter.” Composition offers several benefits that include visual simplification, visual direction, layering of story or subtext, and a holistic design framework (see Composition Functions earlier).
Laying a classical painting down flat and imagining the composition framework as navigable pathways gives us an instant solution for translating visual concepts to interactive environments.
Translating composition techniques to games is fairly straightforward despite the unique way in which the medium is experienced. We need only lay a classical painting down flat and imagine the composition framework as navigable pathways to get a feeling for the possibilities. The lines that viewers would implicitly detect with their eyes when looking at a classical painting or comic, now become pathways along which players can traverse through a three-dimensional environment.
Adaptive composition is the subject of the first half of Interactive Stories and Video Game Art (where it’s referred to as dynamic composition). Adaptive composition is a high-level framework that implies that the composition can respond dynamically to gameplay and narrative context—creating heightened physical empathy and sensory experiences. As you will recall from previous sections, the functional elements used to embed a composition are little more than lines, shapes and volumes that informed the design and placement of cultural symbols. In the context of game design, you can think of adaptive composition as the fundamental elements needed for the whiteboxing stage of development—the preliminary prototype stage where key gameplay elements are blocked-in with placeholders to test if the game’s basic design supports the desired experience before final assets are created. It’s upon this foundation of adaptive composition that layers of detail will be added (character design, light, color, voice acting, animation, etc.) by the various disciplines involved in the development process.
The visual elements that are used to embed shapes in compositions across disciplines like painting, photography, film and comics (the deliberate placement of figures, gestures and silhouettes, drapery, shadow edges, and environment props) are translated to the boldest elements we can find in a video game space. Elements must be bold to account for interactivity and multiple perspectives. This does not mean that the elements used in other media cannot be applied to games, but they’re often too small and subtle to be noticed in the midst of gameplay, and should therefore be treated as secondary.
Visual elements used to indicate shapes in painting or comics compositions translate to bold elements (i.e. character and environment shapes, and pathways) in the context of games to account for interactivity and multiple perspectives.
Unlike traditional art forms, interactivity means that player actions are bound to the visual and artistic experience and must also be considered in the context of composition, even though they occur outside the picture frame. An important consideration for game design is that each element of adaptive composition is pervasive—meaning that it is consistently present during gameplay—to ensure that players are always aware of it despite a constantly shifting viewpoint within the game world. To get the most out of dynamic composition it’s also important to treat each element as adaptive—tuning the elements along the shape spectrum to amplify or subvert the aesthetics in relation to gameplay and/or narrative. The eight elements of dynamic composition are as follows:
- Interactive Motion Lines (formerly, Lines of Movement)
- Context Actions (formerly, Player Gestures)
- Environment Shapes
- Rules and Mechanics (formerly, Dialogue)
- Character Shapes and Poses
The goal of these eight elements, used in conjunction with shape language, is to arrive at the same level of visual clarity and covert storytelling as we’ve explored in traditional art and contemporary media like film and comics (see Part 1).
More importantly, adaptive composition focuses on aspects of a game system that promote and shape player movement— the orchestration of which is more effective when rules and game mechanics are a secondary consideration in support of these aesthetic goals.
From left: the static, vertical composition lines in The Baptism of Christ (c.1450) by Piero della Francesca are echoed in the virtual 3D forests of Firewatch (2016) by Campo Santo.
Using adaptive composition, the game screen can be crafted to resonate with round friendliness (i.e. Animal Crossing series), the poised stability of vertical straight and horizontal lines (i.e. Superbrothers: Sword & Sworcery EP), or vibrate with triangular aggression (i.e. Devil May Cry series), which must be experienced in motion through interactivity for the full aesthetic effect. Be sure to see the later references to ABZÛ in this context.
What makes identifying the eight elements of adaptive composition possible is the ubiquity of shape language, without which we would have to contend with an infinitely complex Duchamp-ist universe. Instead we rely on 3000+ years of humanity’s direct study of Nature, which evolved into steadfast methodologies that transcend engineering, art and design fields in real and virtual contexts. Shape language and cultural symbols are not mutually exclusive but prioritizing shape language over Duchamp's teachings gives us a manageable framework that every player and individual on the development team can intuitively appreciate irrespective of cultural background and expertise.
The adaptive composition framework illustrated with reference to Jesper Juul’s puzzle piece metaphor.
5.1 The Magic Circle and Defining the User Experience
The eight elements of the adaptive composition framework are visually presented above in the form of an octagonal puzzle (inspired by Jesper Juul’s puzzle piece metaphor), with each element serving as a sensory input to the player located at the center. The octagonal puzzle is set within a square that represents the game boundary—allowing the high-level framework to be tessellated with reality and transmedia, like board games, location-based games, films and comics. Interactive motion lines have been emphasized because of the element’s higher importance. The aim here is to make it easier to identify how component parts of games (sensory inputs) influence the player’s physical movements (physical outputs) and emotions.
Illustration of the magic circle metaphor that defines play in binary terms.
(Image source: Gaming Conceptz)
The overarching concept highlights the shortcomings of the popular magic circle metaphor, which defines play in binary terms—you’re either acting within a space designated for play, or performing the same actions in a real-world, non-play context—and doesn’t account for the rich, fluidly interconnecting boundaries that games share with transmedia and reality. As we’ve found throughout this essay, games have indisputable connections with creative disciplines as varied as film, comics, board games, virtual reality (VR), augmented reality (AR), cross reality (XR), theatre, architecture, etc. The adaptive composition framework can be applied to all these art forms, requiring only minor conceptual adjustments to fit each medium. For instance, character shapes in location-based games are a combination of the player's real world physique and attire, and their in-game avatar [see the following analysis of a location-based game titled, Crossover Strategies].
Unlike the magic circle, which players simply enter or leave, the adaptive composition framework, presented as a puzzle piece, highlights that gameplay consists of elements (sensory inputs) that players must actively construct by bridging the boundary between the game system, reality and transmedia. The puzzle is complete only once these connections are made.
We’ve already encountered a similar observation in Raph Koster’s quote, in which Raph states that the essential considerations for making long-lasting games (human psychology, etc.) are “elements that arise from outside the game’s rules and from outside the ‘magic circle.’” [Italics added for emphasis]
We've already explored two elements of adaptive composition— character shapes and poses, and environment shapes—in some depth throughout Shape Language for Game Development, earlier. We'll briefly explore two additional elements—audio, and interactive motion lines—to demonstrate the versatility of shape language and the adaptive composition framework.
5.2 Shape Language and In-Game Music
Audio in the context of adaptive composition focuses on sound that supports gameplay—when players are in control of a video game character. Audio is unquestionably a nonvisual medium, which makes it difficult to imagine how it fits with shape language. To solve this issue, we can look to the visual manifestation of music that we know as dancing—the act of physically turning something that is intangible (sound) and making it tangible (physical movement). Music has such a primal influence on us that it can trigger and choreograph physical activity on a subconscious level. For instance, we’ve all had the experience of listening to a song and realizing that our foot has been tapping the music’s rhythm without our full awareness. When we dance our whole body becomes instinctively engaged in the act of translating sound into form and movement (although some people are more naturally gifted than others!).
There are several components of audio to consider in support of gameplay but we’ll focus here on melodic contours—the shape that notes and pitches describe across a musical score. Melodic contours fall into three primary categories that echo our shape spectrum: curved (sinusoidal), square and triangular (illustrated above). As with character shapes and poses, melodic contours can pull on our kinesthetic strings when we play (or dance) to evoke corresponding physical gestures. Adaptive gameplay music, therefore, serves to cue players to an appropriate mode of interaction. The player’s gestures represent an asynchronous interpretation of music that is strongly influenced by the aesthetics of melody (and tone, rhythm and tempo) but not coordinated with the same precision as in dance. To the benefit of game designers, music can suggest more nuanced movements than game design constraints may allow.
The work of game audio designers echoes that of silent film accompanists, whose job it is to musically portray the protagonist’s movements and amplify actions with sound effects.
While we can aptly relate player activity to that of a dancer responding to music, the role of game audio designers has a clear resemblance to silent film accompanists—like those that played to Buster Keaton (1895–1966) and Charlie Chaplin (1889–1977) films. Audio designers working in video games face the same challenges of portraying the playable character’s movements through music and amplifying actions with sound effects. A composer must work with animators, game designers, and scriptwriters to ascertain in advance the playable character’s varied emotional states, stride tempo, and abilities to determine the music that will accompany players as they experience the game’s narrative—much like the accompanist deciding which song will best fit the action of a silent film.
Exercise: Drawing Melodic Contours
Music clips from The Legend of Zelda: Ocarina of Time (1998) soundtrack by Koji Kondo. Copyright by Nintendo.
Listen to the above three excerpts from the The Legend of Zelda: Ocarina of Time (1998) soundtrack with a pencil and paper in hand. While listening, draw lines that correspond to the melodic contours that you perceive in-between the beat. You can go back and forth over the pencil lines, as the drawing itself is not important—only the physical experience of drawing, which may feel as if you are dancing across the page. Pay particular attention to the sensations in your drawing arm and how they change between music clips. Do the melodic contours evoke movements that you could imagine as analog stick inputs on a controller, relating to character movements in leisurely locations, spiritual sanctuaries and enemy dungeons in the Ocarina of Time? Would you use any of the following adjectives to describe your lines: cheerful, relaxed, respectful, tranquil, zippy, edgy or aggressive? How about button presses? How frequently and with what intensity would you imagine pressing action buttons? Are these sensations evoked by the music’s beat or other musical elements?
The accompanist’s spontaneous accentuations and impromptu notes that underline special events are equivalent to ingame sound effects that are designed by audio teams but activated spontaneously by players during gameplay as their character “dances” across the screen. As David Kanaga, the composer behind Proteus (2013), and Panoramical (2015), puts it:
"There's all these really detailed time structures happening in games. So when I'm doing music I try to think, ok, the game already has time structures in place, and the goal of the music oughta be to just hug that as tightly as possible."
— David Kanaga
5.3 Shape Language and Interactive Motion Lines
Interactive motion lines in the context of adaptive composition are not concerned with “local” animations, like the movements of a character’s hand or facial expressions, because such animations are often too subtle to be noticed in the midst of action. Our primary concern is for broader animations that remain readable at every resolution. Motion lines are the dynamic, in-game equivalent of lines used to compose paintings like Niccolini-Cowper Madonna by Raphael, and Hellboy comics by Mike Mignola.
A character’s interactive motion lines are the backbone of adaptive composition around which all other elements should be orchestrated. A video game character’s motion lines define the player’s primary means of expression in virtual space via the game’s input device, which translates tactile, real-world gestures into nonmaterial, virtual art. Motion lines are generated spontaneously by the player’s physical outputs but constrained to the aesthetics defined by the development team for the playable character.
Frame from Ballet Rotoscope (2011), by Masahiko Sato and EUPHRATES.
The image above is a frame from an experimental video titled Ballet Rotoscope (2011) by Masahiko Sato and EUPHRATES, which investigates computer-generated geometry. The experiment nicely illustrates the concept of interactive motion lines using live footage. If there’s only one exercise that you try, be sure to do the following Motion Lines Study, Part 1 exercise, which consists of drawing the interactive motion lines experienced by players of Journey, INSIDE (2016), and Vanquish (2010).
Exercise: Motion Lines Study, Part 1
Using a pencil and paper, watch each of the above gameplay clips and draw the motion lines that you perceive for each character. You can think of motion lines as the trails created by Light Cycles in Tron (1982), even though such trails are generally not visualized. The focus of the exercise is on the physical sensations of drawing, which are directly linked to the physical movements of a player controlling the character with an analog stick. How do the signature motion lines for each character aesthetically differ between gameplay clips? Do the physical sensations you feel correspond to shape language? Naturally, the concept of motion lines also applies to NPCs.
The aesthetic quality of motion lines can be subtly modulated by adjusting line tension, the character’s speed, and continuity (starts and stops) of motion.
Gameplay in INSIDE (2016), by Playdead, presented in a comic page format.
The exercise of drawing motion lines can be used as a method to visualize narrative as a continuous thread that binds the various gameplay sequences into a coherent whole with a focus on the player’s physical gestures, as illustrated by INSIDE, above. The horizontal gameplay section (top row), evokes nervous back-and-fourth movement as the player evades the guards. In the vertical gameplay section (right
column), the player’s physical outputs are linked to the submarine’s spotlight, which is responsive and “twitchy” compared to the flowing movements of the submarine itself (yellow line). Note that the overlays below have a strong conceptual link to the composition lines in Hellboy comics (see Part 1), and that tracking interactive motion lines is applicable to all game viewpoints, including 2D, 3D and 4D (see Miegakure by Marc Ten Bosch) views.
StarCraft II: Heart of the Swarm (2013) by Blizzard Entertainment. Original video by RedAlert2Havoc.
Real-time strategy (RTS) games like Blizzard’s StarCraft heavily rely on shape language to convey functionality of game units. In the above video clip from StarCraft II: Heart of the Swarm (2013) you’ll see that resource gathering units move smoothly to and fro, while attack units move erratically with jagged motion lines. In all the visual noise, these distinct movements allow players to discern unit types peripherally—with jagged movements attracting more visual attention—without having to focus directly on a given unit. This effect is achieved principally because “collisions” are switched off for workers, and their animations speed up and slow down smoothly. In contrast, acceleration and deceleration of attack units is abrupt, and they have their collisions switched on, and must therefore make additional movements to avoid objects and each other. It doesn't matter whether the movements were intentionally composed or are a byproduct of game design. What's important is appreciation that game design and the aesthetics of motion lines are not mutually exclusive (especially for art and narrative-driven games).
ABZÛ (2016) by Giant Squid.
The masterfully orchestrated motion lines and shape language in ABZÛ (2016) also deserves in-depth analysis—such as the aesthetic differences between free exploration, current riding, and careful circumvention of exploding mines as well as the beautiful, natural shape language of fish shoals, as seen below.
What Journey, INSIDE, Vanquish and ABZÛ demonstrate is that the delivery of an interactive story and life-like characters can only ever be as fluid as the motion lines.
5.4 Interactive Motion Lines and Ludonarrative Dissonance
Motion lines may seem like an obvious consideration for game developers but implementing them correctly to heighten the player’s physical empathy towards a life-like playable character may fail for one of two reasons.
The first reason occurs when the development team loses sight of motion lines—prioritizing other, less essential aspects of game experience. This can easily happen, especially when huge development budgets and team sizes are involved. Take, for instance, the gameplay sequence below from Halo 4 (2012). Using a pencil and paper, draw the interactive motion lines of Master Chief, while the playable character pilots a Ghost vehicle. You’ll find that the sequence is very linear, evoking little lateral movement. The few turns that Master Chief performs are gentle because of the Ghost’s slow turning speed.
Halo 4 (2012) by 343 Industries. Original video by MythicTyrant.
The above gameplay sequence represents one of the dramatic highpoints of Halo 4’s narrative (the player has just unwittingly released the Didact and the universe is in peril), so the physical experience should be closer to Vanquish if players are to feel like they’re empathizing with a life-like Master Chief. Instead there’s a decided lack of tension that is at odds with the situation despite so much eye-candy and sound effects in the sequence. Academically this is known as ludonarrative dissonance, which is when the user experience is disconnected from the narrative.
Averting ludonarrative dissonance is a key benefit of adaptive composition because the high-level framework helps to manage the overwhelming complexity inherent in game development with an emphasis on motion lines and how they affect player gestures and emotions. Wherever player-character empathy is central to the artistic experience, game mechanics should be designed in support of motion lines.
The second reason that motion lines may not match the intended play experience is due to platform limitations, not developer oversight. For instance, ludonarrative dissonance can occur in Proteus if players come to the game in an agitated state of mind. Proteus is a meditative experience with free-form exploration and no specific goals for players to accomplish. The gently billowing plants and slow movement of the disembodied character sets the pace at which players can explore the environment. Nothing on-screen encourages players to make erratic, jagged gestures—like shaking the camera back and forth—but the game’s mouse and keyboard input means that the character’s pace is developer-defined but the camera isn’t. You can purchase Proteus here and try it yourself.
The mouse and keyboard controls of Proteus (2013) by David Kanaga allow players to make gestures that may not be inline with the author’s intended experience.
It's true that most individuals playing Proteus will come to the game with the right mindset but it's nonetheless important for designers to carefully curate the aesthetics of each element of adaptive composition and assess platform limitations to ensure that the potential for ludonarrative dissonance is reduced. Otherwise players may get the sense that their virtual head belongs to a Quake III Arena (1999) contest, while their pace references a character from Animal Crossing (2001). In the case of mouse and keyboard interfaces, curating motion lines is, admittedly, more difficult than with console controllers.
5.5 Formal Game Design: Single and Binary Control Schemes
Where the adaptive composition framework aspires to have the greatest disruptive effect on game design concerns interactive motion lines in formal gameplay. Formal gameplay is here defined as gameplay with explicit objectives and consistent rules that players must master to win. Such games are identified by their heavy UIs and end-of-level stat screens, which are hallmarks of pen-and-paper role-playing game (RPG) that enumerate the player’s progress towards a gameplay objective. Playable characters in formal games are aesthetically constrained to a very limited range of motion lines because their control schemes are bound to an intricate system of interactions.
For example, Chess is a traditional game in the formal gameplay category. Maintaining the angular, V-shaped move permitted by knight pieces is essential for the development of gameplay strategies. Players are not interested in empathizing with the knight, so switching its motion lines to an aesthetically gentle arc mid-game would be an intrusive feature—diluting the objective of Chess and making the game too complex to play.
New Super Mario Bros. Wii (2009) by Nintendo. Original video by Sirloin.
The same is true of formal games like New Super Mario Bros. Wii (2009) by Nintendo (above), where Mario’s jump arcs—the character’s signature ability—align to the circle shape concept. Mario’s character controls must remain consistent, so that designers can structure gameplay in a predictable manner and ensure the game’s intricate system of interactive possibilities doesn’t break.
Changing the aesthetics of a character’s interactive motion lines can have detrimental consequences for the game system. Characters from left: Mario by Nintendo, and Ninja Gaiden’s Ryu Hayabusa by Tecmo.
Dynamically altering Mario’s friendly, up-beat jump arcs to an aggressive and vengeful aesthetic would exponentially increase the complexity of balancing the game’s system—affecting Mario’s ability to successfully navigate the environment. As in Chess, the player’s mastery of New Super Mario Bros. Wii’s game system is the experiential goal, not emotional empathy, and Nintendo does a better job of creating exciting challenges for players to master than most other developers.
Super Smash Bros. Wii U (2014) by Nintendo. Original video by WiiUGamer12.
If players wish to experience Mario with more aggressive, Vanquish-type movements they must play a game like Super Smash Bros. Wii U (2014), featured above. Combing the motion lines of New Super Mario Bros. Wii with Super Smash Bros. Wii U into one game would be a surefire way to create a broken or unmanageable game system from a designer’s perspective.
The desert “gameplay vignette” in Uncharted 3: Drake’s Deception (2011), by Naughty Dog, allowed the game’s designers to insert a control scheme with a different aesthetic to the core gameplay.
Despite their narrative inclinations and a definite wish to elicit player-character empathy, series like Uncharted and God of War are fundamentally formal games abiding to the same creative constraints as Chess and the Mario series. To create a sense of aesthetic variety, such games are often designed with binary character control schemes: a fast and responsive control scheme for combat, and another, more ponderous control scheme for puzzle-exploration sequences. These two schemes reflect interaction modes—not the playable character’s emotional states—and are conveyed pictorially (battle axe at the ready = combat mode). Character development is largely passed on to non-interactive cutscenes, and gameplay vignettes—like the desert sequence in Uncharted 3, above, which is set aside from core gameplay.
Characters in series like God of War (2018), by Santa Monica Studio, feature single or binary control schemes—especially evident in the transitions between battle and post-battle aesthetics. Original video by MKIceAndFire.
The single or binary character controls of such games are most noticeable at the tail end of combat where we would expect to find motion lines that reflect the emotional ups and downs of a real person. First, try to imagine what it would feel like to engage in life-threatening combat and the range of emotions—from anger and heavy regret to euphoria and buoyant relief—you’d experience during and after battle. Now replay a combat sequence in a game like God of War (2018), applying the earlier Motion Lines Study, Part 1 exercise. Ignore automated actions that players trigger but don’t have direct control over, like local animations for spins or attack moves. Where we’d expect to experience nuanced motion lines that realistically model the dynamic physical tension of a real person on a heroic quest we, instead, tend to find only single or binary aesthetic control schemes. The same is true of the Uncharted series, where Nathan Drake will plough through countless enemies and then instantly revert to a puzzle-exploration mode replete with sarcastic wit the moment a bloody battle has concluded.
It’s important to stress that this is not developer oversight. Single and binary character control schemes simply highlight that emotional empathy is not the main goal of gameplay in such titles. Sophisticated combat systems, environment puzzles and levelling-up are central to the experience, which the developers achieve exceptionally well.
5.6 Informal Game Design and Gameplay Vignettes
Gameplay vignettes like Uncharted 3’s desert sequence are classified under informal gameplay, which prioritizes freeform play or pure aesthetic experience. An incredible amount of work goes into concealing the intersections between formal and informal gameplay sequences of such titles. However, the aesthetic constraints dictated by formal game design principles, derived from pen-and-paper RPGs, create a misleading hurdle that prevents the motion lines during combat and puzzle-exploration from being explored with the same emotional range as informal gameplay and cutscenes. This is the essence of what Raph Koster was referring to by writing that “psych tends not to be that amenable to math.” What is needed is the introduction of adaptive composition to complement the stat-heavy gameplay inherent in formal game design—with shape language taking the role of the traditional gamemaster who verbally weaves an aesthetic accent in-between each roll of the player’s dice.
“The Decent” in Journey, by that thatgamecompany, is a masterpiece of adaptive game design aesthetics, evoking motion lines with various “textures.” Original video by GadgetGirlKylie.
Games that entirely forgo formal game design principles (often described as “interactive art” or “art games”) have greater freedom to modulate interactive motion lines to evoke a wide spectrum of physical experiences and sophisticated in-game narratives. There is a middle ground, however, between formal and informal game design, which is illustrated by groundbreaking games like Journey, which utilize all the elements of adaptive composition to evoke a poetic sequence of physical gestures, as illustrated by “The Decent” gameplay sequence above. Compare the interactive motion lines in The Decent to the earlier God of War gameplay sequence. Which character—Journey’s traveler or God of War’s Kratos—has a wider range of interactive “textures” and the feeling of emotional depth? Naturally, short play times make it easier for designers to create games composed entirely of unique gameplay vignettes with aesthetically varied motion lines—a feature that makes games like The Beginner’s Guide (2015) and That Dragon, Cancer (2016) all the richer.
That Dragon, Cancer by Numinous Games is based on Amy and Ryan Green’s autobiographical experience of raising their son, Joel, who was diagnosed with terminal cancer at the age of twelve months. As discussed during an IGDA event on Death and Consequence in Game Design, the developer’s of That Dragon, Cancer wanted to convey that such tragic situations can also contain moments of positivity—much like a tragicomedy that offers humor to offset an emotionally oppressive situation. For instance, thinking of a loved one can trigger the memory of a funny situation. The game’s designer, Josh Larson, also highlighted that people deal with death in different ways. Players of That Dragon, Cancer find that Amy remains doggedly optimistic while Ryan’s tendency is to confront the facts—often withdrawing into an empty space of contemplation.
That Dragon, Cancer (2016), by Numinous Games, presents players with a sequence of varied gameplay vignettes that explore every facet of dealing with death—from moments of optimism and joy (left) to withdrawal and sorrow (right).
Conveying such multifaceted experience of death would not be possible if Josh and the team had focused on one core gameplay mechanic—as is typical in most video games. That Dragon, Cancer is therefore composed of a sequence of gameplay vignettes that explore the emotionally complex nature of losing a loved one. Fleeting moments of happiness, triumph and optimism are contrasted with moments of sorrow and reflection using gameplay mechanics to convey each aesthetic. For example, a Mario Kart-style driving sequence (above left) elicits joyous activity from players but is contrasted by moments of relative stillness when players are tasked with exploring a desolate hospital or find Ryan adrift in an emotional abyss (above right). Each gameplay vignette works in combination with the whole to create a poignant reflection on death.
Since motion lines can be modulated along the shape spectrum, we can consider implementing increasingly nuanced adaptive character control schemes that respond aesthetically to narrative context and player actions. Established visual feedback techniques, used to pictorially convey interaction modes, can be used with new emphasis on the character's emotional state.
Single and binary control schemes can result in an interactive form of uncanny valley that stifles the illusion of lifelike characters in AAA narrative-driven games—caused by the grating of high-end graphics, motion capture and voice acting against aesthetically flat in-game controls. Where physical empathy and emotional depth is the goal, math based systems would be more effective if they also served adaptive shape language, not just RPG-type stats, to increase the tactile believability and emotional depth of playable characters.
Shape sensitivity is critical for developing a sophisticated command of interactive motion lines—gaming’s unique artistic ingredient—as well as bridging the gap between formal and informal game design. The adaptive composition framework gives us conceptual tools to translate traditional composition to the interactive medium of games, where the player is an integral component of the artwork. Conceptualizing the adaptive composition framework in the form of Jesper Juul’s puzzle piece metaphor highlights gaming’s aesthetic connections to reality and art forms across transmedia, with shape language’s sensory inputs serving to define the user experience. We saw that adaptive in-game music can also be aligned to the shape spectrum (see Exercise: Drawing Melodic Contours), and that interactive motion lines (see Exercise: Motion Lines Study, Part 1) are the backbone of interactive experiences, which enables players to physically empathize with life-like virtual characters and embody the video game artwork.
The aesthetic and functional appreciation of shape language and composition that we’ve accumulated over the course of this essay gives us a solid foundation to tackle the ever-persistent debate: are games art?
6. GAMEPLAY IS ART
With so much historic material to support gaming’s connection to art, it’s a wonder that the “are games art?” debate still continues. To help settle the argument it is worthwhile likening gameplay and interaction to Abstract Expressionism—a Modern Art movement that developed in the 1940s to explore spontaneous and automatic creative methods. Abstract expressionist figures like Jackson Pollock (1912-1956), shifted emphasis away from the art object (i.e. a painting) towards the artist and the physical aspects of creation.
In the below-left photo by Hans Namuth (1915-1990) you can see Pollock using his iconic drip painting technique that allowed the artist to make spontaneous creative decisions using gestures that wouldn’t look out of place in a Nintendo Wii commercial. The accompanying painting from 1950 titled, Autumn Rhythm (Number 30), illustrates the results of Pollock’s technique—consisting of lines of varying aesthetic quality that were a direct response to a myriad of spontaneous decisions, including the variable amount of paint on the brush, the flow of dripping paint, existing lines on the canvas, tension in the body, random thoughts, etc.
From left: Jackson Pollock photographed in the act of painting (c. 1950) by Hans Namuth, and the result of his drip painting technique in a painting titled, Autumn Rhythm (Number 30). (Image source: Wikipedia)
Aside from performing arts, this method of working was unprecedented for its pure visual depiction of the artist’s movements and creative process. This unity of aesthetics between the artist, the act of painting and the artwork gives us vital conceptual tools to address the games-art question. In a philosophical treatise on game art titled, On simulation, aesthetics and play: Artifactual Playground (2012), philosopher and game designer, Douglas Stanley, presents a common line of argument against games as art, which contends that visuals in art and games are the result of two irreconcilable intents. Visual artworks like paintings are created solely to serve as aesthetic objects, while art in games is a byproduct of rules and game mechanics designed to mimic real-life experiences, which Douglas terms “simulated metaphors.” Take, for instance, the simulated metaphor of war that underlies a game of Chess, and the visual byproduct of gameplay: A Chess board with pieces arranged in various combinations.
Douglas goes on to compare the artwork of Piet Mondrian (1872-1944) and the arcade classic, QIX (1981), pictured below—highlighting their similarity in visual aesthetics but differing creative intents. The argument is that, unlike Mondrian’s painting, QIX is designed to simulate fast-paced competition over territory against a computer-controlled enemy. The visual similarities between Mondrian’s art style and QIX are therefore incidental because QIX’s visuals are the byproduct of a simulated metaphor—not a singular consideration for shape language, composition, color and value. Such conclusions are bewildering in a traditional art context because, it should go without saying, that metaphors can be used to mimic every aspect of humanity.
From left: Broadway Boogie Woogie (c. 1942) by Mondrian, and QUIX by Taito.
The simulated metaphor of Tennis for Two (1958)—the first video game ever created—is a faithful representation of tennis, despite the game’s primitive graphics. So why not consider the possibility of a game that simulates an artist’s creative process—seeing that games champion active participation? Aren’t all art forms an abstraction of reality that the artist must simulate? In the case of Mondrian this is certainly the case, since the artist received a traditional art training before evolving his iconic style using a process described as follows:
“I want to come as close as possible to the truth [Nature] and abstract everything from that, until I reach the foundation (still just an external foundation!) of things.”
— Piet Mondrian
Tennis for Two (1958) by William Higinbotham.
Resolving the conflict of intent between art and gameplay is easy if we imagine changing the gameplay of QIX, so that its simulated metaphor mimics Mondrian’s artistic methods. To do so we must appreciate that the relationship between artist (author) and the act of painting (agency) changes in the context of games because the game development team authors the game system but players act as agents to activate and perform the artwork. Once we’ve established this relationship between developer-author and player-agent, we can better perceive the possibilities of creating gameplay that simulates the gestures of an artist.
From left: Mondrian in the process of creating Broadway Boogie Woogie in 1942 (image via Counterlight's Peculiars blog; original author of photo unknown), and Spring Sun (c. 1910). (Image source: Wikipedia)
We know that Mondrian’s creative process was to cut out and position squares of construction paper that were replaced with considered and precise paint strokes once a pleasing visual composition had been arranged (above left). The artist’s creative decisions were driven by an impulse to distill the essence from earlier traditional paintings that focused on landscapes in a combination of naturalistic and impressionistic art styles (above right). Using the adaptive composition framework, we have the conceptual tools, and intent, to create a “Mondrian simulation” that will evoke Mondrian-like decision making and gestures from players—the foremost intention being to simulate Mondrian the artist, not a Mondrian painting. Such a simulation could reference the gentle territory gameplay of Flower (2009), by thatgamecompany, transferred to a linear aesthetic.
SPOILER ALERT: the following paragraph contains a potential spoiler for Gone Home (2013) by Fullbright.
Exercise: Discuss "Are Games Art"?
Gone Home (2013) by Fullbright features gameplay that is transformed by the game’s suspense-horror aesthetic to elicit emotions that the simulated metaphor alone could not elicit.
Is a Mondrian simulation more deserving of an art label than QIX because it simulates a real-life artist? Must simulated metaphors model the subject to have artistic value? For instance, Gone Home simulates a detective case but the game’s use of suspense-horror cultural symbols evokes feelings of vulnerability that help players empathize with the social issues and teenage angst of the narrative. Can Gone Home’s gameplay be labelled art even though the visuals and narrative don’t match the simulated metaphor? Is a Mondrian simulation true to the artist only if the visual aesthetics accurately reproduce the artist’s paintings, or is evoking Mondrian-like physical gestures sufficient?
A really exciting discovery occurs when we compare the work of abstract expressionists to visualizations of player movement in video games, like the one above for Super Mario Bros. (1985). Movement tracking is generally used in game development for user testing purposes but we can appropriate the tools to study interactive motion lines in the context of simulated metaphors. What the examples below illustrate is the art of interaction—the visual link between player gestures and the player’s interaction with a game system that references Abstract Expressionism.
From left: visualizations of player movement in the “early” game, and the “late” game, in League of Legends (2009) by Riot Games. (Images sourced via The New York Times)
The images above visualize the movements of 100,000 players in 10,000 games of League of Legends (2009), showing over six years of cumulative player time. The visualizations look similar to abstract expressionist art, wouldn’t you agree? As the authors of Watch 10,000 League of Legends Games in 30 Seconds (The New York Times 2014) note: “while each match is complex, the same patterns emerge over time.” This is an extraordinary observation because it demonstrates the level of aesthetic control and consistency that developers have on the interactive experience of players—visually proving the value of our theoretical Mondrian simulation. It seems that, when viewed up-close, the player’s movements are similar to Jackson Pollock’s line-work. When viewed afar, the player’s collective movements reveal a concealed composition, which is not the result of a single author, but the collaborative effort of the game’s developers and players.
Images from Ubisoft’s DNA tracking system used to visualize multiple players moving through Assassin’s Creed III (2012) by Ubisoft. (Images sourced via Gamasutra, Game Telemetry with DNA Tracking on Assassin's Creed by Jonathan Dankoff)
Ubisoft has a custom movement tracking tool called DNA, which it uses to analyze player movement during play-testing prior to release as well as post-launch, as described in Game Telemetry with DNA Tracking on Assassin's Creed (Gamasutra 2014) by Jonathan Dankoff. In the interests of creating game systems that reference the subtler movements of an artist’s creative process, it would be an interesting challenge to iterate on such tools by developing ways to visualize the aesthetic qualities of movement and player gestures in finer detail—referencing the Ballet Rotoscope video from Shape Language and Interactive Motion Lines and synaesthetic experiences like Panoramical by Fernando Ramallo and David Kanaga.
Exercise: Motion Lines Study, Part 2
Columns from left: Helen Frankenthaler working in the studio (photograph sourced via Widewalls) and Around the Clock with Red (1983); Norman W. Lewis at work (photograph sourced via Pennsylvania Academy of the Fine Arts) and Composition 1 (1945); Kazuo Shiraga painting in the artist’s studio in 1960 (photograph via The Greenstein Lab, copyright Amagasaki Cultural Center, courtesy of Fergus McCaffrey) and Work (c. 1990).
Study the above images to appreciate the aesthetic connections between artist and artwork that abstract expressionists sought to establish. The mid-20th century onwards saw an increase in diversity, represented here by Helen Frankenthaler (1928-2011), Norman W. Lewis (1909-1979), and Kazuo Shiraga (1924-2008). For additional context, we know that Frankenthaler often painted with a technique called soak stain (paint that was heavily diluted with turpentine), which allowed the artist to achieve the soft, watercolor-like edges. Lewis painted in a fairly conventional manner—sitting or standing at the easel with a paint brush in hand. Shiraga gained recognition creating mud paintings by using the entire body to make impressions in wet mud—a technique that was later adapted for oil paint.
Following on from the Motion Lines Study, Part 1 exercise, revisit some memorable gameplay narrative moments. Before playing, use the abstract expressionist analogy above to consider the emotional and physical state of your playable character and draw a corresponding line with an aesthetic quality that illustrates your concept. Once that’s done, play the game sequence and consider whether your physical gestures echo what you’d expect the character to be feeling. Additionally, is there an aesthetic change to the motion lines across the gameplay sequence?
Player movement tracking in Rubberband Racing
by Max Knoblich. (Image sourced via Gamasutra)
It is safe to conclude that a simulated metaphor (gameplay) is indeed art—art that is modelled on the creative process and authored by a development team before being activated by players. That the lines visualized by tracking tools are not generally visualized in-game (aside from games like QIX, the character’s trail in ABZÛ or the trailing scarf in Journey) is not important, since classical compositions were also purposefully concealed (see Concealing the Composition). The question is only whether game developers care more about formal gameplay (rules, objectives and stats) or informal gameplay experiences and interactive motion lines that they evoke, which will determine whether a simulation's output is treated as an "incidental byproduct" or the deliberate aesthetic result of a carefully orchestrated adaptive composition.
By using shape language and composition principles, combined with abstract expressionist concepts, it is clear that an appreciation of games as art requires going beyond surface representation. Instead, we must take-on the classic role of the artist-craftsperson to appreciate the creative process itself.
As the above examples imply, the sensations of painting a painting or sculpting a sculpture are no longer the exclusive domain of the artist. Whenever you interact with a video game you are interacting with an aesthetic system that evokes artist-like gestures and decisions. You can therefore think of each game as a unique stencil set of lines and shapes curated by the game development team, which the player-audience activates during gameplay.
The more we understand about shape language and composition, and the sensory connections between visuals, music and and physical gestures, the greater the harmony will emerge between the component parts of our games.
The adaptive composition framework.
The above sections on Shape Language for Game Design and The Adaptive Composition Framework, offered just a brief overview to four of the eight elements of adaptive composition, illustrated above: character shapes and poses, environment shapes, audio and interactive movements lines. There is a lot more to learn about each element that—together with shape language and the shape spectrum—gives us greater control of the game development process and orchestration of aesthetics that respond dynamically to gameplay and narrative context. These principles are timeless, applicable to VR, AR, XR and all future expressive media. The high-level framework nicely complements or enhances other popular game design methodologies like Jesse Schell's "lenses" [see The Art of Game Design: A Book of Lenses (CRC Press 2008)]. Equally, designers will find that existing game mechanics and system designs remain valid and mostly unchanged. What the framework does is give these mechanics and systems an aesthetics purpose—championing the big picture over details.
Like the traditional artist who should not mistake the brushwork for the painting, obsessing over the player’s mastery of gameplay techniques should never overshadow their aesthetic experience of the story.
The adaptive composition framework also helps to demonstrate game design’s artistic value. Not the conventional notion of game art as a visual “byproduct” of gameplay but of gameplay as art—the rules, mechanics, etc.—and its connections to traditional art principles.
Roman statue of two girls playing Knucklebones (3rd century BC)
framed within a circular composition inspired by ancient Greek sculpture.
The ancient Greek philosophers and Islamic artists were among the first to begin ascribing aesthetic value to primary shapes, influenced by their advanced mathematical knowledge that resulted in the Platonic solids (see Part 1), which form the conceptual basis of Greco-Roman and Islamic art. This pioneering work linking math and aesthetics went on to influence ancient Buddhist art, which arguably holds the most relevance to game design through the concept of embodiment. These concepts, inherited from ancient art, allowed us to conclude that “psych” is indeed amenable to math as experienced through exercises like, Motion Lines Study, Part 1). Even “ludo” (play) has its origins in antiquity. With shape language and composition being such a central topic across all art forms, why aren’t we talking about it in the context of game design?
Artist Bobby Chiu in conversation with Nathan Fowkes on composition.
Duchamp’s teachings hold some answers, since the artist-philosopher has had a tremendous influence on contemporary art thinking. You’ll now find it hard to receive a formal art education, in a classic “academy” sense, outside of a handful of traditional academies and specialist art courses that focus on production techniques for illustration, film, animation, and game art, etc., where visual storytelling is still the focus. It’s here that the importance of shape language and composition endures, attended by celebrated artists like Nathan Fowkes, whose clients include DreamWorks, Blue Sky and Disney. In a Schoolism.com interview with Bobby Chiu, Nathan uses Mondrian as an example to stress the importance of composition as follows:
“What’s the job of our brain? It’s to take all of the noise that we see and experience, and draw out from it the meaningful relationships […], wash away everything else. That’s being a human being. That’s basic survival. And that’s exactly what [Mondrian] did in his work. He distilled all of the noise away until the only thing left were pure, simple, meaningful relationships. […] No matter what we do, that is our job, to find the meaningful relationships that makes those pictures really hold together with the greater sense of purpose.”
— Nathan Fowkes
Bicycle Wheel by Marcel Duchamp, turned upside-down, as a commentary on the persistence of shape language: the dynamic circle, stable square and slicing triangle.
Contrary to popular belief, formal art training teaches unique skills like shape sensitivity that require years to develop, just like a trained musician’s nuanced sensitivity to sound. This misconception is why Duchamp's views often appeal to individuals outside of applied art fields, and why shape language is occasionally dismissed as irrelevant or simplistic.
The truth is that the games industry at large would benefit if everybody along the development pipeline had an understanding and appreciation for traditional art principles. Shape language’s basis in Nature means that it is understood across cultures—not to mention, every individual on the development team. Through shape language, which we’ve found embedded across engineering, art and design fields, we’ve seen that primary shapes—the circle, square and triangle—support Raph Koster’s definition of games as “systems built to help us learn patterns for survival.”
Judging from the international examples of shape language and composition in creative disciplines as diverse as film, comics, photography, and interior design, it appears that coded survival lessons are a crucial component of all art and storytelling mediums as well as play. In fact, its scientifically believed that storytelling could be an evolutionary mechanism that helped keep our ancestors alive. The existential purpose of stories evokes a comforting sense of control over the world—teaching audiences to find patterns in chaos and meaning in randomness. This is perhaps the best definition for shape language and composition—aligning perfectly with Raph’s definition of games.
Working with shape language doesn't mean that an artwork must consist of pure circles, squares and triangles. It means that an artist-game designer uses their heightened shape sensitivity to tune an artwork so that it resonates (covertly or overtly) with a universal, primal language—irrespective of whether the shapes and feelings evoked can be accurately defined.
Take a moment to look back through your drawings for each exercise in this essay to appreciate the aesthetic differences that can be achieved despite each pencil stroke aligning to one of the primary shapes. This is the kind of line quality an artist expresses in a life drawing class. Through shape language and adaptive composition, we can re-visit traditional dramatic structures with a fresh focus on physical empathy and embodiment, which is the topic of the second half of Interactive Stories and Video Game Art.
The benefits of using shape language include:
- Visual clarity, afforded by the distinct aesthetics of the circles, squares and triangles
- Explicit utilitarian value, afforded by shape language’s basis in Nature
- Increased creativity, (creativity defined as the ability to make abstract connections between seemingly disparate things) because shape sensitivity increases awareness of aesthetic connections throughout Nature and Art
The utilitarian basis of shape language takes on high-level emotional themes when placed in an artistic context. Irrespective of an object’s cultural meaning, aligning its broader shape to the circle, square or triangle will consequently align it to one of the following aesthetic themes:
Circle: movement, energy, grace, friendliness, continuity
Square: balance, strength, restraint, tranquility, rest
Triangle: force, instability, aggression, pain, agitation, tension
The benefits of composition include visual simplification, visual direction, story structure and a holistic design framework. We inherit all these benefits when we adapt composition to game design—a straightforward process that requires the acknowledgement of players within the composition structure, and that each supporting element is pervasive, dynamic (adapting to the narrative or gameplay context) and visually broad (so as to remain visible at all zoom levels and from every perspective). These are not superficial considerations exclusive to game art. Each element of adaptive composition involves close collaboration between all the various disciplines involved in game development—suggesting new possibilities for advanced adaptive and procedural storytelling techniques that shape the player’s aesthetic experience on the fly.
By conceptualizing the adaptive composition framework in the form of Jesper Juul’s puzzle piece metaphor, we can also reflect on the definition of games, their boundaries, and how they interface with reality and across transmedia experiences—something that would not be possible without the elegantly simple affordances of shape language versus an infinitely complex Duchamp-ist universe. However, shape language and culturally learned symbols are not mutually exclusive because the former works best when used to covertly influence how audiences experience the latter.
Using these techniques doesn't ensure a masterpiece—in the same way that understanding music notation doesn't turn a novice into Mozart—but it does make the game development process feel more like tuning a familiar instrument with subtle controls that affect the player-audience. These collective insights enable game designers to generate subtler forms of dramatic tension that don’t rely on escalation of gameplay difficulty, which is the wider industry’s go-to for orchestrating in-game narrative experiences. Leading design by physical gestures will no doubt blend the distinctions between formal and informal game design and dissolve the predominance of single and binary character control schemes, which lead to an interactive form of uncanny valley in high-end AAA games.
It just so happens that the first ever video game, Tennis for Two, simulated a sports activity. This does not mean that all games must be decedents of sports and competition. We can equally think of games as decedents of the artist’s creative process in sculpture and painting without denying the value of objective-driven gameplay. Animation went through a similar path of self-discovery by overcoming supposed limitations set by the medium’s early pioneers and critics that couldn’t imagine the medium useful for anything other than comic gags, as explained by Walt Disney (1901-1966):
“At first the cartoon medium was just a novelty, but it never really began to hit until we had more than tricks. You have to have pathos [empathy] in the thing.”
— Walt Disney
Replace “cartoon” with “video game,” and “tricks” with “incidental byproducts” and you have a fairly good idea of the untapped possibilities that we’ve yet to discover using art-driven frameworks and the exponential increase of subject matter that they afford.
Peanuts comic strip (1960) by Charles M. Schulz.
The above Peanuts comic strip perfectly summarizes what makes video games such a unique art form. Charlie Brown explains that an appropriate stance is vital to getting the most out of feeling depressed: avoid straightening up and holding your head high because that will inadvertently make you feel better. By combining game design with shape language and adaptive composition techniques, we have the design tools to heighten sensory inputs to influence player gestures and, as a result, player emotions in increasingly subtle ways that surpass any art form that has come before.
Interactive Stories and Video Game Art: A Storytelling Framework for Game Design is available direct from CRC Press or via Amazon and Barnes & Noble. For updates about future essays and work, you’re welcome to follow me on Gumroad, Twitter, Facebook and Instagram or via my homepage.
A special thank-you to Julia Ayumi Bopp (MMI Human-Machine Interaction at the University of Basel, Switzerland), Martina Hugentobler, Samuel Smith (ProSapien), Corrinne Lewis (University of Utah), Kayleigh MacLeod, Stefano Maccarinelli (Stelex Software), Edith Solarski-Meier, Henri Suhonen, Lasse Juel Larsen (University of Southern Denmark), Seb Long (Player Research), friends, family, and #swissgames organizations like Pro Helvetia and SAE Institute Zurich that kindly supported my work throughout the years. I’d also like to thank Alastair Ingason, Lalima Singh, David Javet (University of Lausanne GameLab and Tchagata Games), David Canela (MODSORK), Robert Zubek (SomaSim), and Amit Goyal (SuperSike Games) for giving me valuable feedback on sections of this essay, and Benjamin Poole for directing me to Raph Koster's work.
Chris Solarski is an artist-game designer and author living in Zurich, Switzerland. His two books are widely considered essential reading for anybody looking for holistic design tools for orchestrating the emotional and physical experience of players in games and transmedia. Chris' first book—Drawing Basics and Video Game Art—has been translated into Japanese and Korean. His second book—Interactive Stories and Video Game Art—has been described as gaming's equivalent to the screenwriting classic, Story, by Robert McKee and is endorsed by film director, Marc Forster. With kind support from the Swiss Arts Council, Pro Helvetia, Chris has given talks on interactive storytelling and game art internationally, including Disney Research, the Smithsonian's The Art of Video Games exhibition, SXSW, GDC Europe, and FMX. Chris is currently collaborating with internationally renowned artist, Phil Hale, to develop an indie game based on the Johnny Badhair series of paintings. He also lectures at SAE Institute in Zurich and manages the IGDA Switzerland Chapter and Co-Chairs the IGDA Game Art SIG, which he cofounded with Natasha Bulatovic Trygg.