Video in Games: The State of the Industry
By Ben Waggoner and Halstead York
Gamasutra
January 3, 2000
URL: http://www.gamasutra.com/features/20000103/fmv_01.htm

Video is one of the most commonly used and least understood elements of modern computer games. Ever since the CD-ROM offered a medium able to carry significant amounts of audio and video, game developers have worked to incorporate video into their titles. Although there have been many missteps along the way, video is still a critical element of the game designer’s palette. When used well, it can create mood, set up game play, introduce characters, and forward narrative. When used poorly, it can rip you out of the game faster than a direct rocket-launcher hit during a deathmatch.

A Brief History of Video in Games

The first game using video was the arcade classic Dragon’s Lair. However, this game was based around an analog LaserDisc, not digital video. The first game using digital video in any significant way was Sherlock Holmes: Consulting Detective, released back in 1992. It had a budget of over $1 million and was shot by experienced filmmakers with professional actors on real sets. Given the limited quality of the playback technologies of the day, the quality of the presentation was poor on most platforms, but the game used video well and it was a pretty fun beer-and-pretzels game.

The first big commercial hit in the burgeoning interactive movie genre was Trilobyte’s The 7th Guest. In retrospect, the success of this title was probably more due to the novelty of seeing video on a computer, rather than the game’s design. Functionally, the game was a series of barely related logic puzzles that used cut scenes as a reward for solving those puzzles. The video was poorly produced; actors were filmed in front of a blue sheet with a home video camera. It was impossible to make a clean key from this, and so the actors were left with ugly blue halos. The game’s signature effect of transparent ghosts with blue halos walking through rendered environments wasn’t originally intended — rather, it was a last-ditch attempt to make the best use of this artifact. The 7th Guest’s success set up some basic fallacies for the genre: quality doesn’t matter, video is enough of a reward for users, and game play doesn’t have to be tied to the video narrative. Needless to say, this model isn’t seen in games anymore, at least in games in which the actors wear clothing.

The 7th Guest was the first commercial hit of the interactive movie genre and established many of its basic fallacies .

The standout success of the early digital video era was Myst. While reviled by some game purists, Myst remains the most popular game of all time, and immeasurably contributed to expanding the mass market for interactive entertainment and CD-ROM drives in general. Myst used video in an appropriate manner — in short snippets, inside the overall game environment, advancing plot in a plot-oriented game.

The expectation grew in some quarters that "Interactive Movies" would prove to become the dominant form of computer game, and prove to be another outlet for the talents of second-tier stars. Some of them were even quite interesting, such as Phantasmagoria.

Wing Commander III was another turning point in the genre. It melded The 7th Guest’s model of video sequences between game play elements with the plot-integration of Myst, and added big name stars such as Mark Hamill and Malcolm McDowell. And it was moderately successful, both creatively and financially. However, the costs for producing what was effectively a TV miniseries in parallel with game development were very high, and the games had to be hugely successful to break even.

Myst (left) and Wing Commander III (right) were two turning points in the use of video in computer games.

The way video was used in Wing Commander III remains dominant today, even though our video budgets are a small fraction. We see video in short, plot-advancing sequences that alternate with game play and give context to what happens in game play sequences. Most of the budget and energy goes to an impressive opening sequence that sets up the plot and is endlessly repeated at game retailers as an attract loop.

The State of the Industry

Throughout the brief history of our industry, games have striven to be more immersive, more involving, and more dramatic. The term often used is "cinematic." There was even a time when the term "interactive movie" was very much in vogue. Now of course, we look back at this period knowing that most filmmakers don’t know much about games and that game designers don’t necessarily make good films. The half-baked efforts of this period stigmatized the term "full-motion video" (FMV). Indeed, we’ve reached a point where one recent FMV adventure game, A Fork in the Tail, proudly proclaimed that it contained "FMV that doesn’t suck." For the record, the FMV didn’t suck. However, the game did, and for a very simple reason: game players don’t like being passive. Game players like to play, and choosing which sophomoric one liner to use on the scantily clad, direct-to-video harlot in front of you ain’t playing.

While many developers have been licking the wounds they received after mistaking cutscenes for game play, others have forwarded the state of the art. Grim Fandango’s FMV adds camera movement and the pacing of professional editing, while keeping the fabulous feel of its engine intact. In fact, many can’t tell where the video ends and the engine begins. Jane’s has done a wonderful job with FMV, repurposing training films and using a news broadcast format at the beginning and end of campaigns in the Longbow series. Here, video is used to present information in a concise and informative manner while simultaneously creating a technothriller feel in the simulations.

Grim Fandango uses full motion video that is almost indistinguishable from the in game engine as these screenshots from the video (left) and the game (right) show

Son, Put Down That Camera Before Somebody Gets Hurt

A lot has changed since the Hi8-video-and-blue-sheet days of The 7th Guest. Game players today demand a much higher level of creative and technical competence, and developers need to recognize that the production values of their video need to match (and often exceed) the quality of the rest of the game.

Why are the production values of cutscenes and other video elements so critical? Because you’re competing against the consumer’s instinctive understanding of production values, an education furthered every time they go to the movies or turn on a TV.

Too often, a game’s development process starts without much thought for the video other than, "We’ll have a cool video intro" or "We’ll use cutscenes to introduce each level." All too often, this attitude leads to cinematics that seem incongruous and inappropriate for the game. I remember watching the Mechwarrior 2 opening animation, and thinking how great it looked. When I started the first mission, the game’s then state-of-the-art 3D engine seemed cheap and flat. Now, I’ve been playing and working on games for some time, so intellectually, I knew that the Mechwarrior engine was damned fine. However, I couldn’t escape wondering what it would be like to play that game I saw in the intro, with large rocky hills to hide behind and cool textures on the Mechs. In short, Activision presented a bill of goods in the intro that they couldn’t deliver once the player took control of the action.

Command and Conquer uses FMV to step completely outside of the game mechanics and offer views of the game universe outside of what’s shown during game play.

Cinematics should either fit in with the look and feel of the game engine, or use the high visual definition afforded by FMV to reveal details that can’t be seen within the game. Grim Fandango, Blade Runner, and Interstate ‘76 offer excellent examples of FMV that looks like the rest of the game, but offers details and subtleties of motion that are still out of reach of a modern real-time 3D engine. They add depth and character to the games without making the player feel that the game play is a weak and underdeveloped version of the cinematics. The Command & Conquer series and Quake II offer excellent examples of games that use FMV to step completely outside of the game mechanics and offer views of the game universe outside of what’s shown during game play. The cinematics in these games let players look at the world from other perspectives, offering images and information that these players can hold in their minds as they play the game. As we watch a unit’s health go into the red in C&C, we remember the opening scenes of missiles and bullets ripping through flesh and steel. Wandering through the Strogg’s homeworld in Quake II, we think from time to time of that dramatic flyover in the game’s intro and keep wondering just what that big gun we saw is meant for.

Jedi Knight: Dark Forces 2 made poor use of live action and prerendered animation.

On the other hand, LucasArts’ otherwise excellent Jedi Knight: Dark Forces 2 made poor use of live action and prerendered animation in its cutscenes. Far too often, a cutscene would transition into game play, leaving the player completely disoriented by how different the surroundings and characters looked. Either the cutscenes made me feel that the much lower-quality in-game graphics were a gyp, or poor acting made important, dramatic villains laughable. Anybody remember Jerec shooting his tongue out like a frog in the opening video? Goofy.

Which is not to say that live-action video is the downfall of a game. The Wing Commander games, particularly Wing Commander: Prophecy, made excellent use of video to further the plot and add depth to characters. The FMV that bookends most missions achieves its goal: it makes the player feel as though he or she is the lead character in a sweeping B-movie space opera. The deeply underrated Spycraft: The Great Game made excellent use of video, creating a sense of realism and immediacy that couldn’t have been matched with CGI. Similarly, Gabriel Knight: The Beast Within used live-action video brilliantly, and showed just how involving real actors can make a game.

Putting Together the Right Team

For many developers, it has become important to bring in a production company or independent producer at the same time the rest of the asset team is being assembled, particularly if the video will involve a live-action production. However, even if the sequences are 100 percent animated, it’s preferable to have the team creating the assets involved as early as possible.

Just as it is important for the developer to appreciate the intricacies of FMV, so to must the FMV team understand the game on which they’re working. Make sure that they have a real sense of the look and feel of the rest of the game, particularly in-game elements. If the game uses a palette overwhelmed with blues, then the video should match it. Final Fantasy VII’s cutscenes have wildly different styles of character animation than the rest of the game. Its video uses a more traditional anime style, while game play centers around super-deformed characters. The super-deformed style is often used in Japanese animation to convey humor and comic relief. However, I’d go out on a limb and say that the dichotomy in the look is disruptive and antithetical to the somber adventure that the game seems to work so hard to present. Apparently, Square agrees — Final Fantasy VIII doesn’t use the super-deformed design elements.

Ensuring Video Production Success

Creating quality FMV

Far too often, I’ll see video that looks almost professional. For example, Dune 2000 (along with just about every other Westwood title using live action), had crystal-clear interlacing artifacts caused by not removing one of the two fields that makes up a video frame. Just about any digital video professional could have corrected this problem with a mouse click. These artifacts are distracting, and make digital video compression much more difficult.

Black Dahlia, is an example of professional quality video in a computer game

Problems such as this can crop up in every stage of in the production of video, from lighting to direction to compositing to editing. The video in Black Dahlia, a game completely dependent on the success of its cinematic elements is, quite simply, professional. The lighting on actors matches backgrounds. They are well placed within their environment. 3D animation is married sensibly and elegantly to stock footage, as well as to newly shot elements. The bottom line is make sure that you have video professionals working on your FMV segments, otherwise your quality could suffer.

Actors need good direction

Game developers have often shown the ability to pull wooden performances from very talented actors. Many a time, I’ve seen some of my favorite character actors deliver lines with all the conviction of a Creationist giving the keynote address at a Darwinists’ convention. These problems often stem from a fairly simple issue: the talent has been inadequately prepared for the task at hand. If the developer is in charge of the production of the video assets, then the fault lies there.

When making the transition from the stage to film or television, actors often complain about the lack of linearity. Movies are rarely shot in sequence, and an actor used to being able to carry their character from scene to scene can get lost in the technical considerations of production. What’s worse is that the nonlinearity of game plots can make an actor’s role even more difficult when filming game video.

In the game developers’ world, video sequences often exist around game sequences, and each segment might come from and go to many branches. Actors can find it difficult to maintain a strong grasp on what their character is doing or, for that matter, who they are.

This fact can result in flat, unappealing performances. Therefore, if you’re going to be working with actors, empower them with an understanding of the process they’re involved in. Some developers have taken to bringing large flowcharts to the set, so that the actors can see where they are in a given scene, and understand how it relates to the rest of the game. Developers have also learned the value of rehearsals and read-throughs of the script. Many problems with a scene can be caught during this process.

Keep your animation animated

Almost all titles today use 3D animation in their cutscenes, either as background elements for actors or as the sole element in the video. Most modern game development teams have top notch 3D artists on their teams. They are brilliant at taking complicated objects and characters and building low–polygon-count 3D models with depth and life. However, these skills don’t necessarily translate into the ability to produce an animated segment for a videogame.

For example, Jedi Knight: Dark Forces 2 opens with a long, drawn out flythrough over Nar-Shaddaa. Although this is a wonderful example of whatever modeling program the animator used, it’s long and uninvolving. There is nothing wrong with using animation to introduce new settings and hardware. Blizzard has done a very good job of that with Starcraft. You just can’t assume that a camera spinning around a 3D object is inherently dramatic. Without any sort of context, it’s boring.

Post-production preparedness

Digital video post production is often thought of as simply the stage where things that went wrong are corrected — as in, "We’ll fix it in post." However, as mentioned earlier, it’s a lot cheaper do to something correctly the first time. Keep that in mind.

Post production should be thought of as fine tuning what you’ve already done right, not correcting earlier flaws. This is the stage where a scene’s structure and rhythm are honed via video editing, animation and live action are merged, and graphics and subtitles are added.

Interstate '76, with its primitive graphics, was able to really capture the feel of a 1970s action film by mimicking their editing style.

Never underestimate the critical importance of good editing. It can be tempting for some computer animators to create an entire scene purely with camera motion instead of using cuts — almost always a terrible idea. Cutting from different camera angles and subjects is core to our video experience. A great way to understand how this works is to watch TV with the volume off.

Notice how often the cuts happen during a scene. Note how they set up the scene, emphasize facial expressions, and set up a rhythmic pace. Interstate ’76, with its primitive graphics, was able to really capture the feel of a 1970s action film by mimicking their editing style.

Some big differences exist between traditional video post and post for game delivery. First, traditional video is made up of tall, rectangular pixels. Computers use square ones. Second, each traditional video frame is made of two unique picture fields interlaced into one frame, with all the even lines capturing a moment in time a fraction of a second different from the odd lines. Video in games is always progressive scan, with only one image in the frame .

Because RGB computer monitors are much better than analog televisions, games can use film-like color saturation and contrast. The drawback to the high-quality images produced by computer monitors is that a lot of subtle noise that would go unnoticed on a television is often readily apparent on the monitor. This can be especially obvious in shadow tones, and it requires processing by a tool such as Media Cleaner or After Effects to reduce the noise.

Digital video codecs also have limitations that should be addressed in post production. For example, Smacker and other codecs that use a limited palette of on-screen colors benefit from video processing that limits the color palette used in any given frame. Other codecs have trouble with certain types of content, such as rapidly moving complex textures. Perform some sample tests of your post and edits well before final compression, so you can alleviate trouble spots in the video.

The Black Art of Chroma Key Compositing

If you're combining live action and animated elements within a scene, compositing will play a large role in your final product. You'll most likely begin by shooting video on a blue or green screen. The background color is then removed using a chroma keying process. Blue and green are the most popular colors for a removable background because they are easy to remove and rarely appear in foreground subjects. Chroma key production and post-production is complicated, and it's strongly recommended that you allow professionals to handle it. However, if time, budget, or ego considerations get in the way, here are a few pointers:

1. Use a big space.You'll want to keep your talent as far a way from the screen as you can and light as flatly and broadly as you possible (more on this later). For these reasons, you want your space to offer as much room as possible.

2. Give your blue surface a slight curve.A very slight curve on the vertical axis will help soften the light and lessen visible highlights on the screen.

3. Use flat lighting on the screen. The screen should be lit softly from above and possibly the sides. Never point lights at the screen from behind (or too near) the camera. The goal is to introduce no contrast variations. A video tool called a waveform monitor is used to measure the brightness and color of the screen to make sure it is consistent throughout. Any good production group will use one. If you use any glossy materials on your screen, you're sunk.

4. When you light the talent, don't relight the screen. Dramatic and flattering lighting is often antithetical to good bluescreen lighting. Keep the talent far enough from the screen so that you can light them without their lights hitting the screen. Never point talent lights at the screen - you won't be able to key later. And remember to recheck your waveform monitor after you've lit everything.

5. Choose a professional, high-quality tape format. You'll need a format that can handle highly saturated, complicated images. This requirement precludes the use of consumer formats such as VHS and Hi-8. The new DV format is a better choice, although it deals with color space in a somewhat limited fashion, and can create pixelated artifacts around the edges of foreground objects, making the objects difficult to key. Betacam is a viable low end, with Digital Betacam, Digital-S, and film being the preferred format choices.

6. Narrow your depth of field. You may want the talent in focus, but keep the screen as blurry as possible in order to hide the inevitable imperfections in the screen and make keying much simpler.

The original design spec for Journeyman Project 3 called for a blue time-travel suit. When developers realized that wouldn't work in a bluescreen shoot, the suit was quickly repainted green.

7. Kick the talent (no, don't hurt them). A warm-colored light (try an orange gel) placed behind the talent pointing at his or her back will help create a strong, very unblue edge on the talent, which will keep fingers and hair form disappearing in the key.

8. Avoid the key color in any foreground objects. Yes this sounds simple, but trust me - it isn't. Blue shirts, socks, even eyes can disappear in the key, as will anything that reflects blue (particularly white and metallics). If you have to use blue elements (such as the Wing Commander uniforms), shoot on a green screen.

9. Fix problems in preproduction. Generally, what you're paying for with high-end post tools is the ability to screw up in production. Terminator 2 might have cost one third as much if it hadn't been rushed through production so quickly. If you don't have access to high-end tools, following these tips (and others like them) are the only chance you'll have to get a good key. You have time and intelligence, which can offer you almost all of the advantages of money as long as you use them well. Also, I don't recommend putting too much faith in the equipment to hide your mistakes.

10. It's important that the live-action video matches the lighting present in the rendered background elements. Otherwise, even the cleanest composite will stand out, and the video will look fake and gimmicky. Giving the animator a diagram of your lighting setup and a tape of the shoot are a requisite.

The Future of Games

So, where is all this going? Great places. Over the next few years, DVD-ROM will become ubiquitous, as will MPEG-2 playback. These advances will increase the technical quality of video in games beyond our current goal of "broadcast quality." The larger media size of the DVD-ROM will also allow more video storage capacity per disc. A dual-layer, single-sided disc can hold over four hours of extremely high-quality video, with one gigabyte left over for the rest of the game.

Not all video will go over to MPEG-2, though. While great for standalone cut scenes, MPEG-2 is difficult to use within games themselves, where video is displayed in only part of the screen or at the same time that other things are going on. Fortunately, a number of next-generation software-based codecs are hitting the market, such as Duck’s TrueMotion 2X and Rad Game Tools’ Bink. Both provide very high-quality content at reasonable data rates and both should be released by the time you read this.

Today’s video production tools are much better suited to game production. The whole video world is going digital, which massively improved the quality and costs of game video. Most of the major video manufacturers have announced progressive scan cameras, which will eliminate the interlacing conversion problems (thereby doubling effective resolution) that have limited quality since the beginning of the game industry. In addition, the next wave of high-definition cameras will offer the possibility of film-quality video on the desktop. The price of professional-quality equipment is also dropping very rapidly, to the point where today’s $4,000 camera is better for game video than the $40,000 rigs of ten years ago.

As playback quality increases, so will the demands on developers to produce high-quality video. After decades playing the poor cousin to the broadcast world, DVD and next-generation video playback engines will offer us the opportunity to provide the end user with far better that broadcast quality. Game developers have an opportunity to take a leadership position in video quality but in order to do so the must avoid at all costs the following ten disasters when producing video for games.

Ten Digital Video Game Disasters

1. Bad bluescreen. Do things look funny around the edges of your actress’s long hair? Is the background either too visible or not visible enough? That’s the product of bad bluescreen. These sorts of effects can be expensively fixed in post with frame-by-frame tweaking, but it’s much better to get right in the first place. See The 7th Guest.

Interlaced video as used in Dune 2000 is ugly and increases data rates but can be easily eliminated.

2. Interlaced video. In analog video, the even lines are captured 1/60th of a second apart from the odd lines. So, when objects are moving rapidly, you see a stair-stepping pattern. It’s ugly, increases data rates, and is easy to eliminate. Every digital video program out there can deinterlace, so do it, or we’ll make fun of you at E3. See Dune 2000.

3. Uncropped video. In analog video, there’s almost always crud around the edges of the image that isn’t seen on TV. Computer monitors show every last pixel all the way around. So you need to crop out the crud on the edges. See the Longbow series.

4. Wrong aspect ratio. Pixels are always square, right? Wrong. Many high-end digital video standards use 720X486 pixels per frame, where the pixels are taller than they are wide. So if the video is played like that on a computer, everything looks wide. Scale the video so that the pixels wind up square and circles aren’t ovals.

5. Bad actors. It’s an old saw in video production that many good stage actors don’t work out well in front of a camera, because their performances are aimed at people thirty feet away in the audience, not two feet away in a close-up. See almost every FMV game made.

Resident Evil's cutscenes suffered from infamously poor voice acting.

6. Poor voice acting. In many ways, poor voice acting is much more of a problem than poor video. There are two main reasons for this: audio is everywhere in games, and voice acting looks easy. Voice acting is not easy. If it was, it wouldn’t be obvious which computer games had the engineers do voice-overs. Voice acting requires skill and experience, and it’s a lot more expensive to spend eight hours in a recording studio with a volunteer than two hours with a professional.

Ask this question before using someone as a voice actor: "Are you a member of AFTRA?" If they respond yes, then they’re almost certainly a skilled professional. If they give you a speech about how they don’t like unions, at least they know what business they’re in, and might be okay. If they say "What’s AFTRA?" smile and go somewhere else.(AFTRA is the American Federation of Television and Radio Artists.) See the Resident Evil series.

7. Wrong frame rate. Video runs at 30 frames a second, and film at 24 frames a second. The final frame rate of digital video should be an integer fraction of the original frame rate in order to preserve smooth motion. For example, video should play back at 30 FPS or 15 FPS, and film at 24 or 12. Note that if you transfer film to video, you still should restore the frame rate to 24 fps. Media Cleaner Pro does a great job with this conversion. See virtually every game with content shot on film, such as Close Combat.

8. Eight-bit audio. Never, ever, ever use 8-bit audio for anything. Ever. Really. With modern audio codecs, 8-bit always produces bigger files and sounds worse than 16-bit.

9. Mismatched foreground and background elements. So, you want to shoot actors on a bluescreen, then incorporate computer graphic elements in the background? The video looks great, and so do the computer graphics, but why don’t the spaceships look like they’re in the same universe as the talent? Lots of reasons, probably. In order to make elements match, they need to have similar frame rate, grain, motion blur, lighting highlights, and so on. A good animator can make a great looking spaceship. It takes a great animator to make it look like part of the set.

10. Video technical requirements out of sync with the rest of the game. While I like codecs as much as the next guy (well, a lot more than the next guy), it’s a mistake to have the processor requirements for your cutscenes be higher than for the rest of the game. Yet, it happens. Sometimes, testers get so used to skipping past the video that they never check cutscenes on low-end computers. Great looking video doesn’t count if users can’t see it. See Civilization II and Close Combat: A Bridge Too Far.

Ben Waggoner is the chief technologist of Journeyman Digital, a company specializing in pushing the quality envelop for interactive media assets. As the World’s Greatest Compressionist and avid game player, he has an unquenchable need to improve the quality of digital video across the gaming industry.

Halstead York is the head of digital media production for Journeyman Digital. When not producing and consulting, he spreads his digital media gospel through teaching and lectures.

Copyright © 2003 CMP Media Inc. All rights reserved.