The nice thing about the YUV color space is that our eyes can see much finer brightness detail than color detail, so U and V can be stored at a lower resolution than Y. The JPEG format makes use of this initially converting from RGB to YUV and then throw away three quarters of the U and V information, half in each direction. When it does this, JPEG gets a guaranteed 50% compression before it even starts doing its discrete cosine transforms.
The YIQ and YUV color spaces are virtually identical. YIQ is just a thirty-three degree rotation of the UV axes. In fact, you can generate NTSC video from YUV quite easily since the hue ends up in the phase of the color carrier, and since YIQ is just a thirty-three degree hue rotation from YUV, you just need to offset the chroma waveform by thirty-three degrees to compensate.
So why does NTSC carefully specify the YIQ color space if YUV would have been identical? The reason is that the sensitivity of your eyes to color detail is also variable. You can see more detail in the orange-cyan color direction, the I channel of YIQ, than you can in the green-purple color direction in the Q channel. Therefore, you can go one step beyond the UV decimation of JPEG and store Q at a lower data rate than I. This would not have worked well with YUV.
Backwards compatibility is another reason why YIQ was a good choice for the NTSC color space. The first NTSC in March 1941 created the standard for monochrome television. The second NTSC issued the color standards in 1953, by which time there were tens of millions of monochrome TVs, and many television stations. To be successful, as opposed to CBS's failed field sequential color system, the new color system had to be backwards compatible. Owners of color TV sets would want to watch monochrome broadcasts, while color broadcasts needed to be visible on monochrome TV sets. That they managed to make this work at all is simply amazing.
YIQ works well for backwards compatibility because the Y signal is equivalent to the monochrome signal. By carefully encoding the color information of I and Q, so that monochrome receivers would ignore it, the NTSC managed the necessary backwards compatibility.
This magic trick was only possible because our eyes are less sensitive to color detail. The I and Q signals can be transmitted at a much lower rate without our eyes noticing the relative lack of color information. This makes it easier to squeeze I and Q into an already crowded wire. Y was given a bandwidth of 4.2 MHz, which Nyquist tells us is equivalent to roughly 8.4 million samples per second. When I and Q were added, I was given 1.3 MHz of bandwidth, and Q a paltry 0.6 MHz. Thus, the total bandwidth was increased by less than 50%.
The lower frequencies of I and Q were justified by studies of the human visual system, which showed less sensitivity to detail in those areas. However even the 8.4 million 'samples' per second of the luminance channel is not very high. If we work out the data rate in a 640 by 480 frame buffer displayed 30 times a second (or, equivalently, a 640 by 240 frame buffer displayed 60 times a second), we get a figure of 9,216,000 pixels per second. That wouldn't be so bad except that NTSC is a real-time standard. A television spends about 25% of its time doing vertical blanking and horizontal blanking. During this time, when the electron beam is stabilizing and preparing to draw, the 4.2 MHz signal is essentially wasted. So, we really only have about 6.3 million samples with which to display over 9 million pixels per second. This means that if you put really fine detail into your screens, perhaps thin vertical lines or small text, then it is impossible for the television to display it.
Although the luminance signal is allowed to go as high as 4.2 MHz, it can encounter problems long before it gets there. Because the color information is stored at 3.579545 MHz, you can't also store luminance at that frequency. The color frequency was chosen to be a rare frequency for luminance, but if you accidentally generate it then either the encoder will notch filter it out, or the TV will interpret it as color!
If you do put in fine horizontal detail then the best thing that can happen is it may be filtered away, rendering your text unreadable. The alternative is actually worse. If the high frequency detail isn't filtered out then your television may have trouble dealing with the signal. If you are connected to the TV with an RF (radio frequency) connector then the high frequencies will show up in the sound. With a composite connector, they may show up in the chroma information, giving you spurious colors.
Similar problems happen with the I and Q information. Quadrature modulation is basically equivalent to interleaving the I and Q signals into one signal (Blinn 98). I and Q can only be separated at the other end only if their frequencies are each less than half of the color carrier frequency of 3.58 MHz. The original bandwidth limitations of 1.3 and 0.6 MHz for I and Q were chosen to respect this limitation, and because Q is less important. Since 1953 filters have gotten a lot better so that I and Q can now both be transmitted at 1.3 MHz.
Still, 1.3 MHz is not very fast. Color changes in NTSC happen more than three times slower than luminance changes. If you have thin stripes of pure magenta on a 70% green background then the Y value will remain constant, but I and Q will oscillate madly. If that oscillation is filtered out, your thin lines disappear. If it isn't, then the color information will show up as inappropriate brightness information. Luckily, the worst cases tend to be rare such as purple text on a green background, which is not something that people do intentionally.
There are excellent video encoders available which do a great job of filtering out the abusive graphics we throw at them. However filtering is not perfect, and the more you push the limits, the more likely you are to run into troubles. Even if the television encoder in your target console filters your high frequencies down to exactly the proper limits, that's a small consolation if many of your customers hook up TVs that handle the boundary cases badly.
I received the inspiration for this article was when I was staring at a section of one of our console game menu screens displayed on a TV. The area I was staring at appeared to be solid green, yet it was flickering far worse than any interlaced flicker I've ever seen. It turns out that the source image wasn't solid green. The color was changing quite frequently between black and green. Even though the color changes were not showing up at all, they were getting to the TV in some distorted form, and causing serious problems. Since the color carrier takes four fields to repeat itself (227.5 cycles per line * 262.5 lines per field means 59718.75 cycles per field) in the worst case this can cause artifacts which repeat every four fields, or fifteen times a second, and flicker twice as bad as interlaced flicker.
The solution was to remove the fine detail. Since it wasn't visible on the TV anyway, the only effect was that the image stabilized. This served as a dramatic lesson in the importance of avoiding high frequency images.
It is incredibly difficult for a television to separate the chroma information from the luminance information. Comb filters help, but they come with their own set of problems. Probably the most common symptom of poor chroma-luminance separation is called chroma crawl. This happens when high-frequency chroma information is decoded as luminance. This causes tiny dots along vertical edges. These dots appear to crawl up the edge as the four-field NTSC cycle puts the excess chroma in a different place for each field.
The underlying problems with high horizontal frequencies are quite different from those for interlaced flicker, but the solutions are very similar. Avoid high frequencies, and consider filtering your images horizontally to avoid them. Using antialiased fonts and dimmed lines to represent thin lines can dramatically improve your effective resolution.
The most important thing to do is antialiased rendering. The highest frequencies that you will encounter in your game will be on edges that aren't antialiased and on distant textures that aren't mip-mapped. Modern console hardware is fast enough to avoid these problems, and it is important to do so. Occasionally, you will hear people claim that the blurry display of television will hide aliased graphics and other glitches. This is not true. Aliased edges are the biggest causes of chroma crawl and interlaced flicker, and distant textures that aren't mip-mapped cause such random texture lookup that no amount of blurring can cover it.
Learn the signs of excessively high frequencies. If a still image is flickering, try filtering vertically to see if interlace is the problem. If the flicker remains, or if you see color in monochrome areas or chroma crawl along sharp edges, try filtering it horizontally. With a little bit of effort, you can translate an image to YIQ, filter Y, I and Q separately, and then translate it back. This lets you find out exactly where the problem is.