One aspect of advanced rendering we haven't discussed yet is stenciling, a technique that can be useful for developing commercial applications. If you want your 3D applications to stand apart from the crowd, you'd be wise to combine stenciling with the texturing techniques you learned about in earlier chapters. This chapter will detail how to use stenciling and show you the different types of effects you can generate with it.

Many 3D games and simulations on the market use cinema-quality special effects to add to their dramatic impact. You can use stencil buffers to create effects such as composites, decals, dissolves, fades, outlines, silhouettes, swipes, and shadows. Stencil buffers determine whether the pixels in an image are drawn. To perform this function, stencil buffers let you enable or disable drawing to the render-target surface on a pixel-by-pixel basis. This means your software can "mask" portions of the rendered image so that they aren't displayed.

1. Perform a bitwise AND operation of the stencil reference value with the stencil mask.
2. Perform a bitwise AND operation on the stencil-buffer value for the current pixel with the stencil mask.
3. Compare the results of Step 1 and Step 2 by using the comparison function.

By controlling the comparison function, the stencil mask, the stencil reference value, and the action taken when the stencil test passes or fails, you can control how the stencil buffer works. As long as the test succeeds, the current pixel will be written to the target. The default comparison behavior (the value that the D3DCMPFUNC enumerated type defines for D3DCMP_ALWAYS) is to write the pixel without considering the contents of the stencil buffer. You can change the comparison function to any function you want by setting the value of the D3DRENDERSTATE_STENCILFUNC render state and passing one of the members of the D3DCMPFUNC enumerated type.

Creating a Stencil Buffer

Before creating a stencil buffer, you need to determine what stenciling capabilities the target system supports. To do this, call the IDirect3DDevice7::GetCaps method. The dwStencilCaps flags specify the stencil-buffer operations that the device supports. The reported flags are valid for all three stencil-buffer operation render states: D3DRENDERSTATE_STENCILFAIL, D3DRENDERSTATE_STENCILPASS, and D3DRENDERSTATE_STENCILZFAIL. Direct3D defines the following flags for dwStencilCaps:

• D3DSTENCILCAPS_DECR Indicates that the D3DSTENCILOP_DECR operation is supported
• D3DSTENCILCAPS_DECRSAT Indicates that the D3DSTENCILOP_DECRSAT operation is supported
• D3DSTENCILCAPS_INCR Indicates that the
  D3DSTENCILOP_INCR operation is supported
• D3DSTENCILCAPS_INCRSAT Indicates that the D3DSTENCILOP_INCRSAT operation is supported
• D3DSTENCILCAPS_INVERT Indicates that the D3DSTENCILOP_INVERT operation is supported
• D3DSTENCILCAPS_KEEP Indicates that the D3DSTENCILOP_KEEP operation is supported
• D3DSTENCILCAPS_REPLACE Indicates that the D3DSTENCILOP_REPLACE operation is supported
• D3DSTENCILCAPS_ZERO Indicates that the D3DSTENCILOP_ZERO operation is supported
  

Direct3D embeds the stencil-buffer information with the depth-buffer data. To determine what formats of depth buffers and stencil buffers the target system's hardware supports, call the IDirect3D7::EnumZBufferFormats method, which has the following declaration:

HRESULT IDirect3D7::EnumZBufferFormats (
REFCLSID riidDevice,
LPD3DENUMPIXELFORMATSCALLBACK lpEnumCallback,
LPVOID lpContext
);

If the method succeeds, it returns the value D3D_OK. If it fails, the method returns one of these four values:

• DDERR_INVALIDOBJECT
• DDERR_INVALIDPARAMS
• DDER_NOZBUFFERHW
• DDERR_OUTOFMEMORY

The code in listing 1 determines what stencil buffer formats are available and what operations are supported and then creates a stencil buffer. As you can see, this code notes whether the stencil buffer supports more than 1-bit -- some stenciling techniques must be handled differently if only a 1-bit stencil buffer is available.

Listing 1. Formats and Operations of Stencil Buffers

HRESULT CMyD3DApplication::CreateStencilBuffer()
{
g_bCanOnlyDoOneBitStencil=FALSE;

DWORD dwStencilCaps =
m_pDeviceInfo->ddDeviceDesc.dwStencilCaps;

if( (!(dwStencilCaps & D3DSTENCILCAPS_INCR) &&
!(dwStencilCaps & D3DSTENCILCAPS_INCRSAT)) ||
(!(dwStencilCaps & D3DSTENCILCAPS_DECR) &&
!(dwStencilCaps & D3DSTENCILCAPS_DECRSAT)))
{
// Must do 1-bit stencil buffer.
g_bCanOnlyDoOneBitStencil=TRUE;
}
else
{
// Prefer sat ops that cap at 0/max, but can
use other
// ones as long as enough stencil bits.
g_StencIncOp=(dwStencilCaps &
D3DSTENCILCAPS_INCRSAT)?
D3DSTENCILOP_INCRSAT:D3DSTENCILOP_INCR;
g_StencDecOp=(dwStencilCaps &
D3DSTENCILCAPS_DECRSAT)?
D3DSTENCILOP_DECRSAT:D3DSTENCILOP_DECR;
}

m_pddsRenderTarget->DeleteAttachedSurface( 0,NULL );

// Get z-buffer dimensions from the render target.
// Set up the surface description for the z-buffer.
DDSURFACEDESC2 ddsd;
D3DUtil_InitSurfaceDesc( ddsd );
m_pddsRenderTarget->GetSurfaceDesc( &ddsd );
ddsd.dwFlags = DDSD_WIDTH | DDSD_HEIGHT
| DDSD_CAPS | DDSD_PIXELFORMAT;
ddsd.ddsCaps.dwCaps = DDSCAPS_ZBUFFER;
ddsd.ddsCaps.dwCaps2 = 0;
ddsd.ddsCaps.dwCaps3 = 0;
ddsd.ddsCaps.dwCaps4 = 0;
ddsd.ddpfPixelFormat.dwFlags = DDPF_ZBUFFER | DDPF_STENCILBUFFER;

if( m_pDeviceInfo->bHardware )
ddsd.ddsCaps.dwCaps |= DDSCAPS_VIDEOMEMORY;
else
ddsd.ddsCaps.dwCaps |= DDSCAPS_SYSTEMMEMORY;

// Get an appropriate pixel format from an
enumeration of
// the formats.
m_pD3D->EnumZBufferFormats(
(*m_pDeviceInfo->pDeviceGUID),
EnumZBufferFormatsCallback,
(VOID*)&ddsd.ddpfPixelFormat );

assert(ddsd.ddpfPixelFormat.dwStencilBitDepth!=0);

g_bCanOnlyDoOneBitStencil=g_bCanOnlyDoOneBitStencil ||
((1<

g_dwMaxStencilValue=
(1<

// Leave g_bUseOneBitStencil set for the
window-resize case.
if( !g_bUseOneBitStencil )
g_bUseOneBitStencil=g_bCanOnlyDoOneBitStencil;

SetMenuStates();

// Create and attach a z-buffer.
if( FAILED( m_pDD->CreateSurface( &ddsd,
&m_pddsDepthBuffer,
NULL ) ) )
return E_FAIL;

if( FAILED(m_pddsRenderTarget->AddAttachedSurface(
m_pddsDepthBuffer ) ) )
return E_FAIL;

// The SetRenderTarget() call is needed to rebuild
internal
// structures for the newly attached z-buffer.
return m_pd3dDevice->SetRenderTarget
( m_pddsRenderTarget, 0L );
}

//------------------------------------------------------
// Name: EnumZBufferFormatsCallback
// Desc: Enumeration function to report valid pixel
// formats for z-buffers
//------------------------------------------------------
static
HRESULT WINAPI EnumZBufferFormatsCallback(
DDPIXELFORMAT* pddpf,
VOID* pddpfDesired )
{
if( NULL==pddpf || NULL==pddpfDesired )
return D3DENUMRET_CANCEL;

// If the current pixel formats match the desired
// ones (DDPF_ZBUFFER and possibly
// DDPF_STENCILBUFFER), copy it and return. This
// function isn't choosy--it accepts the first
// valid format that comes along.

if( pddpf->dwFlags == ((DDPIXELFORMAT*)
pddpfDesired)->dwFlags )
{
memcpy( pddpfDesired, pddpf,
sizeof(DDPIXELFORMAT) );
return D3DENUMRET_CANCEL;
}

return D3DENUMRET_OK;
}

Clearing a Stencil Buffer

The IDirect3DDevice7 interface includes the Clear method, which you can use to simultaneously clear the render target's color buffer, depth buffer, and stencil buffer. Here's the declaration for the IDirect3DDevice7::Clear method:

HRESULT IDirect3DDevice7::Clear(
DWORD dwCount,
LPD3DRECT lpRects,
DWORD dwFlags,
D3DVALUE dvZ,
DWORD dwStencil
);

The IDirect3DDevice7::Clear method still accepts the older D3DCLEAR_TARGET flag, which clears the render target using an RGBA color you provide in the dwColor parameter. This method also still accepts the D3DCLEAR_ZBUFFER flag, which clears the depth buffer to a depth you specify in dvZ (in which 0.0 is the closest distance and 1.0 is the farthest). DirectX 6 introduced the D3DCLEAR_STENCIL flag, which you can use to reset the stencil bits to the value you specify in the dwStencil parameter. This value can be an integer ranging from 0 to 2n-1, in which n is the bit depth of the stencil buffer.