Configuring the Stenciling State

You control the various settings for the stencil buffer using the IDirect3DDevice7::
SetRenderState method. Listing 2 shows the stencil-related members of the D3DRENDERSTATETYPE enumerated type.

Listing 2. Control Settings of Stencil Buffer

These are the definitions for the stencil-related render states:

• D3DRENDERSTATE_STENCILENABLE Use this member to enable or disable stenciling. To enable stenciling, use this member with TRUE; to disable stenciling, use it with FALSE. The default value is FALSE.
• D3DRENDERSTATE_STENCILFAIL Use this member to indicate the stencil operation to perform if the stencil test fails. The stencil operation can be one of the members of the D3DSTENCILOP enumerated type. The default value is D3DSTENCILOP_KEEP.
• D3DRENDERSTATE_STENCILZFAIL Use this member to indicate the stencil operation to perform if the stencil test passes and the depth test (z-test) fails. The operation can be one of the members of the D3DSTENCILOP enumerated type. The default value is D3DSTENCILOP_KEEP.
• D3DRENDERSTATE_STENCILPASS Use this member to indicate the stencil operation to perform if both the stencil test and the depth test (z-test) pass. The operation can be one of the members of the D3DSTENCILOP enumerated type. The default value is D3DSTENCILOP_KEEP.
• D3DRENDERSTATE_STENCILFUNC Use this member to indicate the comparison function for the stencil test. The comparison function can be one of the members of the D3DCMPFUNC enumerated type. The default value is D3DCMP_ALWAYS. This function compares the reference value to a stencil-buffer entry and applies only to the bits in the reference value and stencil-buffer entry that are set in the stencil mask. (The D3DRENDERSTATE_STENCILMASK render state sets the stencil mask.) If the comparison is true, the stencil test passes.
• D3DRENDERSTATE_STENCILREF Use this member to indicate the integer reference value for the stencil test. The default value is 0.
• D3DRENDERSTATE_STENCILMASK Use this member to specify the mask to apply to the reference value and each stencil-buffer entry to determine the significant bits for the stencil test. The default mask is 0xFFFFFFFF.
• D3DRENDERSTATE_STENCILWRITEMASK Use this member to specify the mask to apply to values written into the stencil buffer. The default mask is 0xFFFFFFFF.

The D3DSTENCILOP enumerated type describes the stencil operations for the D3DRENDERSTATE_STENCILFAIL, D3DRENDERSTATE_STENCILZFAIL, and D3DRENDERSTATE_STENCILPASS render states. Here's the definition of D3DSTENCILOP:

typedef enum _D3DSTENCILOP {
D3DSTENCILOP_KEEP = 1,
D3DSTENCILOP_ZERO = 2,
D3DSTENCILOP_REPLACE = 3,
D3DSTENCILOP_INCRSAT = 4,
D3DSTENCILOP_DECRSAT = 5,
D3DSTENCILOP_INVERT = 6,
D3DSTENCILOP_INCR = 7,
D3DSTENCILOP_DECR = 8,
D3DSTENCILOP_FORCE_DWORD = 0x7fffffff
} D3DSTENCILOP;

These members serve the following purposes:

• D3DSTENCILOP_KEEP Indicates that you don't want the entry in the stencil buffer updated. This is the default operation.
  
  
• D3DSTENCILOP_ZERO Sets the stencil-buffer entry to 0.
• D3DSTENCILOP_REPLACE Replaces the stencil-buffer entry with the reference value.
• D3DSTENCILOP_INCRSAT Increments the stencil-buffer entry, clamping to the maximum value.
• D3DSTENCILOP_DECRSAT Decrements the stencil-buffer entry, clamping to 0.
• D3DSTENCILOP_INVERT Inverts the bits in the stencil-buffer entry.
• D3DSTENCILOP_INCR Increments the stencil-buffer entry, wrapping to 0 if the new value exceeds the maximum value.
• D3DSTENCILOP_DECR Decrements the stencil-buffer entry, wrapping to the maximum value if the new value is less than 0.
• D3DSTENCILOP_FORCE_DWORD Forces this enumeration to be compiled to 32 bits; this value isn't used.

Let's walk through some code that uses the stencil buffer while rendering a scene. This code is from a sample that shows how to draw shadows. For now, don't worry about how all this code generates shadows-the algorithm is described later in the chapter.

The shadow-rendering code starts out by disabling the depth buffer and enabling the stencil buffer:

//--------------------------------------------------
// Name: RenderShadow
// Desc:
//------------------------------------------------
HRESULT CMyD3DApplication::RenderShadow()
{
// Turn off depth buffer and turn on
stencil buffer.
m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_ZWRITEENABLE,
FALSE );
m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_STENCILENABLE,
TRUE );

Next the code sets the comparison function that performs the stencil test by calling the IDirect3DDevice7::SetRenderState method and setting the first parameter to D3DRENDERSTATE_STENCILFUNC. The second parameter is set to a member of the D3DCMPFUNC enumerated type. In this code, we want to update the stencil buffer everywhere a primitive is rendered, so we use D3DCMP_ALWAYS:

//
// Set up stencil comparison function,
reference value, and masks.
// Stencil test passes if ((ref & mask)
cmpfn (stencil & mask))
// is true.
//
m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_STENCILFUNC,
D3DCMP_ALWAYS );

In this sample, we don't want the stencil buffer to change if either the stencil buffer test or the depth buffer test fails, so we set the appropriate states to D3DSTENCILOP_KEEP:

m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_STENCILZFAIL,
D3DSTENCILOP_KEEP );
m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_STENCILFAIL,
D3DSTENCILOP_KEEP );

The settings in listing 3 are different depending on whether a 1-bit or a multibit stencil buffer is present. If the stencil buffer has only 1 bit, the value 1 is stored in the stencil buffer whenever the stencil test passes. Otherwise, an increment operation (either D3DSTENCILOP_INCR or D3DSTENCILOP_INCRSAT) is applied if the stencil test passes. At this point, the stencil state is configured and the code is ready to render some primitives.

Listing 3. Multibit Stencil Buffer

if(g_bUseOneBitStencil)
{
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILREF,
0x1 );
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILMASK,
0x1 );
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILWRITEMASK,
0x1 );
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILPASS,
D3DSTENCILOP_REPLACE );
}
else
{
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILREF,
0x1 );
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILMASK,
0xffffffff );
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILWRITEMASK,
0xffffffff );
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILPASS,
g_StencIncOp );
}