From Pompi Pompi Gamers and Developers Blog 

Abstract
Suggesting a technique of implementing a Geometry based Anti Alias(GAA), instead of using the built in 3D graphics AA. This technique has the potential to improve performance and quality of the AA under certain conditions. In short, I am using a geometry based or fins geometry outline technique, and use it to draw translucent smooth edges with the same color of the original 3D model.

Pre Requirements
In order to implement this algorithm, one need to have a geometry based or fins geometry Non Photo Realistic outline shaders implemented. 
This type of shader use a mesh of fins. For every edge or intersection between two triangles in the original mesh, there is a fin in the fins mesh. When rendering this fins mesh, we select in the shader which fins will be visible and which will not. The fins that are visible are those that one of the two triangles the fin belongs to is facing the camera and the other doesn't. This cause that the viewable fins are the outline fins of the projected mesh on the screen space.
Some articles explaining how to do this, can be found here:
Single Pass GPU Stylized Edges
NPR with Pixel and Vertex Shader

How it works
Once you have the fins geometry, either by precalculating or using the geometry shader, you need to add additional information to the vertexes of the fin. If you are drawing a black fading outline using the fin geometry, you already have a V coordinate that is 0 where the fin is touching the 3D model and 1 at the farest point of the fin from the 3D model. For the faded black outline, we use this V coordinate to determine what alpha value we want the fin to have in the pixel.

The additional information we need to add, is all the information we have in the 3D model vertex where the fin come out from, which is used to calculate the final color of the correspondent triangles. For instance, if the vertex keeps a normal to calculate diffuse lighting, we need to keep this normal in the correspondent fin vertex as well.
We pass this information from the fin vertex to the pixel shader and on the pixel shader we calculate the color just like we would calculate the color in the pixel shader of the 3D model.
The next step is to multiply the fin color with an alpha, much like we did with the faded black alpha, and we get the anti aliased edge.
The results are shown in the following images of a ball and a scene from the game Labyrinthica: The quest of lima. For comparison I have included a render with no AA and a render with AA from an NVIDIA Geforce 8600 GTS set at 16Q.
Also a render of GAA outline without the 3D model.

Aliased ball

Built in Anti Alias

GAA ball

GAA only

Performance, quality and issues
GAA might prove to have better or worse performance than the built in AA of the graphics card.
The main bottlenecks of GAA are the vertex shader, in case no geometry shader is available, and part of the bottlenecks that the 3D model suffer from depending on the shader it use to calculate color. For instance, a use of texture might cause a memory bus bottleneck. On the other hand, GAA will most likely be calculated for only small part of the pixels on the screen, unlike the graphics card AA which is calculated for all the pixels in the screen.
In terms of quality, GAA has several differences from built in AA. You can control the fin's thickness, making it possible for smoother and more pleasing AA. It does not make textures or models' surfaces blurred or smoothed. And you can add a black outline for NPR rendering almost for free.

GAA might not work for edges with very high curvature or surface with unsmooth normals or unsmooth parameters used for calculating the color. Problems may arise due to the fact that the color on a specific vertex of a mesh is not the same color as the pixel next to it. Because the data at the pixel is the interpolation of 3 vertexes and not the exact data in the closest vertex. This might cause problems such as back facing normals on the fin.