Rendering Plants with Smooth Edges

When we are rendering many objects on screen at the same time, we only have a few hundred polygons to allocate to each individual plant. We use intersecting alpha-mapped planes to give the impression of much greater detail. Here is a picture of what our desert tree looks like with the alpha channel used for color instead of transparency:

If we just turn on blending, it starts to look like a tree, but there are serious artifacts around the leaves. The transparent pixels in the nearest leaves are occluding the background leaves. The following pictures are zoomed in ~200% for clarity; click to see the full image.

We can fix that problem by turning off depth-writing. But now you can see the background leaves in front of the foreground ones!

We could draw everything in the correct order by sorting each quad from back to front, but this is slow, and intersecting quads would have to be split. In practice, the most common solution is to turn off blending, and turn on alpha-testing. That is, for each pixel it checks if the alpha value is above some threshold, and if so, draws it at 100% opacity.

Now we have everything drawn in the correct order, and it looks like a very detailed tree, but we have lost the smooth, blended edges. This is especially problematic when viewed from far away. Here is an enlarged distant view with alpha-testing used on the left, and blending on the right.

One way to get smooth edges using alpha testing is to use the multisample buffers. By using a different stippling pattern in each alpha buffer, you can combine them to get fairly smooth gradients.

However, this option is not supported well on many 3D cards, and only works well with 4x or more samples per pixel. What we are doing right now in Overgrowth is drawing the plant using alpha testing, and then drawing it again using blending with depth-writing disabled. This is not perfectly accurate, but it is compatible, easy, and fast.

Please let me know if you have encountered this problem before, or know of a better solution!