In Overgrowth we are going to use baked lightmap shadows, but it is still important that the algorithm is very fast. First, I would like the shadows to update in real-time in the editor as objects are transformed. Second, to calculate ambient occlusion we have to accumulate shadows from hundreds of different virtual light sources, which is impractical if the basic shadowing algorithm is slow.
To calculate lightmaps efficiently, my current plan is to use hardware-accelerated depth-map shadowing to render the shadows for each object directly into its lightmap texture. Most graphics cards don't support textures large enough to render a depth map for the whole scene at an acceptable level of detail, so we will divide each shadow update pass into a number of smaller patches, which we will update one at a time.
Here are some screenshots of my current test, which calculates a high-detail shadow patch in real-time around the camera. I used Hale's Foothold map because it contains complex enough geometry to really test the shadow system. Cast shadows are rendered in grey, and backfaces are rendered in black. The backfaces don't tell us much about the shadows, but I thought they made the pictures more interesting!
The next step is to save these shadows into lightmap textures and combine them with the general lighting equations. I think once the direct and indirect shadows are in for the entire scene, our engine will start to look a lot more polished!