I hate to jump in, but I have simple question. Don't you need some sort of specific light source (like the sun or a spot light) to get correct shadows? Even if the source is a light emitter (or several)? Does just the environment sphere create shadows even though the light comes from everywhere? Does the HDRi create shadows from the light parts of the image?

Mike