A side note, in response to ice-boy's question: a specular (only) light does not produce shadows.

This is one IBL (with AO) and one infinite specular light with raytrace shadows switched on just to demonstrate the point.

(edit) with separate diffuse and specular lights, you just need to be a bit careful that the specular highlights remain consistent with the directional lighting.

"If I were a shadow, I know I wouldn't like to be half of what I should be."
Mr Otsuka, the old black tomcat in Kafka on the Shore (Haruki Murakami)