There is no option for shadows on IBL.  You would like your shadows on Infinity lighting which, for daylight, is your primary light source.  For test renders, set back your settings but not so far back that raytracing suffers.  Once you work out your lighting for the scene, then scale up your settings for the final render.  Remember, you are only working with 2 lights.  The renders should go quickly.