Have you set your key light (the point light in the upper right corner) to inverse square falloff? I have, so when I scale back the IBL  intensity I do see a major difference (since the point light has a smaller range of luminance and is not constant).