With that it mind, your specular light should coincide with the direction of the IBL light's main light source - from the right in the image. Right ... I'm seeing that they almost have to be in exactly the same position for the effect to be believable. I'm also finding that the default settings are too strong as well.