I ran some test renders using your settings and began scaling back on IBL intensity, all the way back to 25%.  And I found that it changed very little in all my images.  Have you looked at various IBL intensities in your experiments?  I will do further testing to verify my conclusions.