IBL affects diffuse only, so there's no specularity included. If you want to add just specularity without any other changes to the light, use one or two spec only lights to get wet highlights.