IBL lights only emit/produce diffuse light.

The usual technique is to add a single infinite light set for "Specular only" (i.e. set Diffuse to Black in material room for that light), and/or reduce the intensity of the IBL (and possibly any other lights) to avoid washing out the colours.
You will need to experiment with combinations of settings to see what works best for you.