My only petpeeve with IBL is it's tendency to fill shadowed areas unrealistically.  Using it at low intensity values will compensate but it takes some tweaking.  Try placing a structure with openings in an outdoor environment and render the scene with IBL only then IDL only.  You will find that the IBL will light the interior evenly while IDL will only take light sets into consideration.