Do you perhaps have AO enabled both on the VSS prop and on your IBL? VSS implements it by default, so you'll want to make sure it's not enabled on your light. Also, this might not be relevant, but how many raytracing bounces in your render settings?