There is no one "right" answer for this - it is very scene and HDRI map dependent.  This fix is playing with the following options. The are interelated so while only one may fix the issue, you may need to use a combination of them:

1. increase Photon Accuracy
2.  Increase Photon count
3. Enable "Improved Edges"
4. Engage "Full Ray Tracing"

While this can occur in with all HDRI maps, the low resolution maps  are more prone to creating this problem. If you think of every pixel in the HDRI map as a light bulb, the maps with greater resolution have more "light bulbs" to illuminate your scene accurately.