the graininess depends on the dynamic range of the HDRI. If you look at the sphere renders that accompany the HDRI's at HDRIHaven you will note that the ones that produce sharp shadows will be the more dynamic and produce more noise wen rendered. If you just use a JPG you'll have even less grain and can cut your total samples down for a quick render. This image uses a small jpg of a room and an area light placed where the window is to get some highlights back in the render. It only used 40 samples (1600). So HDRI's of overcast daylight, low dynamic range HDRI's and low res jpg's produce the least noise.

W10, Ryzen 5 1600x, 16Gb,RTX2060Super+GTX980, PP11, 11.3.740