Did you try and adjust the Shadow Min Bias on the Ray Traced light? It might not eliminate the problem but it may make it less visible. Also, the quality settings on reflect/refract have a lot to do with the "grainyness" of the result, and so does pixel samples and Minimum shading rate in the Render Options. It could be some combination of all of these working together to produce the final result.