Any idea how I can fix this?

With Shadows

Without Shadows

Obviously, I'm trying to get rid of that "dirty" effect.

Properties for the Main Light

The main light is the only shadow casting light in the scene. There are only two other lights, infinite lights, just for fill.
I've tried messing with the shadow min bias and blur radius, to no good effect.

Here are the render settings, in case that's of any help.