After removing the "Ambient" value from the shader of the mostly invisible Construct, a whole more samples seem to be able to find a light, the whole scene takes much less time to render.

I've checked with different scenes as well. Self lit polygons can make removing the grain much harder if they are unreachable from the scene. Especially in dark scenes this can make huge difference. After checking some old props and scenery I bought here, I found a lot of shaders with some ambience in it, even in places where I would not expect this.

A ship in port is safe; but that is not what ships are built for.
Sail out to sea and do new things.
-"Amazing Grace" Hopper

Avatar image of me done by Chidori.