Next test: 1 point light, 100%.  Only light in scene, parked over the chandalier. 12 seconds. Shadows on: 35 seconds with Blur radius 1, 13 seconds with it set to 0.  Nice, sharp shadows as you'd expect from a point source.

Interestingly, even without shadows getting away from the damn infinite lights makes it better (due, as you pointed out BB, even without shadows the light is actually coming from somewhere).