You are right.  I did a bit of playing around with loosening the fit inside VF so the clothing was slightly further away from the body, and the simulation time went way down.

So for those who are having simulation time problems, it appears to be a result of the default fit being a bit too close to the body.  Loosen the fit and the improvement can be dramatic.