Look at the model, look for intersecting polygons, look for very high polygon count.

Are you colliding with hair, fingers or other intricate objects? If so there are work arounds to speed that up.

I've found that generally the cloth room moves pretty quick, if it slows down there is usually a good reason and the cause can be eliminated or at least minimized.