I have the same if use high poly clothes or high poly figure.
Try select collide against only important body parts (deselect eyes, center of mass etc). Sometimes it works. Moreover do not calculate dynamic against hands with fingers. If you wish move clothes with hand use "dummy" transparent box parented to hand.

I'm affraid that best solution is a new computer ;)