You're not new to this, so I'm assuming you've already unticked Luke's hair (which for some reason ticks itself) as a collision object, as well as head and hands and feet. And you've checked to make sure no part of the cloth intersects with the target figure (even 'round the back). The other thing I tend to do is load textures - especially hi-res ones - after the sim to conserve on RAM.

Monterey/Mint21.x/Win10 - Blender3.x - PP11.3(cm) - Musescore3.6.2

Wir sind gewohnt, daß die Menschen verhöhnen was sie nicht verstehen
[it is clear that humans have contempt for that which they do not understand] 

Metaphor of Chooks