Check the mesh subdivision on the conforming shorts.  That may be part of the problem if the resolution is too low.  Reduce the number of iterations for the fit simulation.  Higher values cause unnecessary stretching.  You might need to constrain those parts most affected.  

Have you tried a close fit of the shorts and converted them to a prop first without running a fit simulation?  Afterward, perform a Fitting Room simulation with the prop and a conversion for L'Homme.  Or import the OBJ, save it as a prop then do the Fitting Room.  

I'm just throwing out some options you could try to make it work.