My guess would be that this is because the simulator is folding polygons diagonally, resulting in highly non-planar polys; looking at the preview image, one can see several polys that have opposing vertices on the fold, rather than adjacent ones. I had the same problem with a very simple bedsheet simulation I tried to do.

Sorry I can't provide a solution to the problem, other than recommending an even higher resolution, which would entail exporting the high-res square and subdividing it in a modeling program, then re-importing it. I haven't used simulation until very recently, so I don't know what tricks there are to avoiding this without leaving Poser; I'm also looking forward to whatever answers you get.

Good luck!