The problem is that your cloth is not "flattenable". In the real world, clothes are made from flat pieces of fabric. As a crude example, suppose you modeled a hemisphere in your favotire 3d app. Now, if you try to flatten it with a cloth simulator, it won't work, because it is inherently non-flat. The only way to flatten such a thing is to changed the triangle edge lengths appropriately, or to "cut" it strategically. That is why real clothes have darts. Of course, this doesn't really solve your problem, but it may lead you to a solution.