I've found that converted conforming clothes often include things like buttons and belt loops that fly off into space during cloth simulations.  Generally, they tend to fly away from the figure and cloth item, but in your case, you may have one or two aberrant polies trying to intersect with others.  Unfortunately, they're usually very small and hard to find. 

My suggestion is to run a simple 15 frame simulation in which the cloth crumples to the floor.  Then look for any little "bits" hovering around the cloth item.