It's because the Weave node is too regular and has no twist in the yarns that make it up. Real fabric behaves the way it does because fibers are twisted about each other. This affects not only the weave of the fabric (A silk taffeta and a boucle, for instance), but the behavior of the light interacting with the fabric, because at its smallest part, the light is interacting with a twisted object instead of a flat one.

Honestly, the weave you have set up in your picture does not bring to mind a high thread count fabric, either. There's a lot of spacing between the woven threads. Even moreso than in an organza. This looks more like a gauze fabric, which I'm fairly certain you're not wanting to achieve.

If you're set on using the Weave node, use it only for the Bump, not for the diffuse. Unhook it from the clouds. Don't use it on the Displacement, either, unless you're doing a macro shot where it REALLY matters. Instead, save Displacement for such details as seams and embroidery stitching.

花 | 美 | 花美 | 花火 
...It's a pun.