Unless you're working on a game res model, where normal maps are king, I'd do as suggested and use Normal maps for fine details like pores or weaves in clothing. Displacement is great for the larger forms and details that need to change the silhouette of a character/prop in order to look right. Distance from the viewer is another factor to consider - if something is far away from the viewer in a scene, a displacement map would be a waste of resources, making a bump or normal map the better way to go. Meanwhile, if something is close to the viewer in a scene you may need to use both a displacement map and a normal/bump map.  Ultimately, it's a judgement call on your part when to use which or if to use them in conjunction with each other.