If you want your characters hands to interact with a cloth item, I recommend you create and use gloves as collision objects. The characters fingers snag on cloth items, creating undesirable results.

Here's a quick video showing the results...

https://www.youtube.com/watch?v=qawKvP515RQ

Of course you would hide the gloves during render...

Y'all have a great day.