Here's another (untried) suggestion.

Leave the figure out of the way, and work on the duster first. Use the cloth square, and set a few vertices in the middle to the choreographed group. Move the unclothified square around in a dusting motion, then simulate.

That should give you the duster, probably after some experimentation and swearing. ;)

Then you can animate the figure's hand to follow the cloth. Set IK on that arm to help with that.