Without seeing the map itself I can only make some educated guesses as to what might work the best. The Magic Wand is a difficult tool to use when trying to create complex selections. You can use it effectively but it usually means very low sampling rates, pixel level selection and many, many added selections (a lot of Shift clicking). There are masking apps available but in my opinion, while they work, they tend to be more trouble than they're worth or at best, not much better than what you can already do with Photoshop as it is.
A better approach might be to combine methods. Maybe start with a Color Range selection or a channel selection and add or subtract from it various Magic Wand selections or editing it by hand in Quick mask. Another thing to consider is duplicating your image and changing it's mode to LAB. Since it creates a different set of channels (based on different information) it may yield a better channel to start from. Another thing to try might be to use adjustment layers to try and radically change the image enough to separate the hair from the skin for the purpose of being able to extract a better selection/mask from either channels or with Color range selection. You may still end up doing a bit of hand tweaking in Quick Mask but any oft these approaches is bound to get you closer to what you want than trying to grab what you want with the Magic Wand, which as it turns out in many cases, is not all that magical.

Deep