try this, IIRC (it's been a while since I had to use a lip mask)... run the mask into K's of the glossy node. See if this helps.

There's also some extra bells and whistles that you can add via some blender nodes.

If I use glossy on lips in a render I usually just postwork any problems in photoshop.