You will need to create a lip mask: that is to say, on a texture map, outline the lips only, inside that area, make a white area over the majority of the lips and blur the edge of it into black, but not running into or touching the edge of the lip area, but as close as you can get it. Use that image then as the map for the specular. It will make the glossiness fade out toward the edge of the lips and make a gradual fade rather than a stark line.

Laurie