Well the problem here is the shader not only defines the colors but the material's interaction with light.

Lips with heavy lipstick are no longer skin. The shader for lips is skin, but the shader for the layer of lipstick on top of the skin is very different. On top of that (literally) you have to model the gloss - the thin layer of clear liquid which creates strong sharp reflections. So there's actually a two-layer material (gloss, lipstick) that has to be layered on top of our 3-layer skin shader, but it has to suppress the top (specular) layer of the skin shader (since the skin/air boundary is now intercepted by the lipgloss and lipstick). If the lipstick is completely opaque, you can just shut off the skin shader underneath. But if you want translucent lipstick, then you have to leak some energy to the next layers, and get back the right amount of diffuse reflection and SSS from the skin. This is very tricky stuff. Just a rough guess, but truly realistic lip gloss and lipstick add another 20 nodes to the shader.

That's why I'll be building this for you in the Pro version. Not only is it tricky setting up the masks, but you also have to build an entirely different shader within the skin shader, and then the mask has to decide which shader to use. It's more than just setting up a Blender node to choose a color.

And you're getting the hard edge because material zones don't overlap and blend. So you actually have to have the dual-material (skin vs. lipstick) on the face and the lips, and make your lip mask do a nice soft blurred edge for the transition.

---

Renderosity forum reply notifications are wonky. If I read a follow-up in a thread, but I don't myself reply, then notifications no longer happen AT ALL on that thread. So if I seem to be ignoring a question, that's why. (Updated September 23, 2019)