When I was prototyping my makeup shaders for V4 people immediately noticed this and said I needed to fix the hard transition.

So, in my opinion, the solution is to prepare a mask. You need to make a UV mapped white-on-black control map that covers the lip area. Once you have the map drawn, you need to blur the edges. Whether you blur them to extend past the lip zone, or to contract into the lip zone, that's up to you.

In any case, you then apply your lip shading technique controlled by that map. For example, plug it into the Glossy Ks so that the gloss amount fades nicely at the edge of the lip.

If you've blurred past the lip zone, into the surrounding face, you need to apply the lip gloss effect to the face, too.

For this reason, it is probably better to make a contracted mask, one that fades to 0 before extending past the lip zone. That way you don't have to manage the lip shader on the face, too.

Here are some renders of the result of this technique.  Pay particular attention to the corners of the lips - you can easily see the control map at work there.

Creating control maps like this and then working out how to connect them to your shader effects can be a lot of work. That's why I'm supposed to be getting off my butt and finish my system, so you can just buy it. :)

---

Renderosity forum reply notifications are wonky. If I read a follow-up in a thread, but I don't myself reply, then notifications no longer happen AT ALL on that thread. So if I seem to be ignoring a question, that's why. (Updated September 23, 2019)