From the sounds of it, maybe researching 'Vector Displacement' might give pointers and trigger something in you.

Only reason I say that is because of how Vector Displacement behaves compared to Standard Displacement, and it sounds like what you're trying to do is pretty much get at what would essentially be the same sort of crevice information as Vector displacement uses, same trick if you like.  That new Vector Displacement stuff allows undershoots etc, pretty cool, and it definitely needs to know the sort of stuff you're getting at.

Not saying it's the same, but those 2D Vector Displacement maps end up creating 3D crevices and protrusions and even under and overshoots, so the way it works might trigger something in you.