In Poser, a zero value (black) represents zero displacement. (I believe that most other apps use mid-grey to represent zero displacement).

So your mid-grey background is creating positive displacement over the whole face.

Before plugging your displacement map into the displacement channel on the PoserSurface node, you have to subtract the exact background RGB value (128,128,128 in your case), using a Math node in the Advanced Material Room view.

"If I were a shadow, I know I wouldn't like to be half of what I should be."
Mr Otsuka, the old black tomcat in Kafka on the Shore (Haruki Murakami)