
You can simply put a color_math node set to subtract in between, like in this screenshot. Then you can set the color_2 of that node to whatever color you want to be the neutral color (=no displacement) - anything brighter than that will push out, anything darker than that will push in.