Actually I think it's just me misinterpreting what to expect.

Math node or no math node, putting the displaced part of the map is always going to create a step when butted up to a mesh that doens't have one.  When I use the math node I only get the step at displaced parts of the map, so yup, I get it now.

For some reason I had it in my head that using the math node would somehow blend everything - duh!