It might be coming from "fix seams" if your using it.

You may try not using fix seams and see if that helps.

Fix seams creates a ribboned pattern at the seam which works fine for color but the changes in brightness will effect the displacement map. I know this seems weird since its outside the perimeter of the map, but it still effects it