As vilters has suggested, your test results are rendered meaningless unless you account for the minimum shading rate and map size for your image based displacement, try a MSR setting of 0.05 (in object properties and render settings) and a map size of 4096. check the results.