In a nutshell, it's because gamma encoding is not used to produce numbers in displacement maps - it isn't how they are representing heights. They're linear values and so you should not apply any gamma correction when reading them. Gamma encoding is not used to produce normals - it isn't how they represent angles. They're linear values and so you should not apply any gamma correction when reading them. Gamma encoding is not used to produce specular coefficients - it isn't how they represent specularity. Gamma encoding is not used to produce roughness maps - it isn't .... blah blah blah.

Only color images that were intended from the outset to be copied directly into sRGB monitor pixels have gamma encoding, so these are the only images where it is appropriate to apply gamma correction before using them for shader calculations.

---

Renderosity forum reply notifications are wonky. If I read a follow-up in a thread, but I don't myself reply, then notifications no longer happen AT ALL on that thread. So if I seem to be ignoring a question, that's why. (Updated September 23, 2019)