Why do you have a Scatter node in the lower right-hand corner of your Material Room screencap?  You should be connecting a Diffuse map to that SeparateRGB node.  That might be causing the added glossiness you're seeing.  Personally, the Scatter Distance values are way too high but they work for you without causing self-illumination then leave them the way they are.