I always wondered about diffuse maps myself,like to give stuff the illusion of dirty. For your issue couldn't you just use an AO atmosphere?

"Many are willing to suffer for their art. Few are willing to learn to draw."