For this last render, I colored the specular color chip gray, or 191,191,191, while retaining all the modified values already used for the previous image.  It's purpose is to basically match the light source, a pure white infinity light.  I does soften the specularity a little more.

Of course, I could refine this even more by introducing a specular map, if available.  But, I feel, that primarily the light source should be the first consideration before attempting to tweak shader nodes to achieve what you want.