Whatever node is producing your bump - whether it's a texture map or some sort of prodecure - usually has an amplitude that is added (or multiplied?) to the main node's bump value, so maybe balancing those two values can give you something closer to what you're looking for. You could also increase the relative scale of your texture, or apply your bump to the main node's bump but use noise or something in the alternate specular.