Don't overlook the light's ShadowCam's effect on all this.

Bear in mind that simply zooming-in a light's ShadowCam will have the same effect as increasing the ShadowMap Size, - i.e. increasing the shadow's resolution, - without increasing the render time required when using the larger ShadowMap size.

...consequently zooming the ShadowCam in, while reducing the ShadowMap size, can speed up the renders while retaining the same shadow sharpness.  The ShadowCam's field of view generally ends up generating shadows in areas not actually in your rendering camera's field of view.    (zooming in alone doesn't reduce the render time, just the smaller ShadowMap size)