Quote -

In the timeline click the keyframe where the camera stops, right click, keyframe --> keyframes options, set to "Linear Out". Click the keyframe where the camera should start moving again, right click, keyframe-->keyframe options, set to Linear In.

Thanks for the tip, I tried that already.... BUT I tried it again after you mentioned it and I realized that my camera is tracking an object...that is moving... I thought it was static between my two hold frames but I checked it again and it turned out that it was also smoothly interpolating between two slightly different positions making my camera harder to control. :(
So I made sure my tracked object is absolutely static between the two hold frames and now it seems that I can better handle the camera.
Thanks all for your help!