Everything is interpolated from the previous keyframe. Which, based on your description, would probably be frame 1. If you don't want the camera to start zooming from the beginning, you'll have to set another keyframe somewhere in the middle of your animation sequence.