I think it's because your camera height is locked to a certain height above the object right below it. If you move an object under the camera, the camera will move up to keep the same distance betweeen it and the new object below it. To deactive it, just click on the little lock in the object properties panel when the camera is selected.