Not sure it was the best solution but decided to make the camera linked to a small object that I move around the room.  That seemed to reduce some of the problems I had earlier. At least this way its not as easy to mistakenly move one of the camera movement points.