After you have the keyframe graph open, one quick cure is to place a Spline Break (or several) on the troublesome parameter, around the point where things start to go wrong. This will allow the movement to be natural-looking (splinish) on both sides of the break, but will prevent the movement before that point from "pulling" the movement afterward. Spline Break is one of the buttons in the lower left part of the graph frame.

My python page
My ShareCG freebies