I was using the Main Camera as it loads default: 38mm.

I suppose if you were going to do the Hitchcock (Spielberg steals the technique and uses it on a closeup and he gets a credit?) dolly-zoom move you would have simply have to animate your fall-off distances relative to the animation of your focal with matching keyframes. That or use another program with more straight forward lighting.