If the lip sync is correct but consistantly off , I resynch in a video editor when doing the final clip.
I have used mimic , which generates an animated pose file, and on the animation timeline you can select and move those keyframes back to match the sound file.