You probably just need the section you'll see as the background portion at the high resolution.

The more "staged" set up I was contemplating was aimed at addressing this.

If you're stitching photos, maybe there's a way of stitching at higher density for the portion you'll have a background? But lower density elsewhere?

Likewise I expect in Vue one could render the whole spherical panorama at lower res, then a portion at the higher res, then blow up the low res spherical panorama to the size of the full hi res, maybe blur it a bit, and then overlay the hi res portion, at the correct place?

Would involve more postwork of the panorama texture... but would save a lot of render time I expect? Plus some maths to calculate the angles / proportions to render the high res portion at, so that it is a tile on the surface of the total sphere...