Hmm... it even happens, when the model is unwelded ? Strange...
I sometimes had this kind of problem, when the cam was inside an object. Even worse, when passing a door.
So, for architecture, I don't "dig" rooms into massive house blocks, but construct them (as does the mason) from floor slabs, individual walls and so on, for each storey,
and keep them apart, don't bake nor bool'em.
This allows also, to duplicate and offset, to have different mats for the different faces.
But if this effect happens,
even when all elements are unwelded, so the cam isn't inside something, the reason must be elsewhere :(