Honestly, without pictures its hard to say if you are shooting too high or not.

Something else to consider.. could the walls be broken down so only the parts showing would have to be loaded? In most renders, the back of the castle might not even be seen. (Plus having removable wall allows camera control inside the castle much better.)

Think of it less like a solid structure, and more like a movie set, with parts that come apart for filming.

 

Way too many people take way too many things way too seriously.