I'd guess that most of the extraneous polygons are in details of the individual bricks that are hidden, i.e. you only need to have the pillars modelled when they're exposed to the outside, the same for  just about all the internal faces.