If all of the buildings are to be deformed together like that (and that is a good idea); they will have to be relatively close to the same mesh resolution. I mean if you deform a hi-poly model and a lo poly model with the same deformers, the hi poly will be much more distorted than the lo poly one.
I still think it would best for the individual modeller to add some irregularities on there own because:
a) Some specific irregularities are better, IMO, than strictly noise irregularities and;
b) It will add character to the individual structures and;
c) If the structures already have some wear and tear on them, they won't need as much deforming, and that will give some breathing space for the mesh resolutions.

Friends don't let friends use booleans.