yeah
you'd need to bring in the buildings in as a separate layer and work on that.

i wouldn't use the eraser tool, i'd use the masking tool cause it's non-destructive and you can change your mind if you want (can't do that if you've eliminated the pixel information).

look at old buildings to see how the decay looks like. it shouldn't be too difficult to replicate.

:)
retrocity