erm.
it would only be habitable during the summer?

see, you've got one chimney on one side of the building. the flue size would be good for 1-2 fireplaces tops.... there's nothing on the other side at all, that would be relying on the 1-2 fireplaces you would have.. and for the size of the building, your looking at least 4 rooms plus hallways on the upper levels, maybe a ballroom on the ground level.

one of the fireplaces would be the kitchens in base of the building. so only 1 for the persons living there.  plus you've got windows in what would be the Flue run......