File size doesn't really tell us anything - especially as you haven't said what format you are saving in. It sounds as if it's getting bigger because there is less background, so the compression doesn't shrink it down as much - if so that has absolutely no bearing on quality.