Maybe it's because the rays are passing through more than one layer of glass. I read in a thread ages ago that overlapping transparent+refractive textures cause massive increases in render time. You could try setting the refraction back to '100%' (no effect) and see if this improves things... or even remove the panes of glass along the back of the shelter?