It's not so much tree lab that is the problem, more Bryce itself. The way to get around the out of memory error is to keep the scene file below 350mb.

Having many trees doesn't help much, but you could render just trees for background and then use the render on a 2D vertical face.