To get a proper render mask make sure the trees aren't grouped, otherwise you'll get a complete black screen. If they are grouped do not select the group, select the trees individually. Easiest way to do that is the shape icons at the bottom, click the tree icon and all the trees will be selected. Same applies to getting render masks for metaballs.