I don't understand Photoshop masks and so rarely use them; it means that I've come up with some useful useful self-taught alternatives that work well with Bryce renders, I think. My two cents' worth:

Use a cloud brush as the eraser, that way the 'erasings' will not be too solid and will look more natural.

The biggest single tip I can come up with is to use Bryce to make mask renders of the various elements in your picture, say near, middle and background objects, and use these to produce seperate layers for each element of the picture in Photoshop over a background of the flat, basic Bryce render.  You can then paint your FX in front of, behind and even between your scene elements; it works especially well with smoke/steam/clouds and light rays.  I've attached a few samples where I used this technique with Bryce.

http://www.renderosity.com/mod/gallery/index.php?image_id=1617658

http://www.renderosity.com/mod/gallery/index.php?image_id=1616421

Caution - this one's got gore and slight nudity:

http://www.renderosity.com/mod/gallery/index.php?image_id=1393832

And one with Poser, the principle's the same:

http://www.renderosity.com/mod/gallery/index.php?image_id=1736193

My self-build system - Vista 64 on a Kingston 240GB SSD, Asus P5Q Pro MB, Quad 6600 CPU, 8 Gb Geil Black Dragon Ram, CoolerMaster HAF932 full tower chassis, EVGA Geforce GTX 750Ti Superclocked 2 Gb, Coolermaster V8 CPU aircooler, Enermax 600W Modular PSU, 240Gb SSD, 2Tb HDD storage, 28" LCD monitor, and more red LEDs than a grown man really needs.....I built it in 2008 and can't afford a new one, yet.....!

My Software - Poser Pro 2012, Photoshop, Bryce 6 and Borderlands......"Catch a  r--i---d-----e-----!"