As far as i remeber you have to put the mask in the alphachannel or apply a mask object to the layer - adding a mask layer is the little round whie icon in the layer window - hope it makes some kind of sense :-)

Regards
Arne