I know what you mean, the edges appear pixelated even with anti-alias. If it's really bad I use a smudge or blurring tool. Most of the time I render to .TIF and use the mask channel to float the figure in a layer, then I feather the edges. -JH.