Susan Kitchen's book on Bryce 4 mentions one way to achieve the effect. You use the mask render feature and two renders. Its not an ideal situation, as it requires using photoshop (or similar) and working the layers. I think Bryce 3 originally intended to implement a limited antialias function, but it never worked.