Quote - (smooth polygons and raytracing artifacts)I know why, I think. Remember that shadows see the original un-curved polygon, while what is being rendered is a displaced point on the polygon. If the displacement is negative, as happens only with concavity, then the point being rendered is UNDER the original un-bent polygon. Thus, it shadows itself.

An interesting experiment would be to apply a uniform positive displacement in addition to the smoothing. If the uniform displacement was larger than any smoothing-induced negative dispalcement, then the displaced surface will always be "above" its original position. Self-shadowing should not happen in that case.

That sounds very reasonable.  This is troubling.  It seems to me that with character figures, this may cause some problems as you observed earlier, but with conforming clothing models that contain a lot of defailed convoluted geometry, it may cause many more.  In any case I'm going to try some clothing tests with the fantasy suit model I've been working on with Whichway's approach of greatly increased sample radius to see if they come out well.  This may be a good compromise between speed and freedom from artifacts, and a pretty agreeable render time.

Whichway your render with a colored sphere was a good test, thanks for that.

My Freebies