Using 3d clouds or really high altitudes can sometimes give that ragged look, although your clouds don't look that high. One technique that a lot of us use is to render at twice the required size and then re-sample the image down later, followed by a bit of unsharp mask. This can give very nice results. I hate to say it, but if you want really smooth Terragen images, you need to fork out the money. You should call in to the Terragen forum too.