First image: 2024 resolution basic. Second image: 2024 with areas brushed away. Third image: subsampled to 1024 resolution. Ok, here's my take, all in bryce: Create a new terrain. Set it to 512 or 1024 resolution. Paint whatever terrain you want. Duplicate it once. Move the resulting terrain a little more than 1.5 Bryce units down from the first one. Enter the terrain editor for the new terrain. Increase the resolution to 2048 (if you have a sufficient amount or RAM) or to 1024 (if you have less than 400 mb of RAM) Clip the terrain a little bit on the bottom. Drag basic noise until it looks like you have just applied a strong depth of field. (not too strong) Click slope noise 3 times. Click height noise until your "trees" are as tall as you want them. Exit the terrain editor. Apply a green material to the spikey terrain, and a mountainy material to the first terrain. Do a test render. If the trees are too tall, sink the terrain down further until you have them where you want them. Now you can apply effects to the trees. Use the regular brush to clear areas. (strong stroke to erase completely, light stroke to thin out areas) You an also use the other filters: gassian edges, square edges, etc.) Use the mounds effect to create bunched tree areas. Sawtooth for ski trails or logging paths. Drag subcontours to the left to create a really dense forest. FYI, Hold down space while clicking an effect to paint with that effect on the canvas.
Excellant work, the dual terrain idea has several applications. I've done trees that way too and it works pretty well for the background. There's another trick you can do with the dual terrains. 1. Set up your terrain, then duplicate it. Make the second terain just a fraction of a unit lower than the first, but in the same exact location. 2. Give the second terrain a water texture, or one of those blue stone textures. 3. Use the erode tool on the upper terrain. This is a wip and its overdone, but you get the idea.