Big fancy programs like Maya can reduce polys. Once you make a terrain and collapse the mesh you are stuck with the polys you have. I'm assuming you have bryce 6. You have to control the mesh resolution before you boolean.

There is a wireframe resolution button on the right side of the Bryce desktop. You have three settings, Motion, static, and selected. Click on selected and you will see numbers 8, 16, 32, 64, 128. The lower the number the less polys in the terrain. Make a version and save then start by setting selected resolution at 8. Group and collapse and look at the mesh. If it is too smooth go back to the saved version crank the resolution up to 16 and try again.