Tried 16 and 32 bounces, with the same result. Setting a transparency decreased it somewhat, although they were still visible. Changing the background colors on the refract and reflect nodes doesn't seem to do anything.