Interesting effect. Looks like OpenGL's hardware shading is trying to handle a masked displacement that covers and exposes parts of the surface. The hardware shading doesn't do very well.

Here's my Arcade Hotel, which uses a similar technique, in OpenGL with hardware shading on, then rendered in Firefly.

My python page
My ShareCG freebies