I think what your problem is is that in your transmap you didn't make the black BLACK.The transparent part has to be completely no hue sat or value to work,It proably is just a couple of percent too light.Try selecting the black part and cutting it out to a BLACK background color.