The parts of the texture which are going to be transparent should be coloured pure black - and make sure that "apply texture to highlight" is checked. If you aren't using a texture, apply the transmap itself as the texture (black = transparent).