Note that the ToonID value you see in the matroom is actually colors in web-hex format (ie for pure red, it would be FF0000, for pure green it's 00FF00 and for blue its 0000FF, converted to decimal) Maybe that's throwing AE off ? Have you tried setting it to a small number between 1 to 255 ?

As for working with it in PS/nuke, well those apps dont have an explicit ID keyer like that, they just let you key on colors. So what I do is set colors manually in the (poser) ToonID slots and then key them out in comp. Maybe you could try something like that in AE ?