So do you have a color key that you use for each frame? If you are doing your own renderer you can do things like drop your color count to 64 (6 bits) and use the last two bits for alpha. That gives you 4 alpha levels. But still, even if you are chromakeying you can still antialias the art when shrinking then just create a hard edge around the art and set the outside to your chromakey color. Also check you firefly settings. You can try different settings of the gaussian, box post filtering and the pixel precision for post filtering. Also, you can save the frames as TIFFs or PSD files and use the alpha layer as a hard edge maker to create the chromakey area. Also you should check out JPG 2000. It does things like allows you to lossy compress the image layers but lossless compress the alpha layer. Plus it super compacts the files.