A number of things to keep in mind about codecs:

A few are designed for a specific colour depth (like 16-bits), and will generate an error message if you try to feed them 32-bit colour.

Others have limitations for width and height. For instance, Indeo videos must have a width and height evenly divisible by 4.

As for pixelation, some codecs work better than others, depending upon the scene complexity and the amount of compression used.

If you have sufficient room on your hard drive, I'd personally recommend that you render your animations with Full Frames (Uncompressed), then edit and compress the finished video with a third-party utility like VirtualDub.