Concave means the the polygon has a 'cavity' in it.  The polygon on the left in the above image is convex, the one on the right is concave. This can only happen with more than 3 vertices.  

Basically, from a visual standpoint, you can imagine walking around the polygon from vertex to vertex, in either a clockwise or counter-clockwise direction (some apps use a right-handed system, some use a left-handed system).  As you walk, you either turn right or left to get to the next vertex. On the left one (moving clockwise around), each new vertex is "to the right".  But on the right one, most oof them are "to the right", but at that vertex that's pushed in, the angle to the next vertex is "to the left".

Concave polygons are generally frowned upon.. they can cause trouble with back-face culling and depending on the rendering engine, that 'cavity' might even get filled in.

Non-planar polygons also require more than 3 vertices.  The basic definition of planar is if all vertices that make up the polygon lie in the same plane. Once you get above 3 vertices (needed to describe the plane to start with), one or more of them may not be in the same plane... then it's non-planar.

Non-planar polygons can cause surface Normal calculations to be off, because the surface doesn't have just one specific direction it faces - it depends on which triangle of the polygon you're looking at.  Surface Normals are used for Vertex Normal generation, backface-culling, shading and lighting calculations and other things.  So non-planar polygons will typically have some degree of shading/rendering artifacts.  The system Poser uses is pretty forgiving when it comes to these and you'll find that most Poser meshes have tons of non-planar polygons.

Our script for the most part avoids any problems with non-planar or concave polygons, because it only deals with triangles.  But, if your new smoothing stuff is working with our triangles, and it's re-positioning the vertices, then it might be creating concave polygons.  I still hadn't looked at that code to see what it does, but my guess is that you'd want to feed it the original polys and not the tri-polys, so that it at least has some frame of reference of how the polygons are made up (it may well have code in place to keep from making concave polygons as it moves vertices around, but it has to know about all the vertices of the polygon for that to work).

Cinema4D Plugins (Home of Riptide, Riptide Pro, Undertow, Morph Mill, KyamaSlide and I/Ogre plugins) Poser products Freelance Modelling, Poser Rigging, UV-mapping work for hire.