Quote - Well the key to the whole thing is to come up with a set of review criteria that take the subjective aspects out of a review.

and this is the problem i personally see right off the bat.  i've been fighting and fighting my V4 textures right now, and the problem with pretty much all of them is burned in shadow and specular, totally wrong ways of making bump and displacement maps, and a complete misunderstanding of what a specular map actually does.    and all of that traces back to over-reliance on 3d painting tools and photo references.

but to judge by reviews and sales, artifact-free maps are not what people want. quite the opposite, in fact.  i can almost always spot the most popular textures by their highlights on the forehead, nose, and lips.  in fact, many seem to rely on the burned in specular to do what their lighting isn't doing.  and to judge by forum threads and advice, absolutely no one seems to think a specular map should actually define regions rather than low level details.

i think Blackhearted's comment about fit brings up a similar example.  me, i'd rather not have to try to force clothes to fit better everywhere.  but poke through is a pretty simple problem with several quick and easy solutions (Photoshop, morph brush, free 3d software, etc.).  so to me high quality means good fit, and poke-through is pretty irrelevant.

products are full of these kinds of trade-offs.  morphs vs. polycount.  user-friendly mapping  vs. stretching.  control vs.simplicity.  realism vs. resource requirements.  popularity and support vs. originality.  versatility vs. optimized performance.  i don't think it makes sense to make one good and the other bad.  also, the balance is in vastly different places for different people.  there's no way you can address those issues with total objectivity.  some, yeah, but very, very far from complete.

i think honest and specific reviews are good, especially if they're assessments and not evaluations.  that is, what the product is and isn't instead of whether a feature is good or bad. but i think it would be a shame if merchants got discouraged because reviewers just didn't share the same opinions about what was important.