I would like to see video of realistic CGI 3D animated model human character. Why is this so rare in movies?

benwhe posted at 10:44 AM Thu, 23 March 2023 - #2978247

I would like to see video of realistic CGI 3D animated model human character. Why is this so rare in movies?

Because it is hard to do. Which means it is expensive. There's a lot of setup work that is involved. So unless the story somehow benefits from it or really needs it. To cut down on cost and production they will find a way to do without it

I would like more discussion on this. If not movies, then what practical use is a model? What setup work? What future developments are needed to make this practical?

Why is it so rare? Because its really bloody hard to pull off even close to convincingly!

From the day we're born we study faces, learn all their little movements, "micro expressions", small muscle contractions, skin stretching or jiggling in certain ways, etc etc, that are there all the time, but we're not looking for them normally, as soon as they're not their tho, we know something isn't right. All that makes it very nearly impossible to animate a truly convincing CGI replacement, the best ones to date have all been facial mo-cap with CGI over the actors face. I say "very nearly impossible" because, I don't want to discount the possibility of someone taking two years to animate a 5 second clip frame by frame to get every slight detail, but its not going to be cost effective or practical for any project at this point.

It would also help to define what you're looking for, and where you draw the line. Do you count stuff like Davey Jones et all in the Pirates of the Carribean movies? Maybe not, but there's not much left of the actors face by the time VFX is done with the footage. Gollum in the Lord of the Rings/Hobbit films? If you want total replacement, what about Alita in Battle Angel? Not exactly a "natural" human character but a total replacement and a pretty well done one at that. I suppose we could mention Avatar, but, ugh, I'd rather not, not CG's finest moment. 

Total CGi character replacement has been done a few times, first one that comes to mind was Peter Cushing in Stars Wars: Rogue One, which looks pretty convincing, but you can tell it offs, a lot of the movements aren't nearly as "sharp" and tend to "drift" some, and the skin is kind of stiff, especially in the cheeks. Paul Walker in [at least] one of the later Fast and the Furious movies. I'd recommend checking out Corridor Crew's videos on this topic.

https://www.youtube.com/watch?v=2ZKPnuUFwOk

They have a few others, that get into "deep faking" as well [ https://www.youtube.com/watch?v=861gfPVmgdc ], which is the direction that things are more likely to go. Seems like that's giving more realistic results at the moment, than a full CGI replacement is. I'd guess we'll eventually get to a point that its a hybrid of the two methods.

Good discussion Warlock279 thank you. Your second paragraph is about little movements. I understand this would be hard to pull off if the model were manually edited. But can a model be programed to walk? Can a program be written to automatically make those little movements? And if output is not perfect, go back and tweak. Your paragraph does not convince me. But the problem still exists, so maybe you are somehow correct anyway.

You answered my question perfectly. Oh no, you did exactly what I asked for. And now I tweak my question. What I'm looking for is realistic CGI 3D animated model human character, to act in video. To avoid copyright problems, the model should not look like any specific person. The video should not look like any previously published video. But this topic should be about model, not video. Or maybe a better topic would be the automatic animation of a model. Actually, this is animation forum. The references you gave me are about inserting models that look like specific people, into live video. That is close to what I am looking for and is good for reference. Disclaimer. I am not expert in this topic. I do not intend to make a commercial movie.

Short answer then? No, you can't, from scratch create and animate a CGI model that's going to be THAT believable. Its too complicated to be practical at this point in time.

Still image? Absolutely, there are lots of artists out there that could put a CGI model into a still image, and you'd never be able to tell the difference. Animated tho? Nope.


Longer answer? There's a lot of nuances to what you're asking, you seem like you have something specific in mind, but you're a bit vague so I'm not sure what direction to go, perhaps if you laid it out/asked directly, someone could give a clearer, more direct answer, but in any case, here's some thoughts ...

Can a program be written to automatically make the little movements? No, not really. There are just too many, if we can't manually animate them, we can't realistically write a program to account for each and every one, and every permutation thereof, its not necessarily a "math" type function, but that's where machine learning/AI would come in. You might be able to train an AI [eventually] to account for all that stuff.

When I say "little movements" I'm talking things far more granular than the movements you'd find in an action like "walking" [not that the physics sims in clothing wrinkles/etc aren't insane in their own right, and still bleeding edge], but I don't know that you'd necessarily want to "program" a model to walk anyway, you'd probably just want to use motion capture for something like that, [assuming you didn't just do a head replacement over an actor's body, you've been vague, I don't know if that's within the realm of what you're asking]. Motion capture, not exactly "animated" or necessarily "from scratch", if that distinction matters for what you're asking, but motion capture would get you a long way in the right direction. Actions like walking [at a distance] have been done that fairly convincing for years now with motion capture. Big actions/movements like that, aren't really the problem. Hands will still be tricky, but I think we're probably at a point with enough physics sims and a complicated enough rig, you could sell those. Its the face tho, its always going to come down the face.

A full facial motion capture setup, would be a good place to start, the more tracking, the better, but that comes with caveats. You'd need an actor for the mocap performance, that had a  similar facial structure to your 3D model, the more differences between the two, the more obvious problems will become. Full cheeks will move different than gaunt cheeks, smile lines, different creasing in the corners of the mouth, wrinkles/folds around the eyes, etc etc, the more of all those things in common between your mocap actor, and your 3D model, the better. Then you'd need an elite animator to clean it all up.

There are just so many tiny details. The ears shifting when the jaw flexes, the nose stretching when the mouth moves, hairs shifting when the brow furrows, those are "obvious" things to take into account. The cornea of your eye sticks out of the "round" of the eyeball, which causes subtle shifts/stretching in the eyelids as the eyes move around, which cause individual eyelashes to change directions slightly. The way the pores in your skin expand/contract/stretch with the skin. That's moderately obvious stuff. Then there are far less obvious things; take your hands, and place them on the back of your head, a little above ear level, then raise and lower your eye brows, you should feel that action on the back of the head which is causes the hair to shift. Is it something you'd ever consciously observe happening on someone else? No, and if just that one thing was missing it probably wouldn't be an issue, but when you've got that, and hundreds of other "micro" details compounding, it starts to slip into the uncanny valley.

Can a program be written to make all those movements? Maybe? A number of the foundations are out there already, kinda, if you really pushed them. For example maybe set up a few dozen different physics sims to capture more of the way a face realistically stretches. I can't remember now, its been ages, but there were some studios/artists working with creating all the individual muscles of the face rigged with physics, over a model of the skull, and then a physics sim for skin over that, all being animated, giving varying degrees of success. Stack multiple hair simulations to get something more natural than what you commonly see. Get an animator, that really, REALLY knows facial animation, and put in probably hundreds of hours per second of animation. Then maybe you could expect results that would fool everyone.

Thousands of computer hours on physics simulations, and hundreds of man hours animating, plus mocap and everything else for every few seconds you needed animated? Its just not practical.

Without really knowing what you're wanting, how granular you would need to get would depend on what the shot[s] required. You could probably-maybe, do it for a mid-long range shot, full body/full frame at 4k resolution. I could see that being feasible. For a mid-shot, waist up at 4k, ehh, maybe, if you really really got down to it that would be possible. For a tight shot tho, head and shoulders at 4k, nah, we're just not there yet.

Keep in mind, this is all from the standpoint of, "what is practical"? If you throw enough time/resources at the project, then sure maybe, but there's just no scenario where anyone is throwing 3 years of work at a 5 second animation. If you did that starting now, with the current technology, by the time you finished, tech will have come along that can do a better job faster either/or you get caught in a loop where you're constantly iteratively incorporating all the new tech that comes along, continually refining, endlessly, so practicality forces us to settle for "good enough for now" which, just isn't quite there yet.


But, so like, you say maybe it could be done if you went ham on it? How would do it? Like, life-or-death, "deliver a convincing sequence or be dropped into the depths of the ocean wearing only cinder blocks shoes"? I suppose I'd try some sort of hybrid approach. Motion capture as a foundation, layer it with heaps of physics/hair simulations to fill in the gaps the mocap didn't pick up, refine that frame by frame and when its almost, ALMOST perfect ... deep fake it? Instead of putting one persons face on another, I'd be using the nearly perfect CGI to train the AI and put that over the actor I was replacing. Would it work? Maybe? I think there might still be some tells, especially if viewed hyper-critically, but on the flip side I'd likely buy myself a year or two longer to live before I had to put on the cinder block shoes, cause it would take forever to do all that.

Maybe if the deep fake using the CGI model worked well enough [I'm spit-balling with that process, I have no idea if you could actually train an AI using a CGI model as the basis for a deep fake like that, I haven't trained an AI before], you'd get to a tipping point where it does start to become practical, where your initial few thousand hours invested gives you the ability to turn out much longer sequences using a deep fake method.

Great. We agree that a video of realistic CGI 3D animated model human character is not practical at this point in time but AI might eventually make the model. And that answers my initial question.  I was a bit vague and after this discussion I am realising the specific thing I had in mind.

For the model: I did want realistic and I plan to get that in the future. I am now in the present and will take what realisim is available. I want human. I do not want space aliens and talking toys.

For AI: I would like end user to be able to use simple script commands to tell the model what to do. Like go there and pick up that. I remember hearing about such an existing program but it had primitive stick-figure models.

AI step up: I would like end user to be able to give model some traits and goals. Then see how it interacts with enviroment and other models. I remember hearing about such an existing program. It is AI simulator for preditor and prey. The characters are depicted as icons. A video can be made of this.  Evolving AIs - Predator vs Prey, who will win?  https://youtu.be/qwrp3lB-jkQ

benwhe posted at 10:46 PM Sat, 25 March 2023 - #4459598

For AI: I would like end user to be able to use simple script commands to tell the model what to do. Like go there and pick up that. I remember hearing about such an existing program but it had primitive stick-figure models.

AI step up: I would like end user to be able to give model some traits and goals. Then see how it interacts with enviroment and other models. I remember hearing about such an existing program.

I think you're describing video games. Good news then, there's not really a reason to be using AI to "drive" the model, the end user doesn't need to use anything as cumbersome as a script to make a character walk, we solved that eons ago; arrows keys or buttons on a controller can do that, and another button push to pick up an object. ;)

So like ... the Sims?



You can ignore just about everything I said in previous posts, none of that is relevant anymore, its no longer even a question. You're not getting ANYWHERE near a realistic model/animation that's going to fool anyone. Look at gameplay [not cut-scenes or pre-rendered video, not that that's fooling anyone either, but actual gameplay] footage from something like Horizon: Forbidden West, or CyberPunk 2077, or God of War 5, those are probably about the peak of realism in real-time graphics right now. They look great, for what they are, but they're not fooling anyone. Bear in mind, those were done with very large teams of artists.

Almost all the stuff I talked about above is exclusive to offline render scenarios, relies on dozens and dozens of hours of render time per frame [after hours and hours and hours of physics calculations], then a hefty amount of time compositing multiple renders, lighting passes, subsurface scattering, occlusion, reflections, specular, etc. If you're thinking about real-time graphics, which you are if you want a user to be able to give a command and watch it executed ["now", not two months after they input the command and have completely forgotten what command they even issued] then the bar for "realism" has to drop, like through the floor, through the floor's floor, and into a very VERY deep crater in the basement, in fact, put cinder blocks shoes on your expectations, and drop them in the ocean.



It sounds like what you're really after is a simulation, that has hyper-realistic graphics? You're going to be handicapped in what you can do by modern computing power. The more advanced the AI in your simulation,and the more "actors" in your simulation, the more you'll have to compromise on the graphics. The more complex the environment they're interacting with, the more you'll have to compromise, etc etc. That's likely the reason what you're recalling had stick-figures [well that, and it was probably done by one person/programmer without help of any artists as an experiment where graphics weren't important].

Yes, video game is a good way to summarise what I said. I liked sim city. Then I got newer version, which required online activation. So I threw it away and lost my like for Maxis and never got the Sims.

Moving on now. I will reset, and ignore just about everything you said in previous posts. Most of that was relevant then, it answered my question.

I still have my target near a realistic model/animation and might hit that in future. I will take what I can get for now. Even a stick figure. And that will fool little children, some think cartoon characters are real. But you probably define fool to mean model is indistinguishable (in eyes of expert) from real object human.

Now, portrait models of human exist, are practical, and can fool. Good start. To fool with animation of human model, AI and computer power need to be improved.

Next sub-topic: What I can actually do now. Renderosity has many models that make nice protrait. But what to do with the model when I get it? I might use some general software to change filetype and color. Maybe do some simple deformations. To do more, I will need special model-specific software and skills for that.

I use linux so that limits my software to blender. Blender GUI is too hard to learn, I prefer script. But my old bpy script will not work on new blender version. One solution is to pay for, get, online activation, and learn: MS windows, Poser, DAZ, etc. I tried most of those.

Or I could make movie using portrait models floating around without animation. This seems to be my best choice for now.

I did use script and various programs to create model of human and house. And I can view that model from various angles and zoom. That is all I can do, I wish I could do more.

Offline render scenarios is OK with me. It looks like not many people want to chat. So I continue. Now in headline news is newly released two weeks ago AI: Generative Pre-trained Transformer 4 (GPT-4). It can write computer code. Looks like the future is coming quickly. Graphics programs often use GUI, but some use code. When will AI meet graphics?

Hi, It must be really hard to do, i know that einstein was recreated in a U.K advert!, but was so short, how about musical concerts, inserts, but would be awesome to see an actor in CGI form, :)

Nice to hear from you ianharding. You say an actor in CGI form. There are many of these CGI models. It is too hard to manually animate (make act) these. I am waiting for AI to do that job. Any updates? And good bye Hollywood. AI will soon take over writing. The human writers are currently on strike.

Thanks for sharing this type of informative article. I have learned some right stuff here. I really like your articles.