NVIDIA’s New AI: Journey Into Virtual Reality!

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Today we are going to see how NVIDIA’s crazy new AI is able to understand basically any human movement. So, how is this even possible? And even more importantly, what is pose estimation? Well, simple, a video of people goes in, and the posture they are taking comes out. Now, you see here that previous techniques can already do this quite well, what’s more, if we allow an AI to read the wifi-signals bouncing around in a room, it can perform pose estimation, even through walls. Kind of. Now, you may be wondering what is pose estimation good for? By the end of this video, you will see that this can help us move around in virtual worlds, the metaverse, if you will. But let’s not rush there yet, because still, there are several key challenges here. One, we need to be super accurate to even put a dent into this problem. Why? Well, because if we have a video, and we are off by just by a tiny bit from frame to frame, this kind of flickering may happen. That’s a challenge. What else is challenging here? Two, foot sliding. Yes, you heard it right. Yes, previous methods suffer from this phenomenon, you can see it in action here. And also, here too. So, why does this happen? It happens because the technique has no knowledge of the physics of real human movements. So, scientists at NVIDIA, the University of Toronto and the Vector Institute fired up a collaboration, and when I first heard about their concept, I thought you are doing what? But, check this out. First, they perform a regular pose estimation. Of course, this is no good, as it has the dreaded temporal inconsistency, or in other words, flickering. And in other cases, often foot sliding too. Now, hold on to your papers, because here comes the magic! Now, they transfer the motion to a video game character, and embed that character in a physics simulation. In this virtual world, the motion can be corrected to make sure they are physically correct. Now, remember, foot sliding happens because of the lack of knowledge in physics, so, perhaps this idea is not that crazy after all! Let’s have a look! Now this will be quite a challenge. Explosive sprinting motions. And…whoa. This is amazing. This, dear Fellow Scholars, is superb pose estimation and tracking. How about this? A good tennis serve includes lots of dynamic motion, and just look at how beautifully it reconstructs this move. Apparently, physics works. Now, the output also needn’t be stickmen, we can retarget these to proper textured virtual characters built from a triangle mesh. And, that’s just one step away from us being able to appear in a metaverse. No head mounted displays are required, no expensive studio, no motion capture equipment is required. So, what IS required? Actually, nothing! Just a raw input video of us. That is insanity. And all this produces physically correct motion. So, that crazy idea about taking people and transforming them into video game characters is not so crazy after all. So now, we are one step closer to be able to work, and even have some coffee together in a virtual world. What a time to be alive! Thanks for watching and for your generous support, and I'll see you next time!