NVIDIA’s Neuralangelo AI: Gaming Anywhere on Earth!

Finally, NVIDIA’s Neuralangelo the paper is  out. And this one is a different paper. We   will talk about this one too. Now, Neuralangelo.   This is NVIDIA’s photogrammetric neural surface   reconstruction tool. What does that  mean? It means that you can scan your   environment with your phone camera, and  out comes 3D geometry that you can use to   create a virtual world for video games, or  have a videoconference that actually takes   place in this environment. Not just  a 2D image, but actually being there. Unfortunately, this is almost impossible.   Why is that? Well, with previous techniques,   we can choose from two different problems. One,  we get intricate details in the virtual version   of the scene, but with that, we also get holes  and other troubles. Or, if the holes are not   acceptable, that is also fine, however, then,  the fine details are gone. Now, hold on to your   papers Fellow Scholars, and have a look at the new  technique. Whoa! We can get all the high-frequency   details and no holes and no major issues appear  in the results. That is fantastic. I did not   expect the results to be this much better just  one more paper down the line. Stunning results. The numerical comparisons here also reveal  how much ahead of its time this technique   is. Don’t forget, these previous methods are  from just one and two years ago. That is a huge   improvement across the board, especially on the  barn scene. Why is that? Presumably because here,   the high-frequency details in the  trees and the roof were completely   gone with previous technique. And  with the new one - so much better! Now all this pairs incredibly well with our  second paper for today, MagicAvatar. Here we   can perform text to motion. And then, motion  to video. That is great, but wait a second. I   hear you asking, Doctor, this is a video, not  3D geometry. So why showcase it here? Well,   that is exactly the point! Now, if  we use Neuralangelo on this video,   we can have a real piece of 3D geometry  in our virtual worlds. How cool is that? This way, we can spice up an  otherwise ordinary Judo video,   and even play with these  characters in a video game world. And the magic of MagicAvatar does not  end there. You can really unleash your   creativity by using video to motion.   We get a little stick figure showing   how the AI understood our motion. Is that  valuable? Oh yes, yes it is! Now we can use   this motion to create a completely new video  with a new actor, but with our own motions. I have a feeling that with all of these amazing  new AI papers, we are entering the world where   anything to anything will be possible. We will  be able to place ourselves in virtual worlds,   and these virtual worlds can be highly detailed  copies of the real world. Or, we will also be   able to create new worlds from scratch not just by  text to image, but by text to geometry. And these   AI-based techniques can also learn to animate  these characters, simulate how they can interact   with new kinds of objects and furniture, and they  can even learn how to simulate physics itself.    This includes fluids, fire, smoke and all kinds  of physics systems. And we will be able to meet   in a virtual version of any place on planet Earth.   Wow. I can only imagine what kind of world we’ll   be living in just two more papers down the line. I  hope you are excited too. What a time to be alive!