NVIDIA’s New AI: Video Game Graphics, Now 60x Smaller!

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. When creating a video game, an animated movie,  or to create any believable virtual world,   among many other things, we need geometry.   Tons and tons of geometry. Both to store,   and to render images of our  characters and the environment. And having lots of geometry  means a big headache. You see,   we can use traditional methods that store  all the data needed for these objects,   and they also render super fast. But, there is a  problem. What is the problem? Well, size! These   can take gigabytes that will eat not only your  storage but your cellular data plan as well. Now, do not despair, modern learning-based  methods are coming to the rescue in the   form of NERFs. That is, Neural Radiance Fields.   Here, we don’t need to store all the geometry,   just a few photos, and miraculously  they can fill in all the missing data,   so … can that really be? Can we fly through  these photos? Yes we can! This is amazing. Now, it gets better, these are instant NERFs.   These can be trained in a matter of minutes,   sometimes seconds, and rendered also in  seconds. And, look, we are light transport   researchers over here, so our keen eyes see  that specular and refractive objects work   really well too. That is fantastic, and folks  at Luma AI have already made an app that can   create these NERFs right on your phone. And  some of these seem either production quality,   or really close to it. Yes, that’s  right, all this comes from a paper   that was published just two years ago. This  is incredible progress in just two years. But wait. So, these NERF representations are  much smaller than traditional techniques,   however, they still put some strain on  your cellular data plan. And, don’t forget,   they require considerable  horsepower to render quickly. But, good news, neural networks are  excellent at compression. For instance,   NVIDIA already has a neural network that can  help us compress even ourselves! You see,   in this earlier work, the technique takes only  the first image from our video, and it throws away   the entire video afterwards! But before that,  it stores a tiny bit of information from it,   which is, data on how our head is moving  over time, and how our expressions change.    That is an absolutely outrageous idea,  except the fact that it works. With this,   our video data for a video call is thus  at least as detailed as the traditional,   state of the art techniques, but uses  a third as much data, or even less! And, get this in this new paper, scientists at  NVIDIA claim to solve both of our NERFs problems   at the same time! Compact and fast NERFs at  the same time! Now I will believe that when   I see it. Can they put the incredible compression  capabilities of neural networks to more use here?    Let’s see. Whoa, look at that! You see a previous  NERF-based method, and here, the new method. And   can that really be? Am I seeing correctly  or am I dreaming? This new method requires   60 times less data to show us this geometry.   And the quality, as described by the signal to   noise ratio is nearly the same. Not the same,  but very close. And 60x cheaper. My goodness. And it gets even better. That was just  the compression part. What about the   fast rendering part? Well, these models  are so small, we can start downloading   multiple versions of them at the same time,  and the coarsest version can be shown to us   after receiving only the first 10 kilobytes  of data. That is equivalent to downloading   less than one second of music. Wow. And then,  as more data arrives, the finer details start   to appear over time. And we are through the  whole process very, very quickly. Incredible. But wait, we are experienced Fellow Scholars over  here, so we know that there are other papers on   this topic. For instance, what about Plenoxels?   How does it compare to that? This is Plenoxels,   which is an amazing paper that we talked  about earlier in this series. By the   way, interestingly, this earlier method did not  use neural networks to achieve what you see here,

which is incredible. And now, look at that! This  new one, look, it can go at least as far with 20   megabytes of data as Plenoxels can go with over a  150. And Plenoxels is not just some ancient paper,   it is from just one year ago. Seeing this  kind of improvement just one year and one   more paper down the line, that is the  power of human ingenuity. I love it. And you see, this might be the future of imagery.   We don’t store the whole geometry of the objects   anymore, we just take a few photos, and let the AI  fill in the rest. Bravo! What a time to be alive! Thanks for watching and for your generous  support, and I'll see you next time!