NVIDIA’s New AI: Wow, Instant Neural Graphics! 🤖

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today we are going to do this, and this. One of these applications is called a NERF.    What is that? NERFs mean that we have  a collection of photos like these,   and magically, create a video where we can fly  through these photos. Yes, typically, scientists   now use some sort of learning-based AI method to  fill in all this information between these photos.    Something that sounded like science fiction  just a couple years ago, and now, here we are. Now, these are mostly learning-based  methods, therefore, these techniques need   some training time. Wanna see how their  results evolve over time? I surely do, so,   let’s have a look together! This NERF paper was  published about a year and a half or two years   ago. We typically have to wait for at least a  few hours for something to happen. Then came   the Plenoxels paper with something that looks  like black magic. Yes, that’s right, this trains   in a matter of minutes. And it was published two  months ago. Such improvement, in just two years. But, here is NVIDIA’s new paper from  about a month ago. And I hear you asking,   Károly, are you telling me that a two month  old paper of this caliber is going to be   outperformed by a one month old paper?   Yes, that is exactly what I am saying. Now, hold on to your papers, and look here,  with the new method, the training takes…what?    Less time than I have to utter this  sentence, because it is already done. So   first, we wait from hours to days, then,  2 years later, it trains in minutes,   and a month later, just a month later, it trains  in a couple seconds. Basically, nearly instantly. And, if we let it run for a bit longer, but still  less than two minutes, it will not only outperform   a naive technique, but will provide better  quality results than a previous method,   while training for about ten times quicker.   That is absolutely incredible. I would say   that this is swift progress  in machine learning research,   but that word will not cut it  here. This is truly something else. But, if that wasn’t enough, NERFing  is not the only thing this one can do. It can also approximate a gigapixel image. What  is that? That is an image with tons of data in it,   and the AI is asked to create a cheaper  neural representation of this image. And,   we can just keep zooming in, and zooming  in, and we still find new details there. Now if you have been holding on to your papers  so far, now, squeeze that paper, because what   you see here is not the result, but the whole  training process itself. Really? Yes, really.    Did you see it? Well, did you blink? Because  if you did, you almost certainly missed it.    This was also trained from scratch, right in front  of our eyes. But it’s so quick, that if you take   just a moment to hold on to your papers a bit more  tightly, and you already missed it. Once again,   a couple papers before, this took several  hours at the very least. That is outstanding. And if we were done here, I would be very happy.   But, we are not done yet, not even close! It can   still do two more amazing things. One, this is a  neural signed distance field it has produced. That   is a mapping from 3D coordinates in a virtual  world to distance to a surface. Essentially,   it learns the geometry of the object better  because it knows what parts are inside, and   outside. And it is blazing fast, surprisingly,  even for objects with detailed geometry. And, my favorite, it can also do neural radiance  caching. What is that? At the risk of simplifying   the problem, essentially, it is learning to  perform a light transport simulation. It took   me several years of research to be able to produce  such a light simulation, so let’s see how long it   takes for the AI to do. Well…let’s see…holy mother  of papers! NVIDIA, what are you doing! I give up.

As you see, the pace of progress in  AI and computer graphics research   is absolutely incredible, and even  better, it is accelerating over time.    Things that were wishful thinking 10  years ago become not only possible,   but are now easy over the span of just a couple  of papers. I am stunned. What a time to be alive! Thanks for watching and for your generous  support, and I'll see you next time!