NVIDIA's Ray Tracing AI - This is The Next Level! 🤯

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today, we are going to take a light simulation  algorithm that doesn’t work too well,   and infuse it with a powerful neural network.   And what happens, bam! This happens. Wow. So, what was all this about? Well, when we  fire up a classical light simulation program,   first, we start out from…yes, absolutely nothing.   And, look, over time, as we simulate the path of   more and more light rays, we get to know something  about the scene. And slowly, over time, the image   cleans up. But there is a problem.   What is the problem? Well, look.    The image indeed cleans up over time, but no one  said that this process would be quick. In fact,   this can take from minutes, in the case of this  scene, to get this, even days for this scene. In our earlier paper, we rendered  this beautiful, but otherwise,   sinfully difficult scene and it took approximately   three weeks to finish. And it also took  several computers running at the same time. However, these days, neural network-based   learning methods are already capable of  doing light transport. Why not use them?    Well, yes they are, but they are not perfect.   And we don’t want an imperfect result. So, scientists at NVIDIA said that this  might be the perfect opportunity to use   control variates. What are those and  why is this the perfect opportunity? Control variates are a way to inject our knowledge  of a scene into the light transport algorithm.    And here is the key. Any knowledge is  useful, as long as it gives us a headstart,   even if this knowledge is imperfect. Here  is an illustration of that. What we do is   that we start out using that knowledge,  and make up for the differences over time.    Okay, so, how much of a headstart can we get with  this? Normally, we start out from a black image,   and then, a very noisy image. Now hold on to your  papers, and let’s see what this can do to help us.    Wow. Look at that. This is incredible! This is not  the final rendering, this is what the algorithm   knows, and instead of the blackness, we can  start out from this. Goodness, it turns out that   this might not have been an illustration, but the  actual knowledge the AI has of the scene. So cool! Let’s look at another example. In this bedroom,  this will be our starting point. My goodness,   is that really possible? The bed and the  floor are almost completely done. The curtain   and the refractive objects are noisy, but  do not worry about those for a second.    This is just a starting point, and it is still  way better than starting out from a black image. To be able to visualize what the algorithm has  learned with a little more finesse, we can also   pick a point in space, and learn what the world  looks like from this point and how much light   scatters around it. And we can even visualize  what happens when we move this point around.    Man, this technique has a ton  of knowledge about these scenes. So, once again, just to make sure. We start out  not from a black image, but from a learned image,   and now, we don’t have to compute  all the light transport in the scene,   we just need to correct the differences.   This is so much faster! But actually, is it? Let’s look at how this helps us in practice. And  that means, of course, equal time comparisons   against previous light transport simulation  techniques. And, when it comes to comparisons,   you know what I want. Yes! I want Eric Veach’s  legendary scene. See? This is an insane scene

where all the light is coming from not here, but  the neighboring room through a door that is just   slightly ajar. And, the path tracer, the reference  light transport simulation technique behaves as   expected. Yes, we get a very noisy image because  very few of the simulated rays make it to the next   room. Thus, most of our computation is going  to waste. This is why we get this noisy image.   And, let’s have a look what the new method  can do in the same amount of time. Wow!    Are you seeing what I am seeing? This is  completely out of this world. My goodness. Okay, but this was a pathological scene  designed to challenge our light transport   simulation algorithms. What about a more typical  outdoors scene? Add tons of incoming light from   every direction and my favorite, water caustics.   Do we get any advantages here? The path tracer   is quite noisy, this will take quite a bit of  time to clean up. Whereas the new technique…oh   my, that is a clean image. How close is it to the  fully converged reference image? Well, you tell   me, because you are already looking at that. Yes,  now, we are flicking between the new technique   and the reference image, and I can barely tell the  difference. There are some, for instance, here,   but that seems to be about it. Can you tell the  difference? Let me know in the comments below. Now, let’s try a bathroom. Lots of shiny surfaces  and specular light transport. And the results are…

look at that! There is no contest here. A  huge improvement across the whole scene. And, believe it or not, you still haven’t seen  the best results yet. Don’t believe it? Now, if   you have been holding on to your papers, squeeze  that paper, and look at the art room example here.

This is an almost unusable image with classical  light transport. And, are you ready? Well, look at   this. What in the world! This is where I fell off  the chair when I was reading this paper. Absolute   madness. Look. While the previous technique  is barely making a dent into the problem,   the new method is already just a few fireflies  away from the reference. I can’t believe my eyes. And this result is not just an  anomaly. We can try a kitchen scene

and draw similar conclusions. Let’s see.   Now we’re talking! I am out of words. Now, despite all these amazing results, of  course, not even this technique is perfect.    This torus was put inside a glass container, and  is a nightmare scenario for any kind of light   transport simulation. The new method successfully  harnesses our knowledge about the scene,   and accelerates the process a great  deal, but, once again, we get fireflies.    These are going to be difficult to get rid of  and will still take a fair bit of time. But   my goodness, if this is supposed to be a  failure case…then yes, sign me up, right now! Once again, the twist here is  not just to use control variates,   the initial knowledge thing,  because in and of itself,   it is not new. I, like many others have been  experimenting with this method back in 2013,   almost ten years ago, and back then, it was  nowhere near as good as this one. So, what is the   twist then? The twist is to use control variates,  and infuse them with a modern neural network. Infusing previous techniques with  powerful learning-based methods is a   fantastic area of research these days. For  instance, here you see an earlier result,   an ancient light transport technique called  radiosity. This is what it was capable of   back in the day. And here is the neural  network-infused version. Way better. I   think this area of research shows a ton of promise  and am so excited to see more in this direction. So, what do you think? What would you use  this for? I’d love to hear your thoughts,   please let me know in the comments below. And when watching all these beautiful results, if  you feel that this light transport thing is pretty   cool, and you would like to learn more about it,  I held a Master-level course on this topic at the   Technical University of Vienna. Since I was always  teaching it to a handful of motivated students,   I thought that the teachings shouldn’t  only be available for the privileged few   who can afford a college education, but the  teachings should be available for everyone.    Free education for everyone, that’s what I want.   So, the course is available free of charge for   everyone, no strings attached, so make sure to  click the link in the video description to get   started. We write a full light simulation program  from scratch there, and learn about physics,   the world around us, and more. If you watch  it, you will see the world differently. Thanks for watching and for your generous  support, and I'll see you next time!