Ray Tracing: How NVIDIA Solved the Impossible!

Dear Fellow Scholars, this is Two Minute Papers  with Dr. Károly Zsolnai-Fehér. Or not quite. To   be more exact, I have had the honor to hold a talk  here at GTC, and today we are going to marvel at   a seemingly impossible problem, and 4 miracle  papers from scientists at NVIDIA. Why these 4?    Could these papers solve the impossible?   Well, we shall see together in a moment.     If we wish to create a truly gorgeous   photorealistic scene, in computer graphics,  we usually reach out to a light transport   simulation algorithm, and then, this happens.   Oh yes, concept number one. Noise! This is   not photorealistic at all, not yet anyway. Why is  that? Well, during this process, we have to shoot   millions and millions of light rays into the scene  to estimate how much light is bouncing around,   and before we have simulated enough rays, the  inaccuracies in our estimations show up as noise   in these images. This clears up over time, but it  may take from minutes to days for this to happen,   even for a smaller scene. For instance, this one  took us 3 full weeks to finish. 3 weeks! Yes,   really. I am not kidding. Ouch.    Solving this problem in real time seems absolutely  impossible, which has been the consensus in the   light transport research community for a long  while. So much so, that at the most prestigious   computer graphics conference, SIGGRAPH, there  was even a course by the name: Ray tracing is   the future and ever will be. This was a bit of a  wordplay yes, but I hope you now have a feel of   how impossible this problem feels.    When I was starting out as a first year PhD  student, I was wondering whether real-time   light transport will be a possibility within  my lifetime. It was such an outrageous idea,   I usually avoided even bringing up the question in  conversation. And boy, if only I knew what we are   going to be talking about today. Wow.    So, we are still at the point where these images  take from hours to weeks to finish. And now,   I have good news and bad news. Let’s go with  the good news first. If you overhear some   light transport researchers talking, this is  why you hear the phrase “importance sampling”   a great deal. This means to choose where to  shoot these rays in the scene. For instance,   you see one of those smart algorithms here,  called Metropolis Light Transport. This is one   of my favorites. It typically allocates these rays  much smarter than previous techniques, especially   on difficult scenes. But, let’s go even smarter!   This is my other favorite Wenzel Jakob’s Manifold   Exploration paper at work here. This algorithm is  absolutely incredible, and the way it develops an   image over time is one of the most beautiful  sights in of all light transport research.     So if we understand correctly,   the more complex these algorithms are, the smarter  they can get, however, at the same time, due to   their complexity, they cannot be implemented so  well on the graphics card. That is a big bummer.    So, what do we do?    Do we use a simpler algorithm and take  advantage of the ever-improving graphics   cards in our machines, or write something  smarter and miss out on all of that?     So now, I can’t believe I am saying this,   but let’s see how NVIDIA solved the impossible  through 4 amazing papers. And that is, how they   created real-time algorithms for light transport.    Paper number one. Voxel Cone Tracing. Oh my, this  is an iconic paper that was one of the first signs   of something bigger to come. Now, hold on to your  papers, and look at this. Oh my goodness. That is   a beautiful, real-time light transport simulation  program. And it gets better, because this one   paper that is from 2011, a more than 10-year-old  paper, and it could do all this. Wow! How is that   even possible? We just discussed that we’d be  lucky to have this in our lifetimes, and it seems   that it was already here, 10 years ago!    So, what is going on here? Is light transport  suddenly solved? Well, not quite. This solves

not the full light transport simulation  problem, but it makes it a little simpler.    How? Well, it takes two big shortcuts. Shortcut  number one: it subdivides the space into voxels,   small little boxes, and it runs the light  simulation program on this reduced representation.     Shortcut number two, it only computes 2 bounces   for each light ray for the illumination. That is  pretty good, but not nearly as great as a full   solution with potentially infinitely many bounces.    It also uses tons of memory, so, plenty of things  to improve here, but my goodness, if this was not   a quantum leap in light transport simulation, I  really don’t know what is. This really shows that   scientists at NVIDIA are not afraid of completely  rethinking existing systems to make them better,   and boy, isn’t this a marvelous example of that.   And remember, all this in 2011. 2011! More than 10   years ago. Absolutely mind-blowing. And one more  thing: this is the culmination of software and   hardware working together. Designing them for  each other. This would not have been possible   without it.    But, once again, this is not the full light  transport. So, can we be a little more ambitious,   and hope for a real-time solution for  the full light transport problem? Well,   let’s have a look together and find out! And here  is where paper number two comes to the rescue. In   this newer work of theirs, they presented an  amusement park scene that contains a total of   over 20 million triangles, and it truly is  a sight to behold. So let’s see, and! Oh,   goodness! This does not take from minutes to days  to compute, each of these images were produced in   a matter of milliseconds! Wow.    And, it gets better. It can also render this  scene with 3. 4 million light sources, and this   method can really render not just an image, but  an animation of it interactively. What’s more,   the more detailed comparisons in the paper reveal  that this method is 10 to 100 times faster than   previous techniques, and it also maps really well  onto our graphics cards. Okay, but what is behind   all this wizardry? How is this even possible?    Well, the magic behind all this is a smarter  allocation of these ray samples that we have to   shoot into the scene. For instance, this technique  does not forget what we did just a moment ago when   we move the camera a little and advance to the  next image. Thus, lots of information that is   otherwise thrown away can now be reused as we  advance the animation. Now note that there are   so many papers out there on how to allocate  these rays properly, this field is so mature,   it truly is a challenge to create something  that is just a few percentage points better   than previous techniques. It is very hard to  make even the tiniest difference. And to be able   to create something that is 10 to a 100 times  better in this environment? That is insanity.     And, this proper ray allocation   has one more advantage. What is that? Well, have a  look at this. Imagine that you are a good painter,   and you are given this image. Now your task is  to finish it. Do you know what this depicts?    Hmm…maybe. But knowing all the details of this  image is out of question. Now, look, we don’t have   to live with these noisy images, we have denoising  algorithms tailored for light simulations. This   one does some serious legwork with this noisy  input, but even this one cannot possibly know   exactly what is going on because there is so much  information missing from the noisy input. And now,   if you have been holding on to your papers so far,  squeeze that paper, because, look. This technique   can produce this image in the same amount of  time. Now we’re talking! Now, let’s give it   to the denoising algorithm, and…yes! We get a  much sharper, more detailed output. Actually,   let’s compare it to the clean reference image.   Yes, yes yes! This is much closer. This really   blows my mind. We are now, one step closer  to proper, interactive light transport!

Now note that I used the   word interactively twice here. I did not say real  time. And that is not by mistake. These techniques   are absolutely fantastic, one of the bigger leaps  in light transport research, but, they still cost   a little more than what production systems can  shoulder. They are not quite real time yet.     And, I hope you know what’s coming.    Oh yes! Paper number three. Check this out!    This is their more recent result on the Paris  Opera House scene, which is quite detailed,   there is a ton going on here. And, you are all  experienced Fellow Scholars now, so when you   see them flicking between the raw, noisy and the  denoised results, you now know exactly what is   going on. And, hold on to your papers, because  all this takes about 12 milliseconds per frame.    That is over 80 frames per second. Yes yes! My  goodness! That is finally in the real time domain   and then some! What a time to be alive!    Okay, so where is the catch? Our keen  eyes see that this is a static scene.    It probably can’t deal with dynamic movements  and rapid changes in lighting, can it? Well,   let’s have a look. Wow! I cannot believe my  eyes. Dynamic movement, checkmark. And here,   this is as much change in the lighting as  we would ever want, and it can do this too.     And, we are still not done yet. At this point,   real time is fantastic, I cannot overstate how  huge of an achievement that is. However, we need   a little more to make sure that the technique  works on a wide variety of practical cases.    For instance, look here. Oh yes! That is a ton of  noise, and it’s not only noise, it is the worst   kind of noise! High-frequency noise. The bane of  our existence. What does that mean? It means these   bright fireflies. If you show that to a light  transport researcher, they will scream and run   away. Why is that? It is because these are light  paths that are difficult to get to, and hence,   take a ton more ray samples to clean up.    And, you know what is coming! Of course, here  is paper number four. Let’s see what it can   do for us. Am I seeing correctly? That is so much  better! This seems nearly hopeless to clean up in   a reasonable amount of time, and this…this might  be ready to go as is with a good noise filter.    How cool is that!    Now, talking about difficult light paths,  let’s have a look at this beautiful caustics   pattern here. Do you see it? Well, of course  you don’t! This region is so undersampled,   we not only can’t see it, it is hard to  even imagine what should be there. So,   let’s see if this new method can accelerate  progress in this region. That is not true. That   just cannot be true. When I first saw this paper,  I could not believe this, and I had to recheck the   results over and over again. This is at the very  least a 100 times more developed caustic pattern.    Once again, with a good noise filter, probably  ready to go as is. I absolutely love this one. Now, note that there are still shortcomings. None  of these techniques are perfect. Artifacts can   still appear here and there, and around specular  and glossy reflections, things are still not as   clear as the reference simulation. However, we now  have real time light transport and not only that,   but the direction we are heading to is truly  incredible, and amazing new papers are popping   up what feels like every single month. Don’t  forget to apply The First Law Of Papers,   which says that research is a process. Do not look  at where we are, look at where we will be two more   papers down the line.    Also, NVIDIA is amazing at democratizing  these tools and putting them into the   hands of everyone. Their tech transfer  track record is excellent. For instance,   their marbles demo is already out there,  and not many know that they already have   a denoising technique that is online and  ready to use for all of us. This one is a   professional grade tool right there. Many  of the papers that you have heard about

today may see the same fate. So, real-time light  transport for all of us? Sign me up right now!   Thanks for watching and for your generous  support, and I'll see you next time!