This AI Creates Beautiful Light Simulations! 🔆

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today we are going to have a look at this insane   light transport simulation paper and  get our minds blown in three chapters.

Chapter 1. Radiosity. Radiosity is an  old light transport algorithm that   can simulate the flow of light within a scene.   And it can generate a scene of this quality.    Well, not the best, clearly, but this  technique is from decades ago. Back in the day,   this was a simple and intuitive attempt  at creating a light simulation program,   and quite frankly, the best we could do. It  goes something like this: slice up the scene   into tiny surfaces, and compute the light  transport between these tiny surfaces. This worked reasonably well for diffuse  light transport, matte objects if you will.    But, it was not great at rendering shiny objects.   And it gets even worse, look. You see these blocky   artifacts here, these come from the fact that the  scene has been subdivided into these surfaces,   and the surfaces are not fine enough  for these boundaries to disappear. But, yes, I hear you asking, Károly,   why talk about this ancient technique? I’ll tell  you in a moment. You’ll see, I promise. So, yes,   radiosity is old, some professors still teach  it to their students, it makes an interesting   history lesson, but I haven’t seen any use of  this technique in the industry in decades now.    If radiosity would be a vehicle, it would be a  horse carriage in the age of high-tech Tesla cars. So, I know what you’re thinking. Yes,  let’s have a look at those Teslas. Chapter 2. Neural rendering. Many  modern light simulation programs

can now simulate proper light transport with  shiny objects, none of these blocky artifacts,   they are, as you see, absolutely amazing.   They can render all kinds of material models,   detailed geometry, caustics, color bleeding,  you name it. However, they seem to start   out from a noisy image, and as we compute the  path of more and more light rays, this noisy image   clears up over time. But, this still takes a  while. How long? Well, from minutes to days. And then, neural rendering entered the  fray. Here you see our earlier paper   that replaced the whole light simulation  program with a neural network that learned   how to do this. And it can create these images  so quickly that it easily runs not in minutes   or days, but as fast as you see here. Yes,  in real time on a commodity graphics card.    Now note that this neural renderer  is limited to this particular scene. With this, I hope that it is  easy to see that we are now   so far beyond radiosity that it sounds like a  distant memory of the olden times. So, once again,   why talk about radiosity? Well, check this out. Chapter 3. Neural Radiosity. Excuse  me, what? Yes, you heard it right.

This paper is about neural radiosity. This  work elevates the old radiosity algorithm   by using a similar formulation to the original  technique, but also, infusing it with a powerful   neural network. It is using the same horse  carriage, but strapping it onto a rocket,   if you will. Now you have my attention.   So, let’s see what this can do together. Look. Hmm! Yes, it can render really  intense specular highlights. And now,   hold on to your papers, and wow. Once again,  the results look like a nearly pixel-perfect   copy of the reference simulation. And, we now  understand the limitations of the old radiosity,   so, let’s strike where it hurts the most.   Yes, perfectly specular, mirror-like surfaces.    Let’s see what happens here. Well, I can  hardly believe what I am seeing here!    No issues whatsoever. Still, close  to pixel perfect. This new paper   truly elevates the good old radiosity  to the next level. So good! Loving it. But wait a second. I hear you asking, yes,  Károly, this is all well and good, but   if we have the reference simulation, why  not just use that? Good question. Well,   that's right, the reference is great, but that  one takes up to several hours to compute, and   the new technique can be done super quickly. Yet,  they look almost exactly the same. My goodness. In fact, let’s look at an equal-time comparison  against one of the Tesla techniques, path tracing.    We give the two techniques the same amount of  time, and see what they can produce. Let’s see.    Now that is no contest. Look. This is Eric Veach’s  legendary scene where the light only comes from   the neighboring room through a door that is only  slightly ajar. Notoriously difficult for any kind   of light transport algorithm. And yet, look  at how good this new one is in tackling it. Now, note that not even this technique is perfect.   It has two caveats. One, the training takes place   per-scene. This means that we need to give  these scenes to the neural network in advance,   so it can learn how light bounces off of this  place, and this can take from minutes to hours.    But, for instance, if we have a video game  with a limited set of places that you can go,   we can train our neural network on all of  them in advance, and deploy it to the players,   who can then enjoy it for as long as they wish.   No more training is required after that. But,   the technique would still need to be  a little faster than it currently is.    And two, we also need quite a bit  of memory to perform all this. And, yes! I think this is an excellent  place to invoke the First Law Of Papers,   which says that research is a  process. Do not look at where we are,   will be two more papers down  the line. And two more papers down the line,   who knows, maybe we get this in real time and  with a much friendlier memory consumption. Now, there is one more area that I think would  be an excellent direction for future work. And   that is about the per-scene training  of the neural network. In this work,   it has to get a feel of the scene before the light  simulation happens. So how does the knowledge   learned on one scene transfer to others? I imagine  that it should be possible to create a more   general version of this that does not need to look  at a new scene before the simulation takes place. And in summary. I absolutely love this  paper. I takes an old algorithm,   blows the dust off of it, and  completely reinvigorates it   by infusing it with a modern learning-based  technique. What a time to be alive! So, what do you think? What would you use  this for? I’d love to hear your thoughts,   please let me know in the comments below. And when watching all these beautiful results, if  you feel that this light transport thing is pretty   cool, and you would like to learn more about it,  I held a Master-level course on this topic at the   Technical University of Vienna. Since I was always  teaching it to a handful of motivated students,   I thought that the teachings shouldn’t  only be available for the privileged few   who can afford a college education, but the  teachings should be available for everyone.    Free education for everyone, that’s what I want.   So, the course is available free of charge for   everyone, no strings attached, so make sure to  click the link in the video description to get   started. We write a full light simulation program  from scratch there, and learn about physics,   the world around us, and more. If you watch  it, you will see the world differently. Thanks for watching and for your generous  support, and I'll see you next time!