Adobe & NVIDIA’s New Tech Shouldn’t Be Real Time. But It Is.

Look at this… absolutely gorgeous. It  contains lots of beautiful glinty particles.   And believe it or not, this one is rendered  on my laptop and in real time. Yup. This   research work is free and open for everyone,  so you can try it too, I’ll tell you how. If you ever looked at fresh snow under a  streetlamp, or metallic car paint in the   bright sun, you see this amazing glinty  explosion. And if you turn your head,   or in this case, the camera around,  oh my, it’s an incredible sight.   But it’s also incredibly difficult to simulate in  a computer program. Why? Because these surfaces   have millions of microscopic, reflective  flakes. If you try to simulate them all,   your computer crashes. If not, you get a  boring, bland object in your games and movies. So how do we compute all this without using  gigabytes of memory or destroying our framerate? Now hold on to your papers Fellow Scholars,  because this incredible new technique can do that,   and it promises more than 280 frames per second,  wow, that’s crazy. On a consumer NVIDIA graphics   card. But you don’t even need that - it runs in  real time on my much less powerful laptop too. Okay, but how? Well, imagine trying to host  the world's biggest party. Usually, you need a   guest list to know where everyone is standing.   Well, this paper says, throw away that list,   brother. You won’t need it. Okay, but how do  we host a party without a guest list? Well,   instead of remembering where every  guest, every glitter particle is,   they use a bouncer. A really muscular  guy. And this guy doesn't need a list.    He uses some mathematical rule to decide exactly  where a guest should be standing the moment you   look at that spot. He generates the  party guests on the fly, instantly! We’ll go into some more details, but I  cannot resist showing you these results   in the meantime. It can even nail  that sun on the ocean look. Now it   does not look nearly perfect because  this is not a whitewater simulation,   so it does not have foam and bubbles. But that’s  not the point here. The point is that you can   rotate the camera around, it looks great, and it  remains temporally stable. What does that mean? Normally, I would say that this means  the technique remembers what it did   a moment ago. But it does not. That is  the key! It is so fast, for every frame,   it can very easily and quickly recalculate the  result. And it is so accurate, it will always   look exactly the same! So no crazy jumps. And  this is super useful because you don’t need   to use a lot of memory to have these millions and  millions of glinty little mirrors on your objects. So when the camera moves, the sparkles  don't flicker like a broken strobe light.    Nope, they shimmer beautifully. Okay, now wait a second. This is  not the first technique that is   able to render glints. So how does  it do against previous techniques? Here is how it does against one of the industry  standard sampling techniques called GGX. This   is an equal time comparison, so both methods  were given the same amount of time. This allows   us to check how fast the noise clears up. But  this new technique makes all this really easy,   because I can stop this at any frame I want,  and look. The new one always seems better than   GGX. Crazy. Why though? Well, because  GGX searches for the sparkles blindly,   so the image stays noisy and takes a long  time to clear up. But the new technique   knows exactly where they are! It  cleans up the image much quicker. Okay, now we said that the muscular bouncer  guy has a secret. What is the secret? How does   he control the crowd so there is a big party  going on, without being overwhelmed with the   guest list? Well, first, he divides the dance  floor into a grid. If you look from far away,   he groups the guests into big blocks and just  tells you, don’t worry about it, there is a party   over there. But as you walk closer, he breaks  those blocks down into smaller VIP sections,   revealing the individual dancers. He  manages the crowd density dynamically   so you never see the empty spaces, but you  also never get overwhelmed by the crowd. So here, the dance floor is the surface  of the object, and the dancers are the   little flakes that create the sparkles.   So you get to simulate as much detail as   needed with the minimum amount of compute  necessary. Absolutely spectacular work.

Now, this car example, it looks alright,  I’ll be honest here. Not the best parameter   setup for my liking. However, it has its  moments. But this reveals an incredible   property of the new technique. It  can be UV-free! What does that mean? Usually, to put a texture on  a 3D object, like this dragon,   you have to unwrap its skin onto a flat  2D image. Just like flattening a piece   of gift wrapping paper. This process is  called UV mapping, and for complex shapes,   it is a nightmare - the paper tears,  stretches, or has ugly seams. No thanks! But here, you don’t even need that.   Why? Because our bouncer doesn't   need to look at a 2D map or a guest list.   No! Instead, he operates in the 3D world,   not on a flat piece of paper. This means you can  have a complex car chassis or a twisting dragon,   doesn’t matter. And look, the sparkles  just magically appear in the right place,   instantly, no ugly seams and no  stretching. Absolutely incredible work. But if you look closely, it teaches us a  great deal more than it seems at first.   You see, the system discards the  massive guest list and uses a bouncer,   a simple math rule to generate them on the  fly. But that is incredible life advice.    This is exactly what I taught my students at  the university. Stop hoarding information.    Don't memorize the encyclopedia.   That’s useless. Learn the principles,   learn the rules. This way, you can derive  the answer in any situation quickly. Also don’t forget, this method works in 3D space,  refusing to flatten the object onto a 2D map to   avoid tearing and stretching. Great life advice.   Maintain your dimensionality. Do not flatten your   3D personality into a 2D label just to make it  easier for others to process you. No! Stay 3D,   Fellow Scholars. I also live my life this way.   This video series is an absolute madhouse,   but it’s my madhouse and I love it. For some  people, it is too much. For you Fellow Scholars,   it is just right…hopefully. And I  am super lucky to have all of you   on our journey. The Papers couldn’t  exist without you. Thank you so much! We can learn so much from  these papers, it’s incredible. Now all this wizardry comes from the heavy  hitter scientists at Adobe Research, NVIDIA,   and Aalto University. And we get  all this for free. I can’t believe   it. How amazing is that? What a time to be alive! Now, before I show you how you can try it right  now, I’ll tell you that not even this technique   is perfect. First, the method is not strictly  energy conserving, meaning the simulation can   artificially gain or lose light energy near domain  boundaries. Now, for video games and movies,   usually not a problem. The difference is so  little. For super scientific experiments, however,   look elsewhere. Second, some parameter pairs are  not independent, so certain combinations can lead   to counterintuitive visual results. Third, if  you want the UV-free wrapping paper property,   then it is a bit slower. So trying to be  accurate not to overstate things here. Okay, and you Fellow Scholars  can try it right now! There is a link to the paper in the video  description, and another one for the demo,   just click and run it in your browser.   Then, if you click and go left to right,   you can get less or more glinty particles. Look at  this. So pretty. And if you go down or up, you can   change the roughness of the surface to rougher. They also give us the full source code,   it can be implemented in about 337 lines of code,  kind of insane. And thus, you can also play with   the parameters, press a button, recompile, and  the world suddenly works a bit differently.