# NVIDIA’s New AI: Impossible Ray Tracing!

## Метаданные

- **Канал:** Two Minute Papers
- **YouTube:** https://www.youtube.com/watch?v=hUVfAVjsfL4
- **Дата:** 28.04.2025
- **Длительность:** 8:50
- **Просмотры:** 280,914
- **Источник:** https://ekstraktznaniy.ru/video/12426

## Описание

❤️ Check out DeepInfra and run DeepSeek or many other AI projects: https://deepinfra.com/papers

📝 The #nvidia paper "3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting" is available here:
https://research.nvidia.com/labs/toronto-ai/3DGUT/

📝 Our Separable Subsurface Scattering paper: https://users.cg.tuwien.ac.at/zsolnai/gfx/separable-subsurface-scattering-with-activision-blizzard/
📝 SSS For Gaussian Splatting: https://sss.jdihlmann.com/

Sources:
https://x.com/jonstephens85/status/1911923445660893321?s=46
https://x.com/jonstephens85/status/1908730004973986175?s=46
https://www.youtube.com/watch?v=KPFPuXm2u7I
https://www.youtube.com/shorts/si1-zFZpeDE

📝 My paper on simulations that look almost like reality is available for free here:
https://rdcu.be/cWPfD 

Or this is the orig. Nature Physics link with clickable citations:
https://www.nature.com/articles/s41567-022-01788-5

🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers poss

## Транскрипт

### Segment 1 (00:00 - 05:00) []

There are two ways of creating and rendering a virtual world. This is what we call rasterization. This is what most computer games have used for many years now. It is super fast and most of the time it looks good enough. But only if you feel that you can live a happy life without super high quality reflections and refractions. But not me because for that you need something more. My favorite ray tracing. Look at these beauties. This you cannot get through rasterization only through ray tracing. But there is a problem. It simulates the path of millions and millions of light rays which takes super long to compute like orders of magnitude longer. Sometimes from minutes to even weeks. Ouch. This one is a beauty. Reflections, refractions, volutric costics. Oh my. But it took us 3 weeks to render. So rasterization fast limited rate tracing a complete solution but slow. So I guess choose one right? Well scientists at Nvidia had an absolutely insane idea. They said why not do both? Well my opinion was before reading this paper don't do both because it is impossible to pull off. These two ideas are like oil and water. They just don't mix. you most likely end up with the disadvantages of both. That is a limited but super expensive solution. No thank you. So, did they pull off the impossible? We'll get to that in a moment. And then splat happens. I mean, a paper called Gaussian splats came out and took the world by storm. What is that? It's a technique that thinks in terms of little Gaussians representing a scene like this as a collection of little bumps, if you will. And note that these scenes can come from real life. You just walk around with your camera and then you can play in a digital version of it. And the cool part is that it gets better. It is incredibly fast, easily faster than real time. So far so good. However, here is the problem. Mirrorike specular reflections are not great. And many of the beautiful ray tracing effects that you just saw earlier don't look so good there. It can also eat up quite a bit of memory which is scarce and expensive on graphics hardware these days. But the limitations don't stop there. It doesn't support any fancy camera models. For instance, fisheye cameras don't even think about it. Also, no camera tricks like rolling shutter. So, there is tons of potential in Gaussian plats. But if I think about the fact that we get no refractions, none of the good stuff that we fellow scholars love so much. This all ranges from very difficult to impossible. But don't despair because here comes the impossible but at the very least insane idea. Let's do both rasterization and ray tracing at the same time. This research work does it by taking Gaussian splatting and adding to it something they call secondary rays. Oh, this means that there are finally rays of light in the system and they are allowed to bounce around. So, are you thinking what I am thinking? Oh, yes. Hold on to your papers, fellow scholars. And look at that. A proper virtual world running in real time and high quality reflections. I am absolutely loving this one. And refractive objects like glass also. Wow. Now, let's whip out that fisheye camera. And yes, finally, this all used to be impossible. And now, it is not only here, but it runs in real time. And here comes my favorite part, the name of this work. You'll love this. They call it the 3D Gaussian unscented transform. 3D GU T. Yep. But that's not how we will call it. No sir, we shall call it 3D gut. Ah, much better. And here is a nice little surprise. Do you know what super important AI area likes weird camera models and rolling shutter training self-driving cars? Oh yes, and this is going to be super useful for that too. But is it really better than previous techniques? Let's see together. Dear fellow scholars, this is two minute papers with Dr. Kohaa. So 3D gut looks amazing. But we are fellow scholars here. We want to see the comparison against previous techniques. Well, it turns out there is one earlier method that can extend Gaussian splats to fisheye cameras. But oof, I see lots and lots of artifacts and distortions here. So, do you have the guts to have a look at the new technique? Let's see. Oh my, that is a huge difference. The

### Segment 2 (05:00 - 08:00) [5:00]

previous one had trouble, especially with objects that are closer to the camera, but this new one loving it. But the story doesn't end here. There is so much more below the surface. We are going to look at this too in a moment. And here comes the best part. They didn't just close it down to sell it to us. Nope. All this is available right now for all of us for free. The source code is out there. I put a link in the video description. So, I hope you fellow scholars have the guts to play with it. Sorry, I promised myself not to do it again and I just did. But in fact, some of you already do play with it. This fellow scholar says he trained it to only 30% and it already looks amazing even through the compression artifacts from social media. Okay, but I hear you asking, "Wait a minute, Caro. What is this beauty? " Well, I said that there is more below the surface. I said that because this other work is about light transport that happens within these objects. That is subsurface scattering. The art of rendering beautiful translucent objects. Now, you can do that in rasterization. Here is a work called separable subsurface scattering and it gives you super fast human skin, marble, milk, you name it. I love working on things like this so much. And it is so simple. When we published it, we even implemented a version of it that fits into 4 kilobytes. That is crazy. It fits into a file that is smaller than half a second of MP3 music. Much smaller. It is available on my website. The link is in the description. And to the best of my knowledge, you can also use it in Unreal Engine as well. So this is subsurface scattering. but for rasterization, not for Gaussian splats. What about that? Well, this amazing new work offers you exactly that. And it gets even better. It supports real lighting, too. So, you have your object, you add an image, and it imagines what that object would look like in these different environments. Super fun. And if you don't like what you got, you can even edit these materials. So you can go from skin to glass to wax. I mean just think about that. Gaussian splats are not old at all. It's about 2 years old. And now through the power of open research and human ingenuity and a hint of AI reflections, refractions and translucency is already possible for Gaussian splats. Yes, now you can get reasonably high quality translucent objects in these virtual worlds too. So next level computer games and virtual worlds are coming and self-driving cars can also learn better. I am kind of stunned that almost nobody is talking about these papers. But this is why two minute papers exist. And don't forget we are all fellow scholars here. So we love to look at these papers for more information. And it goes without saying that everything is available in the video description. So what do you think? What would you fellow scholars use this for? Let me know in the comments below. Fellow scholars, check this out. This is Deep Infra, an AI inference cloud where you can get access to the best open AI models like Deep Seek, Lama, and pay up to 10 times less than elsewhere. Or use my favorite, the amazing Flux texttoage AI. Generate speech from text and more. It is super easy to use. Try it out today at deepinfra. com/papers or click the link in the description and you get $20 of credit for