This Neural Network Learned To Look Around In Real Scenes! (NERF)

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. About two years ago, we worked on a neural rendering system, which would perform light transport on this scene and guess how it would change if we would change the material properties of this test object. It was able to closely match the output of a real light simulation program, and, it was near instantaneous as it took less than 5 milliseconds instead of the 40-60 seconds the light transport algorithm usually requires. This technique went by the name Gaussian Material Synthesis, and the learned quantities were material properties. But this new paper sets out to learn something more difficult, and also, more general. We are talking about a 5D neural radiance field representation. So what does this mean exactly? What this means is that we have 3 dimensions for location and two for view direction, or in short, the input is where we are in space and what are we looking at, and the resulting image of this view. So here, we take a bunch of this input data, learn it, and synthesize new, previously unseen views of not just the materials in the scene, but the entire scene itself. And here, we are talking not only digital environments, but also, real scenes as well! Now that’s quite a value proposition, let’s see if it can live up to this promise! Wow! So good. Love it! But, what is it really that we should be looking at? What makes a good output here? The most challenging part is writing an algorithm that is able to reproduce delicate, high-frequency details while having temporal coherence. So what does that mean? Well, in simpler words, we are looking for sharp and smooth image sequences. Perfectly matte objects are easier to learn here because they look the same from all directions, while glossier, more reflective materials are significantly more difficult, because they change a great deal as we move our head around, and this highly variant information is typically not present in the learned input images. If you read the paper, you’ll see these referred to as non-Lambertian materials. The paper and the video contains a ton of examples of these view-dependent effects to demonstrate that these difficult scenes are handled really well by this technique. Refractions also look great. Now, if we define difficulty as things that change a lot when we change our position or view direction a little, not only the non-Lambertian materials are going to give us headaches, occlusion can be challenging as well. For instance, you can see here how well it handles the complex occlusion situation between the ribs of the skeleton here. It also has an understanding of depth, and this depth information is so accurate, that we can do these nice augmented reality applications where we put a new, virtual object in the scene and it correctly determines whether it is in front of, or behind the real objects in the scene. Kind of what these new iPads do with their LiDAR sensors, but without the sensor. As you see, this technique smokes the competition. So what do you know, entire real-world scenes can be reproduced from only a few views by using neural networks. And the results are just out of this world. Absolutely amazing. What you see here is an instrumentation of this exact paper we have talked about, which was made by Weights and Biases. I think organizing these experiments really showcases the usability of their system. Weights & Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money, and it is actively used in projects at prestigious labs, such as OpenAI, Toyota Research, GitHub, and more. And, the best part is that if you are an academic or have an open source project, you can use their tools for free. It really is as good as it gets. Make sure to visit them through wandb. com/papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights & Biases for their long-standing support and for helping us make better videos for you. Thanks for watching and for your generous support, and I'll see you next time!