# This AI Creates Real Scenes From Your Photos! 📷

## Метаданные

- **Канал:** Two Minute Papers
- **YouTube:** https://www.youtube.com/watch?v=T29O-MhYALw
- **Дата:** 22.09.2020
- **Длительность:** 5:44
- **Просмотры:** 204,383

## Описание

❤️ Check out Weights & Biases and sign up for a free demo here: https://www.wandb.com/papers 
❤️ Their mentioned post is available here: https://app.wandb.ai/sweep/nerf/reports/NeRF-%E2%80%93-Representing-Scenes-as-Neural-Radiance-Fields-for-View-Synthesis--Vmlldzo3ODIzMA

📝 The paper "NeRF in the Wild - Neural Radiance Fields for Unconstrained Photo Collections" is available here:
https://nerf-w.github.io/

Photos by Flickr users dbowie78, vasnic64, punch, paradasos, itia4u, jblesa, joshheumann, ojotes, chyauchentravelworl, burkeandhare, photogreuhphie / CC BY

🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Haro, Alex Paden, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bruno Mikuš, Bryan Learn, Christian Ahlin, Daniel Hasegan, Eric Haddad, Eric Martel, Gordon Child, Javier Bustamante, Joshua Goller, Lorin Atzberger, Lukas Biewald, Michael Albrecht, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Ramsey Elbasheer, Robin Graham, Steef, Sunil Kim, Taras Bobrovytsky, Thomas Krcmar, Torsten Reil, Tybie Fitzhugh.
If you wish to support the series, click here: https://www.patreon.com/TwoMinutePapers

Károly Zsolnai-Fehér's links:
Instagram: https://www.instagram.com/twominutepapers/
Twitter: https://twitter.com/twominutepapers
Web: https://cg.tuwien.ac.at/~zsolnai/

## Содержание

### [0:00](https://www.youtube.com/watch?v=T29O-MhYALw) Segment 1 (00:00 - 05:00)

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Approximately 5 months ago, we talked about a technique called Neural Radiance Fields, or NERF in short, that worked on a 5D neural radiance field representation. So what does this mean exactly? What this means is that we have 3 dimensions for location and two for view direction, or in short, the input is where we are in space give it to a neural network to learn it, and synthesize new, previously unseen views of not just the materials in the scene, but the entire scene itself. In short, it can learn and reproduce entire real-world scenes from only a few views by using neural networks. And the results were just out of this world. Look. It could deal with many kinds of matte and glossy materials, and even refractions worked quite well. It also understood depth so accurately that we could use it for augmented reality applications where we put a new, virtual object in the scene and it correctly determined whether it is in front of, or behind the real objects in the scene. However, not everything was perfect. In many cases, it had trouble with scenes with variable lighting conditions and lots of occluders. You might ask, is that a problem? Well, imagine a use case of a tourist attraction that a lot of people take photos of, and we then have a collection of photos taken during a different time of the day, and of course, with a lot of people around. But, hey, remember that this means an application where we have exactly these conditions: a wide variety of illumination changes and occluders. This is exactly what NERF was not too good at! Let’s see how it did on such a case. Yes, we see both abrupt changes in the illumination, and the remnants of the folks occluding the Brandenburg Gate as well. And this is where this new technique from scientists at Google Research by the name NERF-W shines. It takes such a photo collection, and tries to reconstruct the whole scene from it, which we can, again, render from new viewpoints. So, how well did the new method do in this case? Let’s see. Wow. Just look at how consistent those results are. So much improvement in just 6 months of time. This is unbelievable. This is how it did in a similar case with the Trevi fountain. Absolutely beautiful. And what is even more beautiful is that since it has variation in the viewpoint information, we can change these viewpoints around as the algorithm learned to reconstruct the scene itself. This is something that the original NERF technique could also do, however, what it couldn’t do, is the same, with illumination. Now, we can also change the lighting conditions together with the viewpoint. This truly showcases a deep understanding of illumination and geometry. That is not trivial at all! For instance, when loading this scene into this neural re-rendering technique from last year, it couldn’t tell whether we see just color variations on the same geometry, or if the geometry itself is changing. And, look, this new method does much better on cases like this. So clean! Now that we have seen the images, let’s see what the numbers say for these scenes. The NRW is the neural re-rendering technique we just saw, and the other one is the NERF paper from this year. The abbreviations show different ways of comparing the output images, the up and down arrows show whether they are subject to maximization or minimization. They are both relatively close, but, when we look at the new method, we see one of the rare cases where it wins decisively regardless of what we are measuring. Incredible. This paper truly is a great leap in just a few months, but of course, not everything is perfect here. This technique may fail to reconstruct regions that are only visible on just a few photos in the input dataset. The training still takes from hours to days, I take this as an interesting detail more than a limitation since this training only has to be done once, and then, using the technique can take place very quickly. But, with that, there you go, a neural algorithm that understands lighting, geometry, can disentangle the two, and reconstruct real-world scenes from just a few photos. It truly feels like we are living in a science fiction world. What a time to be alive!

### [5:00](https://www.youtube.com/watch?v=T29O-MhYALw&t=300s) Segment 2 (05:00 - 05:00)

Thanks for watching and for your generous support, and I'll see you next time!

---
*Источник: https://ekstraktznaniy.ru/video/14066*