# Google’s New AI Learned To See In The Dark! 🤖

## Метаданные

- **Канал:** Two Minute Papers
- **YouTube:** https://www.youtube.com/watch?v=7iy0WJwNmv4
- **Дата:** 17.08.2022
- **Длительность:** 8:56
- **Просмотры:** 372,514
- **Источник:** https://ekstraktznaniy.ru/video/13482

## Описание

❤️ Check out Weights & Biases and sign up for a free demo here: https://wandb.com/papers 

📝 The paper "NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images" is available here:
https://bmild.github.io/rawnerf/index.html

❤️ Watch these videos in early access on our Patreon page or join us here on YouTube: 
- https://www.patreon.com/TwoMinutePapers
- https://www.youtube.com/channel/UCbfYPyITQ-7l4upoX8nvctg/join

🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Balfanz, Alex Haro, Andrew Melnychuk, Benji Rabhan, Bryan Learn, B Shang, Christian Ahlin, Eric Martel, Geronimo Moralez, Gordon Child, Ivo Galic, Jace O'Brien, Jack Lukic, John Le, Jonas, Jonathan, Kenneth Davis, Klaus Busse, Kyle Davis, Lorin Atzberger, Lukas Biewald, Matthew Allen Fisher, Michael Albrecht, Michael Tedder, Nevin Spoljaric, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Rajarshi Nigam, Ramsey Elbasheer, Steef, Taras

## Транскрипт

### Introduction []

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today we are going take a collection of photos  like these, and magically, create a video   where we can fly through these photos. And  we are going to do all this, with a twist. So, how is this even possible? Especially  that the input is only a handful of photos.    Well, typically, we give  it to a learning algorithm   and ask it to synthesize a photorealistic video  where we fly through the scene as we please.    Of course, that sounds impossible. Especially  that some information is given about the scene,   but this is really not much. And as you see, this  is not impossible at all - through the power of   learning-based techniques, this previous AI is  already capable of pulling off this amazing trick.    And today, I am going to show you  something even more incredible. Now, did you notice that most of these were shot  during the daytime, and these are all well lit   images. Every single one of them. So, our question  today is, can we perform view synthesis in the   dark? And my initial answer would be a resounding  no. Why? Well, in this case, we not only have to

### Raw Sensor Data [1:26]

deal with less detail in the images. It would also  be very difficult to stitch new views together if   we have images like this one. Luckily, we have  a choice. Instead, we can try something else,   and that is, using raw sensor data instead. It  looks like this. We get more detail, but, uh-oh,   now we also have a problem. Do you see the problem  here? Yes, that’s right. In the raw sensor data,   we have more detail, but also, much more  noise that contaminates this data too. We either have to choose from less detail and  less noise or from more detail more noise. So,   I guess that means that we get no  view synthesis in the dark, right? Well, don’t despair, not everything is lost yet.   There are image denoising techniques that we can   reach out to. Let’s see if this gets any  better. Hmm! It definitely got a lot better,

### Previous Method [2:32]

but I have to be honest, this is not even close to  the quality we need for view synthesis. This one   denoises a single image. However, yes, finally,  there is an opening here! Remember, in the world   of NERFs, we are not using a single image, we  are using a package of images. A package contains   much more information than just one image,  and hopefully, it can be denoised better. So   this is what the previous method could do,  and now, hold on to your papers, and let’s   look at this new technique, called RAWNerf!   Can it pull this off? Wow! Seemingly, it can. So, now, can we be greedy and hope that view  synthesis works on this data? Let’s see.    My goodness. It really does! And, we are not done  yet. In fact, we are just starting. It can do even   more! For instance, it can perform tone mapping  on the underlying data to bring out even more   detail from these dark images, and here comes  my favorite. Oh yes. We can also refocus these   images, and these highly sought after depth  of field effects will start to appear. I love   it. And what I love even more is that we can even  play with this in real time to refocus the scene. This is a very impressive set of features, so   let’s take this out for a spin and marvel  together at 5 amazing examples of what it can do. Yes, once again, this is extremely noisy.   For instance, can you read this street sign?    Not a chance right? And, what about now?

### View Synthesis [4:29]

This looks like magic. I love it. Now let’s start  the view synthesis part, and this looks really   good given the noisy inputs. The previous,  original NERF technique could only produce   this, and this is not some ancient technique.   Nuh-uh. No sir. This is from just 2 years ago,   and today, a couple papers down the line, and  we get this. I can’t believe it. We can even see   the specular highlight moving around around  the badge of the car here. Outstanding. Two, actually, let’s have a closer look at  specular highlights. Here is a noisy image,   the denoised version, and the view synthesis.   And the specular highlights are once again,   excellent. These are very difficult to capture  because they change a great deal as we move the   camera around, and the photos are spaced out  relatively far from each other. This means a   huge challenge for the learning algorithm, and  as you see, this one passes with flying colors. Three, thin structures are always a problem.   Look, an otherwise excellent previous technique   had a great deal of trouble with the  fence here, even in a well lit scene. So,

### Night Photos [5:52]

let’s see. Are you kidding me? Doing the same  with a bunch of nighttime photos? There is not   a chance that this will work. So, let’s  see. Look at that! I am out of words.    Or you know what’s even better, let’s be really  picky, and look here instead, these areas are   even more challenging. And even these work  really well. Such improvement in so little time. Four, as I am a light transport researcher  by trade, I would love to look at it resolve   some more challenging specular highlights. For  instance, you can see how the road reflects the   street lights here and the result looks not  just passable, this looks flat out gorgeous. Now, talking about gorgeous scenes. Let’s look at  some more of those. Five, putting it all together.    This will be a stress test for the new technique.   Let’s change the viewpoint, refocus the scene,   and play with the exposure at the same time.   That is incredible. What a time to be alive!

### Conclusion [7:02]

And you are saying that it does all this from a  collection of 25 to 200 photos? We can shoot these   in seconds! Now, clearly, not even this technique  is perfect, we can see that this does not match   reality exactly, but going from a set of extremely  noise raw images to this is truly a sight to   behold. The previous, two-year old technique  couldn’t even get close to these results. Bravo! And, this is an excellent place for  us to apply the First Law Of Papers,   which says research is a process. Do not look  at where we are, look at where we will be two   more papers down the line. So, what do  you think? What will we be able to do   two more papers down the line? And what would you  use this for? Let me know in the comments below! Thanks for watching and for your generous  support, and I'll see you next time!