# Waymo's AI Recreates San Francisco From 2.8 Million Photos! 🚘 ## Метаданные - **Канал:** Two Minute Papers - **YouTube:** https://www.youtube.com/watch?v=8AZhcnWOK7M - **Дата:** 02.04.2022 - **Длительность:** 6:59 - **Просмотры:** 217,921 ## Описание ❤️ Check out Weights & Biases and sign up for a free demo here: https://wandb.com/papers ❤️ Their mentioned post is available here (Thank you Soumik Rakshit!): http://wandb.me/2min-block-nerf 📝 The paper "Block-NeRF Scalable Large Scene Neural View Synthesis" from #Waymo is available here: https://waymo.com/research/block-nerf/ ❤️ Watch these videos in early access on our Patreon page or join us here on YouTube: - https://www.patreon.com/TwoMinutePapers - https://www.youtube.com/channel/UCbfYPyITQ-7l4upoX8nvctg/join 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible: Aleksandr Mashrabov, Alex Balfanz, Alex Haro, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bryan Learn, Christian Ahlin, Eric Martel, Gordon Child, Ivo Galic, Jace O'Brien, Javier Bustamante, John Le, Jonas, Jonathan, Kenneth Davis, Klaus Busse, Lorin Atzberger, Lukas Biewald, Matthew Allen Fisher, Michael Albrecht, Michael Tedder, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Paul F, Rajarshi Nigam, Ramsey Elbasheer, Steef, Taras Bobrovytsky, Ted Johnson, Thomas Krcmar, Timothy Sum Hon Mun, Torsten Reil, Tybie Fitzhugh, Ueli Gallizzi. If you wish to appear here or pick up other perks, click here: https://www.patreon.com/TwoMinutePapers Thumbnail background design: Felícia Zsolnai-Fehér - http://felicia.hu Meet and discuss your ideas with other Fellow Scholars on the Two Minute Papers Discord: https://discordapp.com/invite/hbcTJu2 Károly Zsolnai-Fehér's links: Instagram: https://www.instagram.com/twominutepapers/ Twitter: https://twitter.com/twominutepapers Web: https://cg.tuwien.ac.at/~zsolnai/ #BlockNeRF ## Содержание ### [0:00](https://www.youtube.com/watch?v=8AZhcnWOK7M) Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today we are going to see if Waymo’s AI   can recreate a virtual copy of San  Francisco from 2. 8 million photos. To be able to do that, they rely on a previous  AI-based technique that can take a collection   of photos like these, and magically, create a  video where we can fly through these photos. ### [0:25](https://www.youtube.com/watch?v=8AZhcnWOK7M&t=25s) Output: Fly-through video This is what we call a NERF-based technique,  and these are truly amazing. Essentially,   photos go in, the AI fills in the gaps, and  reality comes out. And there are a lot of   gaps between these photos, all of which are  filled in with high-quality synthetic data. So, as you see with these previous methods, great  leaps are being made, but one thing stayed more or   less the same: and that is, the scale of these  scenes is not that big. So, scientists at Waymo   had a crazy idea, and they said, rendering  just a tiny scene is not that useful. We have   millions of photos laying around, why not render  an entire city like this? So, can Waymo do that?    Well, maybe, but that would take Waymo'. I am  sorry. I am so sorry, I just couldn't resist. Well, let’s see what they came up with. Look,  these self-driving cars are going around the city,   they take photos along their journey, and…well,  I have to say that I am a little skeptical here.    Have a look at what previous techniques could do  with this dataset. This is not really usable, so,   could Waymo pull this off? Well, hold on to  your papers, and let’s have a look together.    My goodness…this is their fully reconstructed   3D neighborhood from these  photos. Wow. That is superb. And don’t forget, most of this  information is synthetic. That is,   filled in by the AI. Does this mean that? Yes!   Yes it does! It means three amazing things. One, we can drive a different path that  has not been driven before by these cars, ### [2:27](https://www.youtube.com/watch?v=8AZhcnWOK7M&t=147s) Different pathways and still see the city correctly.   Two, we can look at these  buildings from viewpoints ### [2:36](https://www.youtube.com/watch?v=8AZhcnWOK7M&t=156s) New angles that we don’t have enough information about,  and the AI fills in the rest of the details.    So cool! But it doesn't end there, not even close. Three, here comes my favorite. We can also engage   in what they call appearance modulation.   Yes, some of the driving took place at night, ### [2:58](https://www.youtube.com/watch?v=8AZhcnWOK7M&t=178s) Appearance modulation some during the daytime, so we have information  about the change of the lighting conditions.    What does that mean? It means that we can  even fuse all this information together   and choose the time of day for our virtual  city. That is absolutely amazing. I love it. Yes, of course, not even this technique  is perfect, the resolution and the details   should definitely be improved over time.   Plus, it does well with a stationary city,   but with dynamic moving objects, not so  much. But do not forget, the original,   first NERF paper was published just  two years ago, and it could do this.    And now, just a couple papers down the line, and  we have not only these tiny scenes, but entire   city blocks. So much improvement in just a couple  papers. How cool is that. Absolutely amazing. And with this, we can drive and play  around in a beautiful virtual world   that is a copy of the real world around us. And  now, if we wish it to be a little different,   we can even have our freedom  in changing this world   according to our artistic vision. I would  love to see more work in this direction. But wait, here comes the big question. What  is all this good for? Well, one of the answers   is sim2real. What is that? Sim2real means  training an AI in a simulated world, and   trying to teach it everything it needs to learn  there before deploying it into the real world. Here is an amazing example, look, OpenAI  trained their robot hand in a simulation   to be able to rotate these Rubik cubes. And then,  deployed this software onto a real robot hand,   and, look! It can use this simulation knowledge,  and now it works in the real world too.    But sim2real has relevance to self-driving cars  too. Look. Tesla is already working on creating   virtual worlds, and training their cars there. One  of the advantages of that is that we can create   really unlikely and potentially unsafe scenarios,  but, in these virtual worlds, the self-driving AI   can train itself safely. And when they deploy  them into the real world, they will have all   this knowledge. It is fantastic to see Waymo also  moving in this direction. What a time to be alive! So, what would you use this for? What do you  expect to happen a couple more papers down the   line? Please let me know in the comments  below. I’d love to hear your thoughts. Thanks for watching and for your generous  support, and I'll see you next time! --- *Источник: https://ekstraktznaniy.ru/video/13608*