Google’s New AI: Fly INTO Photos! 🐦

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today, we are able to take a bunch of photos,  and use an AI to magically, create a video   where we can fly through these photos. It is  really crazy, because this is possible today,   for instance, here is NVIDIAs method that can be  trained to perform this in a matter of seconds. Now, I said that in these we can fly through  these photos. But here is an insane idea:   what if we used not multiple photos, but just  one photo, and we don’t fly through it, but fly   into this photo. Now, you are probably asking,  Károly, what are you talking about? This   is completely insane, and it wouldn’t work with  these NERF-based solutions like the one you see   here, these were not designed to  do this at all! Look. Oh yes. That. So, in order to fly into these photos, we  would to invent at least 3 things. One,   image inpainting. Look, if we are to fly into  this photo, we will have to be able to look at   regions between the trees. Unfortunately, these  are not part of the original photo, and hence,   new content needs to be generated intelligently.   That is a formidable task for an AI,   and luckily, image inpainting techniques  already exist out there. Here’s one. But inpainting is not nearly  enough. Two. As we fly into a photo,   completely new regions should also  appear that are beyond the image. This   means that we also need to perform image  outpainting, creating these new regions.    Continuing the image, if you will. Luckily, we are  entering the age of AI-driven image generation,   and this is also possible today, for  instance, with this incredible tool. But even that is not enough. Why is that? Well,  three! As we fly closer to these new regions,   we will be looking at fewer and fewer pixels  and from closer and closer, which means…this.    Oh my, another problem. So, we surely can’t solve  this, right? Well, great news - we can! Here is   Google’s diffusion-based solution to super  resolution, where the principle is simple:   have a look at this technique from last year. In  goes a course image or video, and this AI-based   method is tasked with…this! Yes. This is not  science fiction. This is super resolution,   where the AI starts out from noise and  synthesizes crisp details onto the image. So, this might not be such an insane idea after  all! So, does the fact we can do all three of   these separately mean that this task is easy?   Well, let’s see how previous techniques were able   to tackle this challenge. My guess is that  this is still sinfully difficult to do. And…oh   boy. Well, I see a lot of glitches and not a lot  of new, meaningful content being synthesized here.    And note these are not some ancient techniques,  these are all from just two years ago.    It really seems there is not a lot of hope here.   And now, hold on to your papers, and let’s see   how Google’s new AI puts all of these  together, and lets us fly into this photo.    Wow, this is so much better! I love it. Clearly, not perfect, but I feel that this   is the first work where the flying into  photos concept really comes into life. And it has a bunch of really cool features too,  for instance, one, it can generate even longer   videos, which means that after a few seconds,  everything that we see is synthesized by the   AI. Two, it supports not only this boring linear  camera motion, but these really cool, curvy camera   trajectories too. Putting these two features  together, we can get these cool animations   that were not possible before this paper. Now,  the flaws are clearly visible for everyone,   but this is a historic episode where we can  invoke the three Laws of Papers to address them. The First Law Of Papers says that research  is a process. Do not look at where we are,   will be two more  papers down the line. With this concept,   we are roughly where DALL-E 1 was about a year  ago. That is an image generator AI that could

produce images of this quality. And, just one year  later, DALL-E 2 arrived, which could do this! So,   just imagine what kind of videos this will be  able to create just one more paper down the line. The Second Law of Papers says  that everything is connected.    This AI technique is able to learn  image inpainting, image outpainting,   and super resolution techniques at the same time,  and even combine them creatively. We don’t need   3 separate AIs to do these, just one  technique. That is very impressive. And finally, the Third Law Of Papers says  that a bad researcher fails 100% of the time,   while a good one only fails 99% of the time.   Hence, what you see here is always just 1%   of the work that was done. Why is that? Well, for  instance, this is a neural network-based solution,   which means that we need a ton of training data  for these AIs to learn on. And hence, scientists   at Google also needed to create a technique to  gather a ton of drone videos on the internet   and create a clean dataset also with labelings as  well. The labels are essentially depth information   which shows how far different parts of the image  are from the camera. And they did it for more   than 2 million images in total. So, once again,  if you include all the versions of this idea   that didn’t work, what you see here  is just 1% of the work that was done. And now, we can not only fly through photos, but  also fly into photos. What a time to be alive! What do you think? Does this get your  mind going? What would you use this for?    Let me know in the comments below! Thanks for watching and for your generous  support, and I'll see you next time!