This AI Makes Beautiful Videos From Your Images! 🌊

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. In June 2020, OpenAI published an incredible  AI-based technique by the name Image-GPT. The problem here was simple to understand, but  nearly impossible to actually do: so here goes,   we give it an incomplete image, and we ask  the AI to fill in the missing pixels. That is,   of course, an immensely difficult task, because  these images may depict any part of the world   around us. It would have to know a great deal  about our world to be able to continue the images,   so how well did it do? Let’s have a look! This is undoubtedly a cat. But look! See that  white part that is just starting? The interesting   part has been sneakily cut out of the image. What  could that be? A piece of paper? Something else?    Now, let’s leave the dirty work to the machine and  ask it to finish it! Oh yeah, that makes sense. Now, even better, let’s have a look at this  water droplet example too. We humans, know that   since we see the remnants of ripples over there  too, there must be a splash, but the question   is - does the AI know that? Oh yes, yes it  does! Amazing! And the true image for reference. But wait a second. If Image GPT could understand  that this is a splash, and finish the image like   this, then, here is an absolutely insane idea. If  a machine can understand that this is a splash,   could it, maybe, not only finish the  photo, but make a video out of it? Yes,   that is indeed an absolutely insane idea, we  like those around here. So, what do you think,   is this a reasonable question,  or is this still science fiction? Well, let’s have a look at what this new  learning-based method does when looking at   such an image. It would do something  very similar to what we would do,   look at the image, estimate the direction of  the motion, recognize that these ripples should   probably travel outwards, and based on the fact  that we’ve seen many splashes in our lives, if   we had the artistic skill, we could surely fill in  something similar. So, can the machine do it too? And now, hold on to your papers, because  this technique does exactly that.    Whoa! Please meet Eulerian Motion Synthesis.   And it not only works amazingly well,   but look at the output video. It even  loops perfectly. Yum yum, I love it! And it works mostly on fluids and smoke.   I like that! I like that a lot, because   fluids and smoke have difficult, but predictable  motion. That is an excellent combination for us,   especially given that you see plenty  of those simulations on this channel,   so if you are a long-time Fellow Scholar,  you already have a keen eye for them. Here are a few example images, paired  with the synthesized motion fields,   these define the trajectory of  each pixel, or in other words,   regions that the AI thinks should be animated  and how it thinks should be animated. Now, it gets better, I have found three things   that I did not expect to work, but was  pleasantly surprised that they did.   One, reflections, kind of work. Two, fire. Kind of works.   And now, if you have been holding on to your  papers so far, now, squeeze that paper, because   here comes the best one, three, my beard works  too. Yes, you heard it right. Now, first things   first, this is not any kind of beard, this is an  algorithmic beard that was made by an AI, and now,   it is animated as if it were a piece of  fluid using a different AI. Of course,   this is not supposed to be a correct result,  just a happy accident, but in any case, this   sounds like something straight out of a science  fiction movie. I also like how this has a nice   Obi-Wan Kenobi quality to it. Loving it. Thank you  very much to my friend Oliver Wang and the authors   for being so kind and generating these results  only for us. That is a huge honor, thank you. This previous work is from 2019 and creates  high-quality motion, but, has a limited   understanding of the scene itself. And of course,  let’s see how the new method fares in these cases.    Oh yeah, this is a huge leap forward.

And what I like even better here is that new  research techniques often provide different   tradeoffs than previous methods, but are rarely  strictly better than them. In other words,   competing techniques usually do some things better  and some things worse than their predecessors…but   not this. Look, this is so much better across  the board. That is such a rare sight. Amazing. Now, of course, not even this technique is  perfect. For example, this part of the image   should have been animated, but remains stationary.   Also, even though it did well with reflections,   refraction is a tougher nut to crack. Finally,  thin geometry also still remains a challenge. But this was one paper that made the impossible  possible, and just think about what we will   be able to do, two more papers down the  line. My goodness. What a time to be alive! Thanks for watching and for your generous  support, and I'll see you next time!