Google’s New AI Watched 2,500 Videos! But Why?

This is an incredible AI technique where in  goes a photo, and then, out comes a video of   it. But wait, not any kind of video, even  better, a seamlessly looping video. I love   it. What I found really surprising is that  it works quite well on a variety of cases,   not only on simple periodic movements for flowers,  but for even a sleeping cat, and more. Wow. But, hold on to your flowers,  I mean papers Fellow Scholars,   because look. Boing. And a bigger one. And when  I tug here, a smaller one. These boings work   on other scenes too. And all this little world  model from one simple image AI. Incredible. So,   not just animating the image, but we can  even play with it! How is this even possible? This is an AI model that has looked at videos and  motion trajectories that happen in these videos,   and based on that, it tries  to create something that they   call a neural stochastic motion texture.   These motion textures describe how pixels   should move when they are tugged  at. How the boing should happen. But wait a second, if it has this kind of  representation of the motion, maybe it can also   manipulate already existing videos too, right?   Oh yes, look! We can take an existing video,   and even magnify the amount of motion that  is happening there. And I am very impressed,   I just kept looking and looking, and  couldn’t find a single issue with these   videos. It would be very difficult to tell  that they aren’t real. And it gets better,   it can even do a slow motion version of  already existing videos too! So good! Previous techniques were not nearly as good as  this one in creating these animations. Look. This   image is trying to perform the impossible, which  is, showcase motion in one still image. So what   is happening here? Well, we take a small slice of  this photo and try to visualize how this slice is   moving over time. A previous technique did this,  stochastic I2V is from just 2 years ago, and the   new technique. Okay, this looks better, but we  are experienced Fellow Scholars here. We wish   to see how it relates to the real motion. So…oh  my goodness, look at that. No wonder these videos   are so realistic, the synthesized motions are in  some cases, nearly indistinguishable from reality! Now, not even this technique is perfect. Look.   If I pull this away, I now have distorted the   original image a bit too much, and what is behind  here? We do not know. Information is missing. Now,   image inpainting techniques already exist,  so that can be remedied very easily. However,   the next one cannot, or at least, not easily.   Look. And the background also does not work here. Now, I think this will surprise you.   Being a computer graphics researcher,   I have to note that there was an excellent  graphics paper, called Video Textures,   which was a handcrafted technique that could do  something like this…yes you are seeing correctly,   more than 20 years ago. Goodness! Then, the input  was a video and it tried to find connection points   where it could be connected back to the start of  the video and playing it again. Thus creating a   looping video from a non looping one. It was a  little more elaborate than that, but this was   the key idea. And it is a powerful idea. I wrote  my Bachelor thesis and a paper 15 years ago based   on this technique, which was a procedural brush  stroke generation technique that could learn your   brushstrokes and apply it to already existing  works. It was a ton of fun and was made for a   feature-length movie. The paper and video of it  is available in the video description. And now, an   animation from a single photo. Creamy, slow-motion  movement even! Love it. And just imagine what we   will be capable of just one more paper down  the line. My goodness. What a time to be   alive! This was Two Minute Papers with Dr. Károly  Zsolnai-Fehér. Subscribe if you wish to see more.