Google’s New AI: Like OpenAI’s DALL-E 2, But For Video!

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today we are going to use this  new AI research work to create   absolutely magical videos almost out of  thin air. Like this one. And this one. Yes,   with this new work, we can all become  movie directors. Sort of. Loving it. Now, everyone knows that modern AI techniques are  already capable of text to image, that is taking a   piece of text and creating a picture out of it. What fewer people know, but many of you Fellow   Scholars definitely know is that  text to video is now also possible. However, I bet that you didn’t know about  today’s paper! So, what is this about? Well,   whenever I use these systems, I always  feel myself yearning for more artistic   control. I want to specify the pose  for these characters. The framing too,   what should be visible, where the characters  and objects are. And more. That would be lovely. So, in this new paper scientists had  an excellent question - why not give   more information to the AI for these videos?   And that is an excellent question. You see,   this new technique allows us to show  the AI exactly what we are looking   for by performing it ourselves  and giving it that as an input. It goes the following way: imagine that we would  like to have a video of an astronaut surfing,   but obviously, that’s not us. So, first, we record  ourselves surfing, give this video to the AI,   add a text prompt that now an astronaut  should be surfing, and, I am very excited,   let’s see. Oh yes, that is so cool! And, if  we wish to reimagine what this scene would   look like if we were surfing in the desert, not a  problem. Or, we can also think bigger, and ask for   a surfing sloth instead. Look at that. This sloth  is having the time of his life. And I particularly   like this example because this almost feels like  a motion capture video where we just provide the   movement for some other character. And all this  with just one video and one text prompt. Wow. Now wait, this new work also gives us a  ton more tools to play with. For instance,   if we have a video of a bear playing a guitar, a  common occurrence, we can also ask it to make it a   little more artistic. And if we have a cat, we can  make it two, add cowboy hats, or even remove that   hat and have it synthesize a similar video. I also  loved this example. A simple video of this good   boy running in the garden, and we can use it as  the blueprint for our text to image ideas. Look,   it has been changed to a corgi and it is  now in a different environment, or a lion,   or whatever we can imagine. So I hear you asking  - what is so cool about this example? Well,   imagine if we are very picky with the kind of  footage we want to synthesize, we would have to   write in text what the pace, direction and the  exact angle should the walking take place in.    But this paper says, none of that is required,  just show the AI what you wish to see. So good. Now, this is not the first paper to attempt  this, so let’s see what previous methods   were made of. Oh boy. Look. This is  the Video Diffusion Models paper,   VDM in short, and the quality of  the outputs is not the greatest,   but there is an even bigger problem. What is the  problem? Well, temporal coherence. Each adjacent   frame seems so different, for instance,  the color of the clothing is different,   each time. If we weave these together into a  video, we will get a ton of flickering. Not good. So, let’s see the new method. Whoa. This  is so much better. Let’s do another one of   these fun comparisons. Let’s  see…. yes this is CogVideo,   another earlier technique. The prompt is “a cat  running on a single-plank bridge, comic style”.    Wow, I don’t know about this one. Look. I  am not sure if this is a cat, not sure if   this is a bridge, and I am not sure if this is  in a comic style. The rest is fine though. So,   let’s see the new one. Wow. This is  incredible improvement in just…wait,   actually, in how long exactly? When were these  two previous works published? Is this new work

10 years of progress? No-no, not at all. And this  is the part where I fell of the chair when reading   this paper. Hold on to your papers, because  both of these previous works were published   less then a year before this new one. My goodness.   That is amazing progress in just a few months. Now, we are experienced Fellow Scholars over  here, so we are also looking for a little more   details regarding the old vs new technique.   So let’s dig a little deeper…and, this is   a human evaluation on a set of 32 videos,  and are you seeing what I am seeing? This   is one of the biggest improvements in just  a few months that I have ever seen. Almost   everyone favored the new one. Bravo! And just  imagine what we will be able to do just a couple   more papers down the line. Who knows, we all might  become movie directors. What a time to be alive! So, what do you think? What would you use  this for? Let me know in the comments below! Thanks for watching and for your generous  support, and I'll see you next time!