# This New AI Is The Future of Videomaking!

## Метаданные

- **Канал:** Two Minute Papers
- **YouTube:** https://www.youtube.com/watch?v=muNkPjigQEE
- **Дата:** 14.05.2023
- **Длительность:** 6:07
- **Просмотры:** 85,959
- **Источник:** https://ekstraktznaniy.ru/video/13173

## Описание

❤️ Check out Lambda here and sign up for their GPU Cloud: https://lambdalabs.com/papers

📝 The paper "Structure and Content-Guided Video Synthesis with Diffusion Models" is available here:
https://arxiv.org/abs/2302.03011

Try Runway:
https://runwayml.com/

Full video made with runway: https://twitter.com/IXITimmyIXI/status/1649242592876412928

My latest paper on simulations that look almost like reality is available for free here:
https://rdcu.be/cWPfD 

Or this is the orig. Nature Physics link with clickable citations:
https://www.nature.com/articles/s41567-022-01788-5

🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Balfanz, Alex Haro, Andrew Melnychuk, Benji Rabhan, Bryan Learn, B Shang, Christian Ahlin, Eric Martel, Geronimo Moralez, Gordon Child, Jace O'Brien, Jack Lukic, John Le, Jonathan, Kenneth Davis, Klaus Busse, Kyle Davis, Lorin Atzberger, Lukas Biewald, Martin, Matthew Valle, Michael Albrecht, Michael

## Транскрипт

### Intro []

Dear Fellow Scholars, this is Two Minute Papers  with Dr. Károly Zsolnai-Fehér. Dr. Papers. Earlier we talked about two text to video systems,  Google’s amazing Imagen Video, and also NVIDIA’s   videomaker AI. However, to the best of my  knowledge, as of today, there are no easy ways   of trying them out. Now, before we start, just to  make sure: tons of video AI-related techniques are   available right now, many of them for free, I’ll  show you those, but their text to video system is   still in testing phase as of the making of this  video. I will show you some results for that too,   however, good news, it is also expected  to open to the public within a few weeks. So this is Runway. They had a hand in the creation  of the amazing free text to image AI tool,   Stable Diffusion, and they have an incredible  creative suite for all kinds of AI-based video   magic. I will tell you about text to video  in a moment. First, they introduced Gen-1,   this is their video to video system.   Here you can take a video of yourself,   then choose a style, and bam! You have a  completely new, stylized video. I think

### Features [1:11]

just this one feature makes it worth  trying this system as it has of uses,   but it has a ton of other really creative modes  as well. We can mask out a dog, and only change   that part of the video, which will get tracked as  this good boy moves around in the scene. Lovely. It can also remove the background behind us  in a video, and track both the foreground and   background over time. Also, if we have an image  that we would like to expand, it can also perform   text to image with the appropriate constraints  and it will expand it in a way that we often   can’t even tell that something happened  here. We can also erase parts of an image,   and replace them with a text prompt as well.   Now, wait a second, these two are image-related   operations. But we are talking about videos  here. So what is going on? Well, one,   these photos can be used for Gen-1 as a style  to a previously existing video, or even better,   get this. You can also create a video by adding  a few photos that were taken a few seconds apart,   and like magic, it creates a video out of them.   Yes, that means understanding what these photos   depict, and synthesizing every pixel of every  image between these photos. Absolutely incredible. And their suite is full of features we  haven’t even discussed yet. Colorization   with or without using a text prompt,  blurring faces while tracking them,   and so much more. Now, this is what you’ve  been waiting for! Onwards to text to video.    Note once again that this is still in testing  phase, and will hopefully be part of the main   suite within a few weeks. I was lucky enough  to be able to try it and it did not disappoint.    Before you ask, the answer is yes, of course  I generated a video of Fellow Scholars holding   on to their papers. Wow! I was told that it  is not yet that good in generating humans,   and if this is the not good, I can only imagine  how good the launch version will look like! This   is excellent. My first text to video experience  ever. Loving it. What a time to be alive!

### Fluid simulations [3:37]

And here is my other favorite, fluid simulations,  well, in this case, video footage of splashing   liquids which also worked really well.   Some of the results were not as great,   so a little cherry picking was necessary here,  but in the end I got some really tasty results.    I was also told that the system is improved every  week, so by the time you are watching this video,   it will likely have improved a great deal already.   The current system supports videos up to 1500x900   in resolution. Not bad! And that is easily  good enough so that some folks are already   creating incredibly good movies with it that  are several minutes long. Imagine that soon,   every creative person will have access to  such an amazing system and can let their   imaginations run wild. All of us!   This is the future of video making.

### Conclusion [4:33]

So if you wish to try this suite, note  once again - text to video is coming later,   you can sign up for free on Runway’s website.   The link is in the video description. I will note that Runway is a previous  sponsor of Two Minute Papers,   and had nothing to do with this video apart from  letting my try it and answering my questions.    This is their first time seeing it too. So,  what do you think? What will you make with   this? Let me know in the comments below,  and for now, let the experiments begin! Thanks for watching and for your generous  support, and I'll see you next time!