# New AI Research Work Fixes Your Choppy Videos! 🎬 ## Метаданные - **Канал:** Two Minute Papers - **YouTube:** https://www.youtube.com/watch?v=G00A1Fyr5ZQ - **Дата:** 07.08.2021 - **Длительность:** 7:25 - **Просмотры:** 204,481 ## Описание ❤️ Check out Lambda here and sign up for their GPU Cloud: https://lambdalabs.com/papers 📝 The paper "Time Lens: Event-based Video Frame Interpolation" is available here: http://rpg.ifi.uzh.ch/TimeLens.html ❤️ Watch these videos in early access on our Patreon page or join us here on YouTube: - https://www.patreon.com/TwoMinutePapers - https://www.youtube.com/channel/UCbfYPyITQ-7l4upoX8nvctg/join 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible: Aleksandr Mashrabov, Alex Haro, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bryan Learn, Christian Ahlin, Eric Haddad, Eric Martel, Gordon Child, Ivo Galic, Jace O'Brien, Javier Bustamante, John Le, Jonas, Kenneth Davis, Klaus Busse, Lorin Atzberger, Lukas Biewald, Matthew Allen Fisher, Mark Oates, Michael Albrecht, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Ramsey Elbasheer, Steef, Taras Bobrovytsky, Thomas Krcmar, Timothy Sum Hon Mun, Torsten Reil, Tybie Fitzhugh, Ueli Gallizzi. If you wish to appear here or pick up other perks, click here: https://www.patreon.com/TwoMinutePapers Or join us here: https://www.youtube.com/user/keeroyz/join Károly Zsolnai-Fehér's links: Instagram: https://www.instagram.com/twominutepapers/ Twitter: https://twitter.com/twominutepapers Web: https://cg.tuwien.ac.at/~zsolnai/ ## Содержание ### [0:00](https://www.youtube.com/watch?v=G00A1Fyr5ZQ) Segment 1 (00:00 - 05:00) Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Today we are going to take a bad choppy video, and make a beautiful, smooth and creamy footage out of it. With today's camera and graphics technology, we can create videos with 60 frames per second. Those are really smooth, I also make each of these videos using 60 frames per second, however, it almost always happens that I encounter the paper videos that have only from 24 to 30 frames per second. In this case, I put them in my video editor that has a 60 fps timeline, where half or even more of these frames will not provide any new information. That’s neither smooth nor creamy. And it gets worse. Look! As we try to slow down the videos for some nice slow-motion action, this ratio becomes even worse, creating an extremely choppy output video because we have huge gaps between these frames. So, does this mean that there is nothing we can do and have to put up with this choppy footage? No, not at all! Look at this technique from 2019 that we covered in an earlier video. The results truly speak for themselves. In goes a choppy video, and out comes a smooth, and creamy result. So good! But wait, it is not 2019, it is 2021, and we always say that two more papers down the line, and it will be improved significantly. From this example, it seems that we are done here, we don’t need any new papers. Is that so? Well, let’s see what we have only one more paper down the line! Now, look. It promises that it can deal with 10 to 1, or even 20 to 1 ratios, which means that for every single image in the video, it creates 10 or 20 new ones, and supposedly we shouldn’t notice that. Well, those are big words, so I will believe it when I see it. Let’s have a look together! Holy mother of papers! This can really pull this off, and it seems nearly perfect. Wow. It also knocked it out of the park with this one. And all this improvement in just one more paper down the line. The pace of progress in machine learning research is absolutely amazing. But, we are experienced Fellow Scholars over here, so we will immediately ask, is this really better than the previous 2019 paper? Let’s compare them! Can we have side by side comparisons? Of course we can! You know how much I love fluid simulations. Well, these are not simulations, but a real piece of fluid, and in this one, there is no contest. The new one understands the flow so much better, while the previous method sometimes even seems to propagate the waves backwards in time. A big checkmark for the new one. In this case, the previous method assumes linear motion when it shouldn’t, thereby introducing a ton of artifacts. The new one isn’t perfect either, but it performs significantly better. Do not worry for a second, we will talk about linear motion some more in a moment. So how does all this wizardry happen? One of the key contributions of the paper is that it can find out when to use the easy way and the hard way. What are those? The easy way is using already existing information in the video and computing inbetween states for a movement. That is all well and good if we have simple, linear motion in our video. But, look, the easy way fails here. Why is that? It fails because we have a difficult situation where reflections off of this object rapidly change, and it reflects something. We have to know what that something is. So, look, this is not even close to the true image, which means that here, we can’t just reuse the information in the video, this requires introducing new information. Yes, that is the hard way! And this excels when new information has to be synthesized. Let’s see how well it does! My goodness, look at that, it matches the true reference image almost perfectly. And also, look, the face of the human did not require synthesizing a great deal of new information, it did not change over time, so we can easily refer to the previous frame for it, hence, the easy way did better here. Did you notice? ### [5:00](https://www.youtube.com/watch?v=G00A1Fyr5ZQ&t=300s) Segment 2 (05:00 - 07:00) That is fantastic, because the two are complementary. Both techniques work well, but they work well elsewhere. They need each other! So, yes, you guessed right, to tie it all together, there is also an attention-based averaging step that helps us decide when to use the easy, and the hard ways. Now, this is a good paper, so it tells us how these individual techniques contribute to the final image. Using only the easy way can give us about 26 decibels, that would not beat the previous methods in this area. However, look! By adding the hard way, we get a premium quality result that is already super competitive, and, if we add the step that helps us decide when to use the easy and hard ways, we get an extra decibel. I will happily take it, thank you very much! And, if we put it all together, oh yes, we get a technique that really outpaces the competition. Excellent. So, in the near future, perhaps we will be able to record a choppy video of a family festivity, and have a chance at making this choppy video enjoyable, or maybe even create slow-motion videos with a regular camera. No slow-motion camera is required. What a time to be alive! Thanks for watching and for your generous support, and I'll see you next time! --- *Источник: https://ekstraktznaniy.ru/video/13852*