# TecoGAN: Super Resolution Extraordinaire! ## Метаданные - **Канал:** Two Minute Papers - **YouTube:** https://www.youtube.com/watch?v=MwCgvYtOLS0 - **Дата:** 25.08.2020 - **Длительность:** 5:30 - **Просмотры:** 287,910 ## Описание ❤️ Check out Weights & Biases and sign up for a free demo here: https://www.wandb.com/papers ❤️ Their instrumentation of a previous paper is available here: https://app.wandb.ai/authors/alae/reports/Adversarial-Latent-Autoencoders--VmlldzoxNDA2MDY 📝 The paper "Learning Temporal Coherence via Self-Supervision for GAN-based Video Generation" is available here: https://ge.in.tum.de/publications/2019-tecogan-chu/ The legendary Wavelet Turbulence paper is available here: https://www.cs.cornell.edu/~tedkim/WTURB/ 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible: Aleksandr Mashrabov, Alex Haro, Alex Paden, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bruno Mikuš, Bryan Learn, Christian Ahlin, Daniel Hasegan, Eric Haddad, Eric Martel, Gordon Child, Javier Bustamante, Lorin Atzberger, Lukas Biewald, Michael Albrecht, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Ramsey Elbasheer, Robin Graham, Steef, Sunil Kim, Taras Bobrovytsky, Thomas Krcmar, Torsten Reil, Tybie Fitzhugh. If you wish to support the series, click here: https://www.patreon.com/TwoMinutePapers Meet and discuss your ideas with other Fellow Scholars on the Two Minute Papers Discord: https://discordapp.com/invite/hbcTJu2 Károly Zsolnai-Fehér's links: Instagram: https://www.instagram.com/twominutepapers/ Twitter: https://twitter.com/twominutepapers Web: https://cg.tuwien.ac.at/~zsolnai/ #enhance #superresolution ## Содержание ### [0:00](https://www.youtube.com/watch?v=MwCgvYtOLS0) Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Let’s talk about video super resolution. The problem statement is simple, in goes a coarse video, the technique analyzes it, guesses ### [0:08](https://www.youtube.com/watch?v=MwCgvYtOLS0&t=8s) We put special emphasis on temporally coherent detail generation. what’s missing, and out comes a detailed video. However, of course, reliably solving this problem is anything but simple. When learning-based algorithms were not nearly as good as they are today, this problem was mainly handled by handcrafted techniques, but they had their limits - after all, if we don’t see something too well, how could we tell what’s there? And this is where new learning-based methods, especially this one, come into play. This is a hard enough problem for even a still image, yet this technique is able to do it really well even for videos. Let’s have a look. The eye color for this character is blurry, but we see that it likely has a green-ish ### [0:50](https://www.youtube.com/watch?v=MwCgvYtOLS0&t=50s) Our method increases an input video by 4x yielding highly detailed outputs. blue-ish color. And if we gave this problem to a human, this human would know that we are talking about the eye of another human, and we know roughly what this should look like in reality. A human would also know that this must a bridge, and finish the picture. What about computers? The key is that if we have a learning algorithm that looks at the coarse and fine version of the same video, it will hopefully learn what it takes to create a detailed video when ### [1:18](https://www.youtube.com/watch?v=MwCgvYtOLS0&t=78s) Even under-resolved structures in the input lead to realistic and sharp outputs. given a poor one, which is exactly what happened here. As you see, we can give it very little information, and it was able to add a stunning amount of detail to it. Now of course, super resolution is a highly studied field these days, therefore it is a requirement for a good paper to compare to quite a few previous works. Let’s see how it stacks up against those! Here, we are given a blocky image of this garment, and this is the reference image that was coarsened to create this input. The reference was carefully hidden from the algorithms, and only we have it. Previous works could add some details, but the results were nowhere near as good as the reference. So what about the new method? My goodness, it is very close to the real deal! Previous methods also had trouble resolving the details of this region, where the new method, again, very close to reality. It is truly amazing how much this technique understands the world around us from just this training set of low and high-resolution videos. Now, if you have a closer look at the author list, you see that Nils Thuerey is also there. He is a fluid and smoke person, so I thought there had to be an angle here for smoke simulations. And, yup, there we go. To even have a fighting chance of understanding the importance of this sequence, let’s go back to one of Nils’s earlier works, which is one the best papers ever written, Wavelet Turbulence. That’s a paper from twelve years ago. Now, some of the more seasoned Fellow Scholars among you know that I bring this paper up every chance I get, but especially now that it connects to this work we’re looking at. You see, Wavelet Turbulence was an algorithm that could take a coarse smoke simulation after it has been created, and added fine details to it. In fact, so many fine details that creating the equivalently high resolution simulation would have been near impossible at the time. However, it did not work with images, it required knowledge about the inner workings of the simulator. For instance, it would need to know about velocities and pressures at different points in this simulation. Now, this new method can do something very similar, and all it does is just look at the image itself, and improves it, without even looking into the simulation data! Even though the flaws in the output are quite clear, the fact that it can add fine details to a rapidly moving smoke plume is still an incredible feat! If you look at the comparison against CycleGAN, a technique from just 3 years ago, this is just a few more papers down the line, and you see that this has improved significantly. And the new one is also more careful with temporal coherence, or in other words, there is no flickering arising from solving the adjacent frames in the video differently. Very good. And if we look a few more papers further down the line, we may just get a learning-based ### [4:20](https://www.youtube.com/watch?v=MwCgvYtOLS0&t=260s) Additional result, TecoGAN compared to low-res input algorithm that does so well at this task, that we would be able to rewatch any old footage in super high quality. What a time to be alive! Thanks for watching and for your generous support, and I'll see you next time! --- *Источник: https://ekstraktznaniy.ru/video/14081*