Finally, Video Stabilization That Works! 🤳

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today we are going to talk  about video stabilization.    A typical application of this is when we  record family memories, and other cool events,   and sometimes, the footage gets so shaky  that we barely know what is going on. In these cases, video stabilization  techniques can come to the rescue,   which means that in goes a shaky video,  and out comes a smooth video. Well,

that is easier said than done. Despite  many years of progress, there is a great   selection of previous methods that can do that,  however, they suffer from one of two issues. Issue number one is cropping. This  means that we get usable results,   but we have to pay a great price for it, which is,  cropping away a great deal of the video content.    Issue number two is when we get the  entirety of the video, no cropping,   however, the price to be paid for this is that we  get lots of issues that we call visual artifacts. Unfortunately, today, when we  stabilize, we have to choose our poison.    It’s either cropping, or artifacts. Which one  would you choose? That is difficult to decide,   of course, because none of these two tradeoffs are  great. So our question today is, can we do better?    Well, the Law of Papers says that of course, just  one or two more papers down the line, this will be

way better. So, let’s see…this is  what this new method is capable of.    Hold on to your papers, and notice that this will  indeed be a full-size video, so we already know   that probably there will be artifacts. But…wait  a second! No artifacts. Whoa. How can that be? What does this new method do that previous  techniques didn’t? These magical results are a   combination of several things: one, the new method  can estimate the motion of these objects better,   two, it removes blurred images from the videos  and three, collects data from neighboring video   frames more effectively. This leads to a greater  understanding of the video it is looking at. Now, of course, not even this technique is  perfect - rapid camera motion may lead to warping,   and if you look carefully, you may find some  artifacts, usually around the sides of the screen. So far, we looked at previous methods, and the  new method. It seems better. That’s great. But   how do we measure which one is better? Do we  just look? An even harder question would be,   if the new method is indeed better,  okay, but by how much better is it?

Let’s try to answer all of these questions. We  can evaluate these techniques against each other   in three different ways. One, we can look at the  footage ourselves. We have already done that,   and we had to tightly hold on to our  papers, it has done quite well in this test.    Test number two is a quantitative test. In other  words, we can mathematically define how much   distortion there is in an output video, how smooth  it is, and more, and compare the output videos   based on these metrics. In many cases, these  previous techniques are quite close to each other,   and now, let’s unveil the new method.   Whoa. It scored best or second best   on 6 out of 8 tests. This is truly remarkable,  especially given that some of these competitors   are from less than a year ago. That is nimble  progress in machine learning research. Loving it. And the third way to test which technique is  better, and by how much is by conducting a user   study. The authors have done that too! In this,  46 humans were called in, were shown the shaky   input video, the result of a previous method, and  the new method, and were asked three questions.    Which video preserves the most content, which has  fewer imperfections, and which is more stable. And the results were stunning - despite  looking at many different competing techniques,   the participants found the new method to be  better at the very least 60% of the time,   on all three questions. In some cases, even  90% of the time or higher. Praise the papers! Now, there is only one question left. If it  is so much better than previous techniques,   how much longer does it take to run? With one exception, these previous methods take   from half a second to about 7. 5 seconds per frame,  and this new one asks for 9. 5 seconds per frame.    And in return, it creates these absolutely amazing  results. So, from this glorious day on, fewer,   or maybe no important memories will be lost  due to camera shaking. What a time to be alive! Thanks for watching and for your generous  support, and I'll see you next time!