# Ken Burns Effect, Now In 3D!

## Метаданные

- **Канал:** Two Minute Papers
- **YouTube:** https://www.youtube.com/watch?v=FZZ9rpmVCqE
- **Дата:** 08.11.2019
- **Длительность:** 5:08
- **Просмотры:** 206,390
- **Источник:** https://ekstraktznaniy.ru/video/14224

## Описание

❤️ Pick up cool perks on our Patreon page: https://www.patreon.com/TwoMinutePapers

📝 The paper "3D Ken Burns Effect from a Single Image" is available here:
https://arxiv.org/abs/1909.05483

The paper with the Microplanet scene at the start is available here:
https://users.cg.tuwien.ac.at/zsolnai/gfx/gaussian-material-synthesis/
Scene geometry: Marekv

Image credits: Ian D. Keating, Kirk Lougheed (Link: https://www.flickr.com/photos/kirklougheed/36766944501 ), Leif Skandsen, Oliver Wang, Ben Abel, Aurel Manea, Jocelyn Erskine-Kellie, Jaisri Lingappa, and Intiaz Rahim.

🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Alex Haro, Anastasia Marchenkova, Andrew Melnychuk, Angelos Evripiotis, Anthony Vdovitchenko, Brian Gilman, Bryan Learn, Christian Ahlin, Claudio Fernandes, Daniel Hasegan, Dennis Abts, Eric Haddad, Eric Martel, Evan Breznyik, Geronimo Moralez, James Watt, Javier Bustamante, John De Witt, Kaiesh Vohra, Kasia Hayden, Kjartan Olas

## Транскрипт

### Segment 1 (00:00 - 05:00) []

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. Have you heard of the Ken Burns effect? If you have been watching this channel, you have probably seen examples where a still image is shown, and a zooming and panning effect is added to it. It looks something like this. Familiar, right? The fact that there is some motion is indeed pleasing for the eye, but something is missing. Since we are doing this with 2D images, all the depth information is lost, so we are missing out on the motion parallax effect that we would see in real life when moving around a camera. So, this is only 2D. Can this be done in 3D? Well, to find out, have a look at this. Wow, I love it. Much better, right? Well, if we would try to perform something like this without this paper, we’ll be met with bad news. And that bad news is that we have to buy an RGBD camera. This kind of camera endows the 2D image with depth information, which is specialized hardware that is likely not available in our phones as of the making of this video. Now, since depth estimation from these simple, monocular 2D images without depth data is a research field of its own, the first step sounds simple enough: take one of those neural networks then, ask it to try to guess the depth of each pixel. Does this work? Well, let’s have a look! As we move our imaginary camera around, uh-oh. This is not looking good. Do you see what the problems are here? Problem number one is the presence geometric distortions, you see it if you look here. Problem number two is referred to as semantic distortion in the paper, or in other words, we now have missing data. Not only that, but this poor tiny human’s hand is also…ouch. Let’s look at something else instead. If we start zooming in into images, which is a hallmark of the Ken Burns effect, it gets even worse. Artifacts. So how does this new paper address these issues? After creating the first, coarse depth map, an additional step is taken to alleviate the semantic distortion issue, and then, this depth information is upsampled to make sure that we have enough fine details to perform the 3D Ken Burns effect. Let’s do that! Unfortunately, we are still nowhere near done yet. Previously occluded parts of the background suddenly become visible, and, we have no information about those. So, how can we address that? Do you remember image inpainting? I hope so, but if not, no matter, I’ll quickly explain what it is. Both learning-based, and traditional handcrafted algorithms exist to try to fill in this missing information in images with sensible data by looking at its surroundings. This is also not as trivial as it might seem first, for instance, just filling in sensible data is not enough, because this time around, we are synthesizing videos, it has to be temporally coherent, which means that there mustn’t be too much of a change from one frame to another, or else we’ll get a flickering effect. As a result, we finally have these results that are not only absolutely beautiful, but the user study in the paper shows that they stack up against handcrafted results made by real artists. How cool is that! It also opens up really cool workflows that would normally be very difficult, if not impossible to perform. For instance, here you see that we can freeze this lightning bolt in time, zoom around and marvel at the entire landscape. Love it. Of course, limitations still apply. If we have really thin objects, such as this flagpole, it might be missing entirely from the depth map, or there are also cases where the image inpainter cannot fill in useful information. I cannot wait to see how this work evolves a couple papers down the line. One more interesting tidbit. If you have a look at the paper, make sure to open it in Adobe Reader you will likely be very surprised to see that many of these things that you think are still images…are actually animations. Papers are not only getting more mind-blowing by the day, but also more informative, and beautiful as well. What a time to be alive! This video has been supported by you on Patreon. If you wish to support the series and also pick up cool perks in return, like early access to these episodes, or getting your name immortalized in the video description, make sure to visit

### Segment 2 (05:00 - 05:00) [5:00]

us through Patreon. com/TwoMinutePapers. Thanks for watching and for your generous support, and I'll see you next time!