This AI Learned To Stop Time! ⏱

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Today we get to be a paper historian, and witness the amazing progress in machine learning research together, and learn what is new in the world of NERFs. But, first, what is a NERF? In March of 2020, a paper appeared describing an incredible technique by the name, Neural Radiance Fields, or NERF in short. This work enables us to take a bunch of input photos and their locations, learn it, and synthesize new, previously unseen views of not just the materials in the scene, but the entire scene itself. And here, we are talking not only digital environments, but also, real scenes as well! Now that’s quite a value proposition, especially given that it also supported refractive and reflective surfaces as well, these are both quite a challenge. However, of course, NERF had its limitations. For instance, in many cases, it had trouble with scenes with variable lighting conditions and lots of occluders. And to my delight, only 5 months later, in August of 2020, a followup paper appeared by the name NERF in the Wild, or NERF-W in short. Its speciality was tourist attractions that a lot of people take photos of, and we then have a collection of photos taken during a different time of the day, and of course, with a lot of people around. And, lots of people, of course means, lots of occlusions. NERF-W improved the original algorithm to excel more in cases like this. A few months later, on 2020 November 25th, another followup paper appeared by the name Deformable Neural Radiance Fields. D-NERF. The goal here was to take a selfie video, and turn it into a portrait that we can rotate around freely. This is something that the authors call a nerfie. If we take the original NERF technique to perform this, we see that it does not do well at all with moving things and that's where this new deformable variant really shines. And today’s paper not only has some nice video results embedded in the front page, and it offers a new take on this problem and offers quote “Space-Time View Synthesis of Dynamic Scenes”. Whoa, that is amazing. But what does that mean? What does this paper really do? The space-time view synthesis means that we can record a video of someone doing something. Since we are recording movement in time, and there is also movement in space, or in other words, the camera is moving. Both time and space are moving. And what this can do is one, freeze one of these variables, in other words, pretend as

if the camera didn’t move. Or, two, pretend as if time didn’t move.

Or, three, generate new views of the scene while movement takes place.

My favorite is that we can pretend to zoom in and even better, zoom out even if the recorded video looked like this, or, we can also make a really choppy family memory smoother and much more enjoyable. So how does this compare to previous methods? There are plenty of NERF variants around, is this really any good? Let’s find out together! This is the original NERF, we already know about this, and, we are not surprised in the slightest that it’s not so great on dynamic scenes with a lot of movement. However, what I am surprised by is that all of these previous techniques are from 2020, and all of them struggle with these cases. These comparisons are not against some ancient technology from 1985. No-no, all of them are from the same year. For instance, this previous work is called Consistent Video Depth Estimation, and it is from August 2020. We showcased it in this series, and marveled at all of these amazing augmented reality applications that it offered. The snowing example here was one of my favorites. And today’s paper appeared just three months later, in November 2020. And the authors still took the time and effort to compare against this work from just three months ago. That is fantastic. As you see, this previous method kind of works on this dog, but the lack of information in some regions is quite apparent. This is still maybe usable, but as soon as we transition into a more dynamic example, what do we get? Well, pandemonium. This is true for all previous methods. I cannot imagine that the new method from just a few months later could deal with this difficult case…and. Look at that. So much better! It is still not perfect, you see that the we have lost some detail, but witnessing this kind of progress in just a few months is truly a sight to behold. It really consistently outperforms all of these techniques from the same year. What a time to be alive! If you, like me, find yourself yearning for more quantitative comparisons, the numbers also show that the two variants of the new proposed technique indeed outpace the competition. And it can even do one more thing. Previous video stabilization techniques were good at taking a shaky input video and creating a smoother output, however, these results often came at the cost a great deal of cropping. Not this new work, look at how good it is at stabilization, and it does not have to crop all this data. Praise the papers! Thanks for watching and for your generous support, and I'll see you next time!