Can You Put All This In a Photo? 🤳

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Virtual Reality, VR in short, is maturing at a rapid pace and its promise is truly incredible. If one day it comes to fruition, doctors could be trained to perform surgery in a virtual environment, we could train pilots with better flight simulators, teach astronauts to deal with zero-gravity simulations, you name it. And you will see that with today’s paper, we are able to visit nearly any place from afar. Now, to be able to do anything in a virtual world, we have to put on a head-mounted display that can tell the orientation of our head, and often, hands at all times. So, what can we do with this? Oh boy, a great deal. For instance, we can type on a virtual keyboard, or implement all kinds of virtual user interfaces that we can interact with. We can also organize imaginary boxes, and of course, we can’t leave out the Two Minute Papers favorite, going into a physics simulation and playing with it with our own hands. In this previous work, hand-hand interactions did not work too well, which was addressed

one more paper down the line, which absolutely nailed the solution. This followup work would look at our hands in challenging hand-hand interactions, and could deal with deformations, lots of self-contact and self-occlusion. Take a look at this footage. And, look, interestingly, they also recorded the real hand model with gloves on. We might think, what a curious design decision! What could that be for? Well, what you see here is not a pair of gloves, the reconstruction of the hand model by this followup paper. Absolute witchcraft. Now, as you see, we can manipulate objects, or even wash our hands in virtual reality. This is all great when we play in a computer game, because the entire world around us was previously modeled, so we can look and go anywhere, anytime. But what about operating in the real world? What if we wish to look at a historic landmark from afar? Well, in this case, someone needs to capture a 360-degree photo of it. A regular photo will not cut it, because we can’t turn our head around and look behind things. And this, is what today’s paper will be about. This new paper is called Omniphotos, and it helps us produce this 360 view synthesis, and when we put on that head-mounted display, we can really get a good feel of a remote place, a group photo, or an important family festivity. So, clearly, the value proposition is excellent, but we have two questions. One, what do we have to do for it? Flailing. Yes. We need to be flailing. You see, we need to attach a consumer 360-camera to a selfie stick, and start flailing for about 10 seconds. Like this. This is a crazy idea, because now, we created a ton of raw data, roughly what you see here. So this, is a deluge of information, and the algorithm needs to crystallize all this mess into a proper 360 photograph. What is even more difficult here is that this flailing will almost never create a perfect circle trajectory, so the algorithm first has to estimate the exact camera positions and view directions. And hold on to your papers, because the entirety of this work is handcrafted, no machine learning is in sight, and the result is quite general technique, or in other words, it works on a wide variety of real-world scenes, you see a good selection of those here. Excellent. Our second question is, this is, of course, not the first method published in this area, so how does it relate to previous techniques? Is it really better? Well, let’s see for ourselves! Previous methods either suffered from not allowing too much motion, or, the ones that give us more freedom to move around, did it by introducing quite a bit of warping into the outputs. And now, let’s see if the new method improves upon that. Oh yeah, a great deal! Look, we have the advantages of both methods, we can move around freely, and additionally there is much less warping than here. Now, of course, not even this new technique is perfect, if you look behind the building

you see that the warping hasn’t been completely eliminated, but it is a big step ahead of the previous paper. While we look at some more side by side comparisons. One more bonus question: what about memory consumption? Well, it eats over a gigabyte of memory. That is typically not too much of a problem for desktop computers, but we might need a little optimization if we wish to do these computations on a mobile device.

And now comes the best part. You can browse these Omniphotos online through the link in the video description, and even the source code, and a Windows-based demo is available that works with and without a VR headset. Try it out and let me know in the comments how it went! So, with that, we can create these beautiful Omniphotos cheaply and efficiently, and navigate the real world as if it were a computer game. What a time to be alive! Thanks for watching and for your generous support, and I'll see you next time!