# Perfect Virtual Hands - But At A Cost! 👐 ## Метаданные - **Канал:** Two Minute Papers - **YouTube:** https://www.youtube.com/watch?v=MbZ0ld1ShFo - **Дата:** 09.02.2021 - **Длительность:** 6:34 - **Просмотры:** 774,034 ## Описание ❤️ Check out Perceptilabs and sign up for a free demo here: https://www.perceptilabs.com/papers 📝 The paper "Constraining Dense Hand Surface Tracking with Elasticity" is available here: https://research.fb.com/publications/constraining-dense-hand-surface-tracking-with-elasticity/ 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible: Aleksandr Mashrabov, Alex Haro, Alex Serban, Alex Paden, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bruno Mikuš, Bryan Learn, Christian Ahlin, Eric Haddad, Eric Lau, Eric Martel, Gordon Child, Haris Husic, Jace O'Brien, Javier Bustamante, Joshua Goller, Kenneth Davis, Lorin Atzberger, Lukas Biewald, Matthew Allen Fisher, Michael Albrecht, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Ramsey Elbasheer, Robin Graham, Steef, Taras Bobrovytsky, Thomas Krcmar, Torsten Reil, Tybie Fitzhugh. If you wish to support the series, click here: https://www.patreon.com/TwoMinutePapers Károly Zsolnai-Fehér's links: Instagram: https://www.instagram.com/twominutepapers/ Twitter: https://twitter.com/twominutepapers Web: https://cg.tuwien.ac.at/~zsolnai/ #vr #metaverse ## Содержание ### [0:00](https://www.youtube.com/watch?v=MbZ0ld1ShFo) Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. The promise of virtual reality, VR is indeed truly incredible. If one day it comes to fruition, doctors could be trained to perform surgery in a virtual environment, we could train better pilots with better flight simulators, expose astronauts to virtual zero-gravity simulations, you name it. This previous work uses a learning-based algorithm to teach a head-mounted camera to tell the orientation of our hands at all times. Okay, so what can we do with this? A great deal. For instance, we can type on a virtual keyboard, or implement all kinds of virtual user interfaces that we can interact with. We can also organize imaginary boxes, and of course, we can’t leave out the Two Minute Papers favorite, going into a physics simulation and playing with it with our own hands. But of course, not everything is perfect here, however. Look. Hand-hand interactions don’t work so well, so folks who prefer virtual reality applications that include washing our hands should look elsewhere. And in this series, we often say “one more paper down the line, and it will be significantly better”. So now, here is the moment of truth, let’s see that one more paper down the line. Let’s go in guns blazing and give it examples with challenging hand-hand interactions, deformations, lots of self-contact and self-occlusion. Take a look at this footage. This seems like a nightmare for any hand reconstruction algorithm. Who the heck can solve this? And, look, interestingly, they also recorded the hand model with gloves on. How curious! And now, hold on to your papers because these are not gloves…no, no, no. What you see here is the reconstruction of the hand model by a new algorithm. Look. It can deal with all of these rapid hand motions. And what’s more, it also works on this challenging hand massage scene. Look at all those beautiful details! It not only fits like a glove here too, but I see creases, folds, and deformations too - this reconstruction is truly out of this world. To be able to do this, the algorithm has to output triangle meshes that typically contain over a hundred thousand faces. Please remember this as we will talk about it later. And now, let’s see how it does all this magic, because there is plenty of magic under the hood. Let’s look at the five ingredients that are paramount to getting an output of this quality. Ingredient number one is the physics term. ### [2:50](https://www.youtube.com/watch?v=MbZ0ld1ShFo&t=170s) Physics term Without it, we can’t even dream of tracking self-occlusion and contact properly. Two, since there are plenty of deformations going on in the input footage, the deformation term accounts for that. ### [3:01](https://www.youtube.com/watch?v=MbZ0ld1ShFo&t=181s) Deformation term It makes a huge difference in the reconstruction of the thumb here. And if you think “wow, that is horrific”, then you’ll need to hold on to your papers for the next one, which is…three…the geometric consistency term. This one is not for the faint of the heart. You have been warned. Are you ready? Let’s go. Yikes! A piece of advice, if you decide to implement this technique, make sure to include this geometric consistency term so no one has to see this footage ever again. Thank you. With the worst already behind us, let’s proceed to ingredient number four, the photo-consistency ### [3:44](https://www.youtube.com/watch?v=MbZ0ld1ShFo&t=224s) Photo consistency term term. This ensures that fingernail tips don’t end up sliding into the finger. And five, the collision term fixes problems like this to make sure that the fingers don’t ### [3:55](https://www.youtube.com/watch?v=MbZ0ld1ShFo&t=235s) Collision term penetrate each other. And this is an excellent paper, so in the evaluation section, these terms are also tested in isolation, and the authors tell us exactly how much each of these ingredients contribute to the solution. Now, these five ingredients are not cheap in terms of computation time, and remember, we also mentioned that many of these meshes have several hundred thousand faces. This means that this technique takes a very long time to compute all this. It is not real time, not even close, for instance, reconstructing the mesh for the hand massage scene takes more than 10 minutes per frame. This means hours, or even days of computation to accomplish this. Now the question naturally arises - is that a problem? No, not in the slightest. This is a zero to one paper, which means that it takes a problem that was previously impossible, and now, it makes it possible. That is absolutely amazing. And as always, research is a process, and this is an important stepping stone in this process. I bet that two more good papers down the line, and we will be getting these gloves interactively. I am so happy about this solution, as it could finally give us new ways interact with each other in virtual spaces, add more realism to digital characters, help us better understand human-human interactions, and it may also enable new applications in physical rehabilitation. And these reconstructions indeed fit like a glove. What a time to be alive! Thanks for watching and for your generous support, and I'll see you next time! --- *Источник: https://ekstraktznaniy.ru/video/13980*