❤️ Check out the Gradient Dissent podcast by Weights & Biases: http://wandb.me/gd
📝 The paper "GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds" is available here:
https://jeffreyxiang.github.io/GRAM-HD/
❤️ Watch these videos in early access on our Patreon page or join us here on YouTube:
- https://www.patreon.com/TwoMinutePapers
- https://www.youtube.com/channel/UCbfYPyITQ-7l4upoX8nvctg/join
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Balfanz, Alex Haro, Andrew Melnychuk, Benji Rabhan, Bryan Learn, B Shang, Christian Ahlin, Eric Martel, Geronimo Moralez, Gordon Child, Ivo Galic, Jace O'Brien, Jack Lukic, John Le, Jonas, Jonathan, Kenneth Davis, Klaus Busse, Kyle Davis, Lorin Atzberger, Lukas Biewald, Matthew Allen Fisher, Michael Albrecht, Michael Tedder, Nevin Spoljaric, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Rajarshi Nigam, Ramsey Elbasheer, Steef, Taras Bobrovytsky, Ted Johnson, Thomas Krcmar, Timothy Sum Hon Mun, Torsten Reil, Tybie Fitzhugh, Ueli Gallizzi.
If you wish to appear here or pick up other perks, click here: https://www.patreon.com/TwoMinutePapers
Thumbnail background design: Felícia Zsolnai-Fehér - http://felicia.hu
Károly Zsolnai-Fehér's links:
Instagram: https://www.instagram.com/twominutepapers/
Twitter: https://twitter.com/twominutepapers
Web: https://cg.tuwien.ac.at/~zsolnai/
Оглавление (2 сегментов)
Segment 1 (00:00 - 05:00)
Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Today we are going to use an AI to take a collection of photos, and hopefully create a digital human out of them. However, not so fast! For instance, have a look at NVIDIA’s amazing face generator AI. As you see, it can generate beautiful results, and they even show us the illusion of these characters moving around, however, this does not have a strong concept of 3D information. These are 2D images, and 3D consistency is not that great. What is that? Well, look. This person just one frame away is not exactly the same person. For instance, the hair moves around. This means that these images are not art directable, or at least, not easily. As you see here, it has improved a great deal over two years and one more research paper down the line, it’s still not quite there. So, fine details, checkmark, 3D consistency, not quite there. So let’s have a look at solutions that have more 3D consistency. To be able to do that, we can take a collection of photos like these, and magically, create a video where we can fly through these photos. It is really crazy, because this is possible today, for instance, here is NVIDIAs method that can be trained to perform this in a matter of seconds. This is remarkable, especially that the input is only a handful of photos. Some information is given about the scene, but this is really not much. So, can we have a solution with 3D consistency? Yes, indeed, for these, 3D consistency, checkmark. However, the amount of fine details in these outputs is not that great. Do you see the pattern here? Depending on which previous method we use, we either get tons of detail, but no 3D consistency, or we can get our highly coveted 3D consistency, but then, the details are gone. So, is the dream dead? No digital humans for us? Well, don’t despair, and have a look at this new technique that promises one megapixel images that are also consistent. Details and consistency at the same time! When I first saw this paper, I said I will believe it when I see it, so, let’s have a look together. This is StyleNERF, a previous technique, and we see the problems that we now expect - the hair is not consistent, the earring is flickering a great deal, and there are other issues too. So, are the authors of the new paper claiming that they can solve all this? Well, now, hold on to your papers, and have a look at this. This is the new technique. Oh my goodness. The earring, facial features and the hair stay still as we rotate them, and it indeed shows the new angles correctly. I love it! Let’s examine this phenomenon in a little more detail. Look at the hair here with StyleNERF. It looks decent, but it’s not quite there. And I really wonder why that is. Let’s zoom in and have a closer look. Oh yes, upon closer inspection, this hair is a bit of a mess. And, with the new technique, now that is what I call consistency across the video frames. Smooth hair strands everywhere. So good! And what I absolutely loved about this new work is that it outperforms this previous technique, which is from how many years ago? Well, if you have been holding on to your papers, now squeeze that paper, because this work is from not even one year ago, just from 8 months ago. And you already see a meaningful improvement over that. That is insanity. Now, not even this technique is perfect. I don’t know for sure if the hair consistency is perfect here and we are dealing with video compression artifacts, or whether this is still not a 100% there, but this truly is a great step forward. Also, here, you see that the images are still not as detailed as the photos, but this seems to be a roadblock to me that is much easier to solve than 3D consistency. My impression is that with this, the most difficult part of the task is already done, and just one or
Segment 2 (05:00 - 06:00)
two papers down the line, and I am sure we will be seeing even more realistic virtual humans. So, can we enter into a virtual world as a digital human from just a collection of photos? Oh yes! Time to meet our beloved ones from afar, or meet new people, and play some games together. What a time to be alive! So, does this get your mind going? What would you use this for? Let me know in the comments below! Thanks for watching and for your generous support, and I'll see you next time!
