This Neural Network Creates 3D Objects From Your Photos

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. In computer graphics research, we spend most of our time dealing with images. An image is a bunch of pixels put onto a 2D plane, which is a tiny window into reality, but reality is inherently 3D. This is easy to understand for us, because if we look at a flat image, we see the geometric structures that it depicts. If we look at this image, we know that this is not a sticker, but a three dimensional fluid domain. If I would freeze an image and ask a human to imagine rotating around this fluid domain, that human would do a pretty good job at that. However, for a computer algorithm, it would be extremely difficult to extract the 3D structure out from this image. So, can we use these shiny new neural network-based learning algorithms to accomplish something like this? Well, have a look at this new technique that takes a 2D image as an input, and tries to guess three things. The cool thing is that the geometry problem we talked about is just the first one. Beyond that, two, it also guesses what the lighting configuration is that leads to an appearance like this, and three, it also produces the texture map for an object as well. This would already be great, but wait, there is more. If we plug all this into a rendering program, we can also specify a camera position, and this position can be different from the one that was used to take this input image. So what does that mean exactly? Well, it means that maybe, it can not only reconstruct the geometry, light and texture of the object, but even put this all together and make a photo of it from a novel viewpoint! Wow. Let’s have a look at an example! There is a lot going on in this image, so let me try to explain how to read it. This image is the input photo, and the white silhouette image is called a mask, which can either be given with the image, or be approximated by already existing methods. This is the reconstructed image by this technique, and then, this is a previous method from 2018 by the name category-specific mesh reconstruction, CMR in short. And, now, hold on to your papers, because in the second row, you see this technique creating images of this bird from different, novel viewpoints! How cool is that! Absolutely amazing. Since we can render this bird from any viewpoint, we can even create a turntable video of it. And all this from just one input photo. Let’s have a look at another example! Here, you see how it puts together the final car rendering in the first column from the individual elements, like geometry, texture, and lighting. The other comparisons in the paper reveal that this technique is indeed a huge step up from previous works. Now, this all sounds great, but what is all this used for? What are some example applications of this 3D object from 2D image thing? Well, techniques like this can be a great deal of help in enhancing the depth perception capabilities of robots, and of course, whenever we would like to build a virtual world, creating a 3D version of something we only have a picture of can get extremely laborious. This could help a great deal with that too. For this application, we could quickly get a starting point with some texture information, and get an artist to fill in the fine details. This might get addressed in a followup paper. And if you are worried about the slight discoloration around the beak area of this bird, do not despair. As we always say, two more papers down the line, and this will likely be improved significantly. What a time to be alive! This episode has been supported by Lambda. If you're a researcher or a startup looking for cheap GPU compute to run these algorithms, check out Lambda GPU Cloud. I've talked about Lambda's GPU workstations in other videos and am happy to tell you that they're offering GPU cloud services as well. The Lambda GPU Cloud can train Imagenet to 93% accuracy for less than $19! Lambda's web-based IDE lets you easily access your instance right in your browser. And finally, hold on to your papers, because the Lambda GPU Cloud costs less than half of AWS and Azure. Make sure to go to lambdalabs. com/papers and sign up for one of their amazing GPU instances today. Our thanks to Lambda for helping us make better videos for you. Thanks for watching and for your generous support, and I'll see you next time!