Semantic Scene Completion From One Depth Image | Two Minute Papers #147

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. This piece of work is an amazing application of deep neural networks, that performs semantic

scene completion from only one depth image. This depth image is the colorful image that you see here, where the colors denote how

far away different objects are from our camera. We can create these images inexpensively with commodity hardware, for instance, Microsoft's Kinect has a depth sensor that is suitable for this task. The scene completion part means that from this highly incomplete depth information, the algorithm reconstructs the geometry for the entirety of the room. Even parts, that are completely missing from our images or things that are occluded! The output is what computer graphics researchers like to call a volumetric representation or a voxel array, which is essentially a large collection of tiny Lego pieces that build up the scene. But this is not all because the semantic part means that the algorithm actually understands what we're looking at, and thus, is able to classify different parts of the scene. These classes include walls, windows, floors, sofas, and other furniture. Previous works were able to do scene completion and geometry classification, but the coolest part of this algorithm is that it not only does these steps way better, but it does them both at the very same time. This work uses a 3D convolutional neural network to accomplish this task. The 3D part is required for this learning algorithm to be able to operate on this kind of volumetric data. As you can see, the results are excellent, and are remarkably close to the ground truth data. If you remember, not so long ago, I flipped out when I've seen the first neural network-based techniques that understood 3D geometry from 2D images. That technique used a much more complicated architecture, a generative adversarial network, which also didn't do scene completion and on top of that, the resolution of the output

was way lower, which intuitively means that Lego pieces were much larger. This is insanity. The rate of progress in machine learning research is just stunning, probably even for you seasoned Fellow Scholars who watch Two Minute Papers and have high expectations. We've had plenty of previous episodes about the inner workings of different kinds of neural networks. I've put some links to them in the video description, make sure to have a look if you wish to brush

up on your machine learning kung fu a bit. The authors also published a new dataset to solve these kind of problems in future research works, and, it is also super useful because the output of their technique can be compared to ground truth data. When new solutions pop up in the future, this dataset can be used as a yardstick to compare

results with. The source code for this project is also available. Tinkerers rejoice! Thanks for watching and for your generous support, and I'll see you next time!