This AI Creates Beautiful 3D Photographs!

dear fellow scholars this is two minute papers with dr carol jonaife we hear more and more about rgbd images these days these are photographs that are endowed with depth information which enable us to do many wondrous things for instance this method was used to endow self-driving cars with depth information and worked reasonably well and this other one provides depth maps that are so consistent we can even add some ar effects to it and today's paper is going to show what 3d photography is however first we need not only color but depth information in our images to perform these you see phones with depth scanners already exist and even more are coming as soon as this year but even if you have a device that only gives you 2d color images don't despair there is plenty of research on how we can estimate these depth maps even if we have very limited information and with proper depth information we can now create these 3d photographs where we get even more information out of one still image we can look behind objects and see things that we wouldn't see otherwise beautiful parallax effects appear as objects at different distances move different amounts as we move the camera around you see that the foreground changes a great deal the buildings in the background less so and the heels behind them even less so these photos truly come alive with this new method an earlier algorithm the legendary patch match method from more than a decade ago could perform something that we call image in painting image impainting means looking at what we see in these images and trying to fill in missing information with data that makes sense the key difference here is that this new technique uses a learning method and does this image in painting in 3d and it not only fills in color but depth information as well what a crazy amazing idea however this is not the first method to perform this so how does it compare to other research works let's have a look together previous methods have a great deal of warping and distortions on the bathtub here and if you look at the new method you see that it is much cleaner there is still a tiny bit of warping but it is significantly better the dog head here with this previous method seems to be bobbing around a great deal while the other methods also have some problems with it look at this too and if you look at how the new method handles it is significantly more stable and you see that these previous techniques are from just one or two years ago it is unbelievable how far we have come since bravo so this was a qualitative comparison or in other words we looked at the results what about the quantitative differences what do the numbers say look at the psnr column here this means the peak signal to noise ratio this is subject to maximization as the up arrow denotes here the higher the better the difference is between one half to two and a half points when compared to previous methods which does not sound like a lot at all so what happened here note that psnr is not a linear but a logarithmic scale so this means that a small numeric difference typically translates to a great deal of difference in the images even if the numeric difference is just 0. 5 points on the ps on r scale however if you look at ssim the structural similarity metric all of them are quite similar and the previous technique appears to be even winning here but this was a method that warped the doghead and in the visual comparisons the new method came out significantly better than this so what is going on here well have a look at this metric lpips which was developed at the uc berkeley open ai and adobe research at the risk of simplifying the situation this uses a neural network to look at an image and uses its inner representation to decide how close the two images are to each other loosely speaking it kind of thinks about the differences as we humans do and is an excellent tool to compare images and sure enough this also concludes that the new method performs best however this method is still not perfect there is some flickering going on behind these fences the transparency of the glass here isn't perfect but witnessing this huge leap in the

quality of results in such little time is truly a sight to behold what a time to be alive i started this series to make people feel how i feel when i read these papers and i really hope that it goes through with this paper absolutely amazing what is even more amazing is that with a tiny bit of technical knowledge you can run the source code in your browser so make sure to have a look at the link in the video description let me know in the comments how it went what you see here is an instrumentation of this exact paper we have talked about which was made by weights and biases i think organizing these experiments really showcases the usability of their system weights and biases provides tools to track your experiments in your deep learning projects their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as open ai toyota research github and more and the best part is that if you have an open source academic or personal project you can use their tools for free it really is as good as it gets make sure to visit them through wnb. com papers or click the link in the video description to start tracking your experiments in 5 minutes our thanks to weights and biases for their long-standing support and for helping us make better videos for you thanks for watching and for your generous support and i'll see you next time