This Neural Network Performs Foveated Rendering

dear fellow scholars this is two minute papers with károly fajir as humans when looking at the world our eyes and brain does not process the entirety of the image we have in front of us but plays an interesting trick on us we can only see fine details in a tiny fovea teeth region that we are gazing at while our peripheral or indirect vision only sees a sparse blurry version of the image and the rest of the information is filled in by our brain this is very efficient because our vision system only has to process the tiny fraction of the visual data that is in front of us and it still enables us to interact with the world around us so what if we would take a learning algorithm that does something similar for digital videos imagine that we would need to render a sparse video with only every tenth pixel filled with information and some kind of neural network based technique would be able to reconstruct the full image similarly to what our brain does yes that sounds great but that is very little information to reconstruct an image from so is it possible well hold on to your papers because this new work can reconstruct a near-perfect image by looking at less than 10% of the input pixels so we have this as an input and we get this Wow what is happening here is called a neuro reconstruction or foliated rendering data or you are welcome to refer to it as foliated reconstruction in short during your conversations over dinner the scrambled text part here is quite interesting one might think that well it could be better however given the fact that if you look at the appropriate place in the sparse image I not only cannot read the text I am not even sure if I see anything that indicates that there is a text there at all so far the example assumed that we are looking at a particular point in the middle of the screen and the ultimate question is how does this deal with a real-life case where the user is looking around well let's see this is the input and the reconstruction witchcraft have a look at some more results note that this method is developed for head mounted displays where we have information on where the user is looking over time and this can make all the difference in terms of optimization you see a comparison here against a method labeled as multi-resolution this is from a paper by the name foliated 3d graphics and you can see that the difference in the quality of the reconstruction is truly remarkable additionally it has been trained on 350,000 short natural video sequences and the whole thing runs in real time also note that we often discuss image in painting methods in this series for instance what you see here is the legendary patch match algorithm that is one of these and it is able to fill in missing parts of an image however in image in painting most of the image is intact with smaller regions that are missing this is even more difficult an image in painting because the vast majority of the image is completely missing the fact that we can now do this with learning-based methods is absolutely incredible the first author of the paper is Anton Kaplan who's a brilliant and very rigorous mathematician so of course the results are evaluated in detail both in terms of mathematics and with the user study make sure to have a look at the paper for more on that we got to know each other with Anton during the days when all we did was light transport simulations all day every day and were always speculating about potential projects and to my great sadness somehow unfortunately we never managed to work together for a full project again congratulations on tone beautiful work what a time to be alive this episode has been supported by Linode is the world's largest independent cloud computing provider they offer affordable GPU instances featuring the Quadro r-tx 6000 which is tailor-made for AI scientific computing and computer graphics projects exactly the kind of works you see here in this series if you feel inspired by these works and you wish to run your experiments or deploy your already existing works through a simple and reliable hosting service make sure to join over 800,000 other happy customers and choose Linode to spin up your own GPU instance and receive a $20 free credit visit linda. com slash papers or click the link in the video description and use the promo code papers 20 during signup give it a try today our thanks to Lee note for supporting the series and helping us make better videos for you thanks for watching and for your generous support and I'll see you next time