StyLit, Illumination-Guided Artistic Style Transfer | Two Minute Papers #91

dear fellow Scholars this is 2minute papers with Caro earlier we have talked quite a bit about a fantastic new tool that we called artistic style transfer this means that we have an input photograph that we'd like to modify and another image from which we'd like to extract the artistic style this way we can for instance change our photo to look in the style of famous artists now artists in the visual effects industry spent a lot of time designing the lighting and the illumination of their scenes which is a long and arduous process this is typically done in some kind of light simulation program and if anyone thinks this is an easy and straightforward thing to do I would definitely recommend trying it after this lighting and illumination step is done we can apply some kind of artistic style transfer but we shall quickly see that there is an Insidious side effect to this process it disregards or even worse destroys our illumination setup leading to results that look physically incorrect today we're going to talk about a flamboyant little technique that is able to perform artistic style transfer in a way that preserves the illumination of the scene these kinds of works are super important because they enable us to take the wheel from the hands of the neural networks that perform these operations and force our will on them this way we can have a greater control over what these neural networks do previous techniques take into consideration mostly color and normal information normals basically encode the shape of an object however these techniques don't really have a notion of Illumination they don't know that a reflection on an object should remain intact and they have no idea about the existence of Shadows either for instance we have recently talked about diffuse and specular material models and setting up this kind of Illumination is something that artists in the industry are quite familiar with the goal is that we can retain these features throughout the process of style transfer in this work the artist is given a printed image of a simple object like a sphere this is no ordinary printed image because this image comes from a photo realistic rendering program which is augmented by additional information like what part of the image is a shadowed region and where the reflection are and then when the artist starts to add her own style to it we know exactly what has been changed and how this leads to a much more elaborate style transfer pipeline where the illumination stays intact and the results are phenomenal what is even more important the usability of the solution is also beyond amazing for instance here the artist can do the stylization on a simple sphere and get the artistic style to carry over to a complicated piece of geometry almost immediately temporal coherence is still to be improved which means that if we try this on an animated sequence it will be contaminated with flickering noise we have talked about a work that does something similar for the old kind of style transfer I've put a link in the video description box for that I'm sure that this Kink will be worked out in no time it's also interesting to note that the first style transfer paper was also published just a few months ago this year and we are already lavishing in excellent follow-up papers I think this demonstrates the excitement of research quite aptly the rate of progress in technology and algorithms are completely unmatched fresh new ideas pop up every day and we can only frown and wander at their Ingenuity as usual please let me know in the comment section whether you have found this episode interesting and understandable if you felt that everything is fine here that is also valuable feedback thank you and by the way if you wish to express your scholarly wisdom our stories open with some amazing quality merch have a look we also have a huge influx of people who became patrons recently welcome thank you so much for supporting 2minute papers we love you too thanks for watching and for your generous support and I'll see you next time oh