Deep Photo Style Transfer | Two Minute Papers #150

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. Let's have a look at this majestic technique that is about style transfer for photos. Style transfer is a magical algorithm where we have one photograph with content, and one with an interesting style. And the output is a third image with these two photos fused together. This is typically achieved by a classical machine learning technique that we call a convolutional neural network. The more layers these networks contain, the more powerful they are, and the more capable they are in building an intuitive understanding of an image. We had several earlier episodes on visualizing the inner workings of these neural networks, as always, the links are available in the video description. Don't miss out, I am sure you'll be as amazed by the results as I was when I have first seen them. These previous neural style transfer techniques work amazingly well if we're looking for a painterly result. However, for photo style transfer, the closeups here reveal that they introduce unnecessary distortions to the image. They won't look realistic anymore. But not with this new one. Have a look at these results. This is absolute insanity. They are just right in some sense. There is an elusive quality to them. And this is the challenge! We not only have to put what we're searching for into words, but we have to find a mathematical description of these words to make the computer execute it. So what would this definition be? Just think about this, this is a really challenging question. The authors decided that the photorealism of the output image is to be maximized. Well, this sounds great, but who really knows a rigorous mathematical description of photorealism? One possible solution would be to stipulate that the changes in the output color would have to preserve the ratios and distances of the input style colors. Similar rules are used in linear algebra and computer graphics to make sure shapes don't get distorted as we're tormenting them with rotations, translations and more. We like to call these operations affine transformations. So the fully scientific description would be that we add a regularization term that stipulates, that these colors only undergo affine transformations. But we've used one more new word here - what does this regularization term mean? This means that there are a ton of different possible solutions for transferring the colors, and we're trying to steer the optimizer towards solutions that adhere to some additional criterion, in our case, the affine transformations. In the mathematical description of this problem, these additional stipulations appear in the form of a regularization term. I am so happy that you Fellow Scholars have been watching Two Minute Papers for so long, that we can finally talk about techniques like this. It's fantastic to have an audience that has this level of understanding of these topics. Just absolutely love it. The source code of this project is also available. Also, make sure to have a look at Distill, an absolutely amazing new science journal from the Google Brain team. But this is no ordinary journal, because what they are looking for is not necessarily novel techniques, but novel and intuitive ways of explaining already existing works. There is also an excellent write-up on research debt that can almost be understood as a manifesto for this journal. A worthy read indeed. They also created a prize for science distillation. I love this new initiative and I am sure we'll hear about this journal a lot in the near future. Make sure to have a look, there is a link to all of these in the video description. Thanks for watching and for your generous support, and I'll see you next time!