AI Learns Painterly Harmonization | Two Minute Papers #249

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. When we show a photograph to someone, most of the time we are interested in sharing our memories. Graduation, family festivities beautiful landscapes are common examples of this. With the recent ascendancy of these amazing neural style transfer techniques, we can take a painting, or any other source image, and transfer the style of this image to our contents. The style is transfered, but the contents remains unchanged. This takes place by running the images through a deep neural network, which, in its deeper layers, learns about high level concepts such as artistic style. This work has sparked a large body of followup research works. Feedforward real-time style transfer, temporally coherent style transfer for videos, you name it. However, these techniques are always about taking one image for content, and one for style. How about a new problem formulation where we paste in a part of a foreign image with a completely different style? For instance, if you feel that this ancient artwork is sorely missing a Captain America shield, or if Picasso's self-portrait is just not cool enough without shades, then this algorithm is for you. However, if we just drop in this part of a foreign image, anyone can immediately tell because of the differences in color and style. A previous, non-AI-based technique does way better, but it is still apparent that the image has been tampered with. But as you can see here, this new technique is able to do it seamlessly. It works by first performing style transfer from the painting to the new region, and then, in the second step, additional refinements are made to it to make sure that the response of our neural network is similar across the entirety of the painting. It is conjectured that if the neural network is stimulated the same way by every part of the image, then there shouldn't be outlier regions that look vastly different. And as you can see here, it works remarkably well on a range of inputs. I hope these scroll animations come out really smooth and creamy. This video took a long time to render in 4K resolution with 60 frames per second, and was only possible because of your support on Patreon. If you wish to help us create better videos in the future, please click the Patreon link in the video description and support the series. To validate this work, a user study was done that revealed that the users preferred the new technique over the older ones in 15 out 16 images. I think it is fair to say that this work smokes the competition. But what about comparisons to real paintings? A different user study was also created to answer this question. And the answer is that users were mostly unable to identify whether the painting was tampered with. Excellent work. The source code is also available, so let the experiments begin! Thanks for watching and for your generous support, and I'll see you next time!