Stable Neural Style Transfer | Two Minute Papers #136

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. Neural style transfer is an incredible technique where we have two input photographs, and the output would be a combination of these two, namely, the content of one and the artistic style of the other fused together. When the first paper appeared on this topic, the news took the world by storm, and lots of speculative discussions emerged as to what this could be used for and how it would change digital arts and the video game industry. It is great fun to use these algorithms and we have also witnessed a recent proliferation of phone apps that are able to accomplish this, which is super cool because of two reasons: one, the amount of time to go from a published research paper to industry-wide application has never been so small, and, two, the first work required a powerful computer to accomplish this, and took several minutes of strenuous computation, and now, less than two years later, it's right in your pocket and can be done instantly. Talk about exponential progress in science and research, absolutely amazing. And now, while we feast our eyes upon these beautiful results, let's talk about the selling points of this extension of the original technique. The paper contains a nice formal explanation of the weak points of the existing style transfer algorithms. The intuition behind the explanation is that the neural networks think in terms of neuron activations, which may not be proportional to the color intensities in the source image styles, therefore their behavior often becomes inconsistent or different than expected. The authors propose thinking in terms of histograms, which means that the output image should rely on statistical similarities with the source images. And as we can see, the results look outstanding even when compared to the original method. It is also important to point out that this proposed technique is also more art directable, make sure to have a look at the paper for more details on that. As always, I've put a link in the video description. This extension is also capable of texture synthesis, which means that we give it a small image patch that shows some sort of repetition, and it tries to continue it indefinitely in a way that seems completely seamless. However, we have to be acutely aware of the fact that in the computer graphics community, texture synthesis is considered a subfield of its own with hundreds of papers, and one has to be extremely sure to have a clear cut selling point over the state of the art. For the more interested Fellow Scholars out there, I've put a survey paper on this in the video description, make sure to have a look! Thanks for watching and for your generous support, and I'll see you next time!