This AI Makes The Mona Lisa Come To Life

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. This work presents a learning-based method that is able to take just a handful of photos, and use those to synthesize a moving virtual character. Not only that, but it can also synthesize these faces from new viewpoints that the AI hasn’t seen before. These results are truly sublime, however, hold on to your papers, because it also works from as little as just one input image. This we refer to as 1-shot learning. You see some examples here, but wait a second…really, just one image? If all it needs is really just one photo, this means that we can use famous photographs, and even paintings, and synthesize animations for them. Look at that! Of course, if we show multiple photos to the AI, it is able to synthesize better output results, you see such a progression here as a function of the amount of input data. The painting part I find to be particularly cool because it strays away from the kind of data the neural networks were trained on, which is photos, however, if we have proper intelligence, the AI can learn how different parts of the human face move, and generalize this knowledge to paintings as well. The underlying laws are the same, only the style of the output is different. Absolutely amazing. The paper also showcases an extensive comparison section to previous works, and, as you see here, nothing really compares to this kind of quality. I have heard the quote “any sufficiently advanced technology is indistinguishable from magic” so many times in my life, and I was like, OK, well, maybe, but I’m telling you - this is one of those times when I really felt that I am seeing magic at work on my computer screen. So, I know what you’re thinking - how can all this wizardry be done? This paper proposes a novel architecture where 3 neural networks work together. One, the Embedder takes colored images with landmark information and compresses it down into the essence of these images, two, the Generator takes a set of landmarks, a crude approximation of the human face, and synthesizes a photorealistic result from it. And three, the Discriminator looks at both real and fake images and tries to learn how to tell them apart. As a result, these networks learn together, and over time, they improve together, so much so that they can create these amazing animations from just one source photo. The authors also released a statement on the purpose and effects of this technology, which I’ll leave here for a few seconds for our interested viewers. This work was partly done at the Samsung AI lab and Skoltech. Congratulations to both institutions. Killer paper. Make sure to check it out in the video description. This episode has been supported by Weights & Biases. Weights & Biases provides tools to track your experiments in your deep learning projects. It is like a shared logbook for your team, and with this, you can compare your own experiment results, put them next to what your colleagues did and you can discuss your successes and failures much easier. It takes less than 5 minutes to set up and is being used by OpenAI, Toyota Research, Stanford and Berkeley. It was also used in this OpenAI project that you see here, which we covered earlier in the series. They reported that experiment tracking was crucial in this project and that this tool saved them quite a bit of time and money. If only I had access to such a tool during our last research project where I had to compare the performance of neural networks for months and months. Well, it turns out, I will be able to get access to these tools, because, get this, it’s free and will always be free for academics and open source projects. Make sure to visit them through wandb. com or just click the link in the video description and sign up for a free demo today. Our thanks to Weights & Biases for helping us make better videos for you. Thanks for watching and for your generous support, and I'll see you next time!