Can We Make An Image Synthesis AI Controllable?

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Not so long ago, we talked about a neural image generator that was able to dream up beautiful natural scenes. It had a killer feature where it would take as an input, not only the image itself, but the labeled layout of this image as well. That is a gold mine of information, and including this indeed opens up a killer application. Look! We can even change the scene around by modifying the labels on this layout, for instance, by adding some mountains, make a grassy field, and add a lake. Making a scene from scratch from a simple starting point was also possible with this technique. This is already a powerful, learning-based tool for artists to use as-is, but can we go further? For instance, would it be possible to choose exactly what to fill these regions with? And this, is what today’s paper excels at, and it turns out, it can do, much, much more. Let’s dive in. One, we can provide it this layout, which they refer to as a semantic mask, and it can synthesize clothes, pants, and hair in many, many different ways. Heavenly. If you have a closer look, you see that fortunately, it doesn’t seem to change any other part of the image. Nothing too crazy here, but please remember this, and now would be a good time to hold on to your papers, because two, it can change the sky or the material properties of the floor. And…wait! Are you seeing what I am seeing? We cannot just change the sky because we have a lake there, reflecting it, therefore, the lake has to change too. Does it? Yes, it does! It indeed changes other parts of the image when it is necessary, which is the hallmark of a learning algorithm that truly understands what it is synthesizing. You can see this effect especially clearly at the end of the looped footage when the sky is the brightest. Loving it. So what about the floor? This is one of my favorites! It doesn’t just change the color of the floor itself, but it performs proper material modeling. Look, the reflections also became glossier over time. A proper light transport simulation for this scenario would take a very, very long time, we are likely talking from minutes to hours. And this thing has never been taught about light transport and learned about these materials by itself! Make no mistake, these may be low-resolution, pixelated images, but this still feels like science fiction. Two more papers down the line, and we will see HD videos of this I am sure. The third application is something that the authors refer to as appearance mixture, where we can essentially select parts of the image to our liking and fuse these aspects together into a new image. This could, more or less be done with traditional, handcrafted methods too, but four, it can also do style morphing, where we start from image A, change it until it looks like image B, and back. Now, normally, this can be done very easily with a handcrafted method called image interpolation, however, to make this morphing really work, the tricky part is that all of the intermediate images have to be meaningful. And as you can see, this learning method does a fine job at that. Any of these intermediate images can stand on their own. I’ll try to stop the morphing process at different points so you can have a look and decide for yourself. Let me know in the comments if you agree. I am delighted to see that these image synthesis algorithms are improving at a stunning pace, and I think these tools will rapidly become viable to aid the work of artists in the industry.

Thanks for watching and for your generous support, and I'll see you next time!