GANPaint: An Extraordinary Image Editor AI

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. This paper describes a new technique to visualize the inner workings of a generator neural network. This is a neural network that is able to create images for us. The key idea here is dissecting this neural network, and looking for agreements between a set of neurons and concepts in the output image, such as trees, sky, clouds, and more. This means analyzing that these neurons are responsible for buildings to appear in the image, and those will generate clouds. Interestingly, such agreements can be found, which means way more than just creating a visualizations like this, because it enables us to edit images without any artistic skills. And now, hold on to your papers. The editing part works by forcefully activating and deactivating these units and correspond to adding or removing these objects from an image. And look! This means that we can take an already existing image, and ask this technique to remove trees from it, or perhaps add more, the same with domes, doors, and more. Wow! This is pretty cool, but you haven’t seen the best part yet! Note that so far, the amount of control we have over the image is quite limited. And fortunately, we can take this further, and select a region of the image where we wish to add something new. This is suddenly so much more granular and useful! The algorithm seems to understand that trees need to be rooted somewhere and not just appear from thin air. Most of the time anyway. Interestingly, it also understands that bricks don’t really belong here, but if I add it to the side of the building, it continues it in a way that is consistent with its appearance. And of course, it is not perfect, here you can see me struggling with this spaghetti monster floating in the air that used to be a tree, and it just refuses to be overwritten. And this is a very important lesson - most research works are but a step in a thousand-mile journey, and each of them tries to improve upon the previous paper. This means that a few more papers down the line, this will probably take place in HD, perhaps in real-time, and with much higher quality. This work also builds on previous knowledge on generative adversarial networks, and whatever the follow-up papers will contain, they will build on knowledge that was found in this work. Welcome to the wonderful world of research! And now, we can all rejoice because the authors kindly made the source code available free for everyone, and not only that, but there is also a web app so you can also try it yourself! This is an excellent way of maximizing the impact of your research work. Let the experts improve upon it by releasing the source code, and let people play with it, even laymen! You will also find many failure cases, but also cases where it works well, and I think there is value in reporting both so we learn a little more about this amazing algorithm. So, let’s do a little research together! Make sure to post your results in the comments section, I have a feeling that lots of high-quality entertainment materials will surface very soon. I bet the authors will also be grateful for the feedback as well, so, let’s the experiments begin! Thanks for watching and for your generous support, and I'll see you next time!