Image Colorization With Deep Learning and Classification | Two Minute Papers #71

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. This work is about adding color to black and white images. There were some previous works that tackled this problem, and many of them worked quite well, but there were cases when the results simply didn't make too much sense. For instance, the algorithm often didn't guess what color the fur of a dog should be. If we would give the same task to a human, we could usually expect better results because the human knows what breed the dog is, and what colors are appropriate for that breed. In short, we know what is actually seen on the image, but the algorithm doesn't - it just trains on black and white and colored image pairs and learns how it is usually done without any concept of what is seen on the image. So here is the idea - let's try to get the neural network not only to colorize the image, but classify what is seen on the image before doing that. If we see a dog in an image, it is not that likely to be pink, is it? If we know that we have to deal with a golf course, we immediately know to reach out for those green crayons. This is a novel fusion-based technique. This means that we have a separate neural network for classifying the images and one for colorizing them. The fusion part is when we unify the information in these neural networks so we can create an output that aggregates all this information. And the results, are just spectacular, the additional information on what these images are about really make a huge impact on the quality of the results. Please note that this is by far not the first work on fusion, I've also linked an earlier paper for recognizing objects in videos, but I think this is a really creative application of the same train of thought that is really worthy of our attention. To delight the fellow tinkerers out there, the source code of the project is also available. The supplementary video reveals that temporal coherence is still a problem. This means that every image is colorized separately with no communication. It is a bit like giving the images to colorize one by one to different people, with no overarching artistic direction. The result we'll get this way is a flickery animation. This problem has been solved for artistic style transfer, which we have discussed in an earlier episode, the link is in the description box. There was one future episode planned about plastic deformations. I have read the paper several times, and it is excellent, but I felt that the quality of my presentation was not up there to put it in front of you Fellow Scholars. It may happen in the future, but I had to shelf this one for now. Please accept my apologies for that. In the next episode, we'll continue with OpenAI's great new invention for reinforcement learning. Thanks for watching, and for your generous support, and I'll see you next time!