This AI Creates Beautiful Time Lapse Videos ☀️

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. A few years ago, we have mainly seen neural network-based techniques being used for image classification. This means that they were able to recognize objects, for instance, animals and traffic signs in images. But today, with the incredible pace of machine learning research, we now have a selection of neural network-based techniques for not only classifying images, but, also, synthesizing them! The images that you see here and throughout this video is generated by one of these learning-based methods. But of course, in this series, we are always obsessed with artistic control, or, in other words, how much of a say we have in the creation of these images. After all, getting thousands and thousands of images without any overarching theme or artistic control is hardly useful for anyone. One way of being able to control the outputs is to use a technique that is capable of image translation. What you see here is a work by the name CycleGAN! It could transform apples into oranges, zebras into horses, and more. It was called CycleGAN because it introduced a cycle consistency loss function. This means that if we convert a summer image to a winter image, and then back to a summer image, we should get the same input image back, or at least, something very similar. If our learning system obeys this principle, the output quality of the translation is going to be significantly better. Today, we are going to study a more advanced image translation technique that takes this further. This paper is amazingly good at daytime image translation. It looks at a selection of landscape images, and then, as you see here, it learns to reimagine our input photos as if they were taken at different times of the day. I love how clouds form, and move over time in the synthesized images, and the night sky with the stars is also truly a sight to behold. But wait, CycleGAN and many other followup works did image translation, this also does

image translation, so, what’s really new here? Well, one, this work proposes a novel upsampling scheme that helps creating output images with lots and lots of detail. Two, it can also create not just a bunch of images a few hours apart, but it can also make beautiful timelapse videos, where the transitions are smooth. Oh my goodness. I love this. And three, the training happens by shoveling 20 thousand landscape images into the neural network, and it becomes able to perform this translation task without labels. This means that we don’t have to explicitly search for all the daytime images and tell the learner that these are daytime images, and these other images are not.

This is amazing because the algorithm is able to learn by itself, without labels, but it is also easier to use because we can feed in lots and lots more training data without having to label these images correctly. As a result, we now know that this daytime translation task is used as a testbed to demonstrate

that this method can be reused for other kinds of image translation tasks. The fact that it can learn on its own and still compete with other works in this area is truly incredible. Due to this kind of generality, it can also perform other related tasks, for instance

it can perform style transfer, or in other words, not just change the time of day, but reimagine our pictures in the style of famous artists. I think with this paper, we have a really capable technique on our hands that is getting closer and closer to the point where they can see use in mainstream software packages and image editors. That would be absolutely amazing. If you have a closer look at the paper, you will see that it tries to minimize 7 things at the same time. What a time to be alive! This episode has been supported by Weights & Biases. Here, they show you how to build a proper Convolutional Neural Network for image classification, and how to visualize the performance of your model. Weights & Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money, and it is actively used in projects at prestigious labs, such as OpenAI, Toyota Research, GitHub, and more. And, the best part is that if you are an academic or have an open source project, you can use their tools for free. It really is as good as it gets. Make sure to visit them through wandb. com/papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights & Biases for their long-standing support and for helping us make better videos for you. Thanks for watching and for your generous support, and I'll see you next time!