Hallucinating Images With Deep Learning | Two Minute Papers #74

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. In an earlier episode, we showcased a technique for summarizing images not in a word, but an entire sentence that actually makes sense. If you were spellbound by those results, you'll be out of your mind when you hear this one: let's turn it around, and ask the neural network to have a sentence as an input, and ask it to generate images according to it. Not fetching already existing images from somewhere, generating new images according to these sentences. Create sentences. Is this for real? This is an idea, that is completely out of this world. A few years ago, if someone proposed such an idea and hoped that any useful result can come out of this, that person would have immediately been transported to an asylum. An important keyword here is "zero shot" recognition. Before we go to the zero part, let's talk about one shot learning. One shot learning means a class of techniques that can learn something from one, or at most a handful of examples. Deep neural networks typically require to see hundreds of thousands of mugs before they can learn the concept of a mug. However, if I show one mug to any of you Fellow Scholars, you will, of course, immediately get the concept of a mug. At this point, it is amazing what these deep neural networks can do, but with the current progress in this area, I am convinced that in a few years, feeding millions of examples to a deep neural network to learn such a simple concept will be considered a crime. Onto zero shot recognition! The zero shot is pretty simple - it means zero training samples. But this sounds preposterous! What it actually means is that we can train our network to recognize birds, tiny things, what the concept of blue is, what a crown is, but then we ask it to show us an image of "a tiny bird with a blue crown". Essentially, the neural network learns to combine these concepts together and generate new images leaning on these learned concepts. I think this paper is a wonderful testament as to why Two Minute Papers is such a strident advocate of deep learning and why more people should know about these extraordinary works. About the paper - it is really well written, there are quite a few treats in there for scientists: game theory and minimax optimization, among other things. Cupcakes for my brain. We will definitely talk about these topics in later Two Minute Papers episodes, stay tuned! But for now, you shouldn't only read the paper - you should devour it. And before we go, let's address the elephant in the room: the output images are tiny because this technique is very expensive to compute. Prediction: two papers down the line, it will be done in a matter of seconds, two even more papers down the line, it will do animations in full HD. Until then, I'll sit here stunned by the results, and just frown and wonder. Thanks for watching, and for your generous support, and I'll see you next time!