NVIDIA’s New AI: Paint Like Bob Ross!

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today we are going to look at  NVIDIA’s new AI research work which,   as they say allows us to paint with words. So, let’s see. Yes, this runs a generative  denoising process, or in other words, it   starts out from a bunch of noise, and over time,  uses our text prompt to rearrange it into an image   that we described. Great. Then, it subjects these  coarse images to a super resolution technique,   which means that in goes this coarse image, and  out comes an image with so much more detail. And hence, can generate images of this quality. Now, wait a minute. We are experienced Fellow  Scholars over here, so we know that OpenAI’s   DALL-E 2 can do this. And Google’s Imagen  can do this. The free and open source stable   diffusion can also do this. We have a number of  papers that can pull this off really well. So,   is NVIDIA a little late to the party? Why  publish this paper? What is new here? Well,   let’s push it to its limits through 3 really  fun experiments and find out together. One, it gives us something that many of us Fellow  Scholars desire. And that is, of course, more   control over the synthesized images! For instance,  here, we wish to create an image of boxing   squirrels. Yes, you heard it right. But wait, here  is the more granular control part: we can draw   exactly where each squirrel and the boxing gloves  go. And, there we go! Loving it. Or, if we wish   to create a rabbit who is also a magician, we can  also specify that it should stand on clouds, and   we noted that it should cast a fireball. Now we  can tell it where exactly that fireball should go. Two, it also follows our instructions really well  when it comes to requesting styles as well. We can   ask it to paint this penguin in the style of many  famous artists, with really cool results. However,   sometimes just saying which artist we are  looking for is not that helpful. You know,   which phase of the artist are we talking about?   Or which particular work should it be based on?    And hold on to your papers, because this  can even help out when our words fail us.    How? Well, in this case, we can also use an  image instead. We can still add a text prompt,   and it will create the new image in the style  of this one. This is especially useful in cases   when we have a style in mind that is really  hard to explain. I love this one. So cool! Three, so how does it compare to the usual  suspects? Well, let’s have a look at some   teapots. Of course, to the surprise of no one,  Stable Diffusion and DALL-E 2 are both capable   of this task, however, look. We did not get a  painting of a panda. And, with the new technique,   look at that! Oh yes! Once again, it follows our  instructions better. Now note that text to image   AIs are not easy to evaluate as all models  can generate a ton of different images for   the same prompt. However, further comparisons  reveal that there indeed is a pattern here. Now, have you noticed? There is a  pattern in this video too. I keep   saying that this new technique  follows our instructions better,   so here is the most important question.   Why? How is all this wizardry possible? Well, this was one of my favorite parts of the  paper. Have a look at this. The authors claim   that as the classical text to image AIs start  out from noise, they follow our prompt closely,   however, later on in the image synthesis  process, not so much. If we change the   prompt for the last few percentage  of the noise diffusion process, look.    Ouch! It completely ignores it. And here comes the  best part: the authors trained multiple separate   denoiser networks that are suited to different  parts of the generation process. Hence, yes,   you guessed it right, these can follow our  instructions better throughout later parts   of the image generation process, and  thus, give us better artistic control. Now I am sure that the next generation of text  to image AIs are going to be even more powerful   two more papers down the line, but this concept  may live on to improve even these subsequent

versions. I am very excited to see if this will  really be the case. What a time to be alive! Thanks for watching and for your generous  support, and I'll see you next time!