Google’s New AI: DALL-E, But Now In 3D! 🤯

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today we are going to see how this new AI  is able to take a piece of text from us,   anything we wish, be it a squirrel  dressed like the king of England,   a car made out of sushi, or a humanoid  robot using a laptop, and, magically,   it creates not an image like previous techniques,  but get this, a full 3D model of it. Wow. This is absolutely amazing. An AI that can not  only create images, but create 3D assets? Yes,   indeed, the result is a full 3D model that  we can rotate around and even use in our   virtual worlds. So, let’s give it a really hard  time and see together what it is capable of. For instance, OpenAI’s earlier  DALL-E text to image AI was capable   of looking at a bunch of images of koalas, and  separately, motorcycles,   and it started to understand the concept of  both, and it was be able to combine the two   together into a completely new image.   That is, a koala, riding a motorcycle. So, let’s see if this new method is also capable  of creating new concepts by building on previous   knowledge. Well, let’s see…oh yes! Here is a  tiger wearing sunglasses and a leather jacket,   and most importantly, riding a motorcycle. Tigers  and motorcycles are well understood concepts,   of course, the neural network had plenty of these  to look at in its training set, but combining the   two concepts together, now that is a hint of  creativity. Creativity in a machine. Loving it. What I also loved about this work is that it makes  it so easy to iterate on our ideas. For instance,   first, we can start experimenting with a  real squirrel, or, if we did not like it,   we can quickly ask for a wooden carving,  or even a metal sculpture of it. Then,   we can start dressing it up,  and make it do anything we want. And sometimes, the results are nearly good enough  to be used as-is even in an animation movie   or in virtual worlds, or even in the worse cases,  I think these could be used as a starting point   for an artist to continue from. That would save  a ton of time and energy in a lot of projects! And that is huge. Just consider all the  miraculous things artists are using the   DALL-E 2 text to image AI and Stable Diffusion  for, illustrating novels, texture synthesis,   product design, weaving multiple images  together to create these crazy movies,   you name it. And now, I wonder what unexpected  uses will arise from this being possible for   3D models? Do you have some ideas?   Let me know in the comments below! And just imagine what this will be capable of just  a couple more papers down the line. For instance,   the original DALL-E AI was capable of this, and  then, just a year later, this became possible. So, how does this black magic work? Well, the  cool thing is that this is also a diffusion-based   technique, which means that similarly to the text  to image AIs, it starts out from a piece of noise,   and refines this noise over time to  resemble our input text a little more.    But this time, this diffusion process  is running in higher dimensions, thus,   the result is not a 2D image, but a full 3D  model. So, from now on, the limit in creating   3D worlds is not our artistic skill, the limit  is only our imagination. What a time to be alive!

Thanks for watching and for your generous  support, and I'll see you next time!