# OpenAI's DALL-E 2 Has Insane Capabilities! 🤖

## Метаданные

- **Канал:** Two Minute Papers
- **YouTube:** https://www.youtube.com/watch?v=eM5jn8vY2OQ
- **Дата:** 10.11.2022
- **Длительность:** 9:20
- **Просмотры:** 107,215
- **Источник:** https://ekstraktznaniy.ru/video/13387

## Описание

❤️ Check out Runway and try it for free here: https://runwayml.com/papers/
Use the code TWOMINUTE at checkout to get 10% off!

📝 The paper "Hierarchical Text-Conditional Image Generation with CLIP Latents" is available here:
https://openai.com/dall-e-2/

☀️My free Master-level light transport course is available here:
https://users.cg.tuwien.ac.at/zsolnai/gfx/rendering-course/

📝 Our Separable Subsurface Scattering paper with Activition-Blizzard:
https://users.cg.tuwien.ac.at/zsolnai/gfx/separable-subsurface-scattering-with-activision-blizzard/

📝 Our earlier paper with the caustics:
https://users.cg.tuwien.ac.at/zsolnai/gfx/adaptive_metropolis/

Reynante Martinez, the master's page:
https://www.reynantemartinez.com/

Rendered images:
LuxCore Render / Sharlybg https://luxcorerender.org/wp-content/uploads/2017/12/Salon22XS.jpg
https://luxcorerender.org/wp-content/uploads/2017/12/SSDark_01b.jpg

Hotel scene:
Badblender - https://www.blendswap.com/blend/30669

Path tracing links on Shader

## Транскрипт

### Teaser []

dear fellow Scholars this is two minute papers with Dr Carol jonai fahir finally this is my happy episode well of course I am happy in every episode but this is going to be my happy episode if you will why is that well buckle up because today we are going to use open ai's Dolly 2 a text to image Ai and we will see what it is made of can it create beautiful light transport effects or not we will see through four beautiful experiments for instance this may sound like science fiction but today we will also see if it can recreate this scene from a true master of digital 3D art so what is the slight thing I keep

### Light Transport [0:48]

talking about a light transport simulation means a computer program that is able to compute the path of light rays to create beautiful images like this and this and our key problem is that initially we only get noisy images and it can take a long time for the simulator to eliminate this noise so can Dolly 2 help with that well how for instance it can perform a

### Variant generation [1:18]

variant generation where in goes one image and the AI synthesizes other similar images this is really cool as it means that the AI has a good understanding of what it sees and can create different variations of it and wait a minute are you thinking what I am thinking oh yes experiment number one denoising let's give this noisy input

### Experiment 1 [1:48]

image from a light transport simulator give it to the variant generator and see if it is able to recreate the essence of the image but without the noise let's see well that is super interesting it did not denoise the image but it did something else it tried to understand what the noise is in this context and found it to be some sort of gold powder how cool is that based on the insights

### Let's try it again! [2:20]

gained here let's try again with a little less noise oh yes this difficult scene would normally take even up to days to compute correctly do you see these light streaks here we would need to clean those up so variant generation it's your turn again and look at that wow we get a noise free image that captured the essence of our input I cannot believe it so good interestingly it did not ignore the last tweaks but it thought that this is the texture of the object and synthesized the new ones accordingly this actually means that Dolly 2 does what it is supposed to be doing Faithfully reproducing the scene and putting a different spin on it so cool and I think this concept could be supercharged by generating such a noisy input quickly then denoising it with one of those handcrafted techniques for these images these are typically not perfect but they may be just good enough to kick start the variant generator I would love to see some more detailed experiments in this direction now what else can this do well

### Experiment 2 [3:40]

experiment number two my favorite caustics oh yes these are beautiful patterns of reflected light that we see a lot of in real life and they produce some of the most beautiful images any light transport simulation can offer yes that's right with such a simulation we can compute this too how cool is that so now let's ask Dolly 2 to create some of these for us and the results are truly Sublime so regular Acoustics check mark and what about those fun heart-shaped caustics when we put a ring in the middle of an open book my goodness the AI understands that and it really works loving it however if you look at those beautiful volumetric caustics when running variant generation on that it only kind of works there are some Rays of Hope here but otherwise I feel that the AI thinks that this is some sort of laser experiment instead and also don't forget about Daniel baskin's amazing results who created these drinks but wait we are light transport researchers here so we don't look at the drink what do we look at yes of course the caustics beautiful and if we are looking at beautiful things time

### Experiment 3 [5:05]

for experiment number three subsurface scattering what is that oh boy subsurface scattering is the beautiful effect of light penetrating our skin milk and other materials and bouncing in inside before coming out again the lack of this effect is why the skin looks a little plasticky in older video games however light transport simulation researchers took care of that too this is from our earlier paper with the Activision Blizzard game development company this is the same phenomenon a simulation without subsurface scattering and this one is with simulating this effect and in real time beautiful you can find the link to this paper in the video description so Canon AI pull this off today that's impossible right well it seems so if I plainly ask for subsurface scattering from Dolly too I did not get any of that however when prompting a text to image AI we have to know not only what we wish to see but how to get it out of the algorithm so if we ask for translucent objects with strong backlighting Bingo Dolly too can do this too so good loving it and now hold on to your papers because now is the time for our final experiment

### Experiment 4 [6:34]

number four reproducing the work of a true Master If the previous experiment was nearly impossible I really don't know what this is here is a beautiful little virtual world from reinante Martinez and it really speaks for itself now let's put it into the variant generator and see what Dolly 2 is made of wow look at that these are incredibly good not as good as the master himself but I think the first law of papers should be invoked here wait what is that the first law of papers says that research is a process do not look at where we are will be two more papers down the line and I have to say I can imagine that we will get comparable images I also love how it thinks that fingerprints are part of the liquid it is a bit of a limitation but a really beautiful one what a time to be

### Indirect Illumination, dispersion, course [7:40]

alive and we haven't even talked about indirect illumination dispersion and many other amazing light transport effects I really hope we will see some more experiments perhaps from you fellow scholars in this direction too by the way I have a Master Level light transport simulation course for all of you free of charge no strings attached and we write a beautiful little simulator that can create this image and more the link is in the video description and this episode has been supported by Runway professional and magical AI video editing for everyone I often hear you fellow Scholars asking okay these AI techniques look great but when do I get to use them and the answer is right now Runway is an amazing video editor that can do many of the things that you see here in this series for instance it can automatically replace the background behind the person it can do in painting for videos amazingly well and can do even text to image to image you name it no wonder it is used by editors post-production teams and creators at companies like CBS Google Vox and many other make sure to go to runwayml. com papers sign up and try it for free today and here comes the best part use the code two minute at checkout and get 10 off your first month thanks for watching and for your generous support and I'll see you next time