This AI Removes Shadows From Your Photos! 🌒

6:02

This AI Removes Shadows From Your Photos! 🌒

Two Minute Papers 22.08.2020 117 257 просмотров 5 463 лайков

Machine-readable: Markdown · JSON API · Site index

Смотреть на YouTube

Поделиться Telegram VK Бот

Транскрипт Скачать .md

Анализ с AI

Описание видео

❤️ Check out Weights & Biases and sign up for a free demo here: https://www.wandb.com/papers ❤️ Their post on how to train distributed models is available here: https://app.wandb.ai/sayakpaul/tensorflow-multi-gpu-dist/reports/Distributed-training-in-tf.keras-with-W%26B--Vmlldzo3NzUyNA 📝 The paper "Portrait Shadow Manipulation" is available here: https://people.eecs.berkeley.edu/~cecilia77/project-pages/portrait 📝 Our paper with Activision Blizzard on subsurface scattering is available here: https://users.cg.tuwien.ac.at/zsolnai/gfx/separable-subsurface-scattering-with-activision-blizzard/ 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible: Aleksandr Mashrabov, Alex Haro, Alex Paden, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bruno Mikuš, Bryan Learn, Christian Ahlin, Daniel Hasegan, Eric Haddad, Eric Martel, Gordon Child, Javier Bustamante, Lorin Atzberger, Lukas Biewald, Michael Albrecht, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Ramsey Elbasheer, Robin Graham, Steef, Sunil Kim, Taras Bobrovytsky, Thomas Krcmar, Torsten Reil, Tybie Fitzhugh. If you wish to support the series, click here: https://www.patreon.com/TwoMinutePapers Károly Zsolnai-Fehér's links: Instagram: https://www.instagram.com/twominutepapers/ Twitter: https://twitter.com/twominutepapers Web: https://cg.tuwien.ac.at/~zsolnai/

Оглавление (8 сегментов)

<Untitled Chapter 1>

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. When we look at the cover page of a magazine, we often see lots of well-made, but also idealized photos of people. Idealized here means that the photographer made them in a studio, where they can add or remove light sources and move them around to bring out the best from their models. But most photos are not made in the studio, they are made out there in the wild where the lighting is what it is, and we can’t control it too much. So, with that, today, our question is what if we could change the lighting after the photo has been made? This work proposes a cool technique to perform exactly that by enabling us to edit the shadows on a portrait photo that we would normally think of deleting. Many of these have to do with the presence of shadows, and you can see here that we can

Foreign shadow removal Facial shadow softening

really edit these after the photo has been taken. However, before we start taking a closer look at the editing process, we have to note that there are different kinds of shadows. One, there are shadows cast on us by external objects, let’s call those foreign shadows

We divide shadows into two types

and there is self-shadowing, which comes from the model’s own facial features. Let’s call those facial shadows. So why divide them into two classes?

We propose two data synthesis models to generate images with foreign and facial shadows

Simple, because we typically seek to remove foreign shadows, and edit facial shadows.

We use synthesized data to train a foreign shadow removal model and a controllable facial shadow softening model

The removal part can be done with a learning algorithm, provided that we can teach it with a lot of training data. Let’s think about ways to synthesize such a large dataset! Let’s start with the foreign shadows. We need image pairs of test subjects with and without shadows to have the neural network learn about their relations. Since removing shadows is difficult without further interfering with the image, the authors opted to do it the other way around. In other words, they take a clean photo of the subject, that’s the one without the shadows, and then, and add shadows to it algorithmically. Very cool! And, the results are not bad at all, and get this, they even accounted for subsurface scattering, which is the scattering of light under our skin. That makes a great deal of a difference. This is a reference from a paper we wrote with scientists at the University of Zaragoza and the Activision Blizzard company to add this beautiful effect to their games. Here is a shadow edge without subsurface scattering, quite dark, and with subsurface scattering, you see this beautiful glowing effect. Subsurface scattering indeed makes a great deal of difference around hard shadow edges, so huge thumbs up for the authors for including an approximation of that.

Example foreign shadow removal training data

However, the synthesized photos are still a little suspect. We can still tell that they are synthesized. And that is kind of the point. Our question is “can the neural network still learn the difference between a clean and a shadowy photo” despite all this? As you see, the problem is not easy - previous methods did not do too well on these examples when you compare them to the reference solution. And let’s see this new method. Wow, I can hardly believe my eyes. Nearly perfect. And it did learn all this on not real, but synthetic images.

Results on foreign shadow removal

And believe it or not, this was only the simpler part. Now comes the hard part. Let’s look at how well it performs at editing the facial shadows! We can pretend to edit both the size and the intensity of these light sources. The goal is to have a little more control over the shadows in these photos, but, whatever we do with them, the outputs still have to remain realistic. Here are the before and after results. The facial shadows have been weakened, and depending on our artistic choices, we can

Results on facial shadow softening

also soften the image a great deal. Absolutely amazing. As a result, we now have a two-step algorithm, that first, removes foreign shadows, and is able to soften the remainder of the facial shadows, creating much more usable portrait photos of our friends, and all this after the photo has been made. What a time to be alive! Now, of course, even though this technique convincingly beats previous works, it is still not perfect. The algorithm may fail to remove some highly detailed shadows, you can see how the shadow of the hair remains in the output. In this other output, the hair shadows are handled a little better, there is some dampening, but the symmetric nature of the facial shadows here put the output results in an interesting no man’s land where the opacity of the shadow has been decreased, but the result looks unnatural. I can’t wait to see how this method will be improved two more papers down the line. I will be here to report on it to you, so make sure to subscribe and hit the bell icon to not miss out on that. Thanks for watching and for your generous support, and I'll see you next time!

Другие видео автора — Two Minute Papers

Ctrl+V

Экстракт Знаний в Telegram

Экстракты и дистилляты из лучших YouTube-каналов — сразу после публикации.

Подписаться

Лучшие методички за неделю — каждый понедельник