Google’s New AI Puts Video Calls On Steroids! 💪

5:55

Google’s New AI Puts Video Calls On Steroids! 💪

Two Minute Papers 15.06.2021 189 844 просмотров 11 019 лайков

Machine-readable: Markdown · JSON API · Site index

Смотреть на YouTube

Поделиться Telegram VK Бот

Транскрипт Скачать .md

Анализ с AI

Описание видео

❤️ Check out Fully Connected by Weights & Biases: https://wandb.me/papers 📝 The paper "Total Relighting: Learning to Relight Portraits for Background Replacement" is available here: https://augmentedperception.github.io/total_relighting/ 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible: Aleksandr Mashrabov, Alex Haro, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bryan Learn, Christian Ahlin, Eric Haddad, Eric Martel, Gordon Child, Ivo Galic, Jace O'Brien, Javier Bustamante, John Le, Jonas, Kenneth Davis, Lorin Atzberger, Lukas Biewald, Matthew Allen Fisher, Mark Oates, Michael Albrecht, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Ramsey Elbasheer, Steef, Taras Bobrovytsky, Thomas Krcmar, Torsten Reil, Tybie Fitzhugh, Ueli Gallizzi. If you wish to appear here or pick up other perks, click here: https://www.patreon.com/TwoMinutePapers Meet and discuss your ideas with other Fellow Scholars on the Two Minute Papers Discord: https://discordapp.com/invite/hbcTJu2 Károly Zsolnai-Fehér's links: Instagram: https://www.instagram.com/twominutepapers/ Twitter: https://twitter.com/twominutepapers Web: https://cg.tuwien.ac.at/~zsolnai/

Оглавление (10 сегментов)

Introduction

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. With the increased popularity of online meetings, telepresence applications are on the rise where we can talk to each other from afar. Today, let’s see how these powerful, new neural network-based learning methods can be applied to them. It turns out, they can help us do everyone’s favorite, which is, showing up to a meeting, and changing our background to pretend we are somewhere else. Now that is a deceptively difficult problem. Here, the background has been changed, that is the easier problem, but, look!

The Problem

The lighting of the new environment hasn’t been applied to the subject. And now, hold on to your papers, and check this out. This is the result of the new technique after it recreates the image as if she was really there. I particularly like the fact that the result includes high-quality specular highlights too, or in other words, the environment reflecting off of our skin. However, of course, this is not the first method attempting this. So let’s see how it performs compared to the competition! These techniques are from one and two years ago, and…they don’t perform so well. Not only did they lose a lot of detail all across the image, but, the specular highlights are gone. As a result, the image feels more like a video game character than a real person.

Reference Information

Luckily, the authors also have access to the reference information to make our job of comparing the results easier. Roughly speaking, the more the outputs look like this, the better. So now, hold on to your papers, and let’s see how the new method performed. Oh yes!

Not Perfect

Now we’re talking! Now, of course, not even this is perfect. Clearly, the specularity of clothing was determined incorrectly, and the matting around the thinner parts of the hair could be better, which is notoriously difficult to get right. But, this is such a huge step forward in just one paper. And we are not nearly done - there are two more things that I found to be remarkable about this work.

Still Images

One, is that the whole method was trained on still images, yet, it still works on video too! And we don’t have any apparent temporal coherence issues, or in other words, no flickering arises from the fact that it processes the video as not a video, but a series of separate images. Very cool.

Background Lighting

Two, if we are in a meeting with someone and we like their background, we can simply borrow it. Look. This technique can take their image, get the background out, estimate its lighting, and give the whole package to us too. I think this will be a game changer.

Game Changer

People may start to become more selective with these backgrounds, not just because of how the background looks, but, because how it makes them look. Remember, lighting off of a well chosen background makes a great deal of a difference in our appearance in the real world, and now, with this method in virtual worlds too. And this will likely happen not decades from now, but in the near future. So this new method is clearly capable of some serious magic. But how? What is going on under the hood to achieve this?

How It Works

This method performs two important steps to accomplish this, step number one is matting. This means separating the foreground from the background, and then, if done well, we can now easily cut out the background and also have the subject on a separate layer and proceed to step number two. Which is, relighting. In this step, the goal is to estimate the illumination of the new scene and recolor the subject as if she were really there. This new technique performs both, but most of the contributions lie in this step. To be able to accomplish this, we have estimate the material properties

Material Properties

of the subject. The technique has to know one, where the diffuse parts are, these are the parts that don’t change too much as the lighting changes, and two, where the specular parts are, in other words, shiny regions that reflect back the environment more clearly. Putting it all together, we get really high-quality relighting for ourselves, and given that this was developed by Google, I expect that this will supercharge our meetings quite soon.

Outro

And just imagine what we will have two more papers down the line. My goodness, what a time to be alive! Thanks for watching and for your generous support, and I'll see you next time!

Другие видео автора — Two Minute Papers

Ctrl+V

Экстракт Знаний в Telegram

Экстракты и дистилляты из лучших YouTube-каналов — сразу после публикации.

Подписаться

Лучшие методички за неделю — каждый понедельник