This Magical AI Cuts People Out Of Your Videos! ✂️

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today we are going to see magical things that  open up when we are able to automatically find   the foreground and the background of a video. Let’s see why that matters! This new technique leans on a previous method  to find the boy, and the dog. Let’s call this   level 1 segmentation. So far so good, but  this is not the state of the art. Yet.   Now, comes level 2 - it also found the shadow  of the boy, and, the shadow of the dog.    Now we’re talking! But it doesn’t stop there!    It gets even better. Level three, this  is where things get out of hand - look,   the dog is occluding the boy’s shadow,  and it is able to deal with that too. So, if we can identify all of the effects that  are attached to the boy and the dog, what can   we do with all this information? Well, for  instance, we can even remove them from the video.    Nothing to see here. Now, a common problem  is that still, the silhouette of the subject   still remains in the final footage,  so let’s take a close look together!    I don’t see anything at all. Wow. Do  you? Let me know in the comments below! Just to showcase how good this removal is, here  is a good technique from just one year ago.    Do you see it? This requires the  shadows to be found manually,   so we have to work with that. And still, in the  outputs, you can see the silhouette we mentioned.    And, how much better is the new method?   Well, it finds the shadows automatically.    That is already mind blowing, and the outputs  are…yes, much cleaner. Not perfect, there is still   some silhouette action, but if I were not actively  looking for it, I might not have noticed it. It can also remove people  from this trampoline scene,   and not only the bodies, but it also removes  their effect on the trampolines as well. Wow. And as this method can perform  all this reliably, it opens up   the possibility for new, magical effects. For instance, we can duplicate this test subject,   and even fade it in and out. Note that it  has found its shadows as well. Excellent! So, it can deal with finding not only  the shape of the boy and the dog,   and it knows that it’s not enough to just find  their silhouettes, but it also has to find   additional effects they have on the footage.   For instance, their shadows. That is wonderful,   and what is even more wonderful is that this  was only one of the simpler things it could do. Shadows are not the only  potential correlated effects,   look. A previous method was able to find the  swan here, but that’s not enough to remove it,   because it has additional effects on the scene.   What are those? Well, look, it has reflections,   and it creates ripples too. This is so much  more difficult than just finding shadows.    And now, let’s see the new method... and! Whoa.   It knows about the reflections and ripples,   finds both of them, and gives us this  beautifully clean result. Nothing to see here. Also, look at this elephant. Removing  just the silhouette of the elephant   is not enough, it also has to  find all the dust around it,   and it gets worse, the dust is changing  rapidly over time. And believe it or not…wow,   it can find the dust too, and remove the  elephant. Again, nothing to see here. And if you think that this dust was the new  algorithm at its best, then have a look at this   drifting car. Previous method. Yes, that is the  car, but you know what I want. I want the smoke   gone too. That’s probably impossible, right?   Well, let’s have a look. Wow. I can’t believe   it. It grabbed and removed the car and the smoke  together…and, once again, nothing to see here. So, what are those more magical things  that this opens up? Watch carefully…it   can make the colors pop here. And, remember, it  can find the reflections of the flamingo, so, it

keeps not only the flamingo, but the reflection of  the flamingo in color as well. Absolutely amazing. And, if we can find the background of a video,  we can even change the background. This works   even in the presence of a moving  camera, which is a challenging problem.    Now, of course, not even this technique is  perfect - look here. The reflections are copied   off of the previous scene,  and it shows on the new one. So what do you think? What would  you use this technique for?    Let me know in the comments, or if you wish  to discuss similar topics with other Fellow   Scholars in a warm and welcoming environment,  make sure to join our Discord channel. Also,   I would like to send a big thank you to the mods  and everyone who helps running this community.    The link to the server is available in  the video description. You’re invited. Thanks for watching and for your generous  support, and I'll see you next time!