This AI Makes The Mona Lisa Speak…And More!

this episode has been supported by lamda dear fellow scholars this is two minute papers with károly on IFA here in an earlier episode we covered a paper by the name everybody dance now in this stunning work we could take a video of a professional dancer then record a video of our own let's be diplomatic less beautiful moves and then transfer the dancers performance onto our own body in the video we call this process motion transfer now look at this new also learning based technique that does something similar where in goes a description of a pose just one image of a target person and on the other side out comes the proper animation of this character according to our prescribed motions now before you think that it means that we would need to draw and animate stick figures to use this I will stress that this is not the case there are many techniques that perform pose estimation where we just insert a photo or even a video and it creates all these stick figures for us that represent the pose that people are taking in these videos this means that we can even have a video of someone dancing and just one image of the target person and the rest is history insanity that is already amazing and very convenient but this paper works with a video to video problem formulation which is a concept that is more general than just generating movement way more for instance we can also specify the input video of us then add one or at least a few images of the target subject and we can make them speak and behave using our gestures this is already absolutely amazing however the more creative minds out there are already thinking that if we are thinking about images it can be a painting as well right yes indeed we can make the Mona Lisa speak with it as well it can also take a labelled image this is what you see here where the colored and animated patches show the object boundaries for different object classes then we take an input photo of a street scene and we get photorealistic footage with all the cars buildings and vegetation now make no mistake some of these applications were possible before many of which was showcased in previous videos some of which you can see here what is new and interesting here is that we have just one architecture that can handle many of these tasks beyond that this architecture requires much less data than previous techniques as it often needs just one or at most a few images of the target subject to do all this magic the paper is ample in comparison to these other methods for instance the F ID measures the quality and the diversity of the generated output images and is subject to minimization and you see that it is miles beyond these previous works some limitations also apply if the inputs stray too far away from topics that the neural networks were trained on we shouldn't expect results of this quality and we are also dependent on proper inputs for the poses and segmentation maps for it to work well the pace of progress in machine learning research is absolutely incredible and we are getting very close to producing tools that can be actively used to empower artists working in the industry what a time to be alive if you are a researcher or a startup looking for cheap GPU compute to run these algorithms check out lambda GPU cloud I've talked about lambdas GPU workstations in other videos and I'm happy to tell you that they are offering GPU cloud services as well the lambda GPU cloud can train imagenet to 93% accuracy for less than $19 lambdas web based IDE lets you easily access your instance right in your browser and finally hold on to your papers because the lambda GPU cloud costs less than half of AWS and asia make sure to go to lambda labs comm slash papers and sign up for one of their amazing GPU instances today our thanks to lambda for helping us make better videos for you thanks for watching and for your generous support and I see you next time