AI Creates Facial Animation From Audio | Two Minute Papers #185

dear fellow scholars this is two minute papers with károly Jennifer here I think we're in a moment of history we're probably the most important thing we need to do is to bring the country together and one of the skills that I bring to bear new magento yeah sure Bala Azula sonar Conn Washoe ha this is my reality and is this reality of my people for decades for centuries this work is about creating facial animation from speech in real-time hmm this means that after recording the audio footage of us speaking we give it to a learning algorithm which creates a high quality animation depicting our digital characters uttering these words this learning algorithm is a convolutional neural network which was trained on as little as 3 to 5 minutes of footage per actor and was able to generalize its knowledge from this training data to a variety of real-world expressions and words and if you think you've seen everything you should watch until the end of the video as it gets better than that because of two reasons reason number one it not only takes audio input but we can also specify an emotional state that the character should express when uttering these words you do not understand what it is to are those your Asian Footwear cowboy chaps or jolly earthmoving headgear with tenure Suzy have all the more pleasure for yachting chaps or jolly earthmoving headgear number two and this is the best part we can also combine this together with deepmind's wavenet which synthesizes audio from our text input it basically synthesizes a believable human voice and says whatever text we write down and then that sound clip can be used with this technique to make a digital character say what we have written so we can go from text to speech with wavenet and put the speech onto a virtual actor with this work the avocado is a pear shape with leathery skin smooth edible flesh in a large stone the avocado is a pear-shaped fruit with leathery skin to smooth edible flesh and smooth edible flesh and a large stone this way we get a whole pipeline that works by learning and does everything for us in the most convenient way no actors need it for voiceovers no motion capture for animations this is truly incredible and if you look at the left side you can see that in their video there is some two minute papers action going on how cool is that make sure to have a look at the paper to see the three-way lost function the author's came up with to make sure that the results were correctly for longer animations and of course in research we have to prove that our results are better than previous techniques to accomplish this there are plenty of comparisons in the supplementary video but we need more than that since these results cannot be boiled down to a mathematical theorem that we need to prove we have to do it some other way and the ultimate goal is that a human being would judge these videos as being real with a higher chance than one made with a previous technique this is the core idea behind the user study carried out in the paper we bring in a bunch of people present them with a video of the old and new technique without knowing which is which and ask them which one they feel to be more natural and the result was not even close the new method is not only better overall but I haven't found a single case scenario or language where it didn't come out ahead and that's extremely rare in research typically in a maturing field new techniques introduce a different kind of trade-off for instance less execution time but at the cost of higher memory consumption is a classical case but here is just simply better in every regard excellent listen up the train yard will houses is our main objective now let's say you and your men will do that you have to go in and out very quick I want you to get all the ammo and weapons that you can carry and come back as quickly as you can you understand if you enjoyed this episode

and would like to help us make better videos in the future please consider supporting us on patreon you can pick up cool perks like watching these episodes in early access details are available in the video description beyond telling these important research stories we are also using part of these funds to empower other research projects I just made a small write-up about this which is available on our patreon page the link is in the video description make sure to have a look thanks for watching and for your generous support now see you next time