This AI Learned To Isolate Speech Signals

dear fellow scholars this is two minute papers with károly fajir this is a neural network based technique that can perform audio-visual separation before we talk about what that is I will tell you what it is not what we've seen in the previous episode where we could select a pixel and listen to it have a look and now let's try to separate the sound of the cello and see if it knows where it comes from this one is different this new technique

can clean up an audio signal by suppressing the noise in a busy bar even if the source of the noise is not seen in the video it can also enhance the voice of the speaker at the same time let's listen so test is uh do any video any person who you see talking their audio gets cleaned up did everything else gets suppressed or if we have a Skype meeting with someone in a lab or a busy office where multiple people are speaking nearby we can also perform a similar speed separation which would be a godsend for future meetings so we've been trying to train this network to input to embeddings and output 3 yeah this is just an extra experiment for the paper hi guys so we've been trying to train

this network to input to embedding is it out the three yeah this is just an extra experiment for the paper

and I think if you are a parent the utility of this example needs no further explanation how people please navigate to work

I am not sure if I ever encountered the term screaming children in the abstract of an AI paper so that one was also a first here this is a super difficult task because the AI needs to understand what leap motions correspond to what kind of sounds which is different for all kinds of languages age groups and head positions to this end the authors put together a stupendously large data set with almost 300,000 videos with clean speed signals this data set is then run through a multi stream neural network that detects the number of human faces within the video generates small thumbnails of them and observes how they move over time it also analyzes the audio signals separately then fuses these elements together with a recurrent neural network to output the separated audio waveforms a key advantage of this architecture and training method is that as opposed to many previous works this is speaker independent therefore we don't need specific training data from the speaker we want to use this on this is a huge leap in terms of usability the paper also contains an excellent demonstration of this concept by taking a piece of footage from Conan O'Brien's show where two comedians were booked for the same timeslot and talk over each other the result is a performance where it is near impossible to understand what they are saying but with this technique we can hear both of them one by one crystal-clear you see some results over here but make sure to click the paper link in the description to hear the sound samples as well thanks for watching and for your generous support and I'll see you next time