Visually Indicated Sounds | Two Minute Papers #79

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. This name is not getting any easier, is it? It used to be Károly Zsolnai, which was hard enough, and now this... haha. Anyway, let's get started. This technique simulates how different objects in a video sound when struck. We have showcased some marvelous previous techniques that were mostly limited to wooden and plastic materials. Needless to say, there are links to these episodes in the video description box. A convolutional neural network takes care of understanding what is seen in the video. This technique is known to be particularly suited to processing image and video content. And it works by looking at the silent video directly and trying to understand what is going on, just like a human would. We train these networks with input and output pairs - the input is a video of us beating the hell out of some object with a drumstick. The joys of research! And the output is the sound this object emits. However, the output sound is something that changes in time. It is a sequence, therefore it cannot be handled by a simple classical neural network. It is learned by a recurrent neural network that can take care of learning such sequences. If you haven't heard these terms before, no worries, we have previous episodes on all of them in the video description box, make sure to check them out! This piece of work is a nice showcase of combining two quite powerful techniques: the convolutional neural network tries to understand what happens in the input video, and the recurrent neural network seals the deal by learning and guessing the correct sound that the objects shown in the video would emit when struck. The synthesized outputs were compared to the real world results both mathematically and by asking humans to try to tell from the two samples which one the real deal is. These people were fooled by the algorithm around 40% of the time, which I find to be a really amazing result, considering two things: first, the baseline is not 50%, but 0% because people don't pick choices at random - we cannot reasonably expect a synthesized sound to fool humans at any time. Like nice little neural networks, we've been trained to recognize these sounds all our lives, after all. And second, this is one of the first papers from a machine learning angle on sound synthesis. Before reading the paper, I expected at most 10 or 20 percent, if that. The tidal wave of machine learning runs through a number of different scientific fields. Will deep learning techniques establish supremacy in these areas? Hard to say yet, but what we know for sure is that great strides are made literally every week. There are so many works out there, sometimes I don't even know where to start. Good times indeed! Before we go, some delightful news for you Fellow Scholars! The Scholarly Two Minute Papers store is now open! There are two different kinds of men's T-shirts available, and a nice sleek design version that we made for the Fellow Scholar ladies out there! We also have The Scholarly Mug to get your day started in the most scientific way possible. We have tested the quality of these products and were really happy with what we got. If you ordered anything, please provide us feedback on how you liked the quality of the delivery and the products themselves. If you can send us an image of yourself wearing or using any of these, we'd love to have a look. Just leave them in the comments section or tweet at us! If you don't like what you got, within 30 days, you can exchange it or get your product cost refunded. Thanks for watching, and for your generous support, and I'll see you next time!