# Exploring And Attacking Neural Networks With Activation Atlases

## Метаданные

- **Канал:** Two Minute Papers
- **YouTube:** https://www.youtube.com/watch?v=XSWqLb0VyzM
- **Дата:** 30.04.2019
- **Длительность:** 4:05
- **Просмотры:** 36,504
- **Источник:** https://ekstraktznaniy.ru/video/14322

## Описание

📝 The paper "Exploring Neural Networks with Activation Atlases" is available here:
https://distill.pub/2019/activation-atlas/

❤️ Pick up cool perks on our Patreon page: https://www.patreon.com/TwoMinutePapers

🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
313V, Alex Haro, Andrew Melnychuk, Angelos Evripiotis, Anthony Vdovitchenko, Brian Gilman, Bruno Brito, Bryan Learn, Christian Ahlin, Christoph Jadanowski, Claudio Fernandes, Dennis Abts, Eric Haddad, Eric Martel, Evan Breznyik, Geronimo Moralez, James Watt, Javier Bustamante, John De Witt, Kaiesh Vohra, Kasia Hayden, Kjartan Olason, Levente Szabo, Lorin Atzberger, Marcin Dukaczewski, Marten Rauschenberg, Maurits van Mastrigt, Michael Albrecht, Michael Jensen, Morten Punnerud Engelstad, Nader Shakerin, Owen Campbell-Moore, Owen Skarpness, Raul Araújo da Silva, Richard Reis, Rob Rowe, Robin Graham, Ryan Monsurate, Shawn Azman, Steef, Steve Messina, Sunil Kim, Thomas Krcmar, Torsten Reil,

## Транскрипт

### Segment 1 (00:00 - 04:00) []

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. When it comes to image classification tasks, in which the input is a photograph and the output is decision as to what is depicted in this photo, neural network-based learning solutions became more accurate than any other computer program we, humans could possibly write by hand. Because of that, the question naturally arises: what do these neural networks really do inside to make this happen? This article explores new ways to visualize the inner workings of these networks, and since it was published in the Distill journal, you can expect beautiful and interactive visualizations that you can also play with if you have a look in the video description. It is so good, I really hope that more modern journals like this appear in the near future. But back to our topic - wait a second, we already had several videos on neural network visualization before, so what is new here? Well, let’s see! First, we have looked at visualizations for individual neurons. This can be done by starting from a noisy image and add slight modifications to it in a way that makes a chosen neuron extremely excited. This results in these beautiful colored patterns. I absolutely love, love these patterns, however, this misses all the potential interactions between the neurons, of which there are quite many. With this, we have arrived to pairwise neuron activations, which sheds more light on how these neurons work together. Another one of those beautiful patterns. This is, of course, somewhat more informative: intuitively, if visualizing individual neurons was equivalent to looking at a sad little line, the pairwise interactions would be observing 2D slices in a space. However, we are still not seeing too much from this space of activations, and the even bigger issue is that this space is not our ordinary 3D space, but a high-dimensional one. Visualizing spatial activations gives us more information about these interactions between not two, but more neurons, which brings us closer to a full-blown visualization, however, this new Activation Atlas technique is able to provide us with even more extra knowledge. How? Well, you see here with the dots that it provides us a denser sampling of the most likely activations, and, this leads to a more complete bigger-picture view of the inner workings of the neural network. This is what it looks like if we run it on one image. It also provides us with way more extra value, because so far, we have only seen how the neural network reacts to one image, but this method can be extended to see its reaction to not one, but one million images! You can see an example of that here. What’s more, it can also unveil weaknesses in the neural network. For instance, have a look at this amazing example where the visualization uncovers that we can make this neural network misclassify a grey whale for a great white shark, and all we need to do is just brazenly put a baseball in this image. It is not a beautiful montage, is it? Well, that’s not a drawback, that’s exactly the point! No finesse is required, and the network is still fooled by this poorly-edited adversarial image. We can also trace paths in this atlas which reveal how the neural network decides whether one or multiple people are in an image, or how to tell a watery type terrain from a rocky cliff. Again, we have only scratched the surface here, and you can play with these visualizations yourself, so make sure to have a closer look at the paper through the link in the video description. You won’t regret it. Let me know in the comments section how it went! Thanks for watching and for your generous support, and I'll see you next time!