# NVIDIA’s GameGAN AI Recreated PacMan! 👻

## Метаданные

- **Канал:** Two Minute Papers
- **YouTube:** https://www.youtube.com/watch?v=3UZzu4UQLcI
- **Дата:** 20.06.2020
- **Длительность:** 6:45
- **Просмотры:** 204,626
- **Источник:** https://ekstraktznaniy.ru/video/14111

## Описание

❤️ Check out Weights & Biases and sign up for a free demo here: https://www.wandb.com/papers 

The mentioned blog post is available here:
https://www.wandb.com/articles/visualizing-molecular-structure-with-weights-biases

📝 The paper "Learning to Simulate Dynamic Environments with GameGAN" is available here:
https://nv-tlabs.github.io/gameGAN/

Our paper with the neural renderer is available here:
https://users.cg.tuwien.ac.at/zsolnai/gfx/gaussian-material-synthesis/

🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Haro, Alex Paden, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bruno Mikuš, Bryan Learn, Christian Ahlin, Daniel Hasegan, Eric Haddad, Eric Martel, Gordon Child, Javier Bustamante, Lorin Atzberger, Lukas Biewald, Michael Albrecht, Nader S., Owen Campbell-Moore, Owen Skarpness, Robin Graham, Steef, Sunil Kim, Taras Bobrovytsky, Thomas Krcmar, Torsten Reil, Tybie Fitzhugh.
More info if you would lik

## Транскрипт

### Segment 1 (00:00 - 05:00) []

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Neural network-based learning methods are capable of wondrous things these days. They can do classification, which means that they look at an image, and the output is a decision, whether we see a dog or a cat, or a sentence that describes an image. In the case of DeepMind’s AI playing Atari games, the input is the video footage of the game, and the output is an action that decides what we do with our character next. In OpenAI’s amazing Jukebox paper, the input was a style of someone, a music genre, and lyrics, and the output was a waveform, or in other words, a song we can listen to. But, a few hundred episodes ago we covered a paper from 2015, where scientists at DeepMind asked the question, what if we would get these neural networks to output not sentences, decisions, waveforms or any of that sort… what if the output would be a computer program? Can we teach a neural network programming? I was convinced that the answer is no, until I saw these results. So what is happening here? The input is a scratch pad where we are performing multi-digit addition in front of the curious eyes of the neural network. And if it has looked for long enough, it was indeed able to produce a computer program that could, eventually, perform addition. It could also perform sorting, and would even be able to rotate the images of these cars into a target position. It was called a neural programmer-interpreter, and of course, it was slow and a bit inefficient, but no matter, because it could finally make something previously impossible possible. That is an amazing leap. So, why are we talking about this work from 2015? Well, apart from the fact that there are many amazing works that are timeless, and this is one of them, in this series, I always say, two more papers down the line, and it will be improved significantly. So here is the Two Minute Papers Moment of Truth. How has this area improved with this followup work? Let’s have a look at this paper from scientists at NVIDIA that implements a similar concept for computer games. So how is that even possible? Normally, if we wish to write a computer game, we first, envision the game in our mind, then, we sit down and do the programming. But this new paper does this completely differently. Now, hold on to your papers, because this is a neural network-based method, that first, looks at someone playing the game, and then, it is able to implement the game so that it not only looks like it, but it also behaves the same way to our keypresses. You see it at work here. Yes, this means that we can even play with it and it learns the internal rules of the game and the graphics, just by looking at some gameplay. Note that the key part here is that we are not doing any programming by hand - the entirety of the program is written by the AI. We don’t need access to the source code or the internal workings of the game, as long as we can just look at it, it can learn the rules. Everything truly behaves as expected, we can even pick up the capsule and eat the ghosts as well. This sounds like science fiction, and we are not nearly done yet! There are additional goodies. It has memory and uses it consistently, or, in other words, things don’t just happen arbitrarily. If we return to a state of the game that we visited before, it will remember to present us with very similar information. It also has an understanding of foreground and background, dynamic and static objects as well, so we can experiment with replacing these parts, thereby reskinning our games. It still needs quite a bit of data to perform all this as it has looked at approximately 120 hours of footage of the game being played, however, now, something is possible that was previously impossible. And of course, two more papers down the line, this will be improved significantly I am sure. I think this work is going to be one of those important milestones that remind us that many of the things that we had handcrafted methods for will, over time, be replaced with these learning algorithms. They already know the physics of fluids, or in other words, they are already capable of looking at videos of these simulations and learn the underlying physical laws, and they can demonstrate having learned general knowledge of the rules by being able to continue these simulations, even if we change the scene around quite a bit. In light transport research, we also have decades of progress in simulating how rays

### Segment 2 (05:00 - 06:00) [5:00]

of light interact with scenes and we can create these beautiful images. Parts of these algorithms, for instance, noise filtering are already taken over by AI-based techniques, and I can’t help but feel that a bigger tidal wave is coming. This tidal-wave will be an entirely AI-driven technique that will write the code for the entirety of the system. Sure the first ones will be limited, for instance, this is a neural renderer from one of our papers that is limited to this scene and lighting setup, but you know the saying, two more papers down the line, and it will be an order of magnitude better. I can’t wait to tell all about it to you with a video when this happens. Make sure to subscribe and hit the bell icon to not miss any followup works. Goodness, I love my job. What a time to be alive! Thanks for watching and for your generous support, and I'll see you next time!