DeepMind Made A Superhuman AI For 57 Atari Games! 🕹

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Between 2013 and 2015, DeepMind worked on an incredible learning algorithm by the name Deep Reinforcement Learning. This technique looked at the pixels of the game, was given a controller and played much like a human would… with the exception that it learned to play some Atari games on a superhuman level. I have tried to train it a few years ago and would like to invite you for a marvelous journey to see what happened. When it starts learning to play an old game, Atari breakout, at first, the algorithm loses all of its lives without any signs of intelligent action. If we wait a bit, it becomes better at playing the game, roughly matching the skill level of an adept player. But here's the catch, if we wait for longer, we get something absolutely spectacular. Over time, it learns to play like a pro, and finds out that the best way to win the game is digging a tunnel through the bricks and hit them from behind. This technique is combination of a neural network that processes the visual data that we see on the screen, and a reinforcement learner that comes up with the gameplay-related decisions. This is an amazing algorithm, a true breakthrough in AI research. However, it had its own issues. For instance, it did not do well on Montezuma’s revenge or Pitfall because these games require more long-term planning. Believe it or not, the solution in a followup work was to infuse these agents with a very human-like property… curiosity. That agent was able to do much, much better at these games…and then got addicted to the TV. But that’s a different story. Note that this has been remedied since. And believe it or not, as impossible as it may sound, all of this has been improved significantly. This new work is called Agent57, and it plays better than humans, on all 57 Atari games. Absolute insanity. Let’s have a look at it in action and then in a moment, I’ll try to explain how it does what it does. You see Agent57 doing really well at the Solaris game here. This space battle game is one of the most impressive games on the Atari as it contains 16 quadrants, 48 sectors, space battles, warp mechanics, pirate ships, fuel management, and much more, you name it. This game is not only quite complex, but it also is a credit assignment nightmare for an AI to play. This credit assignment problem means that it can happen that we choose an action, and we only win or lose hundreds of actions later, leaving us with no idea as to which of our actions led to this win or loss, thus, making it difficult to learn from our actions. This Solaris game is a credit assignment nightmare. Let me try to bring this point to life by talking about school. In school, when we take an exam, we hand it in, and the teacher gives us feedback for every single one of our solutions and tells us whether we were correct or not. We know exactly where we did well, and what we need to practice to do better next time. Clear, simple, easy. Solaris, on the other hand, not so much! If this were a school project, the Solaris game would be a brutal, merciless teacher. Would you like to know your grade? No grades, but he tells you that you failed. Well, that’s weird, okay. Where did we fail? He won’t say. What should we do better next time to improve? You’ll figure it out bucko! Also, we wrote this exam 10 weeks ago, why do we only get to know about the results now? No answer. I think in this case, we can conclude that this would be a challenging learning environment even for a motivated human, so just imagine how hard it is for an AI! Hopefully this puts into perspective how incredible it is that Agent57 performs well on this game. It truly looks like science fiction. To understand what Agent57 adds to this, it was given something called a meta-controller that can decide when to prioritize short and long term planning. On the short term, we typically have mechanical challenges, like avoiding a skull in Montezuma’s revenge or dodging the shots of an enemy ship in Solaris. The long term part is also necessary to explore new parts of the game, and have a good strategic plan to eventually win the game. This is great because this new technique can now deal with the brutal and merciless teacher who we just introduced. Alternatively, this agent can be thought of someone who has a motivation to explore the

game and do well at mechanical tasks at the same time and can also prioritize these tasks. With this, for the first time, scientists at DeepMind found a learning algorithm that exceeds human performance on all 57 Atari games. And please, do not forget about the fact that DeepMind tries to solve general intelligence, and then, use general intelligence to solve everything else. This is their holy grail. In other words, they are seeking an algorithm that can learn by itself and achieve human-like performance on a wide variety of tasks. There is still plenty to do, but, we are now one step closer to that. If you learn only one thing from this video, let it be the fact that there are not 57 different methods, but one general learning algorithm that plays 57 games better than humans. What a time to be alive! I would like to show you a short message from a few days ago that melted my heart. This I got from Nathan, who has been inspired by these incredible works and he decided to turn his life around, and go back to study more. I love my job, and reading messages like this is one of the absolute best parts of it. Congratulations Nathan and note that you can take this inspiration and greatness can materialize in every aspect of life, not only in computer graphics or machine learning research. Good luck! If you're a researcher or a startup looking for cheap GPU compute to run these algorithms, check out Lambda GPU Cloud. I've talked about Lambda's GPU workstations in other videos and am happy to tell you that they're offering GPU cloud services as well. The Lambda GPU Cloud can train Imagenet to 93% accuracy for less than $19! Lambda's web-based IDE lets you easily access your instance right in your browser. And finally, hold on to your papers, because the Lambda GPU Cloud costs less than half of AWS and Azure. Make sure to go to lambdalabs. com/papers and sign up for one of their amazing GPU instances today. Our thanks to Lambda for helping us make better videos for you. Thanks for watching and for your generous support, and I'll see you next time!