DeepMind’s AI Trained For 5 Years... But Why?

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today we are going to see how DeepMind’s  incredible AI teaches these virtual   characters to move, play football/soccer  and so much more. Now when reading this   paper I expected that they would learn the  basics, but what happened afterwards, I did   not expect at all. They learned some absolutely  amazing tricks and even more surprising things,   and yes, we are going to look at all of  them, but first, the humble beginnings. Now, this is a physics-based game, so to move,  these AI agents need to use their joints and   produce just the right kind of forces and torque  on them to create sophisticated movements and   amazing plays. So, let’s see. Oof. Unfortunately,  we are seeing none of that here. Why? Well, of course, because they haven’t  had enough time to learn yet. So,   are these just the humble beginnings, or  is this it? Is this the limit of their AI? Not quite! But first, they need to learn to move.   How? Well, scientists at DeepMind say by imitating   motion capture data. In other words, this tries  to copy how real humans move. This is not bad, at   least it can now perform some basic movements, so  that’s great. However, that is not nearly enough,   look. It cannot control the ball yet. Not even  close. But once again, let’s come back 5 days   later and now, can it pull it off? Now, hold  on to your papers, and…oh yes! Yes it can,   and with flying colors! It can now follow a  moving target, dribble, kick, and so much more. Now, get this, we will put these agents  into a training camp for no less than 5   years and see how they do. We started out  like this, and now. Holy mother of papers,   these little AIs can play! And they can do  so much more than just moving around. Look,   they can do these little tricks where the  ball bounces back from an invisible wall,   and not only that, but this player tries to pass  to the other AI this way, and will they score?    Oh yes! Fantastic. I have to say, these are  now pretty good players. Good job, little AI! However, wait a second, what about the 5 years  part? Did scientists at DeepMind have to wait   for 5 years for this paper and hope that something  good comes out of it? Well, not quite! You see,   in real life, one second passes exactly in  one second. That is not new information to   anyone. However, if we have a quick computer,  in a simulation, one second in the game can   be simulated in a matter of milliseconds in  real life, thereby speeding up time itself.    Not our time, of course, but the time that the  AI lives in. So this 5-year training camp only   took 3 days in real life. That’s not that much.   Hm. Are you thinking what I am thinking? Yes,   if it’s so quick, let’s run it  for even longer. See what happens. They ran it for 50 days, and let’s see the  level of plays these can now do. And when   I saw these results, I almost fell off the  chair. They have become even better. Let’s   see what they have learned. Now, they not  only can think a few steps ahead, but they   are even anticipating the behavior of their  teammates and position themselves accordingly. They also learned to use body to  body contact to their advantage.    High kicks are also being performed to catch the  defenders off-guard, and I absolutely loved how   they developed these quick turns to get away  with the ball. This is incredible. I love it. And they can even play their way out of  really difficult situations. Look at this one.    Oh boy, the defending red player would have to  clear this ball without scoring an own goal,   so it would have to kick the ball in this  direction, and the margin of error is razor   thin. So far, good thinking, but can it  pull this off? Wow. The saves they are capable of are  also outstanding. And as we don’t   really have a referee around, these  AIs haven’t exactly been encouraged

to be gentle with each other,  so this can happen too. Ouch. And here is one of my favorite moments, after  a little altercation, the red player falls,   and it almost seems like throwing a  little tantrum. This is super fun,   especially that he was the one who  started it. I am sure some of you   Fellow Scholars will point out that this  is not unlike some real football players. And of course, this little AI appears so injured  it cannot get up, but later when it is really   needed to be there to defend, look! A miracle  happened! And of course, once again, this guy   just cannot help it. This is one of the most  fun papers I’ve read in a while. So cool. However, we are experienced  Fellow Scholars over here,   so we would like to know so much  more than what we’ve seen here. So,   let’s look under the hood and see some  more about what they have really learned. These are the curves that I am looking  for, yes, and I see that their Elo   ratings increase over time. This means that  as they learn more, they get better. That is   very reassuring, however, I still haven’t seen  what my heart desires yet, so let’s continue. Now we’re talking! More detailed data. This is  so cool to see, look. Over time, they can get up   quicker, on average, they use their joints better  to run faster, and yes! There we go. Division of   labor. Yes yes! Wow. That is what my heart  desires. This is where we struck gold. This is   the graph that shows that these two players  learned not only to move properly, but how to   work together as a team. This is also reflected in  their passing frequency, which is increasing over   time. But the division of labor, that is where the  real magic happens. You see, this is one of those   elusive qualities that is quite difficult to learn  properly, often even for humans, and it really   shows here too. How? Well, look, this is one of  the very few cases in the paper that does not show   steady linear growth. At times, it even worsens  over time. After one day of training, the agents   know just enough to score a goal alone, and thus  they become quite selfish. We all know that guy,   don’t we? But, over time, they realize that as a  team, they can perform even better. Wow. What an   incredible piece of work. And this is the moment  where I got goosebumps when reading this paper. I   love moments like this and I am so happy to share  it with you. Remember, this is where we started,   and this is where we ended up, and all this  through the power of modern learning algorithms. Yes, this is Two Minute Papers, the corner  of the internet where we look at a blue   curve and make happy noises. So good. If  you enjoyed this, make sure to subscribe   and perhaps even hit the bell icon if you  wish to see more amazing papers like this.    So, what do you think? What would you use  this for? Let me know in the comments below! Thanks for watching and for your generous  support, and I'll see you next time!