NVIDIA’s New AI Trained For 5,000,000,000 Steps!

little virtual soldiers to camp  and teaches them how to fight. This is a follow up work to an absolutely  amazing paper called ASE. This is a previous   work with little AI soldiers that trained  for 10 years. 10 years? How? 10 years of   their in-game world that is, which is  fortunately when using a fast machine,   is only 10 days in our time. And they learned  a great deal. They went from a bunch of sorry   little recruits to battle-ready soldiers.   Except this chap. This chap goes like “Sir,   I’ve been training for two years, I’ve had  enough! And now, I shall leave… in style. ” In this new work, these little soldiers trained  for 5 billion training steps on a single graphics   card. It’s not a cheap one, but still, a single  graphics card. So, what do we get for these 5   billion training steps? Can this new one do  anything better than the previous one? Well,   first, it can learn from unlabeled motion  capture data. What is that? Well, a bunch   of dots moving that represents movements  of real humans that have been recorded.

Step number one. Basic training. Here, the  moving dots go in, and a neural network is   asked to create motions that are similar to it. So, are we done? Well, not quite. You see,   we wish to use this in video games, and this is  not controllable yet. Imagine a video game where   you press a button on the controller, and nothing  happens. So now comes step number two. Precision   training! Here, it now needs to perform the  movements, but also listen to our controls,

especially when we use the stick to steer it  in different directions. And step number three,

here we not only give it directions, but intuitive  commands as well, such as striking or running. So, all that is great, but what can it  really do now? Well, let’s see together. For instance, it can interpolate between two  movement types. This means that it works like a   gymnast who has perfected a difficult routine. It  can't jump straight from one move to another. It   needs to transition smoothly to maintain balance  and perform the routine successfully. Thus,   it first starts running, and then, look, it  gradually dreams up a smooth transition from   sprinting to crouching. That is a really  nice and believable transition. Love it. And now comes the best part. Hold  on to your papers Fellow Scholars,   because now you’ll see how it can do these  types of transitions on-demand. That means,   as a reaction to our button  mashing on the controller. So good! However, not even this technique is  perfect. It still has some issues to   be worked out. For instance, check this out. Now,   little AI, move towards the target.   Attack, great and now, celebrate!    Whoa! Careful! Moments like this show that this  work is still experimental. I don’t think you   will see this in a new game tomorrow, but this  is what research is about. Taking something   highly unstructured and using learning-based  techniques to create something useful from it. Now, wait a second, in the intro, I also said  that this is a follow up to the ASE paper. So   how does it compare to that one? Oh my. Are you  seeing what I am seeing? Because I am seeing two   excellent news here. Now, I hear you asking,  okay Doctor, what are the excellent news here?   Well, one, let’s start with the most obvious  one - the controllability of the simulation has   improved leaps and bounds. That is incredible.   And second, what is even more incredible,   this controllability did not come at the expense  of the diversity of the movesets. That is perhaps   the best part of the paper - you see, this is in  a class of neural network-based based techniques   that often suffers from a phenomenon called  mode collapse. What is mode collapse? Well,   imagine asking a gymnast to perform incredible  acrobatics, and this gymnast can do a variety   of things, but unfortunately, just keeps doing  the same thing over and over again. Cartwheels   all day every day. Imagine looking at  an olympic gold-medal athlete who just   keeps doing cartwheels. And, it is not  the case here. This is an athlete that   can do everything. I absolutely love  this. Bravo! What a time to be alive! And finally, some fantastic news for the end -  the source code of this project is available.    I would like to send a big thank you to  the scientists on this project making it   available for all of us, free of charge.   And for now, let the experiments begin! Thanks for watching and for your generous  support, and I'll see you next time!