Microsoft's New Game AI: How Is This Good?

Scientists at Microsoft just found the  holy grail: and that is, of course,   finding a way to play video games at work and even  get paid for it. So, what is going on here? Well,   they have a new AI system that looks at  footage of people playing video games,   and then, when we give it a new situation,  it tries to predict what is about to happen   next. You could call it creating  games, sort of. More on that later. Now, this is not easy at all, even for  these lower resolution videos. Initially,   when you start training, you get something  like this. Note that the resolution of these   videos is pretty low, but, it is enough  so that we can already see the bad news:   the footage quickly strays away into  something completely different. Now,   let’s train it for a bit longer. Look, now  this is a lot better, we are in the same game,   the same zone, jumping is working. We are  getting there. And, if we train for even longer,   wow, look, we just notice now that the previous  one was quite wrong because this should have been   not jumping, but flying. Also, interaction with  objects like this power cell works correctly too. Okay, great, now our question is, what’s  the point? What can we use this for? Dear   Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Well, for instance, you can kind of play  with it. You get put into a virtual world,   and when you use your controller to choose  different directions, it generates the rest of   the level and interactions for you. They call  it modeling a wide range of human behaviors,   I call it going into three possible directions.   Although it is really good progress towards   AI-based game making, I just want to make sure  not to oversell this idea as it stands now. And now comes perhaps the most interesting  part: we can not only play with these games,   once again, sort of, but we can also change them.   For instance, we can put another new object here,   and the AI will make sure that it stays there,  and that it is interactable. We can also add a   new character to the mix, and if we specify it, it  will also do something. Now that is really cool. Just imagine just having  one promo image for a game,   and this can make it come alive. That is  fantastic. But as this is at the moment,   would I play with it? Likely not. I’ll fire  up Civilization or No Man’s Sky instead,   thank you very much. However, there are two things  this might already be really good for. First,   rapid prototyping. You can imagine  what your game would look like,   but ultimately you have to put in a lot of work  to make it happen. But with this, not so much,   you can very quickly get a feel of the game and  then decide whether you wish to work on it or not. Two, if you already have a game, you  always want to tinker with something.    You have your what if questions. What if  we add a barrier here? How would the game   change? Did we break something? Can we still  finish the game at all? These questions can   now be explored super quickly  and perhaps automatically. Loving it. Now, two more questions remain: one, this is  not the only way of using an AI to create a   game. There is something that you can do right  now. You can ask a chatbot like the new Claude   3. 7 to write the code for you for a snake game  that is self aware and even does unexpected   things. Like escaping the matrix, and more. And it gets better, you know, the Two Minute   Papers special, cloth simulation, not a problem.   I’ve never seen anything like this done by an   AI from scratch. What a beauty! And, Grok 3 can  write a simple flying simulator with multiplayer   support. It gets a bit choppy, you know, network  code optimization is not trivial if you have   hundreds of players flying around in real time,  but it is kind of possible. That is mind blowing! So, this is two opposing schools  for AI game making: bottom up,   starting from zero and writing the code. Or,  top down, learning from video and generating   new ones. Which one is the way to go? Well,  both are in their early days, but The First   Law of Papers teaches us not to underestimate  the pace of progress in AI research. You see,   just a couple years ago, this was the  state of AI video generation, and today,   this. Whoa. And I expect that we are going to  look at this pixelated game footage with a smile,   and reminisce about the good old times when  we couldn’t do it in full HD or even higher   resolution. Except that I think it will happen not  two years from now. Possibly in less than a year. Now, second question: can this create  fundamentally new games, something that

goes beyond the kinds of games it was trained  on? I would like to take a guess. Based on what   AI can do in other areas, for instance, it can  solve new, previously unseen problems in the   mathematical olympiad. With that in mind, I would  say, fundamentally new games is a question mark   for the next few years, but interesting new  variants of already existing concepts? That   will surely be possible two more papers down  the line. This is the power of the papers. I am really looking forward to a future where we  can all make our own games like Civilization or   No Man’s Sky, right at home, just by talking, and  all this in real time. What a time to be alive! This is an incredible paper in the prestigious  Nature journal, open access, free for everyone,   thank you and huge congratulations. Once again,  very few people are talking about this paper,   so I really wanted to show it to you. Subscribe  and hit the bell if you wish to see more,   and as always, let me know in the comments  below what you Fellow Scholars think.