# AI Plays Minecraft... After 33 Years of Training! 🤖

## Метаданные

- **Канал:** Two Minute Papers
- **YouTube:** https://www.youtube.com/watch?v=5LL6z1Ganbw
- **Дата:** 27.07.2022
- **Длительность:** 8:02
- **Просмотры:** 660,609

## Описание

❤️ If you wish to support us and watch these videos in early access, check this out:
- https://www.patreon.com/TwoMinutePapers

📝 The paper "MineDojo - Building Open-Ended Embodied Agents with Internet-Scale Knowledge" is available here:
https://minedojo.org/

🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Balfanz, Alex Haro, Andrew Melnychuk, Benji Rabhan, Bryan Learn, B Shang, Christian Ahlin, Eric Martel, Geronimo Moralez, Gordon Child, Ivo Galic, Jace O'Brien, Jack Lukic, John Le, Jonas, Jonathan, Kenneth Davis, Klaus Busse, Kyle Davis, Lorin Atzberger, Lukas Biewald, Matthew Allen Fisher, Michael Albrecht, Michael Tedder, Nevin Spoljaric, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Rajarshi Nigam, Ramsey Elbasheer, Steef, Taras Bobrovytsky, Ted Johnson, Thomas Krcmar, Timothy Sum Hon Mun, Torsten Reil, Tybie Fitzhugh, Ueli Gallizzi.
If you wish to appear here or pick up other perks, click here: https://www.patreon.com/TwoMinutePapers

Thumbnail background image credit: https://pixabay.com/images/id-2019147/
Thumbnail background design: Felícia Zsolnai-Fehér - http://felicia.hu

Chapters:
0:00 Minecraft
0:15 GANCraft
1:31 AI playing games
1:52 NVIDIA tries Minecraft
2:20 But how?
3:12 Can this really work?
3:32 Teaching an AI English
4:19 1 - Exploration
4:48 2 - Building a fence
5:03 3 - Getting a bucket of lava
5:18 4 - Building a portal
5:32 5 - Final boss time
6:02 Long time horizons
6:25 More results
7:09 Does this really work?

Károly Zsolnai-Fehér's links:
Instagram: https://www.instagram.com/twominutepapers/
Twitter: https://twitter.com/twominutepapers
Web: https://cg.tuwien.ac.at/~zsolnai/

#minecraft #minedojo #nvidia

## Содержание

### [0:00](https://www.youtube.com/watch?v=5LL6z1Ganbw) Minecraft

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today we are going to see whether NVIDIA’s  modern AI system they call MineDojo   can learn to play and build  things in Minecraft and more.

### [0:15](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=15s) GANCraft

In an earlier episode, we explored a previous  project that goes by the name GANCraft,   which performed world to world translation.   What is that? Well, simple, we give an AI a   very rough draft of a virtual world, and out  comes a much more detailed and beautiful one.    This almost seems like science fiction. Really,  just dream up something, and it makes it happen. And, it gets better, it created water,  it understands the concept of an island,   and it created a beautiful landscape,   also, with vegetation. Insanity. It even  seems to have some concept of reflections,   although they will need some extra work to get it  perfectly right. It even supported interpolation,   which means that we can create one landscape and  ask the AI to create a blend between different   styles. We just look at the output animations, and  pick the one that we like best. And, with this,   finally, we also have some artistic control over  the mood of the results. Absolutely amazing.

### [1:31](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=91s) AI playing games

But then they thought, why stop there? For  instance, DeepMind has a long history of   using AI systems to master all kinds of  games, from Chess, to Go, to StarCraft 2,   OpenAI has a DOTA2 AI project that is  able to challenge a world champion team,   so, NVIDIA thought, why not try their hands  at Minecraft? Minecraft is an open world game,

### [1:52](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=112s) NVIDIA tries Minecraft

one of the most played games in the world,  which is kind of like a virtual sandbox.    Here, you can build things, explore, but really,  do whatever you wish. It’s a sandbox after all. So scientists at NVIDIA thought, let’s train an AI  to play this game and see what happens. Okay, but

### [2:20](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=140s) But how?

how? Well, most AI systems of today require  lots and lots of training data. Wait a minute.    There is no shortage of that out there in the  internet, that’s for sure. Let me explain. We can have this little AI sit down and  watch hundreds of thousands of Minecraft   tutorial videos on Youtube, 33 years of footage  in total, my goodness, then, have it read over 7   thousand wiki pages, and then also becomes the  ultimate Reddit lurker. Wow. That makes a ton   of sense. Just think about it. It can learn  practical knowledge from the tutorial videos,   encyclopedic knowledge from the wiki, and  it can learn what makes the best creations   the best that are shared on reddit. That sounds  fantastic…on paper. But, there is so much to do

### [3:12](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=192s) Can this really work?

and to learn about this game, can an AI  really get an understanding of all this? Now,   hold on to your papers, and let’s see together  what this AI could learn from all this data!

### [3:32](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=212s) Teaching an AI English

But wait a minute! This is a gamer AI  that uses the controller to move around in   a virtual world. But how do we instruct it?   Do we need to speak robot? If not, well,   it doesn’t even understand English text. What do  we do with that? Well, remember OpenAI’s GPT-3,   which is a neural network model that  has read almost the entirety of the   internet. This one has proper English  knowledge, so, we plug that in, and bam,   now you can read that wikipedia, and it gets  better, because we can now even give it text   instructions. Now let’s see how well it  fares through 5 of my favorite examples. One, we can ask it to explore an ocean monument.   I love this one because the text description

### [4:19](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=259s) 1 - Exploration

is sufficiently vague. How could a machine  understand what exploring means? Well, humans do,   and this learns from humans as there must be  tons of tutorial videos out there on exploration.    And, it seems to me that it was able to  learn the concept correctly. Loving it. Two, it can encircle these llamas with a fence.

### [4:48](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=288s) 2 - Building a fence

That is excellent. It understands what object  it needs to use, where to move, where to look,   and that not even a tiny gap  is allowed. Very impressive.

### [5:03](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=303s) 3 - Getting a bucket of lava

Three, this will be a more dangerous quest.   Now scoop a bucket of lava. Oh my goodness.    Be careful there! Do not fall!   We are getting there, and…got it!

### [5:18](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=318s) 4 - Building a portal

Four, we can even ask it to build a nether  portal. It builds the correct portal   frame and, did it use the correct materials?   Does it work? Yes and yes! Good job little AI!

### [5:32](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=332s) 5 - Final boss time

And, five. It is final boss time. Literally.   Oh yes. Someone was brave enough to ask the AI   to fight an ender dragon, essentially  the final boss of the game,   and the AI was brave enough to try it.   Well, apparently, this is a limited example   as it does not appear to be charging at the AI,  but, the AI seems to know what we are looking for,   and what fighting entails in this  game. Starting this game from scratch

### [6:02](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=362s) Long time horizons

and building up everything to defeat such a  dragon takes a long time horizon, and will be   an excellent benchmark for the next AI one more  paper down the line. I’d love to it perform this   start to end. Make sure to subscribe, if such a  paper appears, I’ll be here to show it to you.

### [6:25](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=385s) More results

While we are looking at more examples of  what it could do, I have to note that we have   barely scratched the surface here.   For instance, it understands text,   yes, but if we attach a speech recognition AI  to this agent, we don’t even need to write.    It can essentially be a little virtual friend.   How cool is that? What a time to be alive! Okay, so this is MineDojo, an AI  agent that understands English,   and can learn to navigate these virtual worlds  so well, we can give it a task, and they would   execute it. And not just some simple ones, we  are talking a wide variety of complex tasks.

### [7:09](https://www.youtube.com/watch?v=5LL6z1Ganbw&t=429s) Does this really work?

Now, all this footage looks great,  but how do we know if this is really   performing these tasks correctly? Well, if  you have been holding on to your papers, now,   squeeze that paper, because they had an  experienced human evaluator look at these results,   and agreed with the AI’s solutions about 97%  of the time. Wow. That is an incredible result. And don’t forget, NVIDIA is amazing at  democratizing these works and putting them   into the hands of everyone, and this  one is an excellent example of that.    If you have some programming knowledge,  you can give it a try right now. So, does this get your mind going? What would you  use this for? Let me know in the comments below! Thanks for watching and for your generous  support, and I'll see you next time!

---
*Источник: https://ekstraktznaniy.ru/video/13503*