Deep Reinforcement Terrain Learning | Two Minute Papers #67

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. This is a followup work to a technique we have talked about earlier. We have seen how different creatures learned to walk, and their movement patterns happened to be robust to slight variations in the terrain. In this work, we imagine these creatures as a collection of joints and links, typically around 20 links. Depending on what actions we choose for these individual body parts in time, we can construct movements such as walking, or leaping forward. However, this time, these creatures not only learn to walk, but they also monitor their surroundings and are also taught to cope with immense difficulties that arise from larger terrain differences. This means that they learn both on character features, like where the center of mass is and what the velocity of different body parts are, and terrain features, such as, what the displacement of the slope we're walking up on is or if there's a wall ahead of us. The used machinery to achieve this is deep reinforcement learning. It is therefore a combination of a deep neural network and a reinforcement learning algorithm. The neural network learns the correspondence between these states and output actions, and the reinforcement learner tries to guess which action will lead to a positive reward, which is typically measured as our progress on how far we got through the level. In this footage we can witness how a simple learning algorithm built from these two puzzle pieces can teach these creatures to modify their center of mass and adapt their movement to overcome more sophisticated obstacles, and, other kinds of advertisites. And please note that the technique still supports a variety of different creature setups. One important limitation of this technique is that it is restricted to 2D. This means that the characters can walk around not in a 3D world, but on a plane. A question whether we're shackled by the 2D-ness of the technique or if the results can be applied to 3D remains to be seen. I'd like to note that candidly discussing limitations is immensely important in research, and the most important thing is often not we can do at this moment, but the long-term potential of the technique, which, I think this work has in abundance. It's very clear that in this research area, enormous leaps are made year by year, and there's lots to be excited about. As more papers are published on this locomotion problem, the authors also discuss that it would be great to have a unified physics system and some error metrics so that we can measure these techniques against each other on equal footings. I feel that such a work would provide fertile grounds for more exploration in this area, and if I see more papers akin to this one, I'll be a happy man. Thanks for watching, and for your generous support, and I'll see you next time!