OpenAI Safety Gym: A Safe Place For AIs To Learn 💪

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. Reinforcement learning is a technique in the field of machine learning to learn how to navigate in a labyrinth, play a video game, or to teach a digital creature to walk. Usually, we are interested in a series of actions that are in some sense, optimal in a given environment. Despite the fact that many enormous tomes exist to discuss the mathematical details, the intuition behind the algorithm itself is remarkably simple. Choose an action, and if you get rewarded for it, try to find out which series of actions led to this and keep doing it. If the rewards are not coming, try something else. The reward can be, for instance, our score in a computer game or how far our digital creature could walk. Approximately a 300 episodes ago, OpenAI published one of their first major works by the name Gym, where anyone could submit their solutions and compete against each other on the same games. It was like Disneyworld for reinforcement learning researchers. A moment ago, I noted that in reinforcement learning, if the rewards are not coming we have to try something else. Hmm... is that so? Because there are cases where trying crazy new actions is downright dangerous. For instance, imagine that during the training of this robot arm, initially, it would try random actions and start flailing about, where it may damage itself, some other equipment, or even worse, humans may come to harm. Here you see an amusing example of DeepMind’s reinforcement learning agent from 2017 that liked to engage in similar flailing activities. So, what could be a possible solution for this? Well, have a look at this new work from OpenAI by the name Safety Gym. In this paper, they introduce what they call the constrained reinforcement learning formulation, in which these agents can be discouraged from performing actions that are deemed potentially dangerous in an environment. You can see an example here where the AI has to navigate through these environments and achieve a task, such as reaching the green goal signs, push buttons, or move a box around to a prescribed position. The constrained part comes in whenever some sort of safety violation happens, which are, in this environment, collisions with the boxes or blue regions. All of these events are highlighted with this red sphere and a good learning algorithm should be instructed to try to avoid these. The goal of this project is that in the future, for reinforcement learning algorithms, not only the efficiency, but the safety scores should also be measured. This way, a self-driving AI would be incentivized to not only drive recklessly to the finish line, but respect our safety standards along the journey as well. While noting that clearly, self-driving cars may be achieved with other kinds of algorithms, many of which have been in the works for years, there are many additional applications for this work: for instance, the paper discusses the case of incentivizing recommender systems to not show psychologically harmful content to its users, or to make sure that a medical question answering system does not mislead us with false information. This episode has been supported by Linode. Linode is the world’s largest independent cloud computing provider. They offer you virtual servers that make it easy and affordable to host your own app, site, project, or anything else in the cloud. Whether you’re a Linux expert or just starting to tinker with your own code, Linode will be useful for you. A few episodes ago, we played with an implementation of OpenAI’s GPT-2 where our excited viewers accidentally overloaded the system. With Linode's load balancing technology, and instances ranging from shared nanodes all the way up to dedicated GPUs you don't have to worry about your project being overloaded. To get 20 dollars of free credit, make sure to head over to  linode. com/papers  and sign up today using the promo code “papers20”. Our thanks to Linode for supporting the series and helping us make better videos for you. Thanks for watching and for your generous support, and I'll see you next time!