# OpenAI GPT-2: An Almost Too Good Text Generator!

## Метаданные

- **Канал:** Two Minute Papers
- **YouTube:** https://www.youtube.com/watch?v=8ypnLjwpzK8
- **Дата:** 16.04.2019
- **Длительность:** 7:33
- **Просмотры:** 155,844
- **Источник:** https://ekstraktznaniy.ru/video/14328

## Описание

❤️ Support the show and pick up cool perks on our Patreon page: https://www.patreon.com/TwoMinutePapers

📝 The paper "Better Language Models and Their Implications" is available here:
https://openai.com/blog/better-language-models/

GPT-2 Reddit bot:
https://old.reddit.com/r/MachineLearning/comments/b3zlha/p_openais_gpt2based_reddit_bot_is_live/

Criticism:
https://medium.com/@lowe.ryan.t/openais-gpt-2-the-model-the-hype-and-the-controversy-1109f4bfd5e8?sk=bc319cebc22fe0459574544828c84c6d

The Bitter Lesson video:
https://www.youtube.com/watch?v=wEgq6sT1uq8

🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
313V, Alex Haro, Andrew Melnychuk, Angelos Evripiotis, Anthony Vdovitchenko, Brian Gilman, Bruno Brito, Bryan Learn, Christian Ahlin, Christoph Jadanowski, Claudio Fernandes, Dennis Abts, Eric Haddad, Eric Martel, Evan Breznyik, Geronimo Moralez, Javier Bustamante, John De Witt, Kaiesh Vohra, Kasia Hayden, Kjartan Olason, Levente Szabo, Lo

## Транскрипт

### <Untitled Chapter 1> []

dear fellow scholars this is two minute papers with károly fajir this is an incredible paper from open AI in which the goal is to teach an AI to read a piece of text and perform common natural language processing operations on it for instance answering questions completing text reading comprehension summarization and more and not only that but additionally the AI has to be able to perform these tasks with as little supervision as possible this means that we seek to unleash the algorithm that they call GPT to read the internet and learn the intricacies of our language by itself to perform this of course we need a lot of training data and here the AI reads 40 gigabytes of Internet text which is 40 gigs of non binary plaintext data which is a stupendously large amount of text it is always hard to put these big numbers in context so as an example to Train similar text completion algorithms AI people typically reach out to a text file containing every significant work of Shakespeare himself and this file is approximately 5 megabytes so the 40 gigabytes basically means an amount of text that is 8,000 times the size of Shakespeare's works that's a lot of text and now let's have a look at how it

### Text Completion [1:28]

fares with the text completion part this part was written by a human quoting in a shocking finding scientists discovered a herd of unicorns living in a remote previously unexplored valley in the end these mountains even more surprising to the researchers was the fact that the unicorns spoke perfect English and the a I continued the text the following way quoting a short snippet of it the scientist named the population after their distinctive horn of its unicorn these four horned silver white unicorns were previously unknown to science whoa now note that this is clearly not perfect if there is even such a thing as a pair continuation and it took ten tries which means that the algorithm was run ten times and the best result was cherry picked and recorded here and despite all of these this is a truly incredible result especially given that the algorithm learns on its own after giving it a piece of text it can also answer questions in a quiet competent manner worry not later in this video I will show you more of these examples and likely talk over them so if you are curious feel free to pause the video while you read the prompts in their completions the validation part of the

### Validation [2:45]

paper reveals that this method is able to achieve state-of-the-art results on several language modeling tasks and you can see here that we still shouldn't expect it to match a human in terms of reading comprehension which is the question-answering test more on that in a moment so there are plenty of natural language processing algorithms out there that can perform some of these tasks in fact some articles already stated that there is not much new here it's just the same problem but stated in a more general manner and with more compute AHA it is not the first time that this happens remember our video by the name the bitter lesson I've put a link to it in the video description but in case you missed it let me quote her Richard Sutton address this situation the bitter lesson is based on the historical observations that won AI researchers have often tried to build knowledge into their agents - this always helps in the short term and is personally satisfying to the researcher but three in the long run it plateaus and even inhibits further progress and for breakthrough progress eventually arise by an opposing approach based on scaling computation by search in learning the eventual success is tinged with bitterness and often incompletely digested because its success over a favored human centric approach so what is the big lesson here why is GPT 2 so interesting well big lesson number one is this is one of the clearer cases of what the quote was talking about where we can do a whole lot given a lot of data and compute power and we don't need to insert too much additional knowledge into our algorithms and lesson number two as a result this algorithm becomes quite general so it can perform more tasks than most other techniques this is an amazing value proposition I will also add that not every learning technique scales well when we add more compute in fact you can see here yourself that even GPT two plateaus on the summarization task making sure that these learning algorithms scale well is a great contribution in and of itself and should not be taken for granted there has been a fair bit of discussion on whether open a I should publish the entirety of this model they opted to release a smaller part of the source code and noted that they are aware that the full model could be used for nefarious purposes why did they do this what is the matter with everyone having an AI with a subhuman level reading comprehension well so far we have only talked about quality but another key part is quantity and boy are these learning methods superhuman in terms of quantity just imagine that they can write articles with a chosen topic and sentiment all day long and much quicker than human beings also note that the blueprint of the algorithm is described in the paper and a top-tier research group is expected to be able to reproduce it so does one release the full source code and models or not this is a quite difficult question we need to keep publishing both papers and source code to advance science but we also have to find new ways to do it in an ethical manner this needs more discussion and would definitely be worthy of a conference style meeting or more there is so much to talk about and so far we have really only scratched the surface so make sure to have a look in the video description I left a link to the paper and some more super interesting reading materials for you make sure to check them out also just a quick comment on why this video came so late off the paper has appeared since there were a lot of feelings and intense discussion on whether the algorithm should be published or not I was looking to wait until the dust settles and there is enough information out there to create a sufficiently informed video for you this of course means that we are late to the party and missed out on a whole lot of views and revenue but that's ok in fact that's what we'll keep doing going forward to make sure you get the highest quality information that I can provide if you have enjoyed this episode and would like to help us please

### Supporting Us on Patreon [7:10]

consider supporting us on patreon remember our motto a dollar a month is almost nothing but it keeps the papers coming and there are hundreds of papers on my reading list as always we are available through patreon. com slash two minute papers and the link is also available in the video description thanks for watching and for your generous support and I'll see you next time