# OpenAI’s GPT-2 Is Now Available - It Is Wise as a Scholar! 🎓 ## Метаданные - **Канал:** Two Minute Papers - **YouTube:** https://www.youtube.com/watch?v=0OtZ8dUFxXA - **Дата:** 01.10.2019 - **Длительность:** 5:24 - **Просмотры:** 95,992 ## Описание ❤️ Check out Weights & Biases here and sign up for a free demo: https://www.wandb.com/papers Weights & Biases blog post (the notebook is available too!) - https://www.wandb.com/articles/visualize-xgboost-in-one-line - https://colab.research.google.com/drive/1SPludDkpAPonmdDdRlIJ7d5jB2FwVfZn#scrollTo=kszahG2PBN5R Try GPT-2 yourself here and post your results in the comments if you've found anything interesting. https://talktotransformer.com/ Check out this GPT-2 implementation too (thanks Robert Miles for the link!) - write something, then tab, enter, tab, enter and so on: https://transformer.huggingface.co/doc/gpt2-large OpenAI's post: https://openai.com/blog/gpt-2-6-month-follow-up/ Tweet source: https://twitter.com/gdm3000/status/1151469462614368256 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible: Alex Haro, Andrew Melnychuk, Angelos Evripiotis, Anthony Vdovitchenko, Brian Gilman, Bruno Brito, Bryan Learn, Christian Ahlin, Christoph Jadanowski, Claudio Fernandes, Daniel Hasegan, Dennis Abts, Eric Haddad, Eric Martel, Evan Breznyik, Geronimo Moralez, James Watt, Javier Bustamante, John De Witt, Kaiesh Vohra, Kasia Hayden, Kjartan Olason, Levente Szabo, Lorin Atzberger, Lukas Biewald, Marcin Dukaczewski, Marten Rauschenberg, Matthias Jost,, Maurits van Mastrigt, Michael Albrecht, Michael Jensen, Nader Shakerin, Owen Campbell-Moore, Owen Skarpness, Raul Araújo da Silva, Rob Rowe, Robin Graham, Ryan Monsurate, Shawn Azman, Steef, Steve Messina, Sunil Kim, Taras Bobrovytsky, Thomas Krcmar, Torsten Reil. https://www.patreon.com/TwoMinutePapers Splash screen/thumbnail design: Felícia Fehér - http://felicia.hu Károly Zsolnai-Fehér's links: Instagram: https://www.instagram.com/twominutepapers/ Twitter: https://twitter.com/karoly_zsolnai Web: https://cg.tuwien.ac.at/~zsolnai/ #OpenAI #GPT2 #GPT3 ## Содержание ### [0:00](https://www.youtube.com/watch?v=0OtZ8dUFxXA) Segment 1 (00:00 - 05:00) Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. OpenAI GPT-2 is a learning-based technique which can perform common natural language processing operations, for instance, answering questions, completing text, reading comprehension, summarization, and more. What is absolutely amazing about this technique is that it is able to perform these tasks with as little supervision as possible. This means that we unleash the algorithm that they call GPT-2 to read the internet, and it will learn our language by itself. To perform this, of course, we need a lot of training data, and here, the AI reads 40 gigabytes of internet text, which is 40 gigs of non-binary plaintext data, which is a stupendously large amount of text. You see here an example of how it fares with text completion, and the results were absolutely amazing. And, not only that, because, hold on to your papers, we can now try it ourselves and my results show that it can be wise as a scholar, and it can also cheat like nobody’s business. So, let’s try something simple here. Well, first, as everyone is talking about job security, I’d like to give it the start of this Two Minute Papers script and see if I am about to be replaced. Let’s see! It continues the intro with “The history of Eastern Europe since 1989 has been an interesting and complex story spanning four decades. On one side you have a regime of Western-backed dictators from the mid-1990s” and so on. Very scholarly indeed. However, I think I might be safe …for now. And now, something that I wanted to try since the inception of GPT-2, which is, feed it a snippet of something that I said about Wavelet Turbulence, which is one of the best papers ever written, and have the AI finish it. I tried to frame it as an exam question and was not disappointed be the results. So the text is, “There was a similar technique by the name Wavelet Turbulence, which is”, I hope you see what I did there…and look what it did! “There was a similar technique by the name Wavelet Turbulence, which is not at all what we are talking about, but I'll let you decide whether it is worth your time to read it. ” Wow! GPT-2 has been caught cheating red handed. I love it. Let’s continue what it had to say: “Anyway, my aim here is to describe how we use a fluid simulation to simulate smoke and liquid effects using just a few simple equations: Solving a few functions gives us a set of derivatives of the particle trajectories, that is equations that describe how and where the particles will come and go across the liquid in the simulation. For simplicity (as a good teacher of high-school students should know) we can use a simple equation, which simulates two particles that collide. ” I am stunned. It recognized that we are talking about fluid simulations, which is already remarkable, but it went much further - The completion is not bad at all and is not only coherent, on topic, but has quite a bit of truth to it. I will have to rethink my previous claim about my job security. The even crazier thing is that the size of this model is about 750 million parameters, which is only half of the size of the original full model, which is expected to be even better. I put a link to this website in the video description for your pleasure, make sure to play with it, this is mad fun. And, GPT-2 will also see so many applications that we cannot even fathom yet. For instance, here you can see that one can train it on many source code files on GitHub and it will be able to complete the code that we write on the fly. Now, nobody should think of this as GPT-2 writing programs for us, this is, of course, unlikely, however, it will ease the process for novice and experts users alike. If you have any other novel applications in mind, make sure to leave a comment below. For now, bravo OpenAI, and a big thank you for Daniel King and the Hugging Face company for this super convenient public implementation. Let the experiments begin! Thanks for watching and for your generous support, and I'll see you next time! --- *Источник: https://ekstraktznaniy.ru/video/14245*