How ChatGPT actually works

Since its release, the public has been playing with ChatGPT and seeing what it can do, but how does ChatGPT actually work? While the details of its inner workings have not been published, we can piece together its functioning principles from recent research. We are going to examine GPT-3's limitations and how they stem from its training process, before learning how Reinforcement Learning from Human Feedback (RLHF) works and understand how ChatGPT uses RLHF to overcome these issues. If you'd like to dive deeper or learn about at some of the limitations of this methodology, take a look at our comprehensive blog post: https://www.assemblyai.com/blog/how-chatgpt-actually-works/ 00:00 Overall look at how ChatGPT works 00:29 Misalignment issues 02:02 OpenAI's solution to misalignment 02:39 Step 1 - Creating the SFT model 03:27 Step 2 - Creating the Reward model 04:24 Step 3 - Finetuning the SFT model with the Reward model 04:56 Model evaluation 05:48 Further reading What is ChatGPT? ChatGPT is the latest language model from OpenAI and represents a significant improvement over its predecessor GPT-3. Similarly to many Large Language Models, ChatGPT is capable of generating text in a wide range of styles and for different purposes, but with remarkably greater precision, detail, and coherence. It represents the next generation in OpenAI's line of Large Language Models, and it is designed with a strong focus on interactive conversations. How was ChatGPT trained? The creators have used a combination of both Supervised Learning and Reinforcement Learning to fine-tune ChatGPT, but it is the Reinforcement Learning component specifically that makes ChatGPT unique. The creators use a particular technique called Reinforcement Learning from Human Feedback (RLHF), which uses human feedback in the training loop to minimize harmful, untruthful, and/or biased outputs. What is the difference GPT-3 vs. ChatGPT? Large Language Models, such as GPT-3, are trained on vast amounts of text data from the internet and are capable of generating human-like text, but they may not always produce output that is consistent with human expectations or desirable values. The alignment problem in Large Language Models typically manifests as: - Lack of helpfulness: not following the user's explicit instructions. - Hallucinations: model making up unexisting or wrong facts. - Lack of interpretability: it is difficult for humans to understand how the model arrived at a particular decision or prediction. - Generating biased or toxic output: a language model that is trained on biased/toxic data may reproduce that in its output, even if it was not explicitly instructed to do so. What is Reinforcement Learning from Human Feedback? The method overall consists of three distinct steps: - Supervised fine-tuning step: a pre-trained language model is fine-tuned on a relatively small amount of demonstration data curated by labelers, to learn a supervised policy (the SFT model) that generates outputs from a selected list of prompts. This represents the baseline model. - “Mimic human preferences” step: labelers are asked to vote on a relatively large number of the SFT model outputs, this way creating a new dataset consisting of comparison data. A new model is trained on this dataset. This is referred to as the reward model (RM). - Proximal Policy Optimization (PPO) step: the reward model is used to further fine-tune and improve the SFT model. The outcome of this step is the so-called policy model. ▬▬▬▬▬▬▬▬▬▬▬▬ CONNECT ▬▬▬▬▬▬▬▬▬▬▬▬ 🖥️ Website: https://www.assemblyai.com/?utm_source=youtube&utm_medium=referral&utm_campaign=yt_mis_34 🐦 Twitter: https://twitter.com/AssemblyAI 🦾 Discord: https://discord.gg/Cd8MyVJAXd ▶️ Subscribe: https://www.youtube.com/c/AssemblyAI?sub_confirmation=1 🔥 We're hiring! Check our open roles: https://www.assemblyai.com/careers ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ #MachineLearning #DeepLearning