The Fundamentals of LLM Text Generation

ever wondered how models like chat GPT come up with their responses while all large language models are trained to predict the most likely next word based on previous context that's not exactly how they operate when they are applied to generating text in fact when chat GPT or other Advanced chatbots answer our prompts the way they select words involves a certain degree of Randomness or stochasticity just because llms are trained to predict the most probable words doesn't mean that they always choose the most likely option in practice so then how do l actually generate text how does Randomness play a role in making the responses more Dynamic and humanik and what techniques do they use to balance between predictable and creative outputs in this video we'll answer these questions and take a closer look at how Randomness helps model generate highquality text in the last part of the video we're going to focus on a more sophisticated approach based on information Theory where instead of simply looking for high likelihood sequences the goal is to find the optimal information content of language it may seem a little

counterintuitive but after model training it turn turns out that in text generation or decoding Randomness is essential to allow a language models to search the Infinite Space of possible text sequences in an effective way in other words we need to use Randomness to allow an llm to take some risks and generate text that feels less predictable more like something a human might say with no Randomness the model tends to stick to the most likely response here you see the same fact about cats repeated across different prompts while accurate the output feels rigid and robotic there's no variation and it's not capturing the range of possible responses that might make the conversation feel more natural by introducing a moderate amount of Randomness the model generates more varied responses the facts are still relevant but now the conversation feels a bit more humanlike however if we push Randomness too far the output can become too unpredictable and stray into nonsense while this might be entertaining in some context and specific applications it's usually not what we want from a language model so intuitively it's pretty clear why we want to introduce Randomness in the text generation process but how do we do this in practice let's start from the beginning the basic and most common stochastic decoding strategies that is text generation algorithms that use some Randomness When selecting words or tokens are called topk top p and temperature sampling so let's start by learning how these work topk sampling is

a simple stochastic decoding strategy that introduces Randomness by sampling the next token from the top K most probable choices where K is a fixed value more precisely it works as follows consider only the top K most probable next tokens for the current sequence normalize the probabilities of these K tokens to some to one creating a new truncated distribution randomly sample a token from this new distribution and append it to the current sequence and then repeat the steps until a termination condition is satisfied now there is a little problem with this method to understand what may go wrong let's take a look at the following two edge cases the first one is that of a fat tail distribution for the next token imagine the next token distribution is very spread out approaching a uniform distribution top Cas sampling would arbitrarily cut off many potentially interesting tokens possibly limiting the diversity of the generated text the second case is that of a P key distribution if the distri distribution is highly concentrated topk sampling might include unnecessary tokens when K is too large or exclude equally probable tokens when K is relatively small these examples highlight the main challenge with topk sampling choosing an optimal K value the ideal K May in fact vary depending on the context and the shape of the probability distribution at each step a fixed K value might be too restrictive in some cases and too permissive in others top P sampling or

nucleus sampling tries to solve these issues by offering a more Dynamic approach to token selection here's how it works let's fix a probability threshold P for example P equals 0. 7 the algorithm first selects the minimum number of tokens ordered by highest probability was cumulative probability meets or exceeds this threshold P it then normalizes the probabilities of these tokens to some to one just like before creating a new truncated distribution finally it samples a token from this new distribution and appends it to the current sequence and the steps are repeated notice the difference with the previous approach while topk always considers a fixed number of tokens top P adapts based on the probability distribution at each particular Step at each step the selected tokens form the nucleus from which the next token is sampled so going back to the two edge cases from before let's see how top P handles them more dynamically than top k for flat distributions where many tokens have similar low probabilities top P would sample from a larger pool of tokens preserving the diversity of possible choices for peaky distributions where a few tokens have high probabilities top would sample from fewer tokens focusing on the most likely options now while topy dynamically adjusts the pool of tokens it doesn't directly control the amount of Randomness in the generation process in fact modern llms often combine topy with another technique called temperature sampling which allows for more precise control over how conservative or creative the model's responses are so

what is temperature sampling the idea behind this method is to adjust directly the sharpness of the probability distribution this is achieved by introducing a parameter T the temperature in the softmax function which is used after the final layer of a transformer to compute the token probabilities the temperature parameter T is directly proportional to the amount of Randomness in the sampling process to understand how temperature affects the distribution let's consider three different cases when the temperature is set below one the probability distribution becomes more concentrated or pecky around the most likely tokens as T approaches zero the distribution becomes increasingly skewed the probabilities of the most likely tokens increase while those of less likely tokens decrease in the extreme case where T is very close to zero the sampling process approximates greedy search and becomes terministic setting T equals 1 leaves the original probability distribution unchanged this is often referred to as pure sampling in this setting the model samples from the full vocabulary according to the prior distribution note that unlike top K or top P sampling no tokens are excluded now in the high temperature regime with temperatures above one the probability is flattened and this increases Randomness this makes it more likely to choose less probable tokens as te increases towards very high values the distribution approaches a uniform distribution with each to can have an equal probability of being selected which would give us essentially a random War generator now this is an important

point to understand even though stochastic decoding strategies like top K top p and temperature sampling introduce a degree of unpredictability compared to deterministic methods they still fundamentally aim to maximize the likelihood of text sequences the added Randomness allows an llm to explore different trajectories in the token selection process and this translates into a more natural humanlike experience when we interact with them in a conversation but from a mathematical perspective the goal of these algorithms is still Guided by the numerical optimization of word sequences with high probability according to the model's estimates so the question is it actually the case that maximizing the likelihood of sequences up to adding some Randomness for diversity is the final and optimal way to generate high quality text with an llm or could there be other quantifiable features or metrics that can guide the model towards producing more expressive and creative sentences one could argue that in human communication there exists a balance between predictability and surprise not so much at the grammatical level with choice of less common words but rather at the level of information content of language this intuitive idea is the starting point for typical sampling an alternative approach to language model decoding that aims to apply principles from information Theory to text generation if natural language can be

cast as a process of information transmission it may be reasonable to assume that for effective communication this information should be encoded in text sequences at an optimal rate by viewing text generation through this lens typical sampling builds upon the following two principles one keep sentences short and information dense in other words we want to maximize the amount of information conveyed in a given message two avoid moments of high information density that is we want to avoid excessively complex or surprising sequences that can be too difficult for The Listener to process imagine if you were to ask me how electromagnetism works and my reply to that would be Max's equations now given that these principles obviously trade off against each other how can we guide anlm to find a sweet spot between them typical sampling proposes generating sentences with the expected information content given prior context to produce text that is informative enough to be engaging yet not so complex as to overwhelm the reader so how do they do this in practice in information Theory entropy provides a mathematical measure of the average information content in a probability distribution so the idea of typical sampling is to Target text with average entropy during decoding in their original paper the researchers mostly tested the validity of this approach on two specific NLP tasks abstractive summarization and story generation and the results look interesting compared to top and top P sampling typical sampling improves performance on this task while at the same time reducing repetitions in these two charts from the paper in particular we can see how repetition values for typical sampling in blue stay relatively low which is a good thing across different regimes for the hyper parameters relativ to nucleus and topk on a final note although the current implementation of typical sampling focuses on token level selection it would be interesting to see whether the same underlining principles could also be applied at different levels of text generation such as sentence level or even at the overall text planning for developers working on any llm based application where the creative aspects of the generated text are important I would say it is definitely a technique worth experimenting with and if you want to learn more about lm's emerging abilities check out Ryan's video on this topic all right see you in the next video as llms get bigger we find that there are critical scales at which they're suddenly able to complete tasks like translation summarization and code completion without being trained to do these tasks these abilities are called emergent abilities emerging at a particular scale but are these abilities truly emergent or they have a simpler explanation that may not be immediately obvious