So Google's Research Just Exposed OpenAI's Secrets (OpenAI o1-Exposed)

do you remember the new open ao1 model where the model thinks before it responds and is now at the level of a PhD well there's new research from Google deep mind that somewhat breaks down this method and shows that the ways we were scaling llms before might not have been the most optimal before we dive into the details let's take a step back and understand the landscape of large language models over the past few years llms like GPT 4 Claude 3. 5 Sonic and others have become incredibly powerful tools capable of generating humanlike text answering complex questions coding tutoring and even engaging in philosophical debates their widespread applications have set new benchmarks for AI capabilities however there's a catch as these models become more sophisticated they also become more resource intensive scaling up model parameters which is essentially making them larger and more complex requires enormous amounts of compute power that means higher costs more energy consumption and greater latency especially when you're deploying these mod models in real time or Edge environment and it's not just the infrastructure pre-training these massive models demands huge data sets and months of training time given these challenges it's clear that we need to think Beyond just making these models bigger this is where the idea of optimizing test time compute comes in so what we're going to take a look at is instead of training a model to be a jack of all trades by making it larger what if we could make a smaller model think longer or more effectively during inference this could revolutionize how we think about deploying AI in Practical settings where resources are limited but performance still matters test time compute versus model scaling to understand this we first need to Define what we mean by test time compute refers to the computational effort used by a model when it's generating outputs rather than during its training phase think of it as the difference between a student studying for an exam and actually taking it training is like the study phase where all the learning happens while test time computation is like the exam phase where that knowledge is put to use to answer questions or solve problems so why is test time compute important well as it stands most large language models like GPT 40 or Claude 3. 5 Sonet are designed to be incredibly powerful right out of the gate which means they need to be big

really big but here's the catch scaling these models to massive sizes has some pretty serious downsides first there's the cost more parameters mean more compute power which translates to higher costs for both training and inference and it's not just about the money it's also about energy consumption running these models requires vast amounts of electricity which isn't exactly great for the environment then there's the deployment challenge huge models are difficult to deploy especially in settings where computational resources are limited like on mobile devices or Edge servers given these challenges the question becomes can we get the same or even better performance without scaling up the model itself that's where optimizing test time compute comes in by allocating computational resources more efficiently during inference we can potentially boost a model's performance without needing to make it bigger the dominant strategy over the past few years has been relatively straightforward just make the models bigger this involves increasing the number of parameters in a model which essentially means adding more layers more neurons and more connections between them this method has proven effective no doubt it's why gpt3 with 175 billion parameters was significantly more powerful than gpt2 with only 1. 5

billion and it's why even larger models like GPT 4 or o continue to push the boundaries of what's possible with natural language processing more parameters generally mean a more capable model that can understand more context generate more coherent and nuanced responses and even perform better on a range of tasks however this bigger is better approach comes with significant costs training a model with hundreds of billions of parameters requires massive data sets sophisticated infrastructure and months of compute time on thousands of gpus not to mention the inference the actual usage of these models in real world applications also becomes computationally expensive every time you ask the model a question or prompt it to generate text it requires a lot of compute power which adds up quickly in production environments this is why companies like open Ai and Google are looking for smarter ways to achieve high performance without just throwing more compute and data at the problem now let's consider the trade-offs between these two approaches scaling model parameters versus optimizing test time compute on one hand scaling model parameters is a Brute Force approach it works but it's costly inefficient and has diminishing returns as models get larger imagine a graph showing compute cost on one axis and performance on the other as you increase model size the performance gains start to Plateau while the costs continue to soar upward not a

great return on investment on the other hand optimizing test time compute offers a more strategic alternative instead of relying on massive models we could deploy smaller more efficient models that you additional computation selectively during inference to improve their outputs think of it like a sprinter conserving energy until the final stretch and then giving it their all when it matters most however this approach isn't without its own challenges for example designing effective strategies to allocate compute during test time is a non-trivial task you need to decide when and how much extra compute to use based on the complexity of the problem at hand but the potential upside is significant you could achieve comparable performance to a much larger model using less computer lower costs and reduced energy consumption what does this all mean in practice the key takeaway here is that there's a balance to be struck in some cases adding more parameters might still be the best approach particularly for extremely complex tasks where Brute Force scale is necessary but in many other cases especially For Less complex tasks or when deploying models in resource constrained environments optimizing test time compute could be a GameChanger and that's exactly what this

deepmind research is exploring how to find that optimal balance and what techniques can help us get the most out of every compute cycle now that we've set the stage by understanding the problem of test time compute versus model scaling let's move on to some of the key Concepts introduced in this paper the researchers have developed two main mechanisms to scale up compute during the models usage phase what we call test time without needing to scale up the model itself the first mechanism is called verifier reward models now that might sound a bit technical so let's simplify it imagine you're taking a multiple choice test and after answering a question you have a friend who is a genius in that subject check your answer your friend doesn't just tell you if the answer is right or wrong they also help you figure out the steps that led to the right answer you could

then use this feedback to improve your next answer that's kind of what a verifier reward model does for large language model and so in technical terms a verifier is a separate model that evaluates or verifies the steps taken by the main language model when it tries to solve a problem instead of just generating an output and moving on the model searches through multiple possible outputs or answers and uses the verifier to find the best one the verifier acts like a filter scoring each option based on how good it is and then helping the model choose the best path forward this process-based approach meaning it evaluates each step in the process not just the final answer helps the model become more accurate by ensuring that every part of its reasoning is sound it's like having a built-in quality Checker that allows the model to revise and improve its answers dynamically in Practical terms this means a model doesn't have to be massive to be smart it just needs a good system to check its work by incorporating verifier reward models we can optimize how models use their compute during test time making them both faster and more accurate without needing to be enormous the second mechanism is known as adaptive response updating think of this like

playing a game of 20 questions if you've ever played you know that each question you ask changes based on the answers you get if you find out the answer is a fruit you stop asking if it's an animal similarly adaptive response updating is about allowing the model to adapt and refine its answers on the Fly based on what it learns as it go here's how it works when the model is asked a challenging question or given a complex task instead of just spitting out one answer it revises its response multiple times each time it does this it takes into account what it got right and wrong in the previous attempt this allows it to zero in on the correct answer more effectively in more technical terms this means that the model dynamically adjusts its response distribution at test time think of response distribution like I said of possible answers the model might give by adapting this distribution based on what it's learning in real time the model can improve its output without needing extra pre-training it's like having the ability to think harder or think smarter when the problem is tough rather than just rushing to a conclusion this approach is powerful because it turns the model from a static responder where it only gives you one answer into a more Dynamic thinker capable of adjusting its strategies based on the problem it faces and again this can be done without making the model itself bigger fig which is a game Cher for deploying these models in Practical real world scenarios now let's bring these

two concepts together with what the researchers call a compute optimal scaling strategy don't worry it sounds more complex than it is at its core compute optimal scaling is about being smart with how we use computing power instead of using a fixed amount of compute for every single problem this strategy allocates compute resources dynamically based on the difficulty of the task or prompt so for example imagine you're running a marathon you wouldn't Sprint the entire way you'd pace yourself you'd run faster in some sections and slow down in others based on the terrain similarly the compute optimal strategy does something like this for models if the model is given an easy problem it might not use much compute at all it can just Breeze through it but if the problem is tough the model will allocate more compute like running faster in a marathon to think more deeply use verifier models or make adaptive updates to find the best answer now how is this different from fixed computation strategies which is what most models use today well most traditional models use the same amount of compute power for every task no matter how easy or hard it's like running at the same speed for an entire Marathon whether you're going uphill or downhill pretty inefficient right compute optimal scaling on the other hand adjust based on need making it much more efficient by using compute adaptively models can maintain high performance across a variety of tasks without needing to be scaled up to gigantic sizes to truly understand the effectiveness of these new techniques for scaling test time compute deep Minds

researchers had to put them to the test using real world data and for this they chose a particularly challenging data set known as the math benchmark so what is the math benchmark imagine a collection of high school level math problems everything from algebra and geometry to calculus and combinatoric these aren't your standard math problems either they're specifically designed to test deep reasoning and problem solving skills which makes them a perfect challenge for large language models the idea is to see if a model can not only come up with the right answer but also understand the steps needed to get there this makes the math benchmark ideal for experiments focusing on refining answers and verifying steps which are the core goals of This research by using this data set the researchers could rigorously evaluate how well the proposed methods perform across a range of difficulty levels from relatively straightforward problems to those that require complex multi-step reasoning the choice of this Benchmark ensures that the findings are robust and applicable to real world tasks that demand strong logical and analytical skills next let's talk about the models themselves for This research the team

used Palm 2 models specifically fine-tuned versions known as palm 2 now Palm 2 or Pathways language model is one of Google's Cutting Edge language models known for its powerful natural language processing capabilities it's a great choice for this study because it already has a strong foundation in understanding and generating complex text which is crucial for solving math problems and verifying reasoning however for This research they didn't just use the off-the-shelf version of palm 2 they took things a step further by fine-tuning these models specifically for two key tasks revision and verification revision tasks this involves training the model to iteratively improve its own answers think of it like a student going through their homework and correcting Mistakes One Step at a Time verification task this is about checking each step in a solution to make sure it's accurate much like a teacher reviewing a student's work to provide feedback on every part of the process by fine-tuning Palm 2 in these specific ways the researchers created specialized versions of the model that are highly skilled at refining responses and verifying Solutions which are crucial abilities for optimizing test time compute now

that we've covered the models and data sets let's dig into the core techniques and approaches that were tested in this research the research has focused on three main areas fine-tuning revision models training process reward models prms for search methods and first up we have fine-tuning revision models the goal here was to teach the model how to revise its own answers iteratively think of it like teaching a student to self-correct their mistakes but here's the Big Catch the model isn't just correcting a single mistake and stopping it's trained to go back and keep improving its answer step by step until it gets it right so how did they do this the researchers used a process called supervised fine-tuning they created data sets of multi- turn rollouts where the model starts with an incorrect answer and iteratively improves it until it gets to the correct one but there were some challenges for one generating high quality training data for this kind of task is tough because the model needs to understand the context of previous answers to make better revision to handle this the re Searchers sampled multiple possible answers and then constructed training sequences that combined Incorrect and correct answers this way the model learns not just to retry but to revise intelligently using the context of what it got wrong previously and the result a model that doesn't just spit out a single answer but can think through and refine its responses like a careful student tackling a tough math problem next we have process reward models prms and adaptive search methods prms help the model verify each step of its reasoning process by predicting how correct each step is based on PR previous data without needing human input this is like solving a puzzle where the model gets automated hints on whether it's on the right path making the search for the correct answer more efficient and accurate instead of waiting until the end to see if it's right or wrong the model can adjust its steps in real time similar to having a guide that helps navigate each turn the research also

explores various search methods like best of n beam search and look ahead search which help the model find the best possible answers by trying different parts best of n is like taking multiple shots and picking the best one beams Search keeps multiple options open and prunes the less promising ones as it goes and look ahead search looks several steps ahead to avoid dead Ends by combining these search methods with prms the model can dynamically allocate computing power where it's needed most achieving better results with less computation and potentially outperforming much larger models this approach allows for smarter more efficient AI that can handle complex tasks without requiring enormous computational resources so taking a look at everything we can see that this strategy called compute optimal scaling adapts the amount of computation based on the difficulty of a task the results

show that using this method models can achieve similar or even better performance while using four times less computation compared to traditional methods in some cases a smaller model using this strategy can even outperform a model that is 14 times larger this approach is somewhat similar to open ai's recent 01 model release which also focuses on smarter compute usage open

ai's 01 model ranks in the 89th percentile on competitive programming problems places among the top 500 in the US on a high level math competition and exceeds human PhD level accuracy on scientific questions 01 improves with more compute both during training and at test time so where we look at things aheading both open Ai and Deep Mind demonstrate that by optimizing how and where computation is used whether during learning or when generating answers AI models can achieve high performance without needing to be excessively large

this allows for more efficient models that perform at or above the level of much bigger ones by being strategic about their computational power so previously the Paradigm where individuals thought that scale is all you need the vibe seems to be shifting away from this as we look to more efficient ways to get smarter models and I think that looking into the future this shows us that the future of AI is going to be an explosive one