Googles ALPHACODE-2 Just SHOCKED The ENTIRE INDUSTRY! Full Breakdown + Technical Report

a quick announcement I did recently make a patreon and that's for those of you who want to just give some extra support because sometimes the onean operation can be a bit daunting so if you do want to join the Discord join the patreon get featured in some videos for your comments vote on what videos I should do next don't forget to check out the link in the description otherwise enjoy the rest of the video so Alpha code 2 by Gemini this was something that I wanted to cover in yesterday's video but I'm sure it deserved its own video because it was a bombshell that was dropped that unfortunately went under the radar and kind of slipped by in terms of people missing this crazy announcement so what we have here guys is a very impressive coding system and this is built on top of a fine-tune version of Gemini fro so Alpha code 2 is essentially built on a fine tuned version of Gemini Pro and what you need to understand about Alpha code 2 is that Alpha code 2 is an entirely separate system and currently isn't available in bod so as you know many people are using bod and looking at their Advanced coding capabilities but please understand that this is not Alpha code 2 as Alpha code 2 has not yet been released and they are working on doing that so even if right now you do think B's coding capabilities have increased please understand that you're not seeing the capabilities of alpha code 2 as it is a separate system so I want to quickly dive into the technical reports first page then I want to show you the actual demo from Google and then I'm going to go into some further details about Alpha code 2 that I think you should know the first paragraph of the alpha code 2 technical report actually houses some very fascinating stuff so it actually does mention the fact that their previous version of alpha code if you didn't know Alpha code was essentially the first AI system to perform at the level of the median competitor in competitive programming a difficult reasoning task involving Advanced maths logic and computer science and essentially what they've introduced is their second iteration of this model and essentially what they've introduced was Alpha code 2 their second iteration of this model and it's a new and enhanced system with massively improved performance powered by Gemini Alpha go 2 relies on the combination of powerful language models and a bespoke search and reranking mechanism which is really cool and we're going to get into that later and when evaluated on the same platform as the original Alpha code we found the alpha code 2 solved 1. 7 times more problems and performed better than 85% of competition participants so let's take a look at that Google demo and then I'm going to go back and do a deep dive into the paper because there's some really cool things that you should know about Alpha code 2 in terms of where Google is going to take the system we build Gemini from the ground up to be natively multimodel including something quite important for both of us programming code Gemini is able to consistently understand explain and generate code that is correct and well written in most programming languages that includes python Java C++ and go it substantially improves coding abilities over previous Palm two models from a benchmark around 200 programming functions in Python it consistently solves about 75% of them in the first try versus around 45% on Palm two if you allow Gemini to check and repair its own answers this number jumps to over 90% which is a huge step forward it can help you create and prototype new ideas in seconds let's give it a try I really like trains and if I wanted to create a transporting location web app I can simply ask and get a working prototype in less than a minute while the code isn't perfect it's really helpful to have a first draft Gemini on its own has the ability to transform software development as we understand it but it can also be deployed as a key component of more sophisticated systems Gemini is great at coding but we've been able to take it even further creating a specialized version that performs remarkably well at competitive programming now why do we care about competitive programming well it is one of the ultimate litmus tests of algorithmic coding abilities so we have thousands of talented programmers from all over the world that come together to compete and try to solve incredibly complex problems that require not only coding but also math and reasoning two years ago we presented Alpha code and it was the first AI system that could compete roughly at the level of the average human competitor today I'm delighted to introduce Alpha good 2 a new and enhanced system with massively improved performance powered by Gemini when we evaluate Alpha code 2 on the same platform as the original Alpha code we solve almost twice as many problems while Alpha code broke through the top half of human competitors on average we estimate that Alpha code 2

performs better than 85% of competition participants let's have a look at our model in action on one of the hardest problems that we faced and I say hard because in the original contest in which the problem appeared less than 0 2% of participants actually solved it the problem is quite difficult it's very abstract so I can't get into too many details but the basic gist of it is that we are tasked with Computing aggregate statistics that account for what appears to be an impossibly large amount of random arrays the really cool thing is that to solve it Alpha c 2 makes use of dynamic programming is an advanced algorithmic technique which basically simplifies a complicated Problem by breaking it down into easier sub problems again and what's really impressive is that not only alphaa 2 knows how to properly implement this strategy but also when and where to use it what the example shows us is that competitive programming is not just about implementation it's also about understanding maths computer science and indeed coding and that makes it an extremely hard reasoning task so it's not very surprising that up till now generally available large language models have scored very poorly on this Benchmark these models are really good at following instructions but Alpha code needs to do more than that it needs to show some level of understanding some level of reasoning designing of code Solutions before it can actually get to the actual implementation to solve the problem and it does all that on problems that it's never seen before another thing that is great about Alpha code is that it performs even better when it collaborates with human coders who can provide grounding basically developers can specify properties that the code samples have to obey and when we do that we see performance increase significantly we think of this kind of interaction between uh programmers and AIS as the future of programming where coders will not just give instructions but actually collaborate with highly capable AI models that can reason about their problems that can propose code designs and that can even help with the actual implementation Alpa 2 was built for competitive programming but we're already working on bringing some of its unique capabilities right into the General Gemini models as a first step towards making this new programming Paradigm available for everyone so that was a very nice comprehensive overview of the alpha code 2 capabilities and I found it really fascinating how well it does on coding and just how much it seems to understand the task due to the fact that it is on a fine Tred version of Gemini so one thing I also did find was the interesting overall system so essentially what we have here is the alpha code 2 system and it's quite different to how normal systems are done so essentially Alpha code 2 has a team of mini brains or model and each one is really good at coming up with different ways to solve the coding problems and essentially they all work together to create tons and tons of possible solutions in effect I'm pretty sure later on in the technical report they state that it is actually millions of possible solution solutions that they do create now essentially when they create many different of these Solutions in this sampling mechanism what they do next is they then essentially get rid of the bad ideas and this is of course the filtering mechanism so some of the solutions that they do come up with obviously won't work so they have a special Checker that throws out any ideas that don't actually solve the puzzle then of course we have a clustering SLG grouping algorithm where they notice which ideas are kind of similar and to make things easier they group the similar ideas together this way they don't waste time on ideas that are almost the same then of course what they do is they essentially for each group of ideas they then pick the very best one and they have a special way to score each idea to figure out which one is the most likely to solve each puzzle SL coding problem and overall what we have here is a very effective system so the one thing I wanted to expand a bit upon was their sampling system because I found out that this is where things do get interesting so essentially Alpha codes 2 sampling you can see that it says we generate up to a million code samples per problem using a randomized temperature parameter for each sample to encourage diversity so essentially they create 1 million different solutions for one single coding problem and of course to make sure that each solution is different from the last they use a special trick so essentially they adjust the robots in imagination to come up with a wide range

of ideas and of course this is done by using something called a randomized temperature parameter so think of it like a turning D that changes how creative or different each solution is then of course we have adding more details so they add and change small details about each solution based on things like how hard the puzzle is or what kind of puzzle is and that makes each solution even more unique then of course they have the fine tuned models or their brains essentially that are really good at coming up with the solutions and it uses all of them equally to get a good mix of ideas so although Alpha code used to think in both Python and C++ languages it now only uses C++ because it makes better quality Solutions in this language then of course we have the picking SL filtering and because they have so many solutions they have to pick the best ones it looks at all the solutions throws out the best ones and then ranks them to find the top 10 and in the end it chooses up the best top 10 solutions to try and solve this problem so it's a really interesting Deep dive to see exactly how the model thinks its entire process and how it gets from problem to solution now with that I did want to look at the future of alpha code because although the system was just released it is interesting to see where this could go now one of the problems that Alpha code does suffer from just like the other advanced systems like GPT 4 it does suffer from cost to run so it says our system requires a lot of trial and error and remains too costly to operate at scale further it relies heavily on being able to filter out obviously bad code samples so it seems that right now I don't think we're going to be getting Alpha code because it's too expensive to run and too costly to operate at the scale that they would need to if they wanted to deploy it worldwide and of course they also do state that a lot more remains to be done before we see systems that can reliably reach the performance of the best human coders so it seems like we are moving towards a future where coders aren't going to be replaced but more so we're going to get coding assistance in the form of AI rather than complete Replacements so one thing I did really want to include was of course the evaluation because it's really important to see how it scores up compared to Alpha code 1 so essentially alpha code 2 was tested on code forces the same platform that used the original Alpha code they chose 12 recent contests with over 8,000 participants from two difficulty levels division two the harder division one and two covering 77 problems in total so for each problem Alpha c 2 generated 1 million potential Solutions which were their candidates and then submitted up to 10 Solutions per problem following a specific selection process this continued until it found the correct solution or ran out of candidates and in terms of the performance results Alpha code 2 solved 43% of these competition problems this is almost double the performance of the original Alpha code which solved 25% in terms of competition rankings Alpha code 2 is estimated to be in the 85th percentile performing better than 85% of the participants in comparison with human competitors it ranks between expert and candidate Master categories on code forces and the estimated ranking does take into account a simulated time penalty in efficiency problem solving in some contestants Alpha code 2 performs better than 99. 5% of the participants and it was found to be 10,000 times more sample efficient compared to the original Alpha code and it requires about 100 samples to reach what Alpha code achieved with a million samples in addition it does seem like Alpha code 2 could get even better because in addition it does seem like Alpha code could actually get quite better because they do use Gemini Pro as their Foundation model and using this Foundation model led to significant increases in performance on two critical components of the system the policy models generating the code samples and the scoring model used to select them so essentially they basically say that we suspect using geminite Ultra as the foundation model instead with its improved coding and reasoning capabilities would lead to further improvements in the overall Alpha code 2 approach so now it's simply a waiting game to see whether or not Alpha code 2 is going to be made more efficiently so that it can scaled up and deployed globally or they're going to just simply improve on it like they did with Alpha go type systems and see if they can potentially get 100% on some of these benchmarks