$125B for Superintelligence? 3 Models Coming, Sutskever's Secret SSI, & Data Centers (in space)...

super intelligence just got valued at $5 billion or should I say safe super intelligence led by none other than the reclusive Ilia satova just got valued at $5 billion with this detail free tweet in just the last few hours and today we also got more news of Gemini 2 grock 3 and not just one but two new $125 billion data centers but all this new seem so desperate right but there is one theme through it all which is computing power even to the point of making data centers in space we as a species are making a giant bet that scaling up language models will unlock true artificial intelligence if the scaling hypothesis Believers are right it's coming soon if they're wrong this could all be viewed as the biggest waste of resources in human history but let's start with the news from just a couple of hours ago that Ilia SATA has raised 1

billion at a $5 billion valuation for his startup safe super intelligence there he is by the way in the middle he's definitely still alive and working on AI if you haven't heard of safe super intelligence don't worry it's actually only 3 months old but as mentioned earlier valued at $5 billion if only my 3-month-old side hustles got valued at $5 billion the world would be a happy place but no more seriously what will those $1 billion in funds be used for well it's that key theme you'll see throughout this video and probably throughout the next 5 years the funds will be used to acquire computing power Ilia s by the way sent a link to this Reuters article so we can pretty much trust its spoton with its details the Salient thing though about the company safe superintelligence is just how little detail they're giving out about what they're working on essentially though the key pitch is this give us a couple of years and then we're going to attempt in one shot to bring you super intelligence by the way they're going to do that with a team that's currently just 10 employees strong but before you dismiss them immediately they are backed by some Heavy Hitters like Sequoia capital and Daniel gross who is a co-founder up to this point though Ilia SATA who is clearly the key person in this Venture hasn't given any real detail about his approach but he did sprinkle some hints into this article satova said that he will approach scaling in a different way to his former employer which would be open Ai and he said everyone just says scaling hypothesis everyone neglects to ask what are we scaling but he went on some people can work really long hours and they'll just go down the same path faster it's not so much our style but if you do something different then it becomes possible for you to do something special before people get completely carried away though I do want to add a little bit of context to some of the claims that satsa has made before he co-led the super alignment team at openai which just over a year ago set themselves the deadline of aligning or making safe super intelligence within 4 years now I'm not against crazy ambition and Alignment is important but what actual progress has been made in that year and a bit yes I have read those blog posts put out by the former members of the team but it doesn't strike me as being a quarter of the way to aligning a super Intelligence on the grounded to fancyful scale it is definitely leaning toward the latter now naturally those

weren't the only grandiose Visions announced in the last 48 hours here is musk 2 days ago claiming to have the most powerful AI training system in the world he mentions it soon having around 200,000 h100 equivalents which are the gpus that go into training larg language models now your first thoughts might be that that's either an idle boast or that it's not really the computing power that matters it's how you use it but I do give comments like that and this one from July more Credence because of the capabilities of grock 2 the Frontier Model produced by musk's X AI team is genuinely a GPT 4 level competitor so it's worth paying attention at least to when he says that they are going to train the most powerful AI by every metric by December of this year and I will give one further hint that claim shouldn't be immediately dismissed and that's from this report yesterday in the information now first it did caveat that 100,000 chip cluster known as Colossus isn't fully operational apparently fewer than half of those chips are currently in operation largely because of constraints involving power or networking gear and more about power constraints in a moment but according to the information open AI CEO Sam Alman has told some Microsoft Executives that he is concerned that mus X aai could soon have more access to computing power than open aai does and remember open aai has access to the Beth Microsoft's compute power it's at this point though that you might be starting to wonder something is it all just about computing power isn't there supposed to be some secret Source at open aai or Google is it really just about raw computing power can we buy our way to Super intelligence well it's not for want of trying there

have been plenty of teams that have tried to build their own Foundation models only to realize that the key ingredient is scale computing power character AI even built up a loyal fan base and had some stars of the industry but couldn't make their own Foundation models work you may also recall efforts by Adept Ai and inflection which produced the pi chatbot the key person Personnel from those teams were snapped up by the likes of Google and Microsoft in short people are trying things as alternatives to Brute Force scaling but not that much is working currently sure you can eek out compute efficiencies and optimizations like GPT 40 the Orca series of models and the five family of models but nothing beats scaling and that might be why companies are betting everything on colossal new data centers

we're talking levels of in investment at the scale that could fund the research to cure entire diseases or perhaps fund the national budgets of mediumsized countries and you might have thought from the title of this video that there's a singular 125 billion supercomputer but there's actually two being planned I should of course add the caveat that it's according to the information via officials that would know about such Investments namely the source is the commissioner of Commerce Josh tigan who said that two separate companies approached him and the governor of North Dakota about building Mega AI data centers these would initially consume around 500 megaw to 1 gwatt of Power with plans to scale up to 5 or 10 gaw of power over several years those numbers of course to most of you will be complete gobbley so for context here is an excellent diagram from epoc AI 5 gaw of power allocated to a single training run would put the Power constraint just above this line here now given that it's expected that these other constraints would kick in at higher levels that would give us just over 10,000 times more compute available compared to that which was used for training GPT 4 now given the broad approximate Deltas between generations of gpts that would be the equivalent of a GPT 6 training run now yes I know that there are quote leaks like this one showing that GPT 5 might have between three and 5 trillion parameters but my adage for such leaks is don't trust and verify and also the number of parameters that goes into a model or the number of tweakable knobs if you like doesn't tell you automatically how much compute is used to train the model data is also a massive Factor there chinchilla scaling laws have long since been left behind and we are massively ramping up the amount of data for a given number of parameters but before we all get too

long in the numbers what am I actually saying here I'm saying that with the amount of money that's being spent and the amount of power that's being provisioned people are factoring in models up to the scale of something like GPT 6 by the way if you're skeptical that any progress has been made compare the performance of the original chat GPT in November of 2022 with Claude 3. 5 Sonet it's pretty night and day Claude 5. 5 Sonic would be quite interesting to behold now just to emphasize how much this scaling is a bet rather than a certainty in terms of the outcome it

will produce here again is Mark Zuckerberg it's one of the trickiest things in the world to plan around is when you have an exponential curve how long does it keep going for yeah and um I think it's likely enough that it will keep going that it is worth investing the um tens or you know 100 billion plus in building the infrastructure to assume if that kind of keeps going you're going to get some really amazing things that are just going to make amazing products but I don't think anyone in the industry can really tell you that it will continue scaling at that rate for sure now you

may have noticed that I've barely mentioned open AI successor language models and new verifier approaches those approaches previously labeled qar or strawberry throw in a bit of an xfactor over the coming months according to this article from again the information openai want to launch strawberry which was previously called qar and check out my video on that for what I think that might be they want to launch that within chat PT as soon as this fall interestingly the only hint they gave of its capabilities was that it could solve a New York Times connections word puzzle now since I read this article I have been trying plenty of those New York Times connections puzzles you've got to create four groups of four words that form a kind of logical set here though is the interesting part if you feeding the these puzzles to GPC 40 as text or as an image it usually can get one or two sets of four words but then what will happen is it'll get stuck and even if you prompt it to try different Arrangements of the remaining words it'll still predict the same things again and again so at the very least open AI must have pioneered a method to get language models out of their local Minima to get them to try different things instead of getting stuck in a rut how that plays out though in terms of true reasoning capability I'm going to wait to test it on SIMPLE bench to find out speaking of experiments though let me quickly introduce you to weave from the legendary weights and biases the sponsors of this video proper evaluations of language models are absolutely crucial as is clearly visualizing the differences between them you'd also ideally want your toolkit to be light weight so you could confidently and quickly iterate on your llm applications so in addition to their free courses and guides do check out weave using the link you can see on screen which will also of course be in the description now though for what some of you have been waiting for the news we

got yesterday that one startup is attempting to build data centers in space such is the need for Reliable energy to power these data centers we are resorting to putting the data centers into space this company Lumen orbit is a y combinator startup and they are aiming 4 gwatt scale their promo video even mentions a possible 5 gwatt data center which again if dedicated to pre-training would allow a GPT 6 style model but I must add in a quick caveat before we all go wild about data centers in space things like this have been tried before Microsoft tried to build data centers underwater the idea was that the C could help cool the data center and save on costs and even though it was described as largely a success apparently it didn't make sense from an operational or practical perspective the cost of Maintenance among other things was simply prohibitive now I don't know about you but it strikes me that the cost of maintaining things in space might be even more but as you may have

already deduced that's not exactly going to stop us reaching GPT 6 scale models why not well we do have the option of geographically Distributing the computers used to train the models in fact according to some sources Microsoft found they more or less had to do that apparently one Microsoft engineer on a GPT 6 training cluster project was asked why not just collocate the cluster in one region well they tried that he said but we can't put more than 100,000 h100s that's roughly the size of that Colossus project that we heard earlier from Elon Musk in a single state without bringing down the power grid as we saw from that Epoch analysis it's the power that's the constraining factor and also possibly water but more on that in a future video but if we distribute the training then the Clusters don't all have to be in the same place so it reduces that local power drain and that approach of distributed training to cut a long story short is where we seem to be heading according to a report out just the day from semi analysis Google open Ai and anthropic are already executing plans to expand their large model training from one site to multiple data center campuses and we already know by the way that Gemini Ultra 1. 0 was trained across multiple data centers so it can be done and before we get to more details on that there was this Hidden Gem in the third paragraph again this article was from today and it said Google's existing models lag behind open AI an anthropic because they are still catching up in terms of synthetic data reinforcement learning and model architecture but the impending release of Gemini 2 will change this semi analysis is a pretty reliable source so that's an interesting comment it seems like we will get Gemini 2 and Gro 3 within a few months and as we heard earlier the strawberry system from open aai in roughly the same time frame we will probably have to wait till next year for open ai's next Flagship though which is code named Orion but just for a moment I want you to forget all the names and the fruit and focus on one key detail if the key ingredients to the performance of language models is their scale we should find out that fact by the end of this year or to put it another way if scaling doesn't work up to the levels of grock 3 and Gemini 2 then what else will if the data centers are getting to the kind of scale where we need satellite pictures to assess how big they are and that doesn't produce true artificial intelligence then well do we have to rely on Ilia satova obviously I'm being cheeky but it's priced into the value I think of many of these companies just the possibility at least that scaling will yield super intelligence so if it doesn't you could expect a reflection of that in the form of a bubble bursting just quickly I can't resist pointing out that if you're in America then the fact that models will be increasingly interconnected across that continent will lead to a kind of interesting philosophical moment Microsoft will quite literally be in the famous words of their CEO above you around you beneath you now of course it almost goes without saying that there will be immense hardware issues in getting this all set up and running smoothly billions of man hours worth of problems to be solved for sure and that's why companies it seems are clamping up about how they're solving these hardware issues the publishing of methods has effectively stopped when open Ai and others tell the hardware industry about these issues they are very vague and high level so as not to reveal any of their dis distributed systems tricks to be clear semi analysis says these techniques are more important than model architecture as both can be thought of as compute efficiency here then is the central claim from semi analysis there is a camp that feels AI capabilities have stagnated ever since GPT 4's release I know many of you watching will feel that this is generally true but only because no one has been able to massively increase the amount of compute dedicated to a single model the word only there is of course an opinion rather than an established fact some of course believe that no amount of scaling will yield true reasoning or intelligence I have my thoughts but honestly I'm somewhat agnostic I genuinely want to know how these future models perform on my simple bench I go into a ton of detail about what I'm creating on my patreon which is called AI insiders oh and also just a couple of days ago I released this video on that Epoch AI research that reminds me actually there was one more thing from that research that I wanted to touch on in this video it came about

halfway through the 20,000 word report and it's right here I don't know why I picked out I just find it really quite pant and interesting to see what these beham moth companies will do because basically what they pledged this is Google Microsoft and Amazon to become carbon neutral by 2030 now what do you predict will happen if it turns out that the scaling hypothesis is true and that AI is immensely profitable and yet it requires this immense power and that will break these targets will they stick to their honorable pledges well we know what samman wants to do which is spend in the order of trillions and finance dozens of new chip factories this was from a separate information report but there was one quote that I found interesting in relation to it samman according to the CEO of tsmc which makes most of these chips the power Nvidia and everyone else he said samman was quote too aggressive for me to believe Remember by the way that pretty much all of this comes down to that Taiwanese company which is why by the way the whole tech industry is so nervous about China invading Taiwan anyway Sam wman is according to the tsmc CEO too aggressive for him to believe and maybe even these 125 billion data centers are also too aggressive only time will tell it's indubitable that a mountain has been identified and that the AI industry is trying to climb it whether they will or indeed whether they're even heading in the right direction only time will tell thank you as ever so much for watching all the way to the end I'm super grateful and have a wonderful day