Grok-2 Actually Out, But What If It Were 10,000x the Size?

36 hours ago the biggest version of grock 2 went online but as of recording there is no paper no model card nothing but a Twitter chatbot to test in other words there's no paper to Gro or understand grock 2 but we can still take the release of grock 2 as an opportunity to discuss whether large language models are developing internal models of the world Epoch AI just yesterday put out this paper on how much scaling we might realistically get by 2030 and what exactly will happen to the internet itself in the meantime the first grock 2

which was announced a week earlier in a blog post where it said it outperformed Claude 3. 5 Sonic and GPT 4 Turbo but wasn't released for testing the only things you could play about with were grock 2 mini a smaller language model and flux the image generating model which is completely separate from xai are of course the creators of the grock family of models now while I found it somewhat newsworthy and intriguing that you could generate mostly unfiltered images using flux I was more curious about the capabilities of grock 2 itself and if you're wondering about those videos in the intro I actually generated the images using ideogram 2 which was released yesterday I find that it's slightly better at generating text than the flux model host posted by grock I then fed those images into Runway gen 3 Alpha along with a prompt of course and generated a 10-second video it's a really fun workflow and yes not quite photorealistic yet but I'll get to that again later in the video but back to the capabilities of grock 2 and they've shown its performance on some traditional llm benchmarks here in this table notice though that they cheekily hid the performance of Claude 3. 5 Sonic on the right you have to scroll a little bit I'm not trying to downplay Gro 2 its performance is pretty impressive the biggest version of grock 2 scored second only to Claude 3. 5 Sonic in the Google proof science Q& A Benchmark and second again to Claude 3. 5 Sonic in the mlu Pro think of that like the mlu 57 subject Knowledge Test minus most of the noise and mistakes and on one math benchmark it actually scored the highest math Vista now any of you who have been watching the channel know that I'm developing my own Benchmark called Simple bench and two senior figures at some of the companies on the right have actually reached out to help me and I'm very grateful for that but of course properly testing grock 2 on simple bench requires the API and I don't yet have access to it now obviously I was too impatient to wait for that so I did Test grock 2 on a set of questions that I used that is not found in the actual Benchmark and honestly grock 2's performance was pretty good mostly in line with the benchmarks at some point of course I will do a full introduction video to simple bench but for now those of you who don't know it tests basic reasoning can you map out some basic cause and effect in space and time if it's not found in your training data humans of course can scoring over 90% in simple bench but llms generally struggle Hard grock 2 though does pass my vibe check and comes out with some pretty well-reasoned answers in many cases it does though still get wrong quite a few questions that Claude 3. 5 Sonic gets right so I think it will fall behind that particular model thanks to one notorious Jailbreaker we now likely know the system prompt for grock 2 that's the message that's fed to the model before it sees your message it's to take inspiration from the guide from Hitchhiker's Guide to the Galaxy and apparently it has to be reminded that it does not have access to internal x/ Twitter data and systems ultimately it goal it seems is to be maximally truthful what many of you will be looking out for is breakthrough performance not yet another model at GPT 4's level for that though we may have to wait a few months to the next generation of models at 10 times the scale of GPT 4 before we leave grock though it seems worth noting the seemingly inoble Trend toward ubiquitous fake images on the internet and honestly I doubt that that's mainly going to come from flux housed within grock although Twitter is where those images might spread I'm actually looking at Google their new pixel 9 phone can quote re imagine

images like this one now having a cockroach you can imagine a restaurant that someone doesn't like and suddenly they can post an image to Trip Advisor with that cockroach now you could actually verify that person with a grudge actually went to that restaurant so how would this be taken down and if you don't think those images would make people click or react well it's already happening on YouTube and of course this applies to videos just as much as it does to images although we don't quite yet have realtime photo realism now you can let me know if you disagree but it feels like we might only be months or at most a year or two from realtime photo realism so you literally wouldn't be able to trust that the person that you're speaking to on Zoom actually looks like they appear to do now I get that one answer is just use common sense and don't trust anything you see online this strikes me as some more isolating that we each have to figure out what's

real in this world there's no sense of shared reality or maybe we need technology to solve some of the ills of technology I was casually reading this 63 page paper from a few days ago and it does strike me as a plausible route to solving some of these challenges forget World coin or fingerprints we might be able to use what's called zero knowledge proofs to provide personhood credentials now if you don't know anything about zero knowledge proofs I've linked a brilliant wired video explaining it but in short this paper did make me somewhat more optimistic that there is at least hope that the Internet won't completely

devolve into madness and then of course there is the good kind of Madness The Madness of creativity Unleashed by tools like cling ideogram and flux here's 20 seconds of biller squinting Mad Max Muppet Style now just five days ago the CEO of Google deep mine Demis Aris said that they were working on a way to trace the original image or text from the training data that led to a particular output and then based on the fraction of the output that came from that Source they could pay the original creators but looking at an output like Mad Max Muppets that just strikes me as an almost impossible task and if you thought it was only us that could get creative listen to gp4 from open aai mimic the voice of the user who is speaking to it do this just for the sake of doing it I think it's really important that's such a pure and admirable approach rather than by recognition or Acclaim it's refreshing to hear that kind of perspective especially in such a Cutting Edge field no and I'm not driven by impact either although if there is impact that's great it's just like imagine being on the edge of the Earth you know just because you could be that's what it feels like to me I just want to be in the space where it's all happening talk about a weird failure mode and why on Earth does it scream no before doing the imitation does this justify the delay to the advanced voice mode from open AI or would you not freak out if it started to imitate your voice most people watching won't really care

how the model speaks to them it's about whether the model is as intelligent as them or as it commonly known is the model generally intelligent and on that point you don't exactly get a clear message from these Labs working on AGI on the one hand last week Demis aabis said that AGI is still underhyped I think it's still underhyped or perhaps underappreciated still even now what's going to happen when we get to AGI and post AGI I still don't feel like um that's people have quite understood how enormous that's going to be and therefore the sort of responsibility of that so it's sort of both really I think it's a little bit overhyped in the in the near term and this is why I think we should take much more note of actions and results rather than predictions and words when a model for example gets better than human performance on my uncontaminated simple bench I will take that as much more of an indicator than a press release or blog post if you want to learn more about the inner workings of simple bench and how I might soon be working with some senior figures to make it go viral do sign up to AI Insiders on my patreon I personally message each and every new member and we have live Regional networking on I think now six continents I'm also always on the lookout for people with Discord or moderating experience because we have hundreds of amazing members with Incredible professional backgrounds I personally can't always think of the best ways to help people connect but one

thing that grock 2 gp4 and many other models like it are definitely missing is scale how much scaling we might realistically get by 2030 and assuming the companies are still willing to fund it the tldr is about 10,000 times the scale of GPT 4 there are numerous bottlenecks to scaling mentioned in the paper but the most constraining are data scarcity chip production capacity and actual power constraints even the most constraining of those bottlenecks still leaves room for models 10,000 times the compute of GPT 4 and I know that seems like an abstract number but you can really feel each 10x increase in data and parameters of a model for example llama 270 billion parameters scores around the level of gpc4 Mini on my simple bench llama 3 405 billion parameters not just had more parameters but was trained on far more data scores around the level of GPT 4 and Claw 3 Opus the obvious question of course is what about a model with a 100

times more parameters trained on data would it feel like a breakthrough or just an incremental Improvement for reference by the way GPT 4 is around 10,000 times the size of gpt2 which can only just about output coherent text for me though it's not just about blindly training on more data or naively expecting scale to solve everything we can't all just draw straight lines on a graph like Leopold Ashen braner does we have to figure out as this paper aspires to do what models are developing coherent internal World models if that's the case then scaled up models won't just quote no more they will have a much richer World model and just feel more intelligent these MIT researchers were trying to find out if language models only rely on Surface statistical correlations as some people think to put it simply if they only look at statistical correlations no amount of scale is going to yield a step change in performance but if they can infer hidden functions that X will cause y they can start to figure out the world more concretely when given these inputs from a puzzle they were also given the programmatic instructions and the resulting outputs they were then tested with only inputs and outputs and asked to predict what program had caused those outputs the experimenters wanted to see if the language model had built a Mini World model and was following the moves as they went along as you might expect it's quite complicated to probe where the language models are developing those kind of causal models and so there was a follow-up paper exploring just that paper's conclusion was that language models are indeed learning latent or hidden Concepts they referen other papers showing that language models perform entity State tracking over the course of simple stories and also reference the famous orell paper about a board game that I talked about in my cors era course obviously we're simplifying here but what they found was that after training on enough data with enough scale in this case over a million random puzzles they found that the model spontaneously developed its own conception of the underlying simulation think of that like a very small incipient World model at the start of these experiments they go on the language model generated random instructions that didn't work think gpt2 by the time we completed training our language model generated correct instructions at the rate of 92. 4% and sometimes if I'm being honest I feel for language models trained on trillions of tokens of internet data they will probably have far richer internal models if non-fiction wasn't so mixed with fiction on the internet sometimes I think we don't necessarily need a new architecture but a data labeling Revolution things like simple bench make clear that if there is a model in current llms it's pretty fragile but that doesn't mean it has to be that way ultimately we simply don't know yet whether llms can even in theory develop enough of a world model to eventually count as an AGI or do they need to will they simply serve as the interface for an AG for example translating our verbal and typed requests into inputs for separate World simulators or will their most common function be for convincing deep fakes let me know what you think and as always have a wonderful day