# GPT 4 Level Open Source in 2024..(Llama 3 Leaks and Mistral 2.0)

## Метаданные

- **Канал:** TheAIGRID
- **YouTube:** https://www.youtube.com/watch?v=uCkYoVb2kKw
- **Дата:** 18.01.2024
- **Длительность:** 22:01
- **Просмотры:** 18,654
- **Источник:** https://ekstraktznaniy.ru/video/14573

## Описание

💬 Access  GPT-4 ,Claude-2 and more - chat.forefront.ai/?ref=theaigrid
🎤 Use the best AI Voice Creator - elevenlabs.io/?from=partnerscott3908
✉️ Join Our Weekly Newsletter - https://mailchi.mp/6cff54ad7e2e/theaigrid
🐤 Follow us on Twitter https://twitter.com/TheAiGrid
🌐 Checkout Our website - https://theaigrid.com/

GPT 4 Level Open Source in 2024..(Llama 3 and Mistral 2.0)
https://twitter.com/soumithchintala/status/1671267150101721090?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1671267150101721090%7Ctwgr%5E2b0d647a742308920f1b32a9f0c96ba9878abd62%7Ctwcon%5Es1_&ref_url=https%3A%2F%2Fcdn.embedly.com%2Fwidgets%2Fmedia.html%3Ftype%3Dtext2Fhtmlkey%3Da19fcc184b9711e1b4764040d3dc5c07schema%3Dtwitterurl%3Dhttps3A%2F%2Ftwitter.com%2Fsoumithchintala%2Fstatus%2F1671267150101721090image%3Dhttps3A%2F%2Fi.embed.ly%2F1%2Fimage3Furl3Dhttps253A252F252Fabs.twimg.com252Ferrors252Flogo46x38.png26key3Da19fcc184b9711e1b4764040d3dc5c07 
https://twitter.com/NickADobos/status/1735045196424040928 
http

## Транскрипт

### Segment 1 (00:00 - 05:00) []

So, open source AI is fast approaching the level of GPT4, but will we be able to get that in the year of 2024? With some recent announcements from notable CEOs and many different companies, I think it's safe to say that 2024 might just be that year. And funnily enough, Sam Alman recently said in an interview that it's actually pretty impossible to catch up to GPT4, but it's the developers jobs to try. So let's take a look at what people are doing and just exactly how they're catching up to GPT4. Main things we do know about open source is of course our favorite models and some of the most popular ones include things like Llama and it's a family of large language models released by Meta's AI that are open source. But the one that steals the show is of course Mistl. Although Arthur Mench, the CEO of Mistral, declared on French radio that Mistral will release an open-source GBT4 level model in 2024. Now, some of you might be thinking, wait a minute, I haven't heard of this guy. It's not Meta's Llama. It's not OpenAI. It's not Google. Who are this team? So, for those of you who are eagle-eyed and who do pay attention in the AI space, you're going to know exactly who this is. So, Mistrol are a wrench AI startup that specializes in compute efficient, powerful, and useful AI models. The company focuses on challenging problems to make AI models more efficient, helpful, and trustworthy. And Mistrol is known for its strong research orientation and providing open models, which means they offer transparent access to their model weights, allowing for full customization by their users. Now the company's products include generative AI platforms and models for generation and embeddings. And one of their notable models is Mixtra which is reported to be six times faster than comparable models while matching or outperforming Llama 2 billion parameters on all benchmarks. Now Mixstral supports multiple languages, has natural coding abilities, and can handle sequences up to 32,000 in length. And Mistral AI provides access to its models through an API or allows users to deploy the models themselves under an Apache 2. 0 license. And of course, their first LLM model, Mistral 7B, is available for free download and use. And despite being free, it's not open source in the traditional sense as the data sets and weights are private. And the company's business model seems to revolve around providing a highly permissive license for their models while maintaining private development and funding. And essentially, this company, Mistral AI, has positions itself as a European alternative to larger AI companies like OpenAI with a civic-minded ethos and a focus on ethical AI practices. They aim to democratize access to advanced generative technology and mitigate societal risks with AI. Overall, essentially what we have here is a game-changing emerging player in the AI field, challenging larger companies by offering transparent, efficient, and powerful AI models and services with a particular emphasis on ethical practices and community engagement. One of the key things about Mistral is that their team is 22 employees, which is incredibly small considering the amount of things that they've done in the AI space. So you can see here it says despite its rise to unicorn status, Mistral AI remains a relatively small company with just 22 employees consisting of co-founder and CEO Arthur Mench. The rest of their names I honestly can't pronounce, but they do have experience at Meta and Google's Deep Mind. So definitely an efficient and a comprehensive team of accomplished AI engineers and researchers working there that have been able to disrupt the entire industry. And if we compare that to the likes of the behemoth that is OpenAI with 770 employees, it is no reason to why people are surprised that they're able to catch up to what this company is doing. But one thing that is interesting is that will OpenAI release some of the stuff that they've been working on this year. That is going to be something we do have to see. Now, in terms of actually comparing the models on certain benchmarks, one of the benchmarks that many people have been looking at where Mistral has been exceeding is the Arena ELO. Now, if you don't know what the Arena ELO is, I'm going to break it down for you. You know how you log on to Chat GPT and you have a general conversation? Well, the Arena ELO is a bit different. You log on to a similar system except every time you put in a message, you get two responses. And all you have to do between both responses is rate which one you think is better. Depending on which one is rated better, the one that ELO of that AI system essentially goes up a bit. Um, and

### Segment 2 (05:00 - 10:00) [5:00]

that's just a very, very simple explanation on how this entire, you know, ELO leaderboard is kind of done. And you can see the votes on this side right here. Now, of course, you can see GBT4 Turbo and GPT4 taking the top three spots, but interestingly enough, above Claude's models and above Google Gemini Pro and above GBT3. 5 Turbo, we can see Mistral Medium coming in at number four, which is honestly rather shocking considering this small AI team only has 22 employees. Yet they've managed to create a model that is pretty much on the level or above the level or some of these other large language model systems. And not only that, their mix 8* 7 billion parameters instruct version 0. 1 is also above Gemini Pro Claw 2. 1 which was recently released and GBT 3. 5 Turbo. So all of those things combined provide us with the sense that Mixtra and Mistral are pretty comprehensive models. And what's absolutely crazy about this is that these models guys are extraordinarily smaller than some of their competitions. So that means that what we do have here is a very innovative, fast company that is able to deploy models and is able to run them efficiently, make changes, open source them, and just really disrupt the entire industry in terms of what we thought was normal. And even if we look at some of these organizations, a lot of these organizations are ones that we already do know. OpenAI, Anthropic, and Google. And of course, Mistral is right there above some of these people. Now, of course, you might be thinking that, wait a minute, these aren't really objective benchmarks. They're just subjective rated by the users. But I think that is a very very important benchmark because one thing that does happen and is an issue, you know, as of recently is that what tends to happen is that sometimes people just fine-tune stuff quickly on evals to beat the high score. And although that is technically I guess you could say does beat it in terms of the eval score, it isn't the best way to assess models just based on objective data because it's going to be real people who are interacting with them um and not just testing the model based on certain things. So I do think that you know leaderboards like this are definitely very important. Now, the recent CEO did actually talk about, you know, raising €385 million. And this is crazy because at their recent funding round, raising €385 million is a huge amount that's going to be, you know, for, you know, training models, for more GPUs, for more server costs. And it goes to show that Mistra is going to be one of those companies that could really disrupt OpenAI's position. And there are some other things that I do also want to discuss, but take a look at this video clip which I actually translated with 11 Labs. In just a few months, we created models that uh so one of the things additionally that does really put things into the mix and really puts things to the test is of course the cost effectiveness of Mistral Medium. And by comparison, we can see that Mistral AI, their medium model, is nearly as good as GPT4. Of course, GPT4 is exceedingly good, but if it is nearly as good as GPT4, and it's at the fraction of the cost of GPT4, this is going to really disrupt the industry because one of the key things that, you know, is stopping GBT4 from being so crazy is the fact that there are rate limits on how much we can use GBT4. And of course, even on the chat, you know, even in the normal user interface, not just on the API rate limits, we have the standard ones where it's only like around 30 messages every 3 hours. And I remember when GPT4 was first released, it was around 25 messages every 4 hours or so. And that's not that much of a production, you know, decrease in terms of the amount of time that GBT4 has existed. So, it's pretty surprising based on how long we've had the model for the price not to come down yet. So maybe there's some kind of inefficiencies on GPT4's end that OpenAI simply haven't solved. And if you recently did watch an interview with, you know, Bill Gates, Sam Alman actually did talk about getting the cost effectiveness of this down because essentially they need to if it's to become, you know, scalable for applications and use cases and if people do actually want to use this on a day-to-day basis for many different things because if you're using an AI system and you love it and then every 3 hours you max out your messages, it's kind of not going to be that great because you're going to have to keep switching models and it's much easier to just use something that you can use cost effectively and constantly. So this shows us that you know being you know literally a fraction of a couple cents um is going to you know pose really big problems for OpenAI if they

### Segment 3 (10:00 - 15:00) [10:00]

don't get that cost down and if Mistral manages to keep you know encroaching on their lead. Now additionally what was interesting was their Mistl 8* 7 billion parameters. So this one was a cuttingedge AI model. So what was crazy about this one and why it took the industry by storm was essentially if we think of it like a highly specialized team of experts where each member is really good at handling specific types of problems. Usually in AI models like GBT each part of the model handles everything equally but in mixture you having a team and each person is an expert in a different field. That's what Mixtra does with its tasks. It has a special system which is the router that decides which expert should handle each piece of information. And the model is unique because it can select from eight different groups of these experts for each bit of information it processes. This selection is sparse now meaning it only chooses a few experts for each tasks overall making it more efficient. And this AI system is pretty crazy because it can have a 32,000 context length. It can also have multilingual text and because of how it's designed, it's great for tasks that require quick thinking which are fast inference related tasks and helpful and also helping to find information from large databases which is RA or retrieval augmented generation and it's also customizable meaning it can be trained for specific tasks or industries and the reason I brought this up by Mistl because the recent benchmarks on Mistral medium have been absolutely crazy but that same arch aritecture that we just talked about here was actually apparently the same architecture that was used in GPT4. So in this interview, George H. Hotz actually discusses the GPT4 architecture. And the reason I found it so fascinating and it wasn't just George Hortz that, you know, pretty much confirmed that this is how GPT4 works. Although we don't have an official statement from OpenAI, it's fascinating because it goes to show that now the cat might be out of the bag. And if this is how GPT4 is managed to be so effective, efficient and able to, you know, beat a lot of other, you know, AI systems in terms of benchmarks, then it does mean that these other AI companies now could realize this and of course train their models in the same way. Glombmed on. Um yeah, we could build. So like the biggest training clusters today, I know less about how GP4 was trained. I know some rough numbers on the weights and stuff, but uh llama trillion. Well, okay. So GBT4 is 220 billion in each head and then it's an eight-way mixture model. So mixture models are what you do when you're out of ideas. Um so you know it's a mixture model. Uh they just train the same model eight times and then they have some little trick. They actually do 16 inferences. But u no it's not like so the multimodality is just a vision model kind of glo glombmed on. I mean the multimodality is like obvious what it is too. You just put the vision model in the same token space as your language model. Oh did people think it was something else? No. The mixture has nothing to do with the vision or language aspect of it. It just has to do with well okay we can't really make models bigger than 220 billion parameters. Uh we want it to be better. Well how can we make it better? We can train it longer and okay we've actually already maxed that out. Uh getting diminishing returns there. Okay. Mixture of experts. Yeah. Mixture of experts. We'll train eight of them. Right. So all right. So you know you know the real truth is whenever a start whenever a company is secretive with the exception of Apple. Apple's the only except whenever a company is secretive it's because they're hiding something that's not that cool. People have this wrong idea over and over again that they think they're hiding it because it's really cool. It must be amazing. It's a trillion parameters. No, it's a little bigger than GPT3 and they did an eightway mixture of experts. Like, all right, dude. Anyone can spend eight times the money and get that. All right. Um, but yeah, so uh coming back to what I think is actually going to happen is yeah, people are going to train smaller models for longer and fine-tune them and find all these tricks, right? Like I you know I think uh opening I used to publish when they would publish stuff about how much better the training has gotten given the same holding compute comp it's gotten a lot from that other people also did confirm that GPT4 is a mixture of experts the co-founder of PyTorch at Meta reaffirmed that leak and he said I might have heard the same I guess info like this is passed around but nobody wants to say it out loud GPT4 8* 220 billion experts trained with different data/task distributions and 16 iter inference. I'm glad that Gioz said it out loud. Then of course additionally with that we do have this part of the article. Then we also do have this where it says what do all the tweets mean? Essentially GBT4 is not a large model but a you know union of smaller models sharing the expertise and each of these models is rumored to be 220 billion parameters. Now what's crazy is that like I said with that if that architecture is out there and other companies are going to be using this it means that we could eventually get systems that are on the level of GPT4. Now additionally there are some other things that do show us that this is

### Segment 4 (15:00 - 20:00) [15:00]

going to be the case. So now, crazily enough, as I was researching and making this video, you can see here that just literally like a couple of hours ago, Hermes 2 beats Mistro Instruct Mixture of Experts and becomes the now best open-source model and open source AI continues to make strides on a daily basis. The latest release from the news team beats the best open-source model. So that is pretty incredible which goes to show that every single day there are strides being made across every company which leads us to greater and greater models. And Elon Musk also made this comment saying that GPT4 level AI on a laptop before too long. Pretty much stating that it's not going to be too long before we get these GBT4 levels of AI running locally on our laptops. And that is something that people have been doing for quite some time now. Then of course we also did get this tweet and this one actually does take a different look at the different aspects that you can have because of course as you do know there always is both sides of any argument to consider and this person brilliantly brings up certain points about open-source models versus GPT4 but bear in mind that this video is in depth. So please do take a look at everything before you know take one kind of stance. So essentially here he says that if you think open source models will beat GPT4 this year, you're completely wrong. I worked at top AI research labs and built open source libraries with more than 5 million monthly downloads. GPT4 is 1 year old and so far no model matches it. And here's why. Number one is the talent. OpenAI recruited top AI engineers with salaries above $1 million. Number two is data. massive proprietary data and human annotated data sets. Number three is team structure. In-person centralized teams work better than decentralized opensource teams. Number four is model versus product. GT4 is not just a model, it's a product. You can't beat it with a better model. And number five is the infrastructure. Public cloud infrastructure is terrible compared to what Google/Demind/ OpenAI has. And it's very hard for open source teams to iterate at the same speed. So when we do actually take a look at some of the points, he does actually make some very good points. I mean, you know, the one thing that he does talk about is the talent. And that is really, really true. One thing that, you know, that isn't talked about enough is that although some of these AI labs are able to, you know, create really comprehensive AI systems that are really, really effective and on par with these top labs, the talent and the genius that is at OpenAI is, you know, simply incredible. I mean, you know, the amount of, you know, persuading that it took to get Satska to, you know, between Elon Musk and Sam Alman, you know, there was like a push and pull between getting him to go from Google to OpenAI. It was absolutely incredible and I know that the top talent, you know, the salaries that they're offered, the compensation that they're offered, um it's simply very competitive out there. So although these AI systems are good, I don't think, you know, some of these independent labs can compare to some of the talent, but that doesn't mean that they aren't able to still get it done because like I said, some of the employees there have actually been at some of the other top labs too, which goes to show that it is still possible for it to be done. Now, another thing is as well is that the setup. Okay, the fourth point he makes here is that GBT4 is not just a model, it's a product and you can't beat it with a better model. And I think that is pretty true. In order to get, you know, Mistral or any of their new models or any other the other company's models like Llama, and we're going to talk about that later, you also need to make sure that the product actually does work. And I think one thing that was good about, you know, the way that they went about Chat GBT in terms of OpenAI is that, you know, maybe previously when they were a, you know, open-source, uh, you know, nonprofit that maybe they could have been beaten, but now they're a company operating for profit in terms of, you know, and they're now closed source. I do think that it's going to be pretty hard to beat them because they do have a lot of the product focused people working there that make the model much more usable and much more userfriendly and that entire distribution allows it to be much more effective in terms of its adoption which is something that these other AI systems just don't currently have and I think that is a key point that even if they do beat the benchmarks they do need that effective distribution and of course being able to distribute that product effectively that they may lack. So I think that this is definitely an important you know tweet because it goes to show that whilst you know certain models might beat it on certain benchmarks the adoption curve might not be there for those models as well. Then of course we have Llama 3 and this was essentially some you know an overheard of conversation and essentially according to a first rumor Llama 3 will

### Segment 5 (20:00 - 22:00) [20:00]

be able to compete with GBT4 but will still be freely available under the Llama license. This was overheard by OpenAI engineer Jason Wei, formerly Google Brain, at a generative AI group social event organized by Meta. Weii says he picked up on a conversation that Meta now has enough computing power to train Llama 3 and 4. Now, Llama 3 is planned to reach the performance level of GBT4, but will remain freely available. And that is going to be pretty incredible because the ramifications of an open-source AI system operating on the level of GPT4 but being open source is going to be pretty crazy. And I can only hope that there aren't some bad axes out there that are going to be using that. And of course, additionally, we do know that jumping from Llama 2 to Llama 3 may therefore be more challenging than simply scaling through more training and may take longer than moving from llama 1 to llama 2. Because if GPT4 is a mixture of experts architecture, then it's likely that these other open- source teams are going to be moving in that direction. And what's interesting was that they recently released code llama, which is based on Llama 2, and it achieves GBT 3. 5 and GBT4 level results depending on the type of me measurement in the human evalar coding benchmark through finetuning. And what's even crazier is that the Financial Times reported in mid July that the main goal of Meta's Llama models is to break OpenAI's dominance in the LLM market. And Meta is likely trying to establish Llama models as an enabling technology in the LM market, similar to what Google has done with Android in the mobile market to launch additional offerings later. So, what do you think about Llama 3 and Llama 4 and Mistra's newer models competing or simply, you know, surpassing GPT4's capabilities? Do you think that they can get it done without all of these advantages that companies like OpenAI and Anthropic have? Or do you think that they're just going to struggle regardless? Either way, it's interesting to know your thoughts and I'll see you in the next AI development.