# GPT-5 Is Slowing DOWN! (OpenAI Orion News)

## Метаданные

- **Канал:** TheAIGRID
- **YouTube:** https://www.youtube.com/watch?v=jL-BIW5BCnw
- **Дата:** 10.11.2024
- **Длительность:** 19:08
- **Просмотры:** 32,015
- **Источник:** https://ekstraktznaniy.ru/video/13770

## Описание

Prepare for AGI with me - https://www.skool.com/postagiprepardness 
🐤 Follow Me on Twitter https://twitter.com/TheAiGrid
🌐 Checkout My website - https://theaigrid.com/


Links From Todays Video:
https://www.theinformation.com/articles/openai-shifts-strategy-as-rate-of-gpt-ai-improvements-slows?rc=0g0zvw

Welcome to my channel where i bring you the latest breakthroughs in AI. From deep learning to robotics, i cover it all. My videos offer valuable insights and perspectives that will expand your knowledge and understanding of this rapidly evolving field. Be sure to subscribe and stay updated on my latest videos.

Was there anything i missed?

(For Business Enquiries)  contact@theaigrid.com

#LLM #Largelanguagemodel #chatgpt
#AI
#ArtificialIntelligence
#MachineLearning
#DeepLearning
#NeuralNetworks
#Robotics
#DataScience

## Транскрипт

### Segment 1 (00:00 - 05:00) []

so open AI may have faced another hurdle on their road map to AGI as apparently they are now shifting their strategy is the rate of gpta improvement is slowing recently there was an article from the information that basically discusses what many people in the AI industry have been discussing for quite some time and that is whether or not these systems and these models are truly effective at learning and if the scaling laws and the various Paradigm surrounding AI will continue to keep going up in intelligence or these models start to Peter out in terms of what they're capable of doing so one of the things that we could actually see here from this article that I find to be interesting because there are just a variety of different things that are discussed in this article that need to be discussed at length to understand exactly where AI is going and there are a few things that a lot of people are getting wrong so I want to clear some things up but it starts by stating that some researchers at the company believe that Oran isn't reliably better than its predecessor at handling certain tasks according to employees so Ayan performs better at language tasks but may not outperform previous models at tasks such as coding according to an open a employee and this could be a problem as Orion may be more expensive for opening ey to run in its data centers compared to other models it has recently released so essentially what they're stating here is that the next model in the series which according to many won't be GPT 5 but will actually be called oryan is scheduled to be apparently unfortunately not that much better than the previous model now I think what they're stating here is that in some areas the model does exceed but in certain areas it just isn't reliably better so one area that they talk about here is the fact that coding is an area where the model isn't that much better and I'm going to dive more into this as we get on the article but I just want to read two more points so then I can explain to you guys what's going on so when they state that the model isn't currently better at the other model at coding I think this is kind of interesting considering the fact that there was also information released not too long ago coming from openi about how they have an actual internal model that apparently is executing code for openai themselves so from another article which is you know additionally from the information they actually State themselves that it says for instance open has worked on a product to handle software engineering tasks that actually might take a human hours or days and to write and execute the code for complex applications based on customers instructions but it isn't clear whether open ey would launch such a product so basically from this article in a video that I covered around 7 days ago this was basically talking about how open AI have their own internal product that handles software engineering tasks so I'm wondering what kind of model are opening ey using that isn't the same with as to what they're using for Orion because apparently that one internally works really well and apparently it's actually really popular internally so it will be interesting to see if this product ever makes it out or whatever kind of abilities that they've got in these really good coding models seem to manage to make it to or now I don't think you know Orion and coding and all that stuff is the main information let's get on to that so basically one of the main problems with this article that a lot of people are having is the fact that this article is talking about how the Orion situation could test a core Assumption of the AI field known as the scaling laws and that's the fact that llms would continue to improve at the same Pace as long as they had more data to learn from additional computing power to facilitate that process so basically if you've been in the AI Community for a while well you'll know that the entire Paradigm around GPT models we've had this Paradigm of the fact that look these models are going to get consistently reliably better as long as we have more data and manage to train them for longer and entally make them bigger and larger okay and essentially because the Orion model isn't that much better than the model that they are seeing now in terms of gp2 40 people are starting to think okay if this is the case then it might mean that these a models are going to be slowing down in terms of what they're going to be doing so this is something that of course many people are concerned about and many people are questioning because the entire Paradigm is how companies are investing this is why they're buying more chips so this is going to be something that is really important for them to know about because of course this is integral to how AI really works now I'm not going to lie I do actually agree with this article contrary to what a lot of people are saying I do agree that the GPT scaling laws are potentially slowing down but this doesn't mean that AI is slowing down at all what I mean when I say that is I mean that like if the GPT scaling laws in terms of like just using chat GPT like gp4 and just training bigger and bigger models was the entire way to improve the models I think that would be opening ey's main focus but we can clearly see now that opening ey's main focus is the new scaling laws okay on test time compute which is of course completely different and that scaling law is completely different to the previous one of where you're just focusing on adding more data so you can see right here it says the industry appears to be shifting its effort to

### Segment 2 (05:00 - 10:00) [5:00]

improving models after their initial training potentially yielding a different type of scaling mod and that is of course something that you know does make sense considering the fact that right now everybody knows that the Paradigm that we're looking at is of course the test time compute Paradigm so this is the Paradigm that you know everyone knows about it's about how AI thinks about its responses and this is how the model actually gets better so I do think that yeah maybe the old way of doing things like just adding more data and then just doing some fine-tuning and some post training those kinds of things probably aren't going to produce massive gains but literally you can see here on this new paradigm the 01 Paradigm which s has you know even talked about the O2 series stating that you know that one is going to get 105% on the GP QA I don't think that series is slowing down at all and I think that matters more than Orion anyways now crazily Sam Alman said that he did expected Orion to be significantly better than the last Flagship model I still do think that Orion is going to be significantly better than the prior model even if it doesn't look like it I still do think that this model is going to be significantly better because every single time that a new model is made we often do find new solutions to existing problems in terms of the efficiency in terms of hallucinations so it does seem that this model even if it's not going to be remarkably better we know that marginal improvements actually do allow for a lot more use cases I want you guys to think about the fact that when clae 3. 5 Sonet actually managed to produce really effective code how many people started to produce software do different things with regards to claw 3. 5 Sonic so I do think that even if this model isn't you know ridiculously better because of course the jump from GPD 3. 5 to GPD 4 was pretty incredible but of course even if that jump isn't as substantial we know that it does unlock a more SL wider variety of different use cases that are going to be really amazing for many individuals in the AI industry now a crazy thing as well that I didn't really see most people talk about which was kind of fascinating was the fact that they spoke how though open AI had only completed 20% of the training process for Orion it was already on par with GPT 4 in terms of intelligence and abilities to fulfill tasks and answer questions this is what samman said so the fact that the model has only completed 20% of his training process and it's already on power with GPT 4 I think that is a substantial statement considering the fact that this means this model is going to you know just based on that I'm not trying to do the mathematics here but if it's only completed 20% of its training process it's quite likely that this model is probably still going to be significantly better than GPT 4 in a variety of different ways that like I said before already is going to unlock many different use cases just making the model even better now of course you can see here they actually talk about how you know Orion's performance ended up exceeding that of Prior models the increase in quality was Far smaller compared to the jump between gpt3 and gbt 4 according to the same employees who have used or tested Orion so it's going to be pretty interesting to see what happens if Orion is delayed if there are certain improvements and it's kind of interesting because I do remember that recently there was an article that spoke about how you know Google were expecting huge results from their Gemini 2 model but haven't been able to get the kind of results that they were hoping for in terms of an increased level of abilities and all these kinds of different things that we're hoping to see so this is where I was actually talking about how the fact that open AI are busy baking more code writing capabilities into their model remember how I just spoke about the fact that you know they already have an internal tool that they use for software engineering tasks that's pretty popular at the company and I'm wondering if that's going to somehow make its way into our products and of course openi are also developing software that can take over a person's computer involving web browser activity applications yada y so basically stting a look open ey is still developing their own software engineers in terms of tools and then of course in terms of AI agents they're still developing software that can take over a computer completely now interestingly enough there's a lot of contradictory information here because outman and CEOs of other AI developers say that they haven't hit the traditional scaling law limits yet so of course if the scaling law limits aren't hit yet that would mean that they're going to still be developing these massive data centers and I think that is of course important because when we look at what people say versus what they do you always have to pay attention to what people are doing versus what they say because that is going to give you a greater indication of what is Truth Versus false because of course if the scaling laws were really slowing down then I think we would definitely see perhaps not this major buildout that we are seeing in terms of AI infrastructure so them stating that look we haven't hit the traditional scaling L limits yet we're still going to be buying these data centers I think it's still important but I do think that a lot of the future infrastructure is just going to be put to inference time compute because we've seen that the gains in terms of those scaling laws are not slowing down in terms of every time we add more comput to the model it seems to get even smarter so it seems like maybe they are adding even more data centers because of course they know that there's going to be an increased demand and of course because they know that inference

### Segment 3 (10:00 - 15:00) [10:00]

type compute is potentially going to become a more important Paradigm now interestingly someone who actually works at opening eye on the reasoning model which is the 01 series actually said something interesting and I want to show you guys two screenshots okay so the first thing that they said was they said after all are we really going to train models that cost hundreds of billions of dollars or trillions of dollars which is what brown said okay and this is noran brown the guy who worked on reasoning at01 which is of course the model that thinks before it responds and he said at some point the scaling Paradigm breaks down so basically he say that look even if these models got a lot better with data and stuff like that it's going to be pretty hard to spend trillions of dollars to train a model because it just isn't feasible now the reason I said I need to show you guys another screenshot is because this actual screenshot of this article I don't want to say it was taken out of context but he actually did respond to the statement here so he actually said that in the Ted Talk that he gave which they selectively quoted in the article I make the case that there won't be a Slowdown in AI progress anytime soon so I think it's interesting to see that you know the ual person who said the statement is basically stating that look this entire AI progress isn't slowing down anytime soon I think we can take that at face value more than what is being said in the article and I do believe that like I said before even if these models you know slow down and I'm talking about the GPT series not The 01 series it doesn't mean that AI progress is going to slow down overall because there's still different ways that you can prompt these models and steal interact with these models that you're going to get out more information you have to remember opening eye isn't the only company that is working on the stuff you've got Google you've got anthropic you've got x. a you've got meta as well so that's four other major companies that are all playing so it means that even if let's say openi hits some kind of Roadblock it's quite likely that other companies are going to manage to still be able to make progress and remember this doesn't mean that AI is slowing down it just means that the scaling laws are now focusing on a different aspect of AI now for those of you wondering about the naming of the model it looks like Orion is probably going to replace the naming of GPT 5 so it says when opening ey releases Ryan by early next year it may diverge from its traditional GPT naming convention for Flagship models further underscoring the changing nature of llm improvements employees said now of course they do talk about a data wall and it says one reason for the GPT slowdown is a dwindling supply of high quality text and other datas that llms can process during pre-training to make sense of the wall so now this is where it actually talks about the data wall which is essentially where in order to make these modal smarter of course one of the things that we do is we do add a lot more data and high quality data now what are the problems that they are facing is of course the fact that high quality data available online on the internet is quite hard to come by considering the fact that a lot of internet data is just simply garbage like a lot of the data that you see just is downright nonsense which means that of course in order to get higher quality data you either have to have humans make that quality data like smart humans or you have to have you know synthetic data with which is curated by these models now both of those things are pretty hard and this is potentially one of the reasons that we do have a GPT era kind of slowdown and I think one of the things that we do have to you know look at when we're looking at this kind of Architecture is the fact that like humans don't need millions and millions of pieces of text to be able to generate high quality text so it will be interesting to see in future paradigms how models manage to get super smart via a more efficient architecture that doesn't require these data hungry models now essentially of course remember how I spoke about you know one way of getting you know higher quality data is by you know introducing synthetic data one of the things that they actually did with Orion which was quite fascinating was that they said was Orion was trained in part on AI generated data produced by open eyes other models including gp4 and recently released reasoning models according to an opening ey employee however such synthetic data as it is known is leading to a new problem in which Orion may end up resembling those order models in certain aspects the employee said so essentially what you have here is something that could be referring to model collapse which is where when models produce data it's based on their training data and based on like their entire world view of how the world is and that's just of course based on the training data and how they were trained with human reinforcement slh human feedback and basically what happens is that if you have a model produced data to another model it's quite likely that the model that you're now training on that data is going to resemble parts of that older model in certain aspects which you know brings us back to the problem which is why synthetic data generation is really hard because you have this I guess you could say kind of family tree of models where the first model is really diverse and then because you use some of that data to train the other model it just becomes less and less diverse over time because it's training on the same data so I think one of the things that you know of course you have to do is you have to get really diverse pieces of data and interestingly enough the article also quotes where they talk about the fact that there's a GPT slowdown SL ASM toote there is this very interesting asymptotic kind of thing that's happening right now um where you know two years ago there was one you know llm that was like way out ahead of everybody else's which was Open the eyes and

### Segment 4 (15:00 - 19:00) [15:00]

sitting here today there's like six um that are like on par with that and interestingly at least for right now they're all s of ASM toting at this at sort of the same point they're kind of hitting the same ceiling um on capabilities now they you know well there's lots of smart people in the industry working to break through those ceilings but you know sitting here today if you just looked at the data charts of performance over time what you would say is there's at least a local topping out of capabilities that's happening right yeah if and if you look at like the improvement from GPT uh 2. 0 to GPT 3 to 3. 5 and then compare that from like 3. 5 to four you know we really slowed down in terms of the amount of improvement and the thing to note on that is the GPU increase was comparable so we're increasing gpus at the same like rate um but we're not getting the intelligence improvements at all now one of the fascinating things that actually saw here and I think this is going to actually be a problem for openi and I don't know how they're going to solve this but uh it will be interesting to see how do but this is basically the fact that like the everyday Joe doesn't need AGI and it says mathematicians and other scientists have said that o01 has been beneficial to their work by acting as a companion that can provide feedback or ideas but the model is currently priced six times higher than non- reasoning models and as a result it doesn't have a broad base of customers set to employees with knowledge of the situation so the model is advancing scientific research but apparently it's too expensive for people to use at this moment in time I do think that this kind of makes sense because of course these models actually do burn through a lot more tokens than traditional models with chat GPT gbt 4 gbt you know mini whatever um when you interact with these models essentially what happens is you interact with the model it produces you know a th000 tokens or whatever but with these other models that the thinking there are actually so many tokens that are going on for the model to think about its response and essentially with that it produces a lot more tokens which of course aren't free they cost a lot and so therefore it means these models are six times higher which does mean that right now it isn't cost effective I do think however that in the future it's quite likely that these models are going to become a lot more cost effective in terms of their pricing and I do wonder how that translate to economic value for individuals around the world will people start using this everywhere and everywhere I think that is quite likely the case considering the fact that is definitely beneficial to scientists so it's going to be interesting to see that okay we've got the 01 series reasoning of models that are really smart and advanced that for like things like physics chemistry maths and then of course you've got the GPT series models that are for things like maybe even coding content creation literature storytelling those kind interestingly enough some people have pointed out that Gary Marcus has claimed Victory and if you haven't been paying attention to Gary Marcus he is what some would call an AI skeptic now I don't think he's an AI skeptic per se considering the fact that he's you know built his own AI company and then sold it to the likes of uber but he said folks over game I won GPT is hitting a period of diminishing returns just like I said it would and I think this one was particularly interesting because Gary Marcus was basically saying that look at the end of 2024 we're going to have a bunch of tbt 4 style models and there's going to be no major improvements so it will be interesting to see exactly what happens with the next series of models Orion maybe the next version of Claude maybe Gemini 2 it will be fascinating to see what happens to that regard now also Peter Linder the VP of product basically said that people underestimate how powerful test time compute is compute for longer in parallel or fork and Branch arbitrarily like cloning your mind a thousand times and picking the best thought so I think of course this is going to be something that becomes more powerful over time because of course if a model is able to generate 10,000 thoughts and pick the best one that's going to become a lot better than just picking the first initial thought and over time I do think that Paradigm will exist and these models will continue to get smarter so overall basically to summarize this video some people think that the Paradigm is slowing down but I think that even if the gbt series Paradigm is slowing down I do think that the 01 series is about to go crazy so that means that even if you only get marginal improvements from here on out there's still going to be ways to prompt the GPT series and there's still going to be improvements discovered over time that make these models more effective so the narrative that AI is slowing down completely isn't really true considering what we know now