OpenAI’s GPT 4.1 - Absolutely Amazing!

Alright, GPT 4. 1. Three new models just  appeared, 4. 1, mini, and nano. This is a   mainly coding-focused AI assistant, previously  if you wanted to create a flash card app from   just one text prompt, I mean, this is okay, it  kinda works — you can create and review your   flash cards. However, if you look at the new one,  the bones are very similar, but the usability is   on a completely different level. This went from  good to great in just one release. Loving it! Note that this video comes in two parts, this  is part one, and part two will be where the   real fun begins. So, these new models form  a new Pareto frontier, which roughly means   that you can choose how much speed you are  willing to sacrifice to get more intelligence. If you are typing something and you  want the AI to autocomplete your text,   it does not need to be smart as Einstein  to do that, but it needs to be super fast,   so you probably need the nano for that. But  for most things, like the flash card app,   you will probably want to invoke the regular 4. 1. But it gets better. On coding tasks, 4. 1  can outperform the previous 4. 5, I know,   and surprisingly, 4. 1 outperforms even those  much slower, thinking AIs on coding benchmarks. Also, wow. The context window is now 1  million tokens long, you can put heaps   and heaps of textbooks in there, thousands of  pages in total and ask questions about them,   and then, what happens? This is called the  needle in a haystack test, and OpenAI says   that it recalls all of it very well. However,  when looking for 8 needles in the haystack,   the accuracy decreases considerably. Respect to  OpenAI for showing a little weakness there too in   the name of Scholarly integrity. Independent  tests also seem to verify that, and here,   Google DeepMind’s Gemini 2. 5 Pro reigns supreme.   I’d love to see some more rigorous testing on this   to get to the bottom of it. This is important,  because even if you don’t put in many textbooks,   remembering your past conversations  and understanding you will be super   important going forward. It better not  miss that marriage anniversary date! And, yes I hear you asking, Károly, GPT 4. 5 is  already out. This is GPT 4. 1. Yes, and look,   it gets better, I mean funnier,  look at this wall of other models.    You could say that the marketing could be a  bit better here and I would agree. However,   this also shows the breakneck pace of innovation  and competition here. More on that in a moment. Now, benchmarks. I enjoy seeing numbers going up  over time as much as the next Scholar. I mean,   having it defeat hundreds of PhD level  questions, mathematical and biological   olympiad level questions is incredible.   These systems do unbelievably well on them. But here is the problem with these benchmarks.   Almost all of these AI assistants are trained on   nearly the whole internet. What does that mean?   That means almost whatever your question is,   they had already seen something  that is similar. And what does that   mean? It means these benchmarks will  mean less and less over time. Thus,   I would not take them too seriously. Perhaps  as one little data point. One little pointer. So then, is testing AI assistants  reliably impossible? Well,   not quite. There is a solution. Sort of. Okay, this was part one. Normally here is where   most other videos end. Elsewhere. But not  here. Here we talk about research papers,   and the wider context around these  papers. For me, that is the coolest part,   that’s what makes Two Minute Papers different.   So here comes the coolest part. Part two. Here, we are going to answer questions like: Can we really know how smart these AI systems are?   Why do they say that it is  devilishly difficult to train them?   How do you make a small textbook  bigger? Yes, you heard it right. Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. So, the problem with benchmarks is that they  ask questions that these AIs already know. How about a test with things they  don’t know? Enter Humanity’s Last Exam,   a paper that just appeared where the authors  ask the smartest people in the world to create   questions that none of the current AI  systems can answer. These are really,   really tough. Questions from  all kinds of disciplines,   classics, ecology, mathematics, computer  science, linguistics, chemistry, you name it. And the result? Now hold on to your papers  Fellow Scholars, and look at that. Other   tests? Easy-peasy. Humanity’s Last Exam? Wow.   They fail spectacularly on that. A stunning

result. And even if you plug in the newer ones,  once again, Gemini 2. 5 Pro pops up again. Google   is getting their mojo back in AI. So, yes,  I would very much like to have more of these   systems tested on Humanity’s Last Exam. I think  we should keep asking for it. Doing my part here. But…wait a second. I hear you asking,  Károly, okay, but this one can be gamed too,   right? You just put these questions, or  similar ones in the training dataset,   yum-yum-yum, it eats it up, and suddenly,  the next version of the AI is incredibly   good at that, because it has already  seen it. Like with other benchmarks. Well, not quite. Because get this, many of the  questions are reserved in a hidden dataset that   is not published anywhere, making this one  a bit more truthful than most other tests. In short: private datasets might  still be a good yardstick to measure   AI progress in the future. Please test  new systems like GPT 4. 1 on those too. And as you see, luckily for us, there  is breakneck competition between these   AI labs to create the best models out there,  often for free. So we are completely spoiled   here. OpenAI came up with ChatGPT  and took the world by storm. But now,   Google DeepMind has Gemini 2. 5 Pro, which is an  absolute powerhouse at a very low price point,   so much so that it might even steal the show  for now. And DeepSeek is also on their tails,   just a tiny bit behind, but it is all  free to use for all of us. Amazing. So all of these, including GPT 4. 1  are amazing gifts for all of us,   especially that these systems are  devilishly difficult to train. Why is that? Now, I don’t have a lot of eye candy for  this part, so I really appreciate that   you are all Fellow Scholars, and you know  that most of the value is in the narrative,   not in the visuals. Huge thank you and  huge respect to all of you for that. Now get this. In an interview,  scientists are OpenAI were asked,   if they had to redo GPT4, a state of the  art flagship system from just two years ago,   what would it take? The answer was  something absolutely incredible,   I couldn’t believe my ears: they said  5-10 people would be enough to do that. And now, for their more recent models,   they need hundreds of people and  all the might and compute of OpenAI. But here comes the crazy part: both compute  and training data is growing like crazy,   but compute grows even faster  than data. What does that mean? It means that data has become the bottleneck.   So the goal now is to use the tons of compute   to squeeze out every drop of information  out of this training data. That means data   efficiency. Do you know what system out there  is really data efficient? The human brain. Oh   yes! That is huge. I mean, not the brain,  but the realization. So let me say again:   we are not compute constrained. It’s nice  to have more graphics cards, of course,   but now what we need most is more human ingenuity  to make better use of the data we already have. Imagine that you need to take an exam and you have  access to a textbook. And you have all the time   in the world, fantastic. You think you are lucky,  until you find out that there is one big problem:   the textbook is really tiny. Not even close  to enough for the exam. Why? Because it only   has two problems in it. But the test will  have a hundred problems. So what do we do? Well, first, what you don’t do is just memorize  the two problems it contains. This won’t help with   the other 98. So, instead you dissect these two  problems. You try to understand the fundamental   principles, the methods, and the reasoning behind  the solutions. And that, Fellow Scholars is going   to be the next chapter for AI too. Lots of  compute, and comparatively very little data. But the training it gets even more  difficult. Why? Well, they have small bugs,   tiny little problems during the training of  an AI system. Is that a problem? No and yes. You see, imagine a dripping faucet in your  house. Nobody cares about it. But note that   the new system is more than a 100 times more  demanding, a 100 times more complex, so remember   that dripping faucet that everyone ignored? Well,  multiply it by a hundred, that is now a broken   pipe that pours water into the foundation of the  whole house and then it slowly starts sinking. Oh yes, that’s a classic.   A small problem magnified   by a 100 is suddenly not a small problem anymore.

And don’t forget, this is just how things are at  the moment. But the landscape is changing really,   really fast. Once again, there is breakneck  competition. Remember Sora, OpenAI’s text to   video AI? When they first showcased it, the news  took the world by the storm. We couldn’t believe   how good it was. And by the time they released  it? I would go so far as to say that now,   it might not be able to compete with  DeepMind’s Veo2 that has appeared since.    And there are so many more models being  published for free, you can’t even keep   track of them. Some of them are 7 billion  parameters, they run almost everywhere. So, these are all amazing gifts to humanity  from OpenAI, I am really thankful for them,   and I am also thankful for the fact that there is  huge competition between many other labs too. The   result? We, the users, the Fellow Scholars  get spoiled. Often for free. Thank you! So   whatever you hear today on AI this, AI that,  this is still just the beginning of humanity’s   AI journey. And it is already so incredibly  capable. Loving it. What a time to be alive! This was super fun, hope you enjoyed  the journey, and consider subscribing   and hitting the bell icon if you wish to see  more like this. And check out our new sponsor,   because it helps you try a bunch of amazing  AI systems for free or for very cheap.