Finally, DeepMind Made An IQ Test For AIs! 🤖

Today we are going to see AI techniques fail  in ways that are ridiculous, it’s going to be   an absolute disaster. Why? Now we know that AI  techniques can generate videos from a piece of   text, I always think, okay, but do they really  understand what they are looking at? Do they   really know the physics of these objects? Oh! I like this question a lot, because it   sounds very philosophical, and sounds pretty  much impossible to answer. But wait a minute,   according to this amazing new paper, there is  an actual answer to this question. Incredible. We can ask questions to test whether these  AIs actually understand what they are seeing,   but not this way. If you visualize  what is inside of a neural network,   you don’t get a lot of useful information,  only a bunch of numbers. Remember,   this is not human intelligence,  this is artificial intelligence. So, how do we ask them? Well, scientists at  Google DeepMind say, let’s show them the start   of the video, and if the AI understands it,  it will be able to tell us what will happen   in the next 5 seconds. We humans know  what is about to happen, but do they? Let’s have a look through 4 experiments, where  each gets more challenging than the last one. First, let’s start simple. A rotating teapot.   Pika 1. 0 says yes, stand aside everyone,   I got this, it will not rotate, but grow a  pedestal. Oh my, that is a complete disaster,   and this is just the simplest question. Now then,  Lumiere says no-no-no, this is a rotating teapot,   and I shall do exactly that. Wait…where were  the handles exactly? Object permanence is not   my strong suit. But, OpenAI’s Sora and Runway’s  Gen3 kind of gets it. Not perfect, but not bad. Now, two, let’s paint something. There is a bit of  rotation and it is clear as day what is about to   happen. Yes, now let’s show the start this with  OpenAI’s Sora because it was right last time,   and…oh my goodness. That is not even close.   Then, Pika 1. 0 says yes, I know what’s going   on here. Zooming in and then something happens.   Also wrong. And if we ask Lumiere…come on man,   are you even trying? Now if we ask VideoPoet, now  this is quite reasonable. Not perfect, but better. Now, three, light versus heavy. A classic. We  expect that the heavy kettlebell object will leave   a larger imprint on the pillow than a light scrap  of paper. Easy, right? Well, let’s see together.    VideoPoet says choose me, I know, I know… the  evil pillow eats the paper, and as a revenge,   we then stab it. Yes, that is what will happen.   Pika 1. 0 says, nothing to see here, we just   zoomin’. Just keep zoomin’. Then OpenAI’s  Sora says…I do not know what it says. Wow,   all of them were absolute disasters. So what do  we do now? Do we make it easier for them? Nope, we   make it even harder! Dear Fellow Scholars, this is  Two Minute Papers with Dr. Károly Zsolnai-Fehér. Four, we start with a match on fire, which we put  into water. What happens then? Runway Gen3 says,   of course, it floats. Lumiere  says nope, you’re dead wrong,   fires clearly exist underwater, and  VideoPoet says that’s also wrong,   of course, an explosion happens. And the  best of them all in this case is Sora,   which kind of gets it, wait a minute…then it  gets lit on fire by the water again. Oh my. And this was just the start.   Scientists tested these AIs   on a heck of a lot more. Solid dynamics,  fluid dynamics, optics, thermodynamics,   magnetism, you name it. So, what is  the result? Man, I am so curious. Wow, look at that. Sora came in dead last,  while the multiframe version of VideoPoet   aced it. However, look. This is still  below 30%, so it only aced it compared to   its competition. But overall, that result is  that they mostly have no idea about physics. Another interesting thing is that they understood  fluid mechanics better than solid dynamics,   which is really interesting, because  if you study the basics of both,   I think fluids are way harder. Not  even close. But not for the AI,   this is not human intelligence, this  is a different kind of intelligence. And these results are really surprising to me  - these techniques can generate lots and lots   of photorealistic footage, but apparently,  visual realism and physical understanding

do not go hand in hand. In other words, the  best, most photorealistic looking systems   don’t necessarily understand  the world around us too well. Now there is another study that shows pictures to  these GPT-like AI assistants and asks questions   about them. Think of this like a visual  IQ test where we ask about temperature,   air pressure, or smash a watermelon  and ask about that. And the results   are…each of them know different areas a bit  better, but all of them are surprisingly poor. But why? I mean, everyone says that  many of these are PhD level AI systems,   so what happened? What went wrong here? Well, two things. One. Physical understanding  differs significantly from the tasks that these   systems are trained for. So, just teach them  more about that, right? Nope, two, shockingly,   as we teach these algorithms more, they don’t  start scoring better on tests like these. So yes,   AIs can do amazing things today,  but they are not human intelligence.    They are a fundamentally different kind of  intelligence that still has a long way to go. I hope you had as much fun with this as I had. I  loved this one. If you wish to see more, subscribe   and hit the bell icon. And what do you Fellow  Scholars think? Let me know in the comments below.