It's Official: AI Makes You Worse At Stuff

Well, it's officially official. The thing that we all knew was happening this whole time is actually happening. Using AI to code makes you worse at coding. Anthropic just finished a landmark trial where they pitted two groups of programmers against each other, one with AI, one without. And they found that the ones with AI scored 17% lower across the board. Now, I'm not going to sit here and tell you that this is a bad thing. I think since the dawn of mankind, humanity has used technology to improve their ability to do things at the cost of the core skill. think hunting or gathering or sewing or fishing or farming or whatever the hell a bunch of other people are currently doing to keep us alive. Thank God for that. The reason I'm making this video is because some people around me believe that that's different. That programming is in a category of its own. Well, it's not. The study definitively shows that AI is like the Doom guy and our coding skills are the demons. Also, even though Anthropic didn't do a whole study on other skills like writing or, you know, thinking, I believe you'd have to be pretty dense not to realize this is happening across the stack. So, what did Anthropic actually do? Well, they examined how quickly software developers picked up a new Python library with and without AI assistance, and whether using AI made them less likely to understand the code that they just wrote. They found that when you use AI assistance, that leads to a statistically significant decrease in what they're terming mastery. On a quiz that covered the concepts they just used, participants in the AI group scored 17% lower than those who coded everything by hand, which is the equivalent of two letter grades. So using AI sped up the task slightly by about 2 minutes which we'll get into. But this did not reach the threshold of statistical significance meaning that although there was a statistically significant decrease in their skill there was not a conccommittent statistically significant increase in their speed. The study design was pretty simple. There was a warm-up coding task of about 10 minutes. Both groups did not use AI in it. Then there was a task where the treatment group was allowed to use AI assistance whereas the control was not. There was a post-task quiz afterwards and a survey. And for both of those, neither group was allowed to use AI. So really, the only difference here in this randomized control trial is that during the task of interest, the treatment group used AI.

And though I'm going to get into this in a second, the results should shock nobody. I mean, obviously using a tool like this that significantly improves your leverage and the ease with which you do the work is going to make you worse at the core skill. When's the last time you shot a bow and arrow to catch a rabbit? We don't have to anymore. No wonder we're not very good at it, huh? So in the study, the left hand side of this graph shows the time that it took the two groups. Blue over here is AI and orange is no AI. So blue took about 23 minutes with these error bars above and below just to show that there's some statistical variability in the results. Yellow that had no AI took approximately 1. 75 minutes longer. Statistically speaking, that is not big enough of a difference to say that there was a very large difference between the groups. Contrast that with this second graph that shows the quiz score. The group that used AI scored about 50% whereas the group that didn't use AI averaged somewhere between 65 to 66. Now, I should note that anthropics not just crapping all over AI usage here. They expressly noted that previous research they've done has showed that AI can help people complete some work tasks by 80%. And that seems like that might be a little contradictory to what they're talking about here. But importantly, the previous study measured productivity on tasks where they already had the relevant skills, whereas this study examines what happened when people learned something new. What they're suggesting is AI, at least its usage in the completion of a task, hinders one's ability to learn while doing the task. To me, this is about as impressive a result as the study that showed that multiple stab wounds were harmful to monkeys. Like obviously if you're trying to learn something, it makes sense to make the learning of that thing difficult. You have to approach it from different angles. You have to attempt different things. And ideally, you would do most of that hard work yourself because how else is your brain going to grow? Kind of like if you were at the gym doing a bicep curl and your buddy all of a sudden starts helping you up. Would you expect to grow the same if you did all that work versus if your buddy helped lift? No, obviously not. Despite this not being what I'd consider to be the most groundbreaking of results, I think there's some solid takeaways. The

first thing, and something that I personally felt back in 2020 when I first started using GPT3, that's the predecessor to a lot of the really smart models that we have today, is that AI improves your leverage, but it does so at the cost of eroding your core skills. The way that I see it is there's two forms of production. There's horizontal production, and then there's vertical production. Now, horizontal production is when you take an approach, one that you've already established, one that already works, learned, and then you just multiply it. Maybe you're a business that came up with a cool way of making soap. You start selling the soap and it works. So, what do you do? You buy more factories. You're not necessarily innovating on anything here. You're not really doing research and development. All you're doing is you are growing, okay, the base of factories capable of producing some cool product. You can also think of programming in the same vein. AI has expanded our ability to grow horizontally. we can produce a lot more of wellocumented wellestablished product but it does all of this at the cost of vertical growth. What is vertical growth? Well, at least right now that is our own ability to apply human ingenuity to a problem and then develop better ways of doing it in the first place. For example, if horizontal growth is going from one soap producing factory to three soap producing factories, vertical growth is coming up with a new method of soap production that allows you to produce what you previously would have required a thousand factories to do so with just one. The unfortunate thing is vertical growth is super hard. It typically occurs in bursts and bits and pieces spread throughout our economy. Whereas horizontal growth is sort of that slow incremental add-on to a pre-established process. The situation where it makes the most sense to use AI is one where you actually already have a fair amount of solid skill built up. It's where basically your foundation or your base is quite strong. That allows you to grow horizontally at a very fast rate. You can produce more writing, produce more code. You could produce more applications, produce more websites. But all of this occurs at a cost because you're now focused on horizontal expansion. You're not doing any vertical expansion. Aka often you're not working on the core skill at all. So

rather than sit here and about how it's the most terrible thing in the world, it's not. It's just the reality of using tools and technology. Better for you to make this trade-off knowingly and willingly rather than have no clue what you're walking yourself into. Because there is this erosion of course skill. I would recommend you just don't rely on models for everything. You pick and choose what parts of your life it makes sense to hand off, let's say, to an artificial intelligence. For instance, coding. It's an economically valuable task. I don't think it's going to necessarily be the primary lever moving forward with models being able to do it so much better and faster than us. Sure, if you could produce things even marginally faster, even 2 minutes out of 25, it might make sense to do so. But personal relationships, how to communicate with your boyfriend and girlfriend, interpersonal dynamics, things like that, things that, to be completely honest, probably aren't going anywhere no matter how good AI gets at producing widgets. Those are things you should probably do yourself. Likewise, critical thinking, major life decisions. These are things that you should have control over. How about talking to your wife? I'd recommend talking to your wife yourself, not passing that on to

Clogbot. Because like it or not, AI is soon going to be better than us at basically all economically valuable skills. And I mean, if this is us and this is where AI is at right now, rest assured, in the blink of an eye, it'll be all the way over here. So, if you build a whole identity around your ability to produce economically valuable skills, then you just have AI do all of that for you. Who are you really? Well, you're the sum of your relationships. You're your character. You're what's between your two ears. So, if you're going to outsource anything, outsource the economic part. Don't outsource the you part. I guess the last major point is just that the nature of work is changing. You know, I was having a conversation with a friend of mine a couple days ago, and he raised a point that made me stop and think. I was saying something along the lines of, you know, with robots being better than us at doing basically everything. They think smarter. They think faster. Their brains run on silicon. They can think at the speed of light, not the speed of our biology. Will there be any future career for people at all that isn't complete And he said, "You know, I bet you if you took a random hunter gatherer out of the woods from 10,000 years ago, or even a farmer from 2 or 3,000 years ago, and you showed him what you do on a day-to-day basis, there's no way in hell that he'd say what you're doing is economically valuable work. " They'd say, "Where's the hunting? Where's the gathering? " All you're doing is sitting at a computer twiddling your thumbs. How's that meaningfully improving the world or your family or your community? So, his take on it was, "We already mostly do jobs. What's the difference between the jobs we do today and then the jobs we're going to be doing in 10, 20, 50, 100 years from now? " That really annoyed me cuz I like being right. But who's to

say that he's not? Hopefully, this at least made you think. Thanks for your time and I'll catch youall in the next