Can Claude Code Learn To Draw In MS PAINT?

So you can see this is how we train kind of claude code to learn to draw in paint and it should able to paint a dog here now playing in snow. Okay, so this is going to be the dog. Let's see how it's turned out. Yeah, at least he got four legs and a tail. And all right, that's a retriever, right? Okay, so today's video is just going to be something I've been thinking about for a while after I saw a video on YouTube that kind of gave me inspiration to try this out. So, I hope this video can kind of give you the same inspiration. Uh, I'm going to play that clip from Mo soon. Uh, that kind of gave me the inspiration to try this out. But first, I just want to uh go to kind of what we are going to try out today before I play you the clip and we come back to this. But basically uh or it's better, I think, if I just play the clip from Mo now and then we come back to see uh what kind of my setup is for today. So, I'm just going to play the clip that kind of inspired me to try this out. The hardest part of building a humanoid robot isn't the legs. It's not the back flips. It's not the eyes. It's not the head. It's not the balance. It's the hands. The hands are the most complex part. And they just can't figure it out. And you know, I got to thinking, I'm like, man, how did nature solve this problem? Like, how does it code up this hand essentially? And if you were to take the standard explanation, we would say that nature doesn't know how to code up a hand. But it tries things. And when it works, when it passes the test criteria, which is natural selection, it keeps that feature. Now that I understand that literally all of like life, all living organisms are essentially vibecoded. They are. And it's only when you vibe code. try random things and put them up against automated criteria that you're able to achieve complexity that would be impossible to do by hand. And so, you know, I'm somewhat of a scientific futurist. And so when I have these thoughts, I get really glued to them and I'm like, "Oh the only way that humans start to build more and more complex software is to stop writing it by hand and to start emulating nature. " As long as we establish automated test criteria, LLMs can do the searching and the mutation and the finding of novel solutions to build towards complexity and scope in software that would have been impossible for us to do by hand. — Okay. So yeah, you can kind of get you kind of get it a bit more now. So basically we just want to give cloud code in our setup like a goal. So our goal today is going to be draw a copy of my drawing in JS Paint. And the tools it has available is of course cloud code. We have our browser navigation that we set up using uh Chrome debugger and it can do screenshots and stuff like that to kind of compare because uh the third part is kind of the test part, right? But for us that's just going to be compared to our image when Claude tries to draw its image. So this is just a simple test I just wanted to do and I'm going to show you at the end some of the learnings I kind of came uh take took away from this. Right. So let's just head over to uh JS Paint and I'm going to show you I'm going to draw something and then we're going to try to copy it by just using this style of uh yeah trying and trying and see if we can do it. Okay. So the setup we have now is basically my browser automation tools. So we can kind of navigate on Chrome and that is so all we have now. We have no set up any skills or anything I think. Uh yeah, no skills. So this is like a fresh start. So now we doesn't really know anything about how we can actually draw on Paint. Right. So what I'm going to do now is I'm going to open up this window. I'm going to go to JS Paint here. So the first thing we need to do is kind of create the goal that the bot is trying to do. So, I thought we could just draw something here. So, let's just draw I thought we can just draw some uh Yeah, let's call this like a shoreline or something. Something like that. Uh, and let's do some guy here fishing or something. So, let's just do something like this and a guy. And let's give him like this fishing rod, right? Something like this. And just a very simple setup. Maybe give him like a hat or something. Yeah, that looks pretty good. Uh, let's do some coloring. So, we can select the fill tool. Let's do some colors. Uh, let's do the ground. Uh, okay. And he needs some Let's just connect this.

Okay. So, here is our fisherman image, right? We kind of have the line and everything. So, this is kind of our baseline comparison. So I'm going to head here now and let's do kind of the prompt on this now. So we can start trying to do this. Okay. So this is going to be the prompt, right? So here is your challenge. You have the tools now to navigate and use Chrome. Your goal is to do as follows. Uh read the image fisherman png. Go to JS uh paint. Draw the exact image. Use screenshot to compare the truth at all times. Build tools when you think you have a completed 95% similarity. You can stop. But no cheating. I'm going to say build tools if you need it. Something like that. And do you understand the task? So let's just take this and go over to cloud code. I guess we can go back here now. Paste it in. And I'm just going to close this and let's start this now. So now comes kind of the fun part because now we kind of want to see how Claude Cove is cla clove is going to try to solve this by kind of using the tool already have by navigating the browser and just keep iterating and learning I guess about u about um JS Paint. I think I want to do exit and do dangerously skip here so we save some time. Okay. So, I'm just going to say continue. Yeah. So, I don't have to approve everything. Okay. So, it's going to start by reading the image and comparing it or kind of understanding what it's supposed to do. So, it's want to check the CDP capabilities. That's pretty good. Okay. So, you can see now we navigated to JS Paint and it's going to do build a drawing script that uses CDP directly to control a mouse and open JS Paint. Yeah, we did that. So it uses the screenshot tool to just see that everything is open here. Okay. So now it says it's going to build a drawing script that uses CDP directly to draw on the canvas. Okay. So remember it already has some ID that it needs to draw the fisherman because we already stated that. Okay. So it's completed the script. So let's see what happens. So this is pretty good. So now I want to see if it actually iterates on this. Uh I think it did the colors. That was perfect. Uh the fishing rod is a bit strange. The face doesn't have any Yeah. eyes or mouth, but let's see what happens now if it's going to do some more iterations on this. Okay, so that was the second attempt. Uh yeah, a bit better. I kind of like the feet here, but uh the rod looks a bit strange and it's not perfect, but not too bad, right? All right, that was not a much I think it the bobber here or the bobber made it a bit better, but I don't think it's kind of doing so much else now, but uh it I think it kind of proved my point. I guess we can do one more drawing just to see. Let's just try something else here. Okay, so I did this masterpiece here, AI agent, and I saved it. And let's see if we can redesign this now. So, we have a new image AI agent 2. Uh, do this one. So, I want to see if we can do any text here. Okay. So, let's see now. All right. Not too bad. The N is upside down. Uh, other than that, pretty good. I guess it got the colors right, but let's see if it kind of does a comparison now. fixes this. And it shouldn't overlap. So, it has to fix that. The N is backwards. Yes, that's correct. Okay. So, let's see if we can kind of flip the N now. And let's see if we do any other changes. Yeah, we flip the N. And that was pretty good. Okay. So, now it's we're going to try to do the letters more handdrawn. Okay. I like the styling better, but the A's turn out really strange. But it looks more like what I did. I got to give it that. But the A letter looks kind of strange. Let's build a comparison tool to measure similarities. Okay, so similarity 78. 1%. So that's pretty cool. The comparison is unfair. It's comparing a pure artwork against a screenshot. So it wasn't too happy about that. But let's see what changes it's going to do now. H I wouldn't say we're much better. a bit better, but still it only gets like a 92. Okay, so this is what it ended up with. I don't know. It kind of cheated here at the end, but so it got a 95%. But I guess this was a good example of what my idea about this was. It's just going to keep trying until it kind of uh yeah, gets what it wanted to be here, like 95%. Uh I would say it cheated a bit here, but uh this was not the idea was just to be inspiration.

So yeah, that's pretty cool, right? So now I just want to show you some of the skills I learned while I trained while I tested this yesterday. Okay, so in this instance, we should have a skill that is called draw on JS Paint, right? And here is kind of some skills I learned while kind of playing around with this. So, I think I can do something like draw an abstract oil painting of a female on a hot summer night. So, here I just had it kind of run in a loop to try to figure out how to kind of draw brush strokes. So, let's see if it can kind of pick up on the skill. Now, uh I might just steer it in that uh direction. Okay, so we kind of used the skill now. Let's see uh what it can paint now. So you can see I learned it to kind of paint one and one brush stroke. Okay. So it added some stars and now it's starting to work on kind of the face I guess. And you can see it's just using these brush strokes painting the eyes. And here is the female. She looks a bit strange. But uh yeah, this is what we Why did it add some hair in her face? But yeah, this is what we kind of trained it on. And it's not the worst drawing, right? So you can see I think the ice looks pretty good. And this is the oil painting technique. So I'm going to show you one more. So let's do uh a dog playing in the snow on a winter day. Just a totally different uh image, right? Not image, painting if you want to call it that. So you can see this is how we train kind of claude code to learn to draw in paint. And it should able to paint a dog here now playing in snow. Okay. So, this is going to be the dog. Let's see how it's turned out. Yeah, at least he got four legs and a tail. And all right, that's a retriever, right? So, yeah, this looks pretty good. A dog playing in the snow. Is it going to add more snow here, though? Yeah, not bad, right? So, that was the two oil painting styles. And I think I did add a portrait style, too. Let's try that just for fun. So, let's just do uh a pencil portrait of a female. I think I added a skill on that too. Yeah. Technique for pencil charcoal portrait. Okay. So, here I kind of wrote the script for the pencil painting portrait here. So, it kind of does the this uh Yeah, this is more detailed painting, right? It's doing the face, adding some shadows. Uh this is a bit more detailed. I don't really like the hair here, but uh I think the face looks pretty good. And that is kind of the other technique we had a look at. Can I zoom in on this? Yeah, I wouldn't say it's a masterpiece, but uh it's pretty good to being like an machine drawing this with brush strokes or pencil strokes, right? But yeah, that's what I wanted to share today. Just a fun experiment you can try out for yourself. And it doesn't have to be painting, right? The goal of this is that it could kind of fit in everywhere on anything you do. As long as you have a goal, you can provide all the tools and you have something to kind of steer the LLM or the system to aim to do something. This could be just um unit test or something like that to pass those. It could be anything, right? So, this is a space we're going to explore more. So, if you like this uh kind of content, give this video a like, subscribe, and I'll see you back for more. So, yeah. Hope you enjoyed it. Hopefully, it gave you some inspiration like uh the video did for me. I'm going to leave a link to Mo's video uh in the description. So yeah, thank you for tuning in and I'll see you again soon.