New Claude AI Takes Control of Your Computer!

wow so anthropic hit us with an announcement today that nobody really expected they released a brand new version of CLA 3. 5 Sonet the best model when it comes to code generation and they even introduced a new API that allows clouds to control your computer meaning the llm can go ahead and click buttons scroll around use the browser do things for you all based on text prompts what Details Matter here what is the availability of this and how can you get your hands on this brand new computer use API which is the first in AI space people try to build things like this but Fric is the first major company to actually release this let's get into it

starting with the model updates because those are quickly covered so they announced two big updates here Claude 3. 5 Sonet is getting a new version with improved benchmarks across the board Sonet was already impressive but they managed to increase the scores across the board especially when it comes to coding and math related issues with the release of o1 preview some people were debating if that model is better for certain coding use cases I was always of the opinion that Sonic 3. 5 was still state-of-the-art and your goto model if you wanted to generate code but with this update they just solidified their position as this benchmarks improved 5 to even 50% in some cases and just keep in mind these are numbers that were already beating some of the other models like 40 before now the Gap is even larger this is not the type of move that open a would do they only release things if they really have to Claud is just pulling even further ahead here with this new son 3. 5 release the good news is that you can actually use this right now in the Cloud Web interface as you can see if I go into here I have the clae 3. 5 Sonet new pre-selected now I'm not sure why they didn't name this CLA 3. 6 or something like that but there you go CLA 3. 5 Sonet new available right here and again even yesterday I thought that this was the best llm to use for code generation now it's even better and the News behind the news here is that all of these agentic Frameworks like repet agent or pythagora that we covered on the channel here going to perform even better as all they do in the background is talk to models like this so that's a Sonet update and by the way the API is equally as fast and costs the same as it did before the second announcement here is about their small model hu being updated to the 3. 5 level and this jump is just massive compared to the numbers before I mean heck some of these numbers come close to GPT 40 performance look on code generation GPT 4 has 90. 2 clae 3. 5 hu is 88 and the speed and the prices of this are matched I mean this is a gbt 40 mini competitor now the thing to be noted here is that this is not available yet they said this will be released over the coming weeks but honestly as this is the least significant part of this announcement for most people that's totally fine and

now let's talk about the big news here which is what they're calling computer use so here's the base concept you go in and you give it a prompt hey go ahead and check out when the sunrise happens tomorrow in my location and then create a calendar event in my calendar for a sunrise hike and it figures out all the steps it takes screenshots along the way it clicks the buttons that it needs to click it Googles what it needs to Google it takes the data it moves it around it can switch between applications and it just gets it done now obviously this is the very first version of this that we have seen out of one of the AI companies now I'm not saying this was not possible before people built these versions that combine multiple capabilities but this is the first one that is officially coming out of an AI lab and it comes with some demo videos which I definitely recommend you watch I'm not going to go through all of them here you can just check them out on the blog post that is linked in the description below but the first one is essentially showing how it takes data from one spreadsheet and then heads on over to the browser and enters it into a CRM something that a human would need to do manually this thing can just automate with a series of prompts now here's the thing whenever it gets a specific task done then he repr prompts it and gets to the next task this is really visible in the second demo here which you can see in the Twitter Frets that I will also link below and this one is where development focused akin to something like repet agent that we talked about here before and it goes ahead and based on one prompt it creates a website but then it also does the critical steps of actually taking that code and copying them over to vs code running the code in there hosting the site locally so you can access it through the browser and then when he checks out the website he actually figures out that there's some mistake on it and he romps the computer use API to fix the mistake inside of the code editor not just the code it updates the whole app and then the last example is similar to something I outlined earlier where it checks out Maps data it goes to Google it finds the information on the sunrise and it creates a new event inside of the calendar now this is all pretty amazing but they do note that hey this is really just step one it will go wrong a lot of the times it will be very limited but the amazing thing about all of this is we have access to it now and what the implications exactly are and where we will see this in action we don't know yet time will show that and once we do I will create one of those AI use case videos you guys seem to love on this channel something like 20 use cases for anthropics computer use but I usually like to take my time with those to really give people to build interesting things so once that happens we will review all the use cases but as I mentioned you can use this thing today

now how do you use it well here a few ways let me show you so this is through API only so you will not find this inside of the claw interface but they did publish a GitHub repo which is anthropic computer used demo now this takes a few steps and will not be feasible for non-technical people you have to run it inside of a Docker container and then you can try it out yourselves but I did a little bit of digging and they actually worked together with repet before the launch of this thing as repet will be integrating this into the product and replit together with the launch of this actually has a space where you can try this thing right now you do need the repet subscription which I believe is $25 a month less if you go yearly but if you have that you can head on over to this link that I will also link below and they have anthropic computer use demo and this thing couldn't be any easier to use all you need to do is go down to the bottom left here when you're logged in head on over to secrets and instead of secrets you just copy paste your API key from a Fric console that you can get in here and once you did that you can actually click run in here and this will start up and you can start using it now you do have to start here in the web View and give it some sort of commands so let's just give it something that would be relevant to me right now as I'm in Brazil for a kite surfing workation type of situation and I'll just tell it's something that is top of mind like find the wind forecast for the next seven days in guyu and then find the car rental site in this region I don't know I'll just run this and let's have a look at what happens here we need to head on over to this output pane to see it actually work and here when I expanded I believe we should see a browser and all of this should be happening autonomously let's see if it worked okay it seems to be working but it wasn't showing me the output so let me try this one more time input the same prompt say run okay the agent is running and it's already using the tools it's taking screenshots as we mentioned it's working with those screenshots and here in the output pane we should actually see it work live which is pretty cool okay so it's inside of Firefox now it should find the website with the wind forast uhuh Okay Google searching wy. com gyo that's good that's actually the app that I use it got spot name right too I mean I gave it to it but fair enough it clicks that and if we check the agent view this is good this is progressing very well you will see that it's actually taking screenshots here it's running prompts it's taking actions it's using the computer to Mouse move to specific coordinates and then it uses the left click to actually go in there you see all of this is happening here in the background fully autonomously and it's already on here so this is very interesting let using the search again and as you can see this works and people will need to experiment with this build with this but it's a very exciting time because this is really the moment of inception of autonomous agents we've been moving in this direction more and open air has been teasing us with things like o1 preview that can reason and advanced voice mode that is a really good voice interface but they never gave us this they didn't give us the ability the power to actually remote control a computer with llm and now we have it from mopic it's the first version and in the next days and weeks will show how useful this thing actually is in practice but think about it learning more about this is a literal superpower if you know how to prompt a little bit and use this thing you can just set up multiple agents and have your own Army of plot 3. 5 Sonet instances run around and do things with computers on the internet you can have them do tasks research you could even have them run a service business if you come up with the right workflow and the right prompts to go along with that I'm super excited about this can't wait to see what else this brings and that is pretty much the update and if you want to learn more about this I'll actually be running a community event on this in about 10 days every month I run llm Innovations and this is the perfect topic so we'll dive deep on this and teach your community members how to set this up for themselves because I personally really believe in this agentic direction I think although these things might be primitive right now if you look close enough you will find the use cases that matter and that can make a difference in your life and me and the team are here to support you on your way okay that's all I got for today leave a comment below with what you will be using this for and I'll see you soon