Who let the robot dogs out?

Today, a lot of the emphasis is on how frontier AI models are transforming software engineering. What we're interested in understanding is how that can begin to translate into the physical world. Robotics is sort of the clear entry point to how you have a mostly software system, start having the ability to reach out into the real world.

Project Fetch is this self-contained experiment where we wanted to measure how much does Claude accelerate humans performing a fairly sophisticated technical task that they do not have experience with? Project Fetch was a one day experiment. The experiment was three phases. All of these tasks were shaped approximately like get this robot dog to fetch a beach ball. There were two teams. These teams were comprised of software engineers and research engineers at Anthropic that had hardly any robotics experience. One team had access to Claude and the other team did not.

Phase one was very simple. It was to use the pre-provided controllers to get the dog to walk out to a beach ball and bring it back to where it started. Alright. I see. It's pretty intuitive. And we're supposed to bring it over by the bone? Yeah, I think. I think the team with Claude took about seven minutes. Alright, go attack that team now. Go attack their dog. Charge! Shoot, guys. They’re destroying us. Oh my god. Wait. We're getting, we're getting destroyed. The team without Claude, I think, took 10 minutes. Oh, sorry. It's going to hit you. Alright. I'm going to do a victory dance.

Phase two was also a game of fetch. But this time the teams had to program their own controller. You have to actually get access to the hardware and design a program that you can write on your laptop that will control the dog. Claude just like one-shotted a whole controller. Alright, I'll do some calisthenics. Nice, nice. Is this for— Oh, this is just control. But that's all we need I guess. This is from the official ROS2 SDK, and I got this installed, but then it's asking for a whole bunch of other packages, and that's all failing. I've never really understood how reliant I am on Claude doing the menial work, finding all the nitty gritty details I don't want to have to figure out. We can't, we can't get nervous about them. You know what, I'm just going to install PIP from the actual container later, so. Oh wait. No, I can't. I know I'm impatient. It's been over a minute. One of the primary bottlenecks of the experiment is that you have this hardware, you have this complicated piece of technology, you have your laptop, and you have to, like, get your laptop talking to this hardware. I am setting my Claude up to create a dog server that all of our computers can connect to see what the dog is seeing. There are many different software libraries on the internet for communicating with this particular robot, and Claude found these things for them, it installed the right things on their computer and it pretty quickly got them access to the dog. Oh shit. So fast. Watch out. Careful now. Try not to run into the table. Uh oh. Turn around. It has a mind of its own. Turn it off, it's alive. Oh shit. Stop, stop, stop. I think that team should be disqualified for hitting another participant. The team with Claude finished phase two in about 2 hours and 15 minutes. Probably the area where we saw the most uplift from Claude was just in the task of connecting to the robot. We think that's really important because it is, in fact, difficult for anyone to identify an arbitrary piece of hardware in the world and figure out how to talk to it and how to control it. I think they got their camera. They got their camera working? Yeah. Shit. Was Claude even helpful for this part? Or are we just slow? Yeah. We're not getting very far, but that's okay. It's a learning experience. The team without Claude really struggled with this and went down a lot of different paths. None of which were especially successful. And we basically had to intervene and be like, alright, here, here is a strategy that we know works. Start from there, and then this will unlock kind of the rest of the phase and experiment for them. Nice. Oh great. Uh, Daniel? Daniel or Kevin?

Phase three of the experiment was a greater degree of autonomy. The task in phase three was to write a program that would get the dog to fetch a beach ball all by itself. Essentially, just press go and have the robot search around, detect the location of the ball, walk to the ball and bring it back, all autonomously. This is like ratcheting up in difficulty kind of by design, but also gesturing at the real problem that we expect frontier models having to solve in the future is essentially this kind of autonomous version where, like if a frontier model wants a robot to do something for it, it needs to be able to solve this very hard problem. The team without Claude in phase three did a good job of the initial task of coming up with a way to track the location of the robot in space. They made progress on the task of detecting the ball, but they didn't really come close to knitting everything together. I miss Claude so much. The team with Claude actually came fairly close to finishing phase three. I think by the end the team with Claude was maybe an hour and a half away from being done.

The results of the experiment were essentially that the team with Claude completed all of the things that they did complete in a couple of hours faster than the team without Claude. In the near term we think that AI models are going to do exactly what we showed in this experiment, which is making it easier for people without a lot of robotics experience to engage meaningfully with robots. Just with this one tool we have, we've dramatically accelerated their ability to do things with this robot. We didn't go like train Claude to uplift humans to do robotics tasks. This is just a thing that fell out of this technology. And then maybe in the long run, this is kind of a leading indicator of where the whole system is going. What today requires the combination of a person and an AI model, tomorrow is likely to just require the AI model. The effects of AI are not just going to be in software, they are hardware and in the physical world as well.