Getting started with Weights & Biases for robotics

Hi, my name is Anu and I'm a machine learning engineer. Right now, I'm working on fine-tuning some vision language action models for a robotics team. And because fine-tuning these models takes so long, sometimes up to a week, my team has started using weights and biases to track all of this. Let me show you how. Once you jump into my workspace, you'll see I am fine-tuning a Groot by Nvidia. This is a VLA, but really weights and biases works with all libraries and all infrastructure and all devices. So, your team can really experiment around with different options until you find what works best for you. You see, in our case, we're using some

video data of a robot performing a task and trying to train our model to do the same thing. Because of that, I'm doing something called hyperparameter tuning. I'm looking at a bunch of different options of parameters and training my model to make sure I can find the best combination that works for our tasks. And weights and biases tracks all of this natively for me. They even give me this really nice chart telling me which of these parameters were the most important. If I want, I've got some really easy controls right here in the UI to pause or stop this training if something goes wrong. Now, for my team

something we really love that I'm working on setting up right now is these rollout simulations. So for every experiment that we run, we take the trained model artifact and then put it in Isaac Sim to create this roll out to see if our robot that we're training is actually learning to get better at that task. And you see all the metrics from that is tracked within here. Just like weights and biases tracks all my training metrics and system metrics already. This helps our team a lot with tracking everything and putting it in one single place. We all can communicate about these projects. Now, whenever I or

somebody in my team runs an experiment, we get a artifact from it that's trained and weights and biases actually does the heavy lifting on tracking all the input artifacts like the data set that's being trained on. You saw some clips from that earlier. Or maybe the input model that I'm using. All the information, the files, the documentation for this lives within here. I can even go and see within this project who and when uploaded this model and every single experiment that's consuming it. So I know who else is fine-tuning this from the model we all started with and go and look at their results as well. Now my team even organizes uh links all our best models into an org byte registry. So I don't keep everything local to my project but give this access to everyone to make sure anyone doing real physical rollouts on the really expensive humanoids we just created has access to the right models to do all of that with and they know what they're deploying is prod ready and has been tested. So this makes our teams communicate really well.

Talking about other teams, I work with this really cool other team too that's doing some really quality work on reinforcement learning. And you see they have a meticulously put together workspace because in weights and biases they can go and customize this to show them exactly the information they care about like these roll out videos from their reinforcement learning because in their process they train a little bit and evaluate at each step. They can actually look at the simulation from each step and understand how their model performed at each step and at exactly what point it goes from dropping the object to actually being able to successfully lift it. I love how meticulous they are with this organization. They even create these amazing reports to track every step of the process they're working on. And so they can share with their entire team exactly how these models are being trained and what metrics they're seeing in eval. And look, this is an autogenerated report. This team is so advanced. They have this really cool automation setup in their workspace that triggers every time a run finishes. So they can have a report. They even have automated emails. I think I need to start implementing some of this stuff, you know. Let's do something really quickly. I already have some of these rollout simulations available here. I'll add them to a report. So when I come back next, I can make sure that all of my work is also tracked within here. And especially when I'm working with this team, I'll be able to make sure I share with them my outcomes as well. So, let's just call it simulation report. And you know, after this, I'm going to come here and add some commentary and more metrics so everyone on my team can understand what this report is intended for. Maybe I'll get to the same level as that other team. Now, before I step away from my computer, this experiment is going to keep running, and I want to make sure this doesn't crash or at least I know when that happens. So what I'll do is

I'll go ahead and set this quick alert from the UI and I'll make sure if the there is an increase in loss of more than 20% I can get a slack alert in our machine learning team channel. This will make sure that whenever this happens at least somebody in my team should be me knows exactly at which step this took place so I can go and correct that next time I'm working on the experiments. Now, I think I'll take a break. Let's hope this experiment runs well and thanks for watching. If you're curious about these projects, they're all linked below so you can see how the reinforcement learning and the fine-tuning evolved over time after this video is posted. And if you're interested in trying this, go to wby. ai and sign up today and test this out.