AI Observability with OpenAI Agents SDK

Okay, so we're going to move on to chapter two of the agents SDK course. In this chapter, we're going to be taking a look at the built-in tracing that comes with the SDK. So, just to give you a quick preview of what tracing looks like when you're using it, it looks like

this. So, this is all built in. I haven't or at least for most of these, I have not done anything much beyond passing in an API key. So this is one thing that agents SDK does very well in my opinion. Just by passing my opening API key and turning tracing on in my dashboard and I'll show you how to do that in a moment. We get all of this information about what is going on within our agents when we're using agents SDK. So this is an example from a future chapter that we are going to be working on which is where we're going to be building this essentially JFK rag pipeline. And in this example we have this JFK document assistant. You can see that this is a chat response. So this is when we're getting the LM to generate some text. In this case I can see that the LM generated some text. It decided to use the JFK file search tool. Here I must have been searching something to do with some guy Valerie. And we can see what was returned. Okay. And we also get to see how long each one of these steps took. So that initial tool call from the LM is about a second. The response time there super fast. It's about well 100 milliseconds there. Then the final response. The final response has a bit more to process because we've just given it a ton of documents to read and it needs to respond. You can see that it went through and you got all this input here and then the output was this. So that's pretty cool. We have all that and you'll see it's super easy to set up. This is a pretty short chapter. So

switching across to our code and make sure I'm using the agents SDK course environment there. If you are in Collab, I will just pre-warn you that tracing doesn't work, at least not by default in Collab as of today. Maybe in the future it will, but if you do want to run this, you will unfortunately just have to run this locally. You can't do this part in Collab. At least not right now. There in the future maybe you can. So we come down. If you are running this anywhere remotely, of course you will need open agents. So that is to install. And the first thing we're going to do is just see how we set this up so that we can go into the OpenAI dashboard and see all these traces because by default you actually if you're part of an organization you can't. This information is by default only accessible by the organization owner. So if you're an

engineer in a larger company, you will need to go through a few steps which I have outlined here. So we first need to click that little settings icon at the top. We're then going to click on data controls. We should find a little section here which allows us to set the log visibility. Right? This includes traces. So you're looking at your response logs, stored chat completions, and traces that will need to be set to be visible. either to everyone or visible for selected projects or just visible so that you as a engineer can see it. So once you have selected the required permissions there you click save. Now once you've gained access you should actually be able to see traces already and you can do so by going to your dashboard. Go to traces and then you see here we have all of these traces. So you should be able to see this page. If you can't, you need to talk to whoever owns your open eye organization and fix that. Once you've

confirmed you can see everything, let's switch back to the notebook and we're going to start creating some traces. So, we will need an open eye API key, of course. So, you just enter that here. Then, we're going to go ahead and create a basic agent. Now, as usual, agent object will have that name, instructions, and model. You will see these in the tracing dashboard. Okay, so we're going to start with tracing prompt agent speak like pirate and the model there will be GT4. 1 nano. So we initialize that come down to here and we're going to say write a one sentence poem. We run that and what we should see if we navigate back to our browser click on the most recent trace and we should see okay we've got that tracing prompt agent. So that's the agent name that we just created and we can see what happened inside there. So it used 43 tokens. It used the 4. 1 nano and this is the April 14th model. The instructions were speak like a pirate. Input from us the user was write a one sentence poem and we can see the assistant output there. So that is pretty self-explanatory. There's nothing. We literally did nothing. We just entered our API key and it worked. So that's great, but there is a little more to this. So I want to go through and we'll just see what else we can

gather from traces or how we can even set our own custom traces. So starting with the custom traces to customize our tracing. We use the trace function from the agents SDK library. And with this we can add or modify any of the parameters that we are tracking. And that will look a little like this. So we'll have this with trace and then we would have any of our code that we want to trace here. And that doesn't necessarily actually even need to be agents SDK code. Right? If you have a particular tool and there's something you want to track with that tool, you can do that using this. although that isn't necessarily the intended use of the traces here. So let's come down a little bit and we're going to import that trace object and I'm doing with trace. I'm setting the workflow name. So I can do this. I can set a default name. I'm going to call this the prompt agent workflow. You can see that I'm setting a group ID here. So if we want to group things together, we can do. I'm going to call this one agent SDK course tracing. So all of our the traces we create in this little chapter we can put under this group ID. And what I'm going to do is run this. Okay. So I'm just running the same thing again just within the trace. Let's run that. Switch over to our browser and we should see an update here. You can see the workflow the name has changed. Prompt agent workflow. Now by default if you don't set the workflow name it will just be agent workflow. If you do set the workflow name this is what will appear right your custom workflow name. Now for all the other information well the group ID actually that we added we'd have to click into here we come across to details I believe and we can find the group ID there. Okay. So that's our agents SDK course tracing. Now if we come up here you can also search. So agents and now we can see you know I've done this a couple of times so you can see all the all of the cases where I have set my course or group ID sorry to be agent SDK course tracing. So I could go into here go into details and you can see that there as well. as well as that you can as you can probably see here search by the workflow name. So I've got web search agent. Let's look for that or filter for that. I filter for that. Let's remove this filter and see how many of those we have. Probably oh just three. Okay. So we have three workflows where I set to web search agent. So you can filter by that as well. Now let's come

down and let's have a look at how we would trace tools. So we are going to be using the web search tool from openAI. We're going to be providing this as a tool obviously to a agent. The agent's name will be tracing tool agent here. Uh you can see that the instructions here are your web search agent searches web information on the user's queries. Okay. And we're using 4. 1 mini here. Nano isn't supported for tool use or it's not recommended for tool use. So it's best to use mini there. run that and I'm going to say what are the current world headlines and this will usually come up with something. Yeah. Well, fun in in some way. One thing that I will just note here is look how long that took. It's very long web search with the built-in opening web search tool. I just wouldn't generally recommend it too much. It's slow and not that good. So it's fine. You can use it, but I would use other tools if you can. So let's see what that looks like in our traces. Okay. So we have this agent workflow. That's the default name. So that must be it. Let's have a look. Okay. Let's take a look in here. So we see okay it there was the web search tool was used. We can see okay user what are the current world headlines? Search the web. It has. Okay. Middle East, Ukraine. It opened a page for me. Okay. Back to this. International relations, natural disasters, science and technology list. That one's nice. Very cool. So, yep. I mean, you know, world news. Lovely. That's actually better than a lot of the world. that that's actually not bad for the open eye web search. A lot of time I get a lot of rubbish from it. So could be worse. So yeah, we can see what happened there. It Oh, and this is just the output, right? So this is like a really big response that we got from our agent with ton of sources, ton of information. So really not too bad actually even for that 13 seconds. That's it could be worse. could also be better. Okay, so we can come down. Now, what you know what we just saw is that we had the called the tracing tool agent. Maybe what we'd like to do is actually give that a more the workflow a more specific name. So in this case, we're going to call that web search agent. It's what we saw before. I'm also going to add metadata. Okay. So, we can add metadata in here to well, we can search or filter based on that metadata. I'll show you that in a moment. So, let's run this and jump across to our browser. So, we can see we have the website agent workflow in here. Okay, it's I think it's still running if I'm No, it has Oh, that was it was quite fast that time. So, 6. 6 6 seconds and yeah we get more news there the we can look in the details we have group ID we can see metadata here and we can filter by that metadata so if I go into here and I write it say tools web search tool and we can see all of the cases where that metadata has been added. So you can you'd add a lot of various metadata in there whatever you need basically which is really helpful. Great. So that is

our quick introduction to the agents SDK tracing. As I think you've seen it's pretty useful and it requires minimal setup on our side. We you know we kind of went in there. We're using the trace context but you don't even need to do that. It's I honestly I pro in most projects I probably wouldn't do that. I would just set my IPI key, allow the traces to be collected and go and review them when I need to review them, which I think is just really helpful. Now, that is it for this chapter. It's a quick one. So, I will see you in the next one. Thanks.