AI Observability with OpenAI Agents SDK

00:00:00.000 | Okay so we're going to move on to chapter two of the Agents SDK course. In this chapter we're going

00:00:05.940 | to be taking a look at the built-in tracing that comes with the SDK. So just to give you a quick

00:00:13.160 | preview of what tracing looks like when you're using it, it looks like this. So this is all

00:00:19.520 | built-in. I haven't, or at least for most of these, I have not done anything much beyond passing in an

00:00:26.560 | API key. So this is one thing that Agents SDK does very well in my opinion. Just by passing my

00:00:33.480 | OpenAI API key and turning tracing on in my dashboard, and I'll show you how to do that in a

00:00:39.420 | moment, we get all of this information about what is going on within our Agents when we're using

00:00:46.940 | Agents SDK. So this is an example from a future chapter that we are going to be working on, which

00:00:54.660 | is where we're going to be building this essentially GFK RAG pipeline. And in this example we have this

00:01:01.580 | GFK document assistant. You can see that this is a chat response. So this is when we're getting the

00:01:08.960 | LLM to generate some text. In this case I can see that the LLM generated some text. It decided to use

00:01:14.040 | the GFK file search tool. Here I must have been searching something to do with some guy here,

00:01:21.720 | Valerie. And we can see what was returned. Okay. And we also get to see how long each one of these steps

00:01:28.960 | took. So that initial tool call from the LLM is about a second. The response time there, super fast,

00:01:35.960 | it's about, well, 100 milliseconds there. Then the final response, the final response has a bit more

00:01:42.600 | to process because we've just given it a ton of documents to read and it needs to respond. You can

00:01:48.640 | see that it went through and got all this input here and then the output was this. So that's pretty cool.

00:01:57.740 | We have all that and you'll see it's super easy to sell. This is a pretty short chapter.

00:02:02.700 | So switching across to our code and make sure I'm using the agents SDK course environment there.

00:02:09.040 | If you are in Colab, I will just pre-warn you that tracing doesn't work, at least not by default,

00:02:16.980 | in Colab as of today. Maybe in the future it will, but if you do want to run this, you will

00:02:24.100 | unfortunately just have to run this locally. You can't do this part in Colab. At least not right

00:02:29.520 | now. In the future, maybe you can. So we come down. If you are running this anywhere remotely,

00:02:37.100 | of course, you will need OpenAI agents. So that is to install. And the first thing I'm going to do is

00:02:43.140 | just see how we set this up so that we can go into the OpenAI dashboard and see all these traces because

00:02:51.900 | by default, if you're part of an organization, you can't. This information is by default only

00:02:58.480 | accessible by the organization owner. So if you're an engineer in a larger company, you will need to

00:03:04.100 | go through a few steps, which I have outlined here. So we first need to click that little settings icon

00:03:10.400 | at the top. We're then going to click on data controls. We should find a little section here,

00:03:16.420 | which allows us to set the log visibility, right? And this includes traces. So you're looking at your

00:03:23.720 | response logs, stored, check completions, and traces. That will need to be set to be visible either to

00:03:31.020 | everyone or visible for selected projects or just visible so that you as a engineer can see it.

00:03:37.800 | So once you have selected the required permissions there, you click save. Now, once you've gained access,

00:03:44.380 | you should actually be able to see traces already. And you can do so by going to your dashboard, go to traces,

00:03:50.620 | and then you see here we have all of these traces. So you should be able to see this page. If you can't,

00:03:57.560 | you need to talk to whoever owns your OpenAI organization and fix that. Once you've confirmed,

00:04:04.180 | you can see everything. Let's switch back to the notebook, and we're going to start creating some traces.

00:04:08.420 | So we will need an OpenAI API key, of course. So you just enter that here. Then we're going to go ahead

00:04:15.060 | and create a basic agent. Now, as usual, agent object will have that name instructions and model.

00:04:22.500 | You will see these in the tracing dashboard. Okay, so we're going to start with tracing prompt agent,

00:04:29.620 | speak like a pirate and the model there will be GT 4.1 nano. So we initialize that, come down to here,

00:04:37.460 | and we're going to say, write a one sentence poem. We run that. And what we should see, if we navigate

00:04:45.860 | back to our browser, click on the most recent trace, and we just see, okay, we've got that tracing prompt

00:04:53.780 | agent. So that's the agent name that we just created. And we can see what happened inside there.

00:04:59.460 | So it used 43 tokens. It used the 4.1 nano, and this is the April 14th model. The instructions

00:05:06.820 | were speak like a pirate. Input from us, the user was write one sentence poem. And we can see the

00:05:13.540 | assistant output there. So that is pretty self-explanatory. There's nothing, we literally did

00:05:20.340 | nothing. We just entered our API key and it worked. So that's great, but there is a little more to this.

00:05:27.460 | So I want to go through and we'll just see what else we can gather from traces or how we can even set

00:05:34.660 | our own custom traces. So starting with the custom traces, to customize our tracing, we use the trace

00:05:43.620 | function from the agent's SDK library. And with this, we can add or modify any of the parameters that we

00:05:52.740 | are tracking. And that will look a little like this. So we'll have this with trace and then we would have

00:05:58.260 | any of our code that we want to trace here. And that doesn't necessarily actually even need to be

00:06:03.940 | agent's SDK code, right? If you have a particular tool and there's something you want to track with

00:06:10.420 | that tool, you can, you can do that using this. Although that isn't necessarily the intended use

00:06:15.940 | of the traces here. So let's come down a little bit and we're going to import that trace object.

00:06:24.100 | And I'm doing with trace, I'm setting the workflow name. So I can, I can do this. I can set a default name.

00:06:29.700 | I'm going to call this the prompt agent workflow. You can see that I'm setting a group ID here. So if

00:06:34.980 | we want to group things together, we can do, I'm going to call this one agents SDK course tracing. So

00:06:39.940 | all of our, the traces we create in this little chapter, we can put under this group ID. And what

00:06:46.180 | I'm going to do is run this. Okay. So I'm just running the same thing again, just within the trace.

00:06:53.220 | Let's run that. Switch over to our browser and we should see an update here. You can see the workflow,

00:07:01.700 | the name is changed, prompt agent workflow. Now by default, if you don't set the workflow name,

00:07:07.620 | it will just be agent workflow. If you do set the workflow name, this is what will appear, right?

00:07:13.540 | Your custom workflow name. Now for all that other information, well, the group ID actually that we

00:07:19.620 | added, we'd have to click in to here. We come across to details, I believe, and we can find a group ID

00:07:27.540 | there. Okay. So that's that agents SDK course tracing. Now, if you come up here, you can also

00:07:33.540 | search. So agents SDK course tracing.

00:07:40.420 | And now we can see, you know, I've done this a couple of times. So you can see all the,

00:07:48.020 | all of the cases where I have set my course or group ID, sorry, to be agents SDK course tracing.

00:07:55.220 | So I could go into here, go into details, and you can see that there as well. As well as that,

00:08:00.580 | you can, as you can probably see here, search by the workflow name. So I've got web search agent. Let's

00:08:07.140 | look for that or filter for that.

00:08:09.140 | Our filter for that, let's remove this filter and see how many of those we have probably. Oh, just three. Okay.

00:08:16.500 | So we have three workflows where I set to web search agent. So you can, you can filter by that as well.

00:08:23.780 | Now let's come down and let's have a look at how we would trace tools. So we are going to be using the

00:08:31.780 | web search tool from open AI. We're going to be providing this as a tool, obviously to a agent.

00:08:38.500 | The agent's name will be tracing tool agent. Yeah. You can see that the instructions here are your

00:08:45.460 | web search agent searches the web information on the user's queries. Okay. And we're using 4.1 mini here.

00:08:52.420 | Nano isn't supported for tool use or it's not recommended for tool use. So it's best to use mini there.

00:08:59.460 | Run that. And I'm going to say, what are the current world headlines? And this will usually come up with

00:09:04.100 | something. Yeah. Yeah. Well, fun in, in, in some way. One thing that I will just know here is

00:09:13.540 | look how long that took. Yes. It's very long web search with the built-in opening our web search tool.

00:09:20.740 | I just wouldn't generally recommend it too much. It's slow, slow, and not that good. So it's fine.

00:09:28.740 | You can use it, but I would use other tools if you can. So let's see what that looks like in our traces.

00:09:36.660 | Okay. So we have this agent workflow. That's the default name. So that must be it. Let's have a look.

00:09:40.980 | Okay. Let's take a look in here. So we see, okay. It, there was the web search tool was used.

00:09:49.300 | We can see, okay, user, what the current world headlines search the web. It has, okay. Middle East.

00:09:56.340 | Ukraine. Ukraine. It opened a page for me. Okay. Back to this. International relations,

00:10:06.260 | natural disasters, science and technology list. That one's nice. Very cool. So, yep. I mean, you know,

00:10:16.740 | world news. Lovely. That was actually better than a lot of the world. That's actually not bad for the

00:10:22.740 | OpenAI web search. A lot of time, I get a lot of rubbish from it. So it could be worse.

00:10:30.740 | So yeah, we can see what happened there. Oh, and this is just the output, right? So this is like a really

00:10:38.180 | big response that we got from our agent with a ton of sources, ton of information. So really not too bad,

00:10:45.940 | actually. Even for that 13 seconds, I suppose. It could be, it could be worse. It could also be better.

00:10:51.140 | Okay. So we can come down now. What, you know, what we just saw is that we had the, it was just called

00:11:00.500 | the tracing tool agent. Maybe what we'd like to do is actually give that a more, the workflow a more

00:11:05.700 | specific name. So in this case, we're going to call it that web search agent. It's what we saw before.

00:11:12.980 | I'm also going to add metadata. Okay. So we can add metadata in here to, well, we can, we can search

00:11:21.860 | or filter based on that metadata. I'll show you that in a moment. So let's run this and jump across

00:11:28.820 | to our browser. So we can see, we have the website agent workflow in here. Okay. It's, I think it's still

00:11:36.580 | running if I'm no, no, no, it has. Oh, that was, it was quite fast that time. So 6.6 seconds. And yeah,

00:11:45.140 | we got, we'll get more news there. The, we can look in the details. We have group ID. We can see metadata

00:11:51.700 | here and we can filter by that metadata. So if I go into here and I write tools, web search tool,

00:12:04.900 | and we can see all of the cases where that metadata has been added. So you can, you'd add a lot of

00:12:09.940 | various metadata in that, whatever you need basically, which is, is really helpful. Great. So that is our

00:12:17.780 | quick introduction to the agent's SDK tracing. As I think you've seen, it's pretty useful and it requires

00:12:26.980 | minimal setup on our side. We, you know, we kind of went in there, we're using the, the trace context,

00:12:34.020 | but you don't even need to do that. It's I, I honestly, I pro in most projects, I probably wouldn't

00:12:41.780 | do that. I would just set my API key, allow the traces to be collected and go and review them when I need to

00:12:48.180 | review them, which I think is just really helpful. Now, that is it for this chapter. It's a quick one.

00:12:56.420 | So I will see you in the next one. Thanks.

00:12:59.860 | Bye.

00:12:59.980 | Bye.

00:13:11.660 | Bye.

00:13:11.660 | you

00:13:13.720 | you

AI Observability with OpenAI Agents SDK

Chapters