WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=gv4w23xYj-0 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/gv4w23xYj-0.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.560 --> 00:00:15.440 Um, before we get started, I just wanted to introduce myself. s3 00:00:15.440 --> 00:00:18.240 I'm Sally Ann, and I'm head of product at Arize. s4 00:00:18.240 --> 00:00:20.600 Uh, been at Arize for like 4 years now. s5 00:00:20.600 --> 00:00:25.320 Um, one of the first projects that I owned actually was building our AI agent Alex into our platform. s6 00:00:25.320 --> 00:00:31.320 So, I've been dealing evals for a little while now, and uh, even building a lot of our evals product. s7 00:00:31.320 --> 00:00:37.160 And so, I think the way that I think about evals is that they're our primary signal for our agents. s8 00:00:37.160 --> 00:00:41.600 Um, and that includes all of our evals at Arize. s9 00:00:41.600 --> 00:00:44.640 So, my goal today is you'll leave with three things. s10 00:00:44.640 --> 00:00:48.640 Um, a clear definition of what an eval is and what it isn't. s11 00:00:48.640 --> 00:00:50.720 Uh, being able to write your own criteria. s12 00:00:50.720 --> 00:01:00.280 So, I'm hopefully going to be able to provide you with like a little bit of a recipe uh, for how you can think about evals and how you can apply that for any of your real agent scenarios. s13 00:01:00.280 --> 00:01:02.960 And then we're actually going to build a working judge in Arize. s14 00:01:02.960 --> 00:01:09.280 Um, and I'll show you a little bit about the life cycle there and how you can think about improving your evals over time. s15 00:01:09.280 --> 00:01:11.080 So, this is a bit of our agenda. s16 00:01:11.080 --> 00:01:12.840 So, again, I'm going to give you a little bit of context. s17 00:01:12.840 --> 00:01:20.200 I don't want to spend too much time in slides, but I want to make sure everyone has a really solid foundation for what an eval is and how you should think about it. s18 00:01:20.200 --> 00:01:26.000 Um, we're going to explore some data in Arize together, um, write our criteria, and then we'll actually go build our eval. s19 00:01:26.000 --> 00:01:28.600 That's the main part of our workshop today. s20 00:01:28.600 --> 00:01:32.560 Um, and then I'll give you a little bit of like where you could take this for what's next. s21 00:01:32.560 --> 00:01:36.480 Um, I love it to kind of have some like working sessions so people can collaborate together. s22 00:01:36.480 --> 00:01:39.600 I'll try to like walk around the room, help folks out. s23 00:01:39.600 --> 00:01:46.960 Um, and then I will show kind of like a one-pager that's in Notion that you can kind of open and follow along as well. s24 00:01:47.000 --> 00:01:48.560 So, just check the room a little bit. s25 00:01:48.560 --> 00:01:55.080 Who here has either like shipped a chatbot or an agent, ever written an eval, or even just opened the Arize product today? s26 00:01:55.080 --> 00:01:57.400 Anyone here? s27 00:01:57.680 --> 00:01:59.240 All right, we got a a few folks here. s28 00:01:59.240 --> 00:02:06.292 So, I think this is going to be great for folks who maybe haven't done any of this and even if you have, I think today should still be pretty helpful. s29 00:02:06.292 --> 00:02:11.812 So, going to start off the context with like why I don't think vibe checks actually scale. s30 00:02:11.812 --> 00:02:15.132 So, this is the vibes problem as I see it. s31 00:02:15.132 --> 00:02:18.972 You know, you have Monday, you maybe change something to your prompt, you ship it. s32 00:02:18.972 --> 00:02:20.492 It overall feels better. s33 00:02:20.492 --> 00:02:24.212 You don't really have a way to know, but overall things are looking good. s34 00:02:24.212 --> 00:02:29.892 You know, Tuesday ship it, a week later now that use case that you shipped a fix for is broken. s35 00:02:29.892 --> 00:02:37.732 And then next week, as the problem continues to get worse and worse, there's no way for anybody to prove whether something was fixed or that it's actually failing. s36 00:02:37.732 --> 00:02:41.452 have vibes to go off of. s37 00:02:41.572 --> 00:02:46.612 What you can't do without evals is you can't detect regressions when you change a prompt. s38 00:02:46.612 --> 00:02:52.012 You can't objectively compare prompt or even different model versions. s39 00:02:52.012 --> 00:03:00.812 You don't know if a new model is any better for your use case and you can't actually put any official quality gates into CI as you're shipping your agent. s40 00:03:00.812 --> 00:03:08.292 So, without evals it's you can imagine it's it's pretty hard to actually know how your agent is performing. s41 00:03:08.292 --> 00:03:13.692 A question I commonly get is is an eval just a unit test? s42 00:03:13.692 --> 00:03:21.852 I think that there's a lot of comparison of AI evals to like kind of traditional software tests and I do think that they're a little bit different. s43 00:03:21.852 --> 00:03:26.372 Cuz first off, with a test you kind of just have like one input and there's one correct output. s44 00:03:26.372 --> 00:03:29.652 You can also read all of the logic in the code. s45 00:03:29.652 --> 00:03:33.132 With an AI agent, you actually cannot read any of that logic. s46 00:03:33.132 --> 00:03:36.492 That actually comes out after in the form of traces. s47 00:03:36.492 --> 00:03:46.692 And so, you kind of need to have this AI eval where you can look at like the day-by-day itinerary in this travel planner example I have here, the budget breakdown or s48 00:03:46.692 --> 00:03:48.652 multiple criteria basically. s49 00:03:48.652 --> 00:03:55.812 So, there's these many valid outputs that you just can't have, you know, a more traditional unit test on like you can with software. s50 00:03:56.852 --> 00:04:03.894 So, basically how I like to think about this is traces are the logs for AI Um, evals are actual tests. s51 00:04:03.894 --> 00:04:10.014 And I always say that uh uh I used to have a meme, I didn't include it today, but the hard pill is you kind of need traces, you need observability s52 00:04:10.014 --> 00:04:13.214 to actually do your evals. s53 00:04:13.294 --> 00:04:17.094 So, this is an example here for why agents make this harder. s54 00:04:17.094 --> 00:04:21.894 So, this is just a sample example of what like a trace could look like. s55 00:04:21.894 --> 00:04:31.334 And for anybody in the room who's maybe not familiar with what a trace is, um essentially it is all of the steps that an AI agent took to be able to get an answer back to a response. s56 00:04:31.334 --> 00:04:37.974 So, a user says something and then what I'm showing here in all of those spans is all the individual steps the agent took uh to get that answer. s57 00:04:37.974 --> 00:04:45.014 So, in this example, there's like a research agent that's going to go perform some research, uh maybe even do like some tool calls there. s58 00:04:45.014 --> 00:04:48.534 You have, you know, your local agent that's maybe retrieving something from a knowledge base. s59 00:04:48.534 --> 00:04:51.094 So, lots of back and forth are happening. s60 00:04:51.094 --> 00:04:54.654 And this is where I mean you can see the logic of your agent. s61 00:04:54.654 --> 00:04:58.454 Um, and we're going to explore some traces in the Arize UI just a little bit. s62 00:04:58.454 --> 00:05:03.254 Um, so you can see there's lots of LM calls, lots of things composed together. s63 00:05:03.254 --> 00:05:14.654 And so, you can imagine just having one single unit test like in software not really going to accurately capture all the complexity that's in an agent and be able to tell us how it's performing. s64 00:05:15.694 --> 00:05:17.774 There's actually lots of different types of evals. s65 00:05:17.774 --> 00:05:20.054 Today, we're going to focus on LM as a judge. s66 00:05:20.054 --> 00:05:26.534 Um, LM as a judge is where we take a secondary LLM and we ask it to grade some aspect of our agent. s67 00:05:26.534 --> 00:05:34.094 Um, and so, there can be things like QA, summarization, relevance, and then you get a label and an explanation out of that. s68 00:05:34.094 --> 00:05:35.454 Explanations are really helpful. s69 00:05:35.454 --> 00:05:39.934 Explanations are kind of like the the reasoning or the thinking behind providing a label. s70 00:05:39.934 --> 00:05:43.934 It gives you a little bit of understanding and not just have to take a label at face value. s71 00:05:43.934 --> 00:05:46.254 But, there's lots of different types of evals. s72 00:05:46.254 --> 00:05:48.214 I want to call out decision models. s73 00:05:48.214 --> 00:05:50.254 Uh this is something that's pretty new. s74 00:05:50.254 --> 00:05:59.014 Uh if anybody kind of took or was a part of the the Jevas a Job storm that happened about 2 weeks ago and OpenAI just last week released their decision model API. s75 00:05:59.014 --> 00:06:03.454 So, I think decision models are going to be a huge part of evals in the future. s76 00:06:03.454 --> 00:06:11.655 Um, those are pretty cheap and fast and it's uh more like a question answer and you just get like a probability and an output whether it's like a true or false or a label. s77 00:06:11.655 --> 00:06:15.495 So, you don't get that explanation that you get with Elements of AI Judge. s78 00:06:15.495 --> 00:06:17.895 The other type of eval is code. s79 00:06:17.895 --> 00:06:20.375 Uh, these are really great for any deterministic tests. s80 00:06:20.375 --> 00:06:29.055 So, maybe you have some eval criteria that's like checking if a word is in an output or making sure that something's in JSON. s81 00:06:29.055 --> 00:06:32.255 Uh, that's going to be really great for code-based evals. s82 00:06:32.255 --> 00:06:36.775 There's also things like Rouge and Blue scores that are popular code evals. s83 00:06:36.775 --> 00:06:39.375 Um, annotations I do see as a form of evaluation. s84 00:06:39.375 --> 00:06:43.015 This is where we're having uh you know, golden truths on golden datasets. s85 00:06:43.015 --> 00:06:45.375 Um, sometimes they're synthetic as well. s86 00:06:45.375 --> 00:06:51.415 Um, agent as a judge is a newer type of eval that we've seen maybe over the last 6 months. s87 00:06:51.415 --> 00:06:55.255 Um, that's where we're going to take a full agent and evaluate something in a different agent. s88 00:06:55.255 --> 00:07:00.495 I think this is really good for super complex um like success criteria. s89 00:07:00.495 --> 00:07:05.775 Um, then we have user feedback and business metrics and those three at the end are going to be more for your production use cases. s90 00:07:05.775 --> 00:07:12.015 But, you'll notice the first four are something that you want to use across your life cycle in development and production. s91 00:07:12.015 --> 00:07:20.375 Um, and that's pretty important because you want this to be able to evolve with your use case and you don't want to like put a lot of investment in something you're using in development and just throw it away. s92 00:07:20.375 --> 00:07:26.615 Those evals can grow with you over time and it gives you that consistent um performance. s93 00:07:29.335 --> 00:07:32.615 All right, let's go ahead and look a little bit at the data. s94 00:07:32.615 --> 00:07:37.255 Um, so we're going to pause here on this slide give everyone a few minutes to go and get started. s95 00:07:37.255 --> 00:07:44.015 That QR code is going to be for the one-pager which has all of the instructions for our workshop today. s96 00:07:44.015 --> 00:07:48.975 Um, but I want everyone here to go ahead to go to you know, app.arise.com. s97 00:07:48.975 --> 00:07:50.375 Uh, we want to log into AX. s98 00:07:50.375 --> 00:07:55.335 So, if you don't have an account um encourage you to make a free um account today. s99 00:07:55.335 --> 00:07:59.135 Um, and then I'll go over the steps from there in just a second. s100 00:07:59.135 --> 00:08:05.375 Um, and for anybody who already I guess might have an arise account, the next step is going to be getting an API key from um a model provider. s101 00:08:05.375 --> 00:08:11.417 So, I'll pause there uh walk around make sure nobody has any issues logging into the platform. s102 00:08:11.417 --> 00:08:14.977 The question was, can you do you have to use like can you use a custom endpoint? s103 00:08:14.977 --> 00:08:18.897 Do you have to use a a common provider for step two? s104 00:08:18.897 --> 00:08:21.657 No, feel free to set up whatever LLM provider you want. s105 00:08:21.657 --> 00:08:25.177 We'll just want something that we can make LLM calls to so we can test our eval out. s106 00:08:25.177 --> 00:08:32.817 But yeah, step two basically go to your Anthropic console, your OpenAI console, whatever provider of your choosing. s107 00:08:32.817 --> 00:08:39.737 You can find your AI providers under settings in the product and we're just going to want to go ahead and add that API key in. s108 00:08:39.737 --> 00:08:45.577 So, let me go to my laptop here and show you all a little bit of how to do this. s109 00:08:46.217 --> 00:08:49.537 So, settings here in the nav. s110 00:08:49.537 --> 00:08:58.897 You're going to go click into that and then under your account settings, let me try to Cool. s111 00:08:58.897 --> 00:09:02.137 You're going to come to AI providers. s112 00:09:02.537 --> 00:09:04.697 And you can see here we have a whole bunch of different providers. s113 00:09:04.697 --> 00:09:09.817 So, just choose one that is your favorite to use, Anthropic, OpenAI, Gemini, doesn't matter. s114 00:09:09.817 --> 00:09:14.857 Just something that we can use to make API calls with. s115 00:09:20.537 --> 00:09:28.377 And I encourage you to use the test integration button when you're doing that just so we can make sure we have a working API key. s116 00:09:28.377 --> 00:09:30.857 Yeah, for for folks you'll find the demo project. s117 00:09:30.857 --> 00:09:31.777 Sorry, I'll go back. s118 00:09:31.777 --> 00:09:40.217 Just trying to make sure everyone has time to to catch up, but you can access the demo project that we're going to use in one of two ways. s119 00:09:40.217 --> 00:09:47.457 The first is on from this when you go to this tracing page here, you'll see see demo project. s120 00:09:47.457 --> 00:09:48.977 I have the LLM agent. s121 00:09:48.977 --> 00:09:51.817 I know some folks might have our older demo model there. s122 00:09:51.817 --> 00:09:54.897 Does not matter which one you have, you should be able to do our exercise today. s123 00:09:54.897 --> 00:09:58.897 So, we just want to be able to look at some traces. s124 00:09:58.897 --> 00:10:07.337 And you know, you can expand the time if you want to see a little bit more data there, but give everyone a few more minutes cuz I only saw s125 00:10:07.337 --> 00:10:10.058 a handful of hands that got the AI provider. s126 00:10:10.058 --> 00:10:16.098 and to see what I can do to see if I can get one that we can share really quickly, too. s127 00:10:16.098 --> 00:10:22.378 All right, so if you have that travel agent demo project opened, this is a travel planning agent. s128 00:10:22.378 --> 00:10:31.098 It takes a request from a user and then can it make follow-up questions, call tools, and basically returns a plan. s129 00:10:31.098 --> 00:10:38.218 What I think good looks like in product terms is it answers the follow-up questions the user asked for and respects the trip's days and budget. s130 00:10:38.218 --> 00:10:40.978 So, this is generally what a user request looks like. s131 00:10:40.978 --> 00:10:48.258 You can see the destination, how long they want to go, what their budget is, and what their kind of general travel style looks like. s132 00:10:48.258 --> 00:10:52.978 And that's an example of what we might see from a user who goes to this travel agent. s133 00:10:52.978 --> 00:10:55.418 So, I'd love for everyone to take a a few minutes. s134 00:10:55.418 --> 00:10:57.658 We want to look at the traces. s135 00:10:57.658 --> 00:10:59.018 We kind of want to just take inventory. s136 00:10:59.018 --> 00:11:00.458 What are users asking for? s137 00:11:00.458 --> 00:11:02.578 What did the agent tool do? s138 00:11:02.578 --> 00:11:05.018 What tools, what are the answers? s139 00:11:05.018 --> 00:11:07.898 And then just kind of make note, was this an answer that you would share? s140 00:11:07.898 --> 00:11:13.258 So, just want to kind of explore the data and get familiar with it. s141 00:11:15.698 --> 00:11:23.778 As yeah, open one or two traces and maybe start thinking a little bit about what you'd want to measure as you're you're looking through the data here. s142 00:11:23.778 --> 00:11:27.258 So, as an example, let me hop on over to the product. s143 00:11:27.258 --> 00:11:29.618 I explained mine to last day. s144 00:11:29.618 --> 00:11:31.738 Let's go ahead and take a look at one of these traces. s145 00:11:31.738 --> 00:11:40.698 So, you can see here, this destination is for Prague, Czech Republic, for 3 days, $800 budget. s146 00:11:40.698 --> 00:11:45.658 The user asked for an option that feels more local, less touristy. s147 00:11:45.658 --> 00:11:52.938 We have some kind of our ideas here and then we can start to view the plan that the the trip planning agent created. s148 00:11:52.938 --> 00:11:57.858 So, you have a detailed day one, day two, day three. s149 00:11:57.858 --> 00:12:03.978 And if you want to see any additional, you know, attributes, any additional data that we have, you can see that in attributes. s150 00:12:03.978 --> 00:12:08.218 And so, a little bit of how to navigate our product here. s151 00:12:08.218 --> 00:12:10.258 So, again, this is the trace tree. s152 00:12:10.258 --> 00:12:12.018 This is every individual step. s153 00:12:12.018 --> 00:12:20.059 If you want to see the data that was passed in for a step, like for example, we want to see the data that went into OpenAI to process our request, s154 00:12:20.059 --> 00:12:24.619 we can switch to that span and then look at the input and output. s155 00:12:24.619 --> 00:12:33.259 So, we're going to get to this in a little while, but we want to pay attention to where the data lives and what data we think might be needed to do any of our evals. s156 00:12:33.259 --> 00:12:40.819 So, we'll spend a few minutes here just exploring and seeing what kind of insights we find. s157 00:12:40.899 --> 00:12:47.459 Getting some questions about where to find the actual data here, so let me just show that one more time. s158 00:12:47.459 --> 00:12:54.339 A lot of times in the new spaces it can be a little hidden, but didn't want folks to spend too much time tracing today when we're here to talk about evals. s159 00:12:54.339 --> 00:12:59.219 So, you'll just want to click the traces option on the nav here. s160 00:12:59.219 --> 00:13:08.859 This will kind of prompt you to get started with the rise, but we're just going to go to see demo project and then you can see it in recently viewed from there. s161 00:13:08.859 --> 00:13:13.379 And you can always expand the time to see a little bit more data there. s162 00:13:13.379 --> 00:13:20.259 Has everyone generally had time to like find the project, take a a quick look? s163 00:13:20.659 --> 00:13:25.379 See if you had nods, we'll give I know. s164 00:13:25.859 --> 00:13:28.579 Like fix that. s165 00:13:29.499 --> 00:13:29.979 All right. s166 00:13:29.979 --> 00:13:41.139 So, basically what I want folks to do now before we get into the the next part of of this, is just like look at the agent and think about how you would measure this agent s167 00:13:41.139 --> 00:13:42.979 or success for this agent. s168 00:13:42.979 --> 00:13:52.019 Think of yourself as the product person, engineer, any real stakeholder of this project and look at the traces, look at how users are interacting with it, how it's responding, s169 00:13:52.019 --> 00:13:57.339 and start making kind of like mental note of what the things are that you would measure. s170 00:13:57.339 --> 00:14:06.459 In the next part of the workshop, we're actually going to start defining what that criteria would be, but high-level as you're going through, think about some of those things that you would measure. s171 00:14:06.779 --> 00:14:09.699 Yes, so the question was like how we should think about reading a a trace here. s172 00:14:09.699 --> 00:14:19.741 So, again for for anyone who may be just not familiar, a trace is essentially a collection of all the steps an agent took to answer a user's question. s173 00:14:19.741 --> 00:14:21.821 So, let's go ahead and open this one here. s174 00:14:21.821 --> 00:14:25.261 So, at the top we call that our root span. s175 00:14:25.261 --> 00:14:28.741 Like the technical definition of that is parent ID equals null. s176 00:14:28.741 --> 00:14:31.981 This means that this span here doesn't have anything ahead of it. s177 00:14:31.981 --> 00:14:33.461 It just sits at the top. s178 00:14:33.461 --> 00:14:37.341 And this is usually where the input and final output live. s179 00:14:37.341 --> 00:14:39.661 And then from there we have our individual spans. s180 00:14:39.661 --> 00:14:42.341 So, each of these is what we call a span. s181 00:14:42.341 --> 00:14:44.221 I like to think of it as a unit of work. s182 00:14:44.221 --> 00:14:48.221 So, let's take a look at this first collection of spans here. s183 00:14:48.221 --> 00:14:56.301 So, from the root span the first thing that this agent did along with calling some other agents it looks like called the research agent. s184 00:14:56.301 --> 00:15:04.301 So, this is a what we call a multi-agent system where there's multiple agents doing different tasks on behalf of the bigger agent. s185 00:15:04.301 --> 00:15:09.421 So, in this first one we have the research agent and that is going to go ahead and call a tool. s186 00:15:09.421 --> 00:15:13.301 So, each of these has like a type you can see it different icons. s187 00:15:13.301 --> 00:15:14.741 This is an agent span. s188 00:15:14.741 --> 00:15:16.261 This is a tool span. s189 00:15:16.261 --> 00:15:19.501 Tool span that's going to be for getting external data. s190 00:15:19.501 --> 00:15:29.861 So, you can imagine like in this one research there's some essential info that we have to grab like things like weather or sites or adequate things like that. s191 00:15:29.861 --> 00:15:33.581 And then from there we're going to use an LLM. s192 00:15:33.581 --> 00:15:37.381 You can see here all of the inputs and outputs that actually went into the LLM. s193 00:15:37.381 --> 00:15:41.221 So, this is your concise travel agent. s194 00:15:41.221 --> 00:15:48.061 And this is using that information from that tool to actually summarize that we can use it in a later step here. s195 00:15:48.061 --> 00:15:52.861 And so you can just go down the trace tree just like that like one step at a time. s196 00:15:52.861 --> 00:16:01.181 You can see how they all connect but each of these is going to be doing a step or unit of work so that it can go ahead and answer. s197 00:16:01.181 --> 00:16:08.221 So, again the goal of this application is to look at a user's request for a trip and plan it for them. s198 00:16:08.221 --> 00:16:15.701 Give them an itinerary of things that they can do, places they can stay all in accordance with their kind of preferences. s199 00:16:15.701 --> 00:16:23.662 And so each of these spans is the units that were done to ultimately get this final kind of itinerary. s200 00:16:23.662 --> 00:16:24.582 Any questions on that? s201 00:16:24.582 --> 00:16:28.982 Happy to answer some questions about tracing while we're we're going through this. s202 00:16:28.982 --> 00:16:32.182 Yeah, so the question is is like there's two LLM spans. s203 00:16:32.182 --> 00:16:36.662 Yeah, so sometimes it's I'll go back and look at the exact trace of what happened there. s204 00:16:36.662 --> 00:16:46.742 Sometimes it's instrumentation of showing the the different like the LLM like choosing the tool to use and then actually like passing that information to the tool. s205 00:16:46.742 --> 00:16:51.262 So let's go ahead and take a look at what happened here. s206 00:16:54.262 --> 00:16:58.222 And some of this is just kind of part of instrumentation here. s207 00:16:58.222 --> 00:17:12.342 So this looks like here this way and I think the first LLM call chose the tool and then I think in this second LLM call that's where the actual data of the tool was passed to the LLM to summarize. s208 00:17:12.342 --> 00:17:17.782 So here it's kind of hard sometimes you have to go down the trace to come back up. s209 00:17:17.782 --> 00:17:23.902 So this LLM span chose this tool and then this LLM span summarized the output of the tool. s210 00:17:23.902 --> 00:17:34.382 So that tool went and grabbed that like temperature data, destination data and then the second LLM basically summarized it so that we can do kind of further work with it. s211 00:17:34.382 --> 00:17:34.942 Yeah, great question. s212 00:17:34.942 --> 00:17:42.382 So the question for for everyone in case they couldn't hear it is is that kind of root node making all of the decisions or is there some flexibility with how that works? s213 00:17:42.382 --> 00:17:43.622 And there's a lot of flexibility. s214 00:17:43.622 --> 00:17:46.422 It comes down to what your agent kind of infrastructure is. s215 00:17:46.422 --> 00:17:56.262 So you can have what's known as kind of like a supervisor or an orchestrating agent and a lot of times that will make a plan for the sub agents to follow s216 00:17:56.262 --> 00:18:02.382 and then sometimes it's just passed kind of to an LLM with a set of tools and each step is goes like one by one. s217 00:18:02.382 --> 00:18:09.302 So depends a little bit about like what framework you're using, what your infra looks like and the general architecture of your agent. s218 00:18:09.302 --> 00:18:12.022 In this case here this is a link graph agent I believe. s219 00:18:12.022 --> 00:18:14.782 So we're not doing any like supervisor architecture really. s220 00:18:14.782 --> 00:18:20.102 Just we have a a few like modular agents that are responsible for different tasks. s221 00:18:20.102 --> 00:18:20.582 Great question. s222 00:18:20.582 --> 00:18:24.023 The question was like what strategic frameworks we support with arise. s223 00:18:24.023 --> 00:18:25.903 We're agnostic to frameworks. s224 00:18:25.903 --> 00:18:33.103 We have a bunch of what we call auto instrumenters available through our tracing package open inference. s225 00:18:33.103 --> 00:18:35.543 We don't have a preference of which framework we want to use. s226 00:18:35.543 --> 00:18:40.463 Our goal is to be really flexible and provide you with really good observability and evals on top of it. s227 00:18:40.463 --> 00:18:47.503 So I can link our docs where you can see all the different frameworks that are supported. s228 00:18:49.423 --> 00:18:50.743 Okay. s229 00:18:50.743 --> 00:18:53.223 Let's keep going a little bit here. s230 00:18:53.223 --> 00:19:02.623 Sounds like everyone had some time to take a look at the traces, get themselves familiar with the data here and hopefully was able to spot some things that they would measure s231 00:19:02.623 --> 00:19:03.783 for the success. s232 00:19:03.783 --> 00:19:10.303 So the first step of defining what your eval is is defining what good means. s233 00:19:10.303 --> 00:19:12.503 This is sometimes tricky for folks to do. s234 00:19:12.503 --> 00:19:19.383 So I've kind of have this little trick here for vague versus concrete criteria. s235 00:19:19.383 --> 00:19:26.183 I generally like to just write down what good means but with really specific criteria. s236 00:19:26.183 --> 00:19:32.183 So instead of just being like helpful, accurate, good response, you want to get a little bit more specific. s237 00:19:32.183 --> 00:19:39.903 So answer that follows the user's follow-ups, matches the requested number of days, keeps totals within the stated budget. s238 00:19:39.903 --> 00:19:45.983 For this application those are all really concrete criteria for what success looks like. s239 00:19:45.983 --> 00:19:53.823 And I encourage you you know if you're ever building this for your application, you know, sit down and and kind of think about what that criteria is. s240 00:19:53.823 --> 00:19:58.983 Another helpful hint that I have is judge one quality per eval. s241 00:19:58.983 --> 00:20:00.143 There's some exceptions to this. s242 00:20:00.143 --> 00:20:08.543 I like to point out that for especially the decision models and agent as a judge, you can really get away with you know doing multiple criteria in one. s243 00:20:08.543 --> 00:20:15.503 But for LM as a judge, there's lots of research that shows that it really only does well with handling one criteria at a time. s244 00:20:15.503 --> 00:20:21.103 So I think it's good practice to just think of one of them and then add more evals over time. s245 00:20:21.103 --> 00:20:23.023 Building evals sets iterative thing. s246 00:20:23.023 --> 00:20:27.945 You don't just build your evals and then like set it and forget and you're set for the life of the application. s247 00:20:27.945 --> 00:20:37.745 You kind of have to iterate and so um you know, when we're defining like what success looks like, I usually write down a handful um and then kind of start to build those and then over time s248 00:20:37.745 --> 00:20:39.465 we can add more and more. s249 00:20:39.465 --> 00:20:46.745 Um I mentioned I worked on Alex, our AI agent um that lives in Arize and when I first built evals, I had a very simple set. s250 00:20:46.745 --> 00:20:49.065 It was like QA correctness. s251 00:20:49.065 --> 00:20:56.065 I think I had some like user frustration, and of course like jailbreak toxicity, those types of safety evals. s252 00:20:56.065 --> 00:20:59.905 Um and now we have 32 evals that run um on Alex. s253 00:20:59.905 --> 00:21:11.465 So you can see how you want to start with something small um and then as your application gets usage, you can ground that in real production data and then evolve your evals to see what else you might need to define success. s254 00:21:11.465 --> 00:21:13.745 I see some questions over there. s255 00:21:13.745 --> 00:21:19.385 Yeah, so the question for the the group is is a generic eval better than a more specific one? s256 00:21:19.385 --> 00:21:20.545 I think it really depends. s257 00:21:20.545 --> 00:21:24.105 I think generally speaking, I say building a specific eval for your use case is better. s258 00:21:24.105 --> 00:21:27.505 But what I would say is an eval is better than no eval at all. s259 00:21:27.505 --> 00:21:35.545 So a lot of folks like at Arize, for example, when they're getting started, we have an eval hub with a bunch of out-of-the-box templates in it and they're really great for starting. s260 00:21:35.545 --> 00:21:45.705 We have things like QA correctness, hallucination, uh session coherence, all sorts of different evals and I think they're great for getting started, but the the key is and we'll go through this a little bit today, s261 00:21:45.705 --> 00:21:50.865 is to kind of align that to your use case um and then create more specific ones over time. s262 00:21:50.865 --> 00:21:59.025 So I think basic ones and reusing them, especially across an org, are great because it's better to have some evals, but then just make them more specific. s263 00:22:02.305 --> 00:22:05.665 All right, so start with one, add more evals later. s264 00:22:05.665 --> 00:22:12.465 Um I think in pairs or whoever the group of around you, um I would like to kind of have y'all jot down some ideas. s265 00:22:12.465 --> 00:22:15.585 Um basically, there is six things we want to write down. s266 00:22:15.585 --> 00:22:19.465 This is going to be really helpful for us actually building our eval. s267 00:22:19.465 --> 00:22:21.385 Um so first, it's going to be the task. s268 00:22:21.385 --> 00:22:35.346 Like what do you want this eval uh to be going off of, uh what success looks like, what failure looks like, um, what they're judging, like what the evals are actually supposed to judge here in this evaluation, s269 00:22:35.346 --> 00:22:41.426 and then, you know, basically three bullets on what pass would look like, um, and what fail looks like. s270 00:22:41.426 --> 00:22:46.146 And remember from this slide over here, the more specific and concrete you can be, the better your eval will be. s271 00:22:46.146 --> 00:22:51.626 So, we're going to use all of this, uh, all these notes that you're writing to actually build our eval in the next step. s272 00:22:51.626 --> 00:22:55.506 So, the more specific we can be, um, the better. s273 00:22:55.506 --> 00:23:04.786 And so, yeah, I think it's a great opportunity to kind of chat with folks around you, see what you're thinking, what they noticed in the traces, and I'll walk around and let me know if you have any questions. s274 00:23:04.786 --> 00:23:11.266 I got a question over here that was like, um, the the way that the data looks on the trace for this application is pretty specific. s275 00:23:11.266 --> 00:23:13.706 It has all like the request info. s276 00:23:13.706 --> 00:23:18.506 Um, and the question was like, does that come out of the box, or do I have to like build that specific to my use case, right? s277 00:23:18.506 --> 00:23:20.826 That was the just guy, make sure. s278 00:23:20.826 --> 00:23:27.066 Um, and the question is is with our out-of-the-box instrumentation, we kind of abstract kind of all of the tracing best practices for you. s279 00:23:27.066 --> 00:23:29.786 So, there's a lot of common attributes that we send in. s280 00:23:29.786 --> 00:23:39.786 So, things like input down value, or the input messages that we send to the LLM, that's pretty standard, but what you can do is called, um, you know, manual instrumentation or hybrid, which allows you to customize. s281 00:23:39.786 --> 00:23:41.986 So, uh, tracing is really flexible. s282 00:23:41.986 --> 00:23:47.226 We want to make sure it's tailored and you get the information you need for your use case, but we try to make it easy at the same time. s283 00:23:47.226 --> 00:23:56.586 Um, and if you have, um, you know, your own applications that you want to trace, on that kind of starter page was our like MPX evals, that call brings in all of our skills, s284 00:23:56.586 --> 00:23:58.466 uh, which is the best way to do tracing these days. s285 00:23:58.466 --> 00:24:01.666 Let your agent figure out how to instrument it, and you can kind of nudge it along. s286 00:24:01.666 --> 00:24:07.146 So, highly recommend that if you're going to send in your own traces, uh, to evaluate later. s287 00:24:07.146 --> 00:24:07.706 All right. s288 00:24:07.706 --> 00:24:12.906 I'm just going to click to the next slide to show you an example of what this could look like, and then I know there were some questions over here s289 00:24:12.906 --> 00:24:15.866 that I'm going to come to. s290 00:24:15.866 --> 00:24:22.626 So, just going to show an example, um, here of like what your criteria you generally want it to look like. s291 00:24:22.626 --> 00:24:24.906 Um, so, this is like the task that we want to judge. s292 00:24:24.906 --> 00:24:27.306 User sends a follow-up about their trip. s293 00:24:27.306 --> 00:24:31.467 Uh, the success is answers the follow-up directly using the trip's details. s294 00:24:31.467 --> 00:24:33.947 Failures is falls back to a generic budget. s295 00:24:33.947 --> 00:24:36.827 And you can see here, this is the pass and fail. s296 00:24:36.827 --> 00:24:47.187 Some folks asked me like why we want to have the bullets and why the one-liner's not enough and uh, the reason for that is basically we're going to use this to write um, our LLM prompt to do our LLM as a judge. s297 00:24:47.187 --> 00:24:51.587 So we want to be really specific so the LLM um, can understand. s298 00:24:51.587 --> 00:24:58.107 The way that I like to think about this is describing to like a coworker or, you know, a stakeholder like what success looks like. s299 00:24:58.107 --> 00:25:01.627 You almost want to treat this criteria uh, creation the same. s300 00:25:01.627 --> 00:25:05.668 So I'm going to answer a few more questions and then we're going to move on to the next step. s301 00:25:05.668 --> 00:25:16.347 Uh, for folks who want to get a little bit ahead of that, um, you can this is just describing make sure that um, if you have too much in your criteria, you can cut it back, but s302 00:25:16.347 --> 00:25:18.427 I think for the sake of time I'm going to move here. s303 00:25:18.427 --> 00:25:22.307 Um, we want to go ahead and start building our eval. s304 00:25:22.307 --> 00:25:24.547 The first step is you're going to want to go to those traces. s305 00:25:24.547 --> 00:25:34.427 Let me go there now and you're going to want to select a few um, traces to actually add to your data set. s306 00:25:34.427 --> 00:25:40.707 Um, so for example, say you wanted to just go ahead and evaluate the input versus the itinerary. s307 00:25:40.707 --> 00:25:43.427 Like does the itinerary have the general input? s308 00:25:43.427 --> 00:25:47.388 You would want to save all of these root spans to a data set. s309 00:25:47.388 --> 00:25:57.627 Um, if you've decided you wanted to maybe test the tool call and see if the tool call returns um, the right information that it was supposed to, you would want to add in all of these s310 00:25:57.627 --> 00:25:59.947 tools uh, calls to your data set. s311 00:25:59.947 --> 00:26:00.947 Um, we have filters here. s312 00:26:00.947 --> 00:26:03.227 It's the best way to identify data. s313 00:26:03.227 --> 00:26:10.427 Um, so one really great example, if it's like root spans, you could do just parent ID equals null or go off of the default traces. s314 00:26:10.427 --> 00:26:18.627 Um, and if you wanted to do like a tool call for example that um, I think it was the essential tool, you could also filter by name that way. s315 00:26:18.627 --> 00:26:27.107 Um, and then you would just want to grab these or go from the span view to, you know, select those and add to a new data set. s316 00:26:27.107 --> 00:26:34.509 So um, that's kind of the the next step and then we're going to head on over um, to our playground to actually build and test our eval. s317 00:26:34.509 --> 00:26:39.309 So, folks can head over there while I answer some questions. s318 00:26:39.309 --> 00:26:44.509 And let me know if there any that come up for adding to the data set and how you should think about filtering. s319 00:26:44.509 --> 00:26:52.349 I want to show some folks kind of the next steps here for time cuz I know we don't have that much left here. s320 00:26:52.989 --> 00:27:02.909 So, everybody hopefully has been able to go through, write their criteria, identify some spans for what they're building. s321 00:27:02.909 --> 00:27:11.709 And then this is essentially kind of the first pass is what you probably saw, which I'm going to do now live for everyone to kind of follow along with. s322 00:27:11.709 --> 00:27:13.789 But we wanted to filter the spans you're judging. s323 00:27:13.789 --> 00:27:14.989 That's what we just walked through. s324 00:27:14.989 --> 00:27:22.549 So, look through the traces, identify what span you need to evaluate, build your filter, add those to a data set. s325 00:27:22.549 --> 00:27:25.309 And now what we're going to do is go into our prompt playground. s326 00:27:25.309 --> 00:27:29.469 That's my favorite place to go to actually test out our prompts here. s327 00:27:29.469 --> 00:27:31.589 So, you can go to the playground. s328 00:27:31.589 --> 00:27:34.949 I already have one set up, but I'm going to go ahead and create a new one. s329 00:27:34.949 --> 00:27:38.829 We have two types of playgrounds in our product. s330 00:27:38.829 --> 00:27:40.429 We're going to use the prompt playground. s331 00:27:40.429 --> 00:27:45.509 That's great for testing eval prompts or you know any LLM prompts for your agent. s332 00:27:45.509 --> 00:27:47.029 We do also have an agent playground. s333 00:27:47.029 --> 00:27:49.589 That's for testing kind of your agent end to end here. s334 00:27:49.589 --> 00:27:52.589 So, not so relevant for today. s335 00:27:52.589 --> 00:27:54.069 So, just give it a name. s336 00:27:54.069 --> 00:27:58.829 I'm going to do AI engine evals. s337 00:27:59.029 --> 00:28:00.349 And then we can go ahead and create it. s338 00:28:00.349 --> 00:28:10.909 So, if you were in our eval hub, you were over here and you created an eval, you kind of have a slide over that looks like this. s339 00:28:10.909 --> 00:28:15.709 You can go ahead in the playground, just add that from the drop down here. s340 00:28:15.709 --> 00:28:18.589 So, you'll see your eval pop up here. s341 00:28:18.589 --> 00:28:22.269 For anybody who's just starting the eval process, we're just going to do blank eval. s342 00:28:22.269 --> 00:28:25.549 And you can actually just start writing your template here. s343 00:28:25.549 --> 00:28:30.029 So, what you're going to basically do is take all that criteria and turn that into a prompt. s344 00:28:30.029 --> 00:28:31.669 A lot of it should be like paste. s345 00:28:31.669 --> 00:28:42.750 You can paste in those like bullets that you had as criteria, but um um a lot of times you want to first give it kind of an intro, like you aren't eval agent s346 00:28:42.750 --> 00:28:48.750 or an eval LLM uh looking at the output accuracy for your application. s347 00:28:48.750 --> 00:28:52.590 Let me actually just load this one in. s348 00:28:52.590 --> 00:28:53.670 Okay, perfect. s349 00:28:53.670 --> 00:28:59.510 Um so, I like to start my evals with an example of what the LLM is supposed to do. s350 00:28:59.510 --> 00:29:01.950 I I always kind of center it in its job. s351 00:29:01.950 --> 00:29:10.510 You're evaluating whether an assistant's output actually accurately captures the details requested um or input in the or provided in the input. s352 00:29:10.510 --> 00:29:12.590 Um you then want to provide the data. s353 00:29:12.590 --> 00:29:14.190 So, this is really important. s354 00:29:14.190 --> 00:29:21.230 When we're looking at those traces, there's probably uh attributes that you want to identify. s355 00:29:21.230 --> 00:29:24.710 You don't really need to worry about writing the specific names in your eval prompt. s356 00:29:24.710 --> 00:29:29.510 We do something called variable mapping um that allows you to connect it to your exact data. s357 00:29:29.510 --> 00:29:31.710 Uh the benefit of that is that your evals can be flexible. s358 00:29:31.710 --> 00:29:40.830 Like, if you've running on a data set calls it one thing, and then your traces call it something different, rather than you need to update every eval, you can just update the task where it's being used. s359 00:29:40.830 --> 00:29:44.110 So, don't worry too much about the name of the variable. s360 00:29:44.110 --> 00:29:48.430 Um and then this is an example of what I have written for mine. s361 00:29:48.430 --> 00:29:50.590 Um these are the check-in particulars. s362 00:29:50.590 --> 00:29:55.470 Those are things like those three bullet points for pass and fail. s363 00:29:55.470 --> 00:30:06.830 Um and then here I have mark only correct if, you know, the output is consistent with the input, incorrect if there's a wrong destination, property type, whatever it is, that's going to be incorrect. s364 00:30:06.830 --> 00:30:11.790 Um and under params here, you'll see if you set it up in the task, that's where we have the labels. s365 00:30:11.790 --> 00:30:16.310 This one here is just correct or incorrect, but you can have, you know, a multi-label problem. s366 00:30:16.310 --> 00:30:19.470 Maybe you do correct, incorrect, partially correct. s367 00:30:19.470 --> 00:30:25.790 I have use cases in Alex where it's actually relevant to me to know when something is somewhat correct, but not fully. s368 00:30:25.790 --> 00:30:28.390 Um you can also do things like not applicable. s369 00:30:28.390 --> 00:30:29.910 Um so, lots of flexibility. s370 00:30:29.910 --> 00:30:42.552 The eval will return only one label, um and then you have some things here around the optimization direction, which says like is good or is correct the good thing or s371 00:30:42.552 --> 00:30:45.512 is a lower score a good thing and then explanations. s372 00:30:45.512 --> 00:30:53.312 Again, those are kind of the reasoning for why the template or by the eval gave a certain response. s373 00:30:53.312 --> 00:30:57.272 And then what you're going to want to do is load in your data set. s374 00:30:57.272 --> 00:31:01.272 So those spans that you created and saved your data set, you can load them in. s375 00:31:01.272 --> 00:31:04.912 And now what we're in the spot to do is actually start testing our eval. s376 00:31:04.912 --> 00:31:08.672 We have all those inputs and outputs that are here. s377 00:31:08.672 --> 00:31:14.312 If you need to map them to different variables, mine happen to be the same, but you can do that here. s378 00:31:14.312 --> 00:31:17.872 And so now what you can do is actually start to run. s379 00:31:17.872 --> 00:31:21.192 I'm going to run with 41 mini, that seems good to me. s380 00:31:21.192 --> 00:31:25.632 And we'll actually start to see the output of our eval right here in the playground. s381 00:31:25.632 --> 00:31:30.632 And this is a great spot to just start iterating on your eval template. s382 00:31:30.632 --> 00:31:39.632 Look at the response, you know, kind of see if there's something that you agree with or disagree and then you can adjust the prompt and then continue to run it over time. s383 00:31:39.632 --> 00:31:45.112 So that's going to be what we do for the next little bit here is try to get to a template in a good spot. s384 00:31:45.112 --> 00:31:56.032 And then what you can do is save that off here in this little button here, save evaluator, and then that will be available to use across the platform for either our demo project s385 00:31:56.032 --> 00:31:59.392 or for any application that you build and bring into our eyes after. s386 00:31:59.392 --> 00:32:08.392 So I'm going to walk around, help some folks write their templates and then I have just, you know, one or two slides here to to close us out. s387 00:32:08.392 --> 00:32:12.032 I did want to kind of point out that we do have some tools to help you. s388 00:32:12.032 --> 00:32:18.912 I had everybody write out their criteria and write their eval and test it, but we do have Alex in the platform that can help you with that. s389 00:32:18.912 --> 00:32:25.392 So I definitely recommend helping or having Alex help you, especially after you've done kind of your initial eval pass. s390 00:32:25.392 --> 00:32:30.231 It can actually run the playground for you, look at the results, update the prompt and repeat. s391 00:32:30.231 --> 00:32:33.912 So that's really helpful for when you're writing evals. s392 00:32:33.912 --> 00:32:38.471 Um pulling out a little bit of the the remainder of the loop here. s393 00:32:38.471 --> 00:32:48.753 So we spent a lot of time on the second loop our second step today, um where you instrument, you write your eval, um but then what the rest of the loop looks like is we have things like signal. s394 00:32:48.753 --> 00:32:55.073 That's our managed agent in the platform that is able to look at your traces, find issues for you, even make fixes. s395 00:32:55.073 --> 00:33:02.513 Um so this becomes signal that our managed agents can then consume and ultimately help improve your application. s396 00:33:02.553 --> 00:33:06.313 Um these are some things I think good teams do next, alignment. s397 00:33:06.313 --> 00:33:16.473 Um alignment is going to be the process of like annotating all of those examples that we added to our data set, and then going to run that eval and seeing if s398 00:33:16.473 --> 00:33:18.993 um it matches the human feedback. s399 00:33:18.993 --> 00:33:24.793 Um online evals will be essentially us putting that into a task and have it running continuously over our traces. s400 00:33:24.793 --> 00:33:34.233 So what that means in our eyes, every time you send in a trace, we'll evaluate it according to all your evals, and then that automatically gets posted, and then you can use that to actually monitor s401 00:33:34.233 --> 00:33:36.033 your application. s402 00:33:36.033 --> 00:33:41.673 Um a key everybody to walk away here with is that every stakeholder evaluates the agent. s403 00:33:41.673 --> 00:33:44.273 This is not just one kind of role's job. s404 00:33:44.273 --> 00:33:52.753 You really want to get everybody in the room who has some kind of stake in the agent and have them basically define what success looks like according to their outcomes. s405 00:33:52.753 --> 00:33:58.793 So uh product, eng, support, um experts, um security, those are all folks. s406 00:33:58.793 --> 00:34:08.113 And then um the key differences between like an metric and an eval is like a metric tells you that something might be broken, but evals tell you what the actual failure mode is. s407 00:34:08.113 --> 00:34:11.673 So you really want to use both there. s408 00:34:11.673 --> 00:34:16.193 Um and then I think what might be fun to try next is we do have all these new decision judges. s409 00:34:16.193 --> 00:34:25.953 Um and so I think it's always interesting to take your elements as a judge, you know, convert it to a decision judge question, um and then maybe compare the performance there, uh see which one might be better. s410 00:34:25.953 --> 00:34:28.393 Um and yeah. s411 00:34:28.393 --> 00:34:38.793 And then again, start small, 20 traces maybe, one dimension, one judge, you know, read the explanations, iterate over it, and then you can expand to more and more evals. s412 00:34:39.433 --> 00:34:44.914 Um and just some resources, we have some docs, um, evaluate your LLM as a judge, your judge docs. s413 00:34:44.914 --> 00:34:48.434 Uh, we do have some deep dives on like the decision model benchmarking. s414 00:34:48.434 --> 00:34:53.154 So, I think somebody had a question of like, when should I use LLM as a judge, when should I use judge? s415 00:34:53.154 --> 00:34:58.634 I think that's a really great use uh, case and uh, yeah, I'll be hanging out for the rest of the day. s416 00:34:58.634 --> 00:35:01.314 Arize has a booth in the expo starting at I believe 4:00 today. s417 00:35:01.314 --> 00:35:06.194 So, if you want to learn more about evals, see a little bit more of our product, definitely come find us. s418 00:35:06.194 --> 00:35:12.714 Um, and if there are any questions, um, happy to answer them and thanks so much for joining me today.