WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=UGuNt2_Oz8s

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/UGuNt2_Oz8s.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.560 --> 00:00:14.360
Well, thank you guys for coming to the workshop.

s3
00:00:14.360 --> 00:00:15.120
Really appreciate you.

s4
00:00:15.120 --> 00:00:17.320
My name is Fouad Ali.

s5
00:00:17.320 --> 00:00:20.080
I am a product manager at Arise.

s6
00:00:20.080 --> 00:00:24.320
Arise is helps you guys build the self-improvement loop for your agents.

s7
00:00:24.320 --> 00:00:34.920
Today today I'm going to be talking through what that loop is, how it gets built, and I'm going to be walking through a little bit of an example agent as well to kind of show

s8
00:00:34.920 --> 00:00:36.920
how that agent works.

s9
00:00:36.920 --> 00:00:38.800
Now we have an hour for this workshop.

s10
00:00:38.800 --> 00:00:42.760
With that in mind, I know that workshops can be hectic, Wi-Fi can go out.

s11
00:00:42.760 --> 00:00:45.120
So I have everything ready on my screen.

s12
00:00:45.120 --> 00:00:46.400
I'll be walking through it.

s13
00:00:46.400 --> 00:00:47.760
It's optional for you guys to follow along.

s14
00:00:47.760 --> 00:00:48.680
Highly recommend you do.

s15
00:00:48.680 --> 00:00:51.960
I personally learn a lot better when I have hands to the keyboard.

s16
00:00:51.960 --> 00:00:54.000
But if you are having issues, don't worry.

s17
00:00:54.000 --> 00:00:58.440
And we also have Lori in the house who is more than happy to kind of help troubleshoot along as well.

s18
00:00:58.440 --> 00:01:00.360
Sound good?

s19
00:01:00.360 --> 00:01:01.360
Cool.

s20
00:01:01.360 --> 00:01:04.680
So I'm Fouad, I work in product at Arise, and we work with a lot of agents.

s21
00:01:04.680 --> 00:01:12.520
And I'm assuming you guys are all very familiar with what agents are and how they fail and how they are non-deterministic.

s22
00:01:12.560 --> 00:01:21.640
And you know, this is sort of like a classic engineering problem that has reared its ugly head, you know, 10 times worse with agents because of the indeterminism, right?

s23
00:01:21.640 --> 00:01:31.960
So when you have agents that can do a lot of things and have a lot of different ways of doing a lot of things, understanding their failure modes becomes trickier.

s24
00:01:31.960 --> 00:01:36.240
And then add on to that the fact that there are thousands of these agents.

s25
00:01:36.240 --> 00:01:37.480
Maybe I have internal agents.

s26
00:01:37.480 --> 00:01:39.440
Maybe I have coding agents.

s27
00:01:39.440 --> 00:01:40.560
You have marketing agents.

s28
00:01:40.560 --> 00:01:44.200
And you have obviously customer-facing agents as well.

s29
00:01:44.200 --> 00:01:47.640
And that all adds a ton of noise to the problem.

s30
00:01:47.640 --> 00:02:02.200
And so how do you actually start to build to a loop where these agents use the feedback that you provide as a human being, customer feedback, and obviously actual evidence, so things like traces,

s31
00:02:02.200 --> 00:02:05.720
um to inform how they can improve.

s32
00:02:05.720 --> 00:02:07.600
Uh and that's what I'll be talking about today.

s33
00:02:07.600 --> 00:02:11.120
Uh Signals are out-of-the-box version of this, which I'll be showing heavily today.

s34
00:02:11.120 --> 00:02:16.280
We have a managed agents platform that lets you kind of define on your own terms what these do.

s35
00:02:16.280 --> 00:02:21.680
Um obviously data sets, evals, experiments, uh all that will be part of this as well.

s36
00:02:21.680 --> 00:02:25.240
Uh but this is what we're trying to solve, and uh apologies for the meme.

s37
00:02:25.240 --> 00:02:31.440
Claude's uh sense of humor is not that good, but um I think it illustrates the use case pretty well.

s38
00:02:31.440 --> 00:02:39.959
Um in the previous world of software engineering, um your SRE would probably be in the first picture, right?

s39
00:02:39.959 --> 00:02:47.920
So, classic situation, PagerDuty uh notification goes out 4:00 a.m. Your agent broke something in production.

s40
00:02:47.920 --> 00:02:58.120
Um in this case, we're we're doing like a little, you know, uh shopping assistant, and shopping assistant is saying, "We sell zero board games to all your customers on production."

s41
00:02:58.120 --> 00:02:59.840
Not a situation you want to be in.

s42
00:02:59.840 --> 00:03:07.400
Uh there have been worse use cases of agents as well, where uh I don't know if you guys are familiar with what happened in Canada with a car dealership, where

s43
00:03:07.400 --> 00:03:14.640
someone was able to get um a uh I think it was like a a Prius or something for like $5 uh through a chatbot.

s44
00:03:14.640 --> 00:03:17.840
They obviously didn't honor the sale, but that's legal's problem now.

s45
00:03:17.840 --> 00:03:28.120
Um and so, these agents can fail in a number of different ways for a variety of reasons, and it becomes really important to try to move to a world where we're in the after stage,

s46
00:03:28.120 --> 00:03:31.800
where the agents can learn how to fix themselves and build that feedback loop into it.

s47
00:03:31.800 --> 00:03:38.000
And I'll talk a little bit about what where humans still add value in in that sort of world as well.

s48
00:03:38.000 --> 00:03:41.360
So, this is what you'll need to follow along.

s49
00:03:41.360 --> 00:03:43.640
Again, like I said, this will be optional.

s50
00:03:43.640 --> 00:03:46.680
I'll be walking through the flow for you guys.

s51
00:03:46.680 --> 00:03:48.320
Um we do have an hour.

s52
00:03:48.320 --> 00:03:50.000
Um we can spend the rest of the time with questions.

s53
00:03:50.000 --> 00:03:52.560
You can spend some of the time trying out the loop.

s54
00:03:52.560 --> 00:03:59.000
Um I would recommend if you have some time I'll I'll put up here for you know 30 seconds ish to a minute.

s55
00:03:59.000 --> 00:04:07.240
Um if you can sign up for a completely free Arise X account uh you'll need a space ID and an API key which I can show you how to get.

s56
00:04:07.240 --> 00:04:16.519
Um you'll need some sort of coding um API or sorry um LLM API provider key OpenAI or Anthropic both work.

s57
00:04:16.519 --> 00:04:20.840
Uh you'll need Node and Python on your machine if you don't have that don't worry we'll we'll still talk you through it.

s58
00:04:20.840 --> 00:04:25.120
Um Cloud Coder Chris will help kind of set this up for you as well.

s59
00:04:25.120 --> 00:04:28.440
Um I assume none of you guys are strangers to those tools either.

s60
00:04:28.440 --> 00:04:29.720
Uh and it's very simple.

s61
00:04:29.720 --> 00:04:33.480
So this is kind of the app we have.

s62
00:04:33.480 --> 00:04:37.200
So it's a chat base toy buying experience.

s63
00:04:37.200 --> 00:04:38.520
It's called Wonder Toys, right?

s64
00:04:38.520 --> 00:04:39.880
Very simple.

s65
00:04:39.880 --> 00:04:43.480
You can ask in the chat about different products that are available.

s66
00:04:43.480 --> 00:04:45.080
You can chat to purchase.

s67
00:04:45.080 --> 00:04:47.080
Um this is all running locally on my machine.

s68
00:04:47.080 --> 00:04:49.600
So this is the agent that we're going to be dealing with today.

s69
00:04:49.600 --> 00:04:56.520
Uh we're going to be walking through how we instrument it, how we set it up so that we can build that feedback loop.

s70
00:04:56.520 --> 00:05:01.040
Um so I'll pause here real quick.

s71
00:05:01.040 --> 00:05:05.520
I'll give you guys 30 seconds to try and do some of this again optional.

s72
00:05:05.520 --> 00:05:10.880
I don't want to you know block the entire workshop on API keys for sure.

s73
00:05:11.120 --> 00:05:11.440
Cool.

s74
00:05:11.440 --> 00:05:20.960
So really three things getting the repo running the actual app making sure you have an API key that actually powers the agent.

s75
00:05:20.960 --> 00:05:24.880
Um so not uh a provider API key like OpenAI or Anthropic.

s76
00:05:24.880 --> 00:05:33.000
Um then this this part here is about installing the skills that helps your coding agent work with our tools uh in a more kind of opinionated manner.

s77
00:05:33.000 --> 00:05:38.360
Um Node and Python just to run the repo and that's about it.

s78
00:05:38.360 --> 00:05:39.560
Cool.

s79
00:05:39.560 --> 00:05:42.120
I'm going to continue in the interest of time.

s80
00:05:42.120 --> 00:05:48.760
So open question for you guys of these you know six Did I count that right?

s81
00:05:48.760 --> 00:05:49.640
Five boxes.

s82
00:05:49.640 --> 00:05:51.800
Five boxes.

s83
00:05:51.800 --> 00:05:55.000
Which of these boxes would you classify your team is in?

s84
00:05:55.000 --> 00:06:01.480
And, you know, the usually it starts from, "Okay, we are tracing what our agents are doing.

s85
00:06:01.480 --> 00:06:06.400
So, we know when they fail, we know what tools they call.

s86
00:06:06.400 --> 00:06:10.840
We can see the entire trace tree for the interaction points of the agent.

s87
00:06:10.840 --> 00:06:12.680
Uh and that's box one.

s88
00:06:12.680 --> 00:06:16.720
Show of hands, does anyone have tracing for their agents right now?

s89
00:06:16.720 --> 00:06:17.480
Cool.

s90
00:06:17.480 --> 00:06:19.280
Okay, we got a bunch, actually.

s91
00:06:19.280 --> 00:06:20.520
It's good.

s92
00:06:20.520 --> 00:06:27.440
Then the next stage of that is, "Okay, well, I know what what the agent is doing, but is it doing the right thing?"

s93
00:06:27.440 --> 00:06:29.240
Um and how do I actually score and evaluate that?

s94
00:06:29.240 --> 00:06:35.160
So, we call those evals in the industry, and so I would say, you know, some people probably are in that box.

s95
00:06:35.160 --> 00:06:37.080
Are Are any of you guys evaluating your agents?

s96
00:06:37.080 --> 00:06:39.480
You have evals running on traces?

s97
00:06:39.480 --> 00:06:40.520
Okay, we got a couple.

s98
00:06:40.520 --> 00:06:41.080
Okay, a lot.

s99
00:06:41.080 --> 00:06:42.720
Yep.

s100
00:06:42.720 --> 00:06:45.480
Then there's someone reading the traces, presumably.

s101
00:06:45.480 --> 00:06:50.200
Uh and that's sort of what we're trying to unlock here, right, with the agents.

s102
00:06:50.200 --> 00:06:56.680
Um but realistically, if you have coding agents internally, there's no way someone's actually reading all these traces, right?

s103
00:06:56.680 --> 00:07:04.280
Or, you know, a chatbot serving millions of customers, no one single human being is going to be able to read those traces, right?

s104
00:07:04.280 --> 00:07:09.880
Then you actually want to be able to save those failures as tests, right?

s105
00:07:09.880 --> 00:07:19.240
So, every time something fails, you capture that, and you're able to use that as a regression test moving forward, so you don't, you know, mess the same thing up twice.

s106
00:07:19.240 --> 00:07:27.600
Um and then finally, you want to prove that you want to make the fix, and you want to prove that you actually fixed the thing, right?

s107
00:07:27.600 --> 00:07:33.360
So, in those last three boxes, have you got anyone kind of in in there, in that stage where they're actually Awesome.

s108
00:07:33.360 --> 00:07:33.680
Cool.

s109
00:07:33.680 --> 00:07:35.440
We got a couple.

s110
00:07:35.440 --> 00:07:35.840
Cool.

s111
00:07:35.840 --> 00:07:46.800
So, how do you actually thread all these things together, and how do you make this workflow as easy as possible, and, you know, have humans in that workflow in their highest value leverage points,

s112
00:07:46.800 --> 00:07:50.880
but not necessarily reading tons of traces, right?

s113
00:07:50.880 --> 00:08:00.960
Um the flow generally goes trace signal, which is our out-of-the-box kind of SRE agent, which I'll show you guys in the platform.

s114
00:08:00.960 --> 00:08:14.000
Signal then runs reads the traces, creates a data set off the failures, proposes an eval for you, automatically fixes what's going wrong, and then runs an experiment that actually proves

s115
00:08:14.000 --> 00:08:17.919
that that fix is um something that's valid.

s116
00:08:17.919 --> 00:08:22.480
And then you review that fix, you choose to merge that fix, and you go back into the loop.

s117
00:08:22.480 --> 00:08:31.280
And every single one of these steps builds additional context as your agent continues to improve, continues to, you know, move into new use cases.

s118
00:08:31.280 --> 00:08:35.960
Let's say I want to add, you know, one-click checkout on this on this agent or something like that.

s119
00:08:35.960 --> 00:08:43.440
Um that then feed not just your product roadmap, but your engineering reliability organization as well.

s120
00:08:43.760 --> 00:08:46.120
So, uh this is kind of what we'll talk through.

s121
00:08:46.120 --> 00:08:47.600
I'll walk through the loop.

s122
00:08:47.600 --> 00:08:58.880
Uh I'll show you a bit of the agent and its traces, signal reviewing the agents, the failure data set, the eval that we build off of that failure data set, fixing it, proving it with an experiment,

s123
00:08:58.880 --> 00:09:07.360
what the agents did themselves, and what still needs a person, um and where you guys can add the most value, and then we'll have some time for questions as well.

s124
00:09:07.360 --> 00:09:11.240
So, Wonder Toys of the Shopping is assistant agent that we're doing today.

s125
00:09:11.240 --> 00:09:15.760
Uh it's got a bug in it um that we planted for the purpose of this demo.

s126
00:09:15.760 --> 00:09:19.320
Although, I hope you guys are not planting bugs in your own agents.

s127
00:09:19.320 --> 00:09:25.480
Um it's built with uh like I said, the repo has a bunch of different agent frameworks out there.

s128
00:09:25.480 --> 00:09:31.120
Um Mostly Fast API, Next.js, Chroma DB, uh and then agent SDKs as well.

s129
00:09:31.120 --> 00:09:34.160
Uh and it sends traces to Arize AX.

s130
00:09:34.160 --> 00:09:36.800
It has a few tools available to it.

s131
00:09:36.800 --> 00:09:43.520
Um and one of those tools, specifically the one with the bug, um, misfires a filter.

s132
00:09:43.520 --> 00:09:49.000
And so, I'll kind of show you at a base level what the tracing experience looks like.

s133
00:09:49.000 --> 00:09:57.800
Um, we made it very, very easy for you to set up traces, so you're not instrumenting things manually to get, you know, the information out of your agents.

s134
00:09:57.800 --> 00:10:04.640
Uh, we have skills set up that wrap our CLI and our MCPs uh, that help you get started uh, with tracing.

s135
00:10:04.640 --> 00:10:12.520
We have provider um, all the major providers uh, instrumented already with auto instrumenters uh, and all the major agent frameworks as well.

s136
00:10:12.520 --> 00:10:15.200
It's all open source, so very easy to get started with tracing.

s137
00:10:15.200 --> 00:10:21.520
Sounds like a lot of you guys are familiar with tracing, and so you'll be very familiar with what the trace tree looks like.

s138
00:10:21.520 --> 00:10:26.520
Um, a user asks, "Show me toys for 9-year-olds."

s139
00:10:26.520 --> 00:10:30.640
Um, and that goes through an entire request path, right?

s140
00:10:30.640 --> 00:10:37.560
So, the agent's called, the agent calls a model, the agent has an end uh, LLM call.

s141
00:10:37.560 --> 00:10:48.080
It has some tools available to it, whether it's searching products, etc. Um, and you can kind of see all the information available um, to ask it what the agent's doing under the hood.

s142
00:10:48.080 --> 00:10:56.560
Uh, so all these attributes kind of show, you know, start time, end time, latency, what the input was, what metadata was passed, what stage in the in the graph it is,

s143
00:10:56.560 --> 00:11:03.920
um, results that it fetches from the DB, uh, and what at every single one of these stages, what information is available to the agent.

s144
00:11:03.920 --> 00:11:14.120
Uh, so this is valuable, but again, you probably don't want to read this at 4:00 a.m. Um, I think it would be kind of tough to read through thousands of these at 4:00 a.m. just to find where an error lies.

s145
00:11:14.120 --> 00:11:24.440
Um, so what we want to do is we want to enable agents to make use of this information, and then see how we can actually close that self-improvement loop.

s146
00:11:28.400 --> 00:11:30.839
So, I kind of walked through the trace.

s147
00:11:30.839 --> 00:11:34.400
If you want to instrument, the AX CLI is what we use.

s148
00:11:34.400 --> 00:11:36.280
Um it has a guided setup.

s149
00:11:36.280 --> 00:11:37.520
It's very easy to get started with it.

s150
00:11:37.520 --> 00:11:46.360
All you need is an API key from Rise and it will take care of translating all those nasty little attributes into the beautiful trace tree that you saw there.

s151
00:11:46.360 --> 00:11:51.760
Uh you'll get a span per tool call with arguments and results uh to the tool call.

s152
00:11:51.760 --> 00:11:57.520
You'll get model calls, prompt token counts, all all the basics that you need to actually monitor what's going on.

s153
00:11:57.520 --> 00:11:58.880
Um as well as grouping.

s154
00:11:58.880 --> 00:12:09.400
So, you can see sessions uh where there's multiple back and forth with a customer instead of just individual traces that are single back and forth with a customer.

s155
00:12:10.040 --> 00:12:15.040
So, enter Signal which is sort of the second step in this platform, right?

s156
00:12:15.040 --> 00:12:25.880
So, once you have these traces, what Signal does is Signal is an always-on agent that is able to read your traces on a schedule.

s157
00:12:25.880 --> 00:12:29.840
Uh and you can configure this in the platform to be kind of like whatever schedule you want.

s158
00:12:29.840 --> 00:12:33.880
Um by default, it's uh set to every 6 hours.

s159
00:12:33.880 --> 00:12:38.520
Uh you point it to a project that's, you know, has live traces coming into it.

s160
00:12:38.520 --> 00:12:46.760
Um you enable it and it reviews those traces and can then do a number of different actions via its skills, right?

s161
00:12:46.760 --> 00:12:54.040
So, [clears throat] the base action that we're going to show today is Signal connected to a repo on GitHub.

s162
00:12:54.040 --> 00:12:58.640
So, this agent Signal is reading another agent's traces.

s163
00:12:58.640 --> 00:13:10.839
It's reading the source code of that agent based on sort of what's actually happening in production traces from that agent and the coding definition, it's able to actually propose

s164
00:13:10.839 --> 00:13:12.600
fixes to what you're seeing.

s165
00:13:12.600 --> 00:13:16.000
Uh so, I'll show you guys what that looks like here real quick.

s166
00:13:16.000 --> 00:13:19.080
So, we have this Wonder Toys agent.

s167
00:13:19.080 --> 00:13:20.320
It's publishing traces in.

s168
00:13:20.320 --> 00:13:21.720
We can see all the user requests.

s169
00:13:21.720 --> 00:13:33.000
You can see I did a lot right before uh this presentation at 1:00 p.m. We can see, you know, kind of base facts like latency, token count, cost uh of running this project.

s170
00:13:33.000 --> 00:13:34.880
And you'll see this signal tab.

s171
00:13:34.880 --> 00:13:45.880
So, what Signal is doing is it's clustering issues it finds with the traces and actually proposing and identifying um a fix for for these issues.

s172
00:13:45.880 --> 00:13:52.360
Um so, for the purpose of this agent, the bug we planted uh is causing redundant search retries.

s173
00:13:52.360 --> 00:14:04.000
And so, we're see what Signal is seeing is that based on the traces that are coming into this project, we are uh causing three sequential LLM calls and high latency

s174
00:14:04.000 --> 00:14:08.520
because of something that's happening with the search products call, right?

s175
00:14:08.520 --> 00:14:17.160
And specifically, what that's happening what's happening is that um the casing on some of these queries for the filtering is not working, right?

s176
00:14:17.160 --> 00:14:27.760
So, what Signal does is instead of you getting paged at 4:00 a.m. uh if you remember the meme in the beginning of this presentation, um Signal's finding this issue,

s177
00:14:27.760 --> 00:14:41.120
analyzing, you know, clusters of these issues, writing an overview and analysis, writing a fix for you, and then what you can do is you can create evals to catch this issue moving forward automatically

s178
00:14:41.120 --> 00:14:42.960
and trigger off those uh evals.

s179
00:14:42.960 --> 00:14:53.200
So, you know, if we run into this issue at any point in the next, you know, 30 days, we automatically fire an alert and that gets triaged by our team immediately.

s180
00:14:53.200 --> 00:14:57.680
You can add failure cases to the data set here as well.

s181
00:14:57.680 --> 00:15:03.640
So, that you have a snapshot uh for future regression tests on exactly this failure mode.

s182
00:15:03.640 --> 00:15:11.200
You'll see a GitHub icon here as well to connect and open a PR or open issues well, so that you can actually get Signal to write a PR.

s183
00:15:11.200 --> 00:15:18.000
I've actually already done that, and so you can see the PR attempts uh that Signal has done to fix this, right?

s184
00:15:18.000 --> 00:15:20.920
So, I'm going to go click into the session.

s185
00:15:21.960 --> 00:15:37.760
And the agent worked for 6 minutes, found the root cause within the coding repo as well, based on the traces, and found that product product categories are stored as lowercase hyphenated slugs,

s186
00:15:37.760 --> 00:15:42.600
but the element is passing human-readable names with title case and spaces.

s187
00:15:42.600 --> 00:15:48.920
Uh so, when someone is searching for car games, for example, without that normalization, we actually never get back any results.

s188
00:15:48.920 --> 00:15:53.280
The agent is then going and searching every single product, right?

s189
00:15:53.280 --> 00:15:59.120
So, you can imagine why that causes the latency spike up to 14 17 seconds.

s190
00:15:59.120 --> 00:16:02.360
So, everyone following along so far?

s191
00:16:02.360 --> 00:16:05.920
We have an agent, we have an issue.

s192
00:16:05.920 --> 00:16:11.560
Signal is reading the traces for you, proposing a fix in connection with GitHub.

s193
00:16:11.560 --> 00:16:20.560
Uh so, actually reading the the definition of this agent, uh and then doing some analysis for you, and kind of making sure that that's available to you.

s194
00:16:20.560 --> 00:16:25.080
So, this is one of many skills you can connect to Signal.

s195
00:16:25.080 --> 00:16:36.400
Um you'll see here on the manage agents platform, you have the ability to use one of our out-of-the-box templates, so Signal is reviewing your project traces on a schedule, clustering failures,

s196
00:16:36.400 --> 00:16:38.000
suggesting fixes.

s197
00:16:38.000 --> 00:16:45.839
We have a cost agent that can flag expensive models doing cheap work, recommend cheaper routing, and actually quantify your savings.

s198
00:16:45.839 --> 00:16:48.960
Uh but, the interesting part is you can actually start from scratch.

s199
00:16:48.960 --> 00:16:58.800
Um And so, what the agent studio allows you to do is you choose an agent config, you point it to a project where traces are coming in, you give it a task,

s200
00:16:58.800 --> 00:17:00.520
and you can connect skills to it as well.

s201
00:17:00.520 --> 00:17:03.920
So, like I mentioned, GitHub is one of those sources.

s202
00:17:03.920 --> 00:17:10.800
Maybe you have um you know, a third-party system that holds your R back information that and you want to check permissions on this agent.

s203
00:17:10.800 --> 00:17:14.880
You want to make sure what the agent is calling is actually within its permission boundaries.

s204
00:17:14.880 --> 00:17:17.000
You can You can a skill to do that.

s205
00:17:17.000 --> 00:17:23.240
You can connect a skill to uh another APM system like DataDog if there's more logs you want to pull in.

s206
00:17:23.240 --> 00:17:30.960
Um there's really no limit to what skills you can define as long as you have an API key and it's callable from the agent harness itself.

s207
00:17:30.960 --> 00:17:37.040
Um and we're actually under the hood running these as a Rise managed sandboxes uh within the platform.

s208
00:17:37.040 --> 00:17:46.680
Uh you can bring your own harness, bring your own sandbox provider as well uh to kind of you know whether you're using Modal, Daytona, E2B, whatever it is or whatever harness you're using as well we support

s209
00:17:46.680 --> 00:17:51.760
Codex, Cursor, Cloud Code, and Open Code with more harness support rolling out.

s210
00:17:51.760 --> 00:17:57.800
So, does everyone kind of understand, you know, we have a managed agent doing this in the platform.

s211
00:17:57.800 --> 00:18:05.160
You can set up your own automations to you know either be triggered on a time schedule or off of an alert.

s212
00:18:05.160 --> 00:18:11.720
So, anytime you see latency spike above 10 seconds, we automatically kick off this agent to do an investigation.

s213
00:18:11.720 --> 00:18:14.040
Um and investigation agents are part of it.

s214
00:18:14.040 --> 00:18:16.000
Kind of solves the SRE use case.

s215
00:18:16.000 --> 00:18:17.320
There's a cost lens to this.

s216
00:18:17.320 --> 00:18:19.600
There's a product classification lens to this.

s217
00:18:19.600 --> 00:18:26.600
Um there are many ways in which you can kind of use this agent platform to operate on your traces.

s218
00:18:26.920 --> 00:18:28.360
Cool.

s219
00:18:28.360 --> 00:18:30.880
Uh I will continue through these.

s220
00:18:30.880 --> 00:18:35.200
So, Signals are reading your traces, filing what it finds.

s221
00:18:35.200 --> 00:18:38.400
It keeps one ticket per problem uh the way a reviewer would.

s222
00:18:38.400 --> 00:18:44.000
So, you're not getting hundreds and hundreds of issues on the same uh piece of evidence.

s223
00:18:44.000 --> 00:18:45.080
Uh it opens those.

s224
00:18:45.080 --> 00:18:46.000
You fix it.

s225
00:18:46.000 --> 00:18:54.560
You can either resolve it if you think it's resolved, you can ignore it um and it will dedupe those issues moving forward so you don't, you know, kind of get inundated with noise.

s226
00:18:54.560 --> 00:19:05.760
Um the add to data set field or uh flow I think is one of the most important parts here because how do you build the actual continuous improvement?

s227
00:19:05.760 --> 00:19:11.480
You have to have a record of what's failed in the past and guard against those as regression checks moving forward, right?

s228
00:19:11.480 --> 00:19:16.720
So, adding these two data sets give you that sort of living fixture quality.

s229
00:19:16.720 --> 00:19:20.840
So, you know, you define test cases in code before you ship code.

s230
00:19:20.840 --> 00:19:25.720
Everyone should be doing that, and hopefully we are all doing that with the help of some agents now.

s231
00:19:25.720 --> 00:19:38.240
Um but this idea of living fixtures that change over time as you add new use cases, as you see new failure modes, is a pretty novel one and that kind of helps with the self-improvement

s232
00:19:38.240 --> 00:19:49.520
uh loop where, you know, you're not beholden to one definition at one time of what has failed and what our failure modes, but you're organically growing that definition of failure mode

s233
00:19:49.520 --> 00:19:51.480
as your agent is failing in production, right?

s234
00:19:51.480 --> 00:20:04.960
So, um think of it like, you know, every time you go to try and ride a bike, you notice something about how you're riding, and then over time, those become so ingrained in your memory that riding a bike feels like nothing.

s235
00:20:04.960 --> 00:20:18.880
You don't even think about the failure modes of um I actually fell off a bike and broke my foot 2 months ago, which is why I'm mentioning this this little anecdote, but typically you don't think about, you know, putting your left foot on left pedal anymore because it's been so ingrained into your learning process.

s236
00:20:18.880 --> 00:20:23.160
And that's what we're trying to kind of accomplish with these agents.

s237
00:20:23.400 --> 00:20:32.040
Um so, on the data set side, once you get these living fixtures, you actually want to codify the behavior on these fixtures, right?

s238
00:20:32.040 --> 00:20:34.080
And that's where the eval comes in.

s239
00:20:34.080 --> 00:20:38.880
Um so, I saw a few show of hands of that people are running evals.

s240
00:20:38.880 --> 00:20:40.680
Uh can I get a quick gauge of the room?

s241
00:20:40.680 --> 00:20:45.120
Who here is running LLM as a judge evals or you're running code evaluators?

s242
00:20:45.120 --> 00:20:46.200
Cool.

s243
00:20:46.200 --> 00:20:51.080
So, if you were in any of the earlier workshops of the Rise, we went into evaluators in great depth.

s244
00:20:51.080 --> 00:20:54.240
Um there are many, many different forms of evaluators.

s245
00:20:54.240 --> 00:20:56.560
Um so, just quick quick overview.

s246
00:20:56.560 --> 00:21:03.560
There are code checks that are, you know, really, really cheap to run that are for deterministic checks, right?

s247
00:21:03.560 --> 00:21:05.240
Is this field present?

s248
00:21:05.240 --> 00:21:07.200
Is this valid JSON?

s249
00:21:07.200 --> 00:21:08.120
Exact rules.

s250
00:21:08.120 --> 00:21:09.680
Was this filter applied?

s251
00:21:09.680 --> 00:21:12.024
Was this tool called with the right arguments?

s252
00:21:12.024 --> 00:21:12.120
[clears throat]

s253
00:21:12.120 --> 00:21:13.989
Very, very easy for that, right?

s254
00:21:13.989 --> 00:21:14.520
[snorts]

s255
00:21:14.520 --> 00:21:24.560
LLM as a judge, evals also help with this use case where you need to have more of a qualitative or nebulous kind of analysis of the trace, right?

s256
00:21:24.560 --> 00:21:32.360
And so, this is using another LLM model to score the results of one model's trace.

s257
00:21:32.360 --> 00:21:37.240
Um obviously costs will be higher here because you're making another model call.

s258
00:21:37.240 --> 00:21:42.480
Um per record, obviously you can, you know, kind of set your own sampling, filtering.

s259
00:21:42.480 --> 00:21:45.840
You don't have to run these on 100% of the traces that you're running at.

s260
00:21:45.840 --> 00:21:51.840
Um there's some trade-offs in terms of eval design that we can discuss, but that's sort of what an LLM as a judge is for.

s261
00:21:51.840 --> 00:21:58.520
Then you have for the even more open-ended use cases, you have this concept of agent as a judge.

s262
00:21:58.520 --> 00:22:10.240
So, where you're not actually sure what you're evaluating, you have a goal in mind, um and you leave kind of the the determination of whether that goal has been achieved

s263
00:22:10.240 --> 00:22:10.960
to the agent.

s264
00:22:10.960 --> 00:22:17.040
So, the agent can actually pick its own rails, it can actually pick its own scores, it can go through that process itself.

s265
00:22:17.040 --> 00:22:22.640
And that's obviously the highest cost method because there's multiple LLM calls happening under the hood.

s266
00:22:22.640 --> 00:22:26.880
Um [snorts and clears throat] decision models have been around for, you know, what?

s267
00:22:26.880 --> 00:22:38.960
A month-ish now, and with Lower actually at Arize has published a ton of material on sort of benchmarking them and how they operate, and what trade-offs are present with decision models.

s268
00:22:38.960 --> 00:22:47.320
I will say they are a very, very cheap way to run some of these checks on your traces versus um using a classic LLM.

s269
00:22:47.320 --> 00:22:53.800
Um and they give you, you know, type type choices, yes, no, choice, score.

s270
00:22:56.360 --> 00:23:00.880
So, once you have sort of this eval signal, right?

s271
00:23:00.880 --> 00:23:06.960
So, you you understand, you know, qualitatively is this in the right direction?

s272
00:23:06.960 --> 00:23:08.080
Is it not in the right direction?

s273
00:23:08.080 --> 00:23:10.160
Is it passing my eval checks?

s274
00:23:10.160 --> 00:23:12.000
And you have the trace data.

s275
00:23:12.000 --> 00:23:18.000
That's all really rich context for a coding agent to go um implement a fix, right?

s276
00:23:18.000 --> 00:23:29.560
So, signal in this case is is running on on an Auri managed sandbox with a coding harness, implement those fixes for you, but if you're in your development phase,

s277
00:23:29.560 --> 00:23:32.680
you can actually pull these traces live into Auri through the CLI as well, right?

s278
00:23:32.680 --> 00:23:38.200
So, it's instant context to feed other agents to actually work on those PRs for you.

s279
00:23:38.200 --> 00:23:44.040
Um and this is where we have kind of the human check and balance in in the room.

s280
00:23:44.040 --> 00:23:52.440
Um so, I would say the the places a human adds the most value here are one, obviously validating the fix, right?

s281
00:23:52.440 --> 00:23:53.840
Reading the PR.

s282
00:23:53.840 --> 00:23:58.920
There are, you know, maybe some of us in the camp of we don't want to validate every human PR.

s283
00:23:58.920 --> 00:24:05.720
I assume if you're in a high-stakes use case like finance, you're very much in the camp that humans read the PR before the merge, right?

s284
00:24:05.720 --> 00:24:09.120
So, that's one very high-leverage point for a human to be here.

s285
00:24:09.120 --> 00:24:18.280
In the data set curation phase as well, so adding that to a data set is part of why we left it manual is are these good enough findings to add to my fixture, right?

s286
00:24:18.280 --> 00:24:23.320
So, that's a manual step in the process, uh and that relies on kind of human judgment as well.

s287
00:24:23.320 --> 00:24:35.760
Um the eval design, so we have agents that kind of help you write these evals, um but the eval design portion is something where humans can definitely be in the mix because you're imparting your taste of what is good

s288
00:24:35.760 --> 00:24:41.040
onto the agent, whether that's in the form of an LLM as a judge or a more deterministic check.

s289
00:24:41.040 --> 00:24:47.720
Um and then folding all this information into the coding agent gives it really rich context for it to work from.

s290
00:24:47.720 --> 00:24:54.280
Um so, just as an example of kind of like how you would create an eval off this, I'm going to go to my traces tab.

s291
00:24:54.280 --> 00:24:58.480
I'm working off of this Wonder Toys project.

s292
00:24:58.720 --> 00:25:02.000
I have signal running and signals found two issues so far.

s293
00:25:02.000 --> 00:25:07.200
One is the orphan spans issue, um, which was actually just my mistake instrumenting this.

s294
00:25:07.200 --> 00:25:08.800
Uh, but it did catch it.

s295
00:25:08.800 --> 00:25:12.440
Um, and two is the redundant search retries.

s296
00:25:12.440 --> 00:25:14.160
That's the bug we planted.

s297
00:25:14.160 --> 00:25:18.040
Um, so I'm able to create a new evaluator with one click off of this.

s298
00:25:18.040 --> 00:25:26.120
And that actually, uh, looks like I need to use a different model.

s299
00:25:28.640 --> 00:25:38.240
That actually opens up Alex, uh, which is our in-house agent that I'm doing a talk about tomorrow, um, that is able to go and create that eval off of that evidence for me, right?

s300
00:25:38.240 --> 00:25:53.677
So, based on the traces that are supported in the evidence here, we can kind of codify in a prompt template what we should be expecting moving forward and how we how we make sure that the eval fires and fails on any mistakes that are made.

s301
00:25:53.677 --> 00:25:54.040
[snorts]

s302
00:25:54.040 --> 00:25:56.320
So, this is the eval I came up with.

s303
00:25:56.320 --> 00:26:09.160
You're a helpful AI bot that checks for redundant search product retries, inspect the complete trajectory including tool calls, see if there's any unnecessary tool call, return a redundant retry when search product is called more than once,

s304
00:26:09.160 --> 00:26:14.080
um, and put the score and classification choice on the trace itself.

s305
00:26:14.720 --> 00:26:24.960
And then you can run this eval on incoming traces that are coming into your project, so you make sure you catch that, you can monitor that, you can graph that, you can create an alert off of that,

s306
00:26:24.960 --> 00:26:27.960
uh, and you can track spend associated with that eval as well.

s307
00:26:27.960 --> 00:26:37.440
So, over time, you know, maybe your development flow has codified this as a regression check and you no longer need to deploy this eval on production data.

s308
00:26:37.440 --> 00:26:41.480
You don't want to spend the model tokens on running this on every trace.

s309
00:26:41.480 --> 00:26:46.440
You can actually view the cost over time and retire that eval as well.

s310
00:26:47.200 --> 00:26:49.200
Cool.

s311
00:26:50.080 --> 00:26:55.800
So, the next part of this is okay, we have a fix.

s312
00:26:55.800 --> 00:26:57.640
Signal found something.

s313
00:26:57.640 --> 00:26:59.440
It analyzed the repo.

s314
00:26:59.440 --> 00:27:01.520
We created an eval to catch it for the next time.

s315
00:27:01.520 --> 00:27:04.000
We added the evidence to a data set.

s316
00:27:04.000 --> 00:27:06.680
So, now we have a living fixture that's growing with time.

s317
00:27:06.680 --> 00:27:09.280
How do we know that the fix is right?

s318
00:27:09.280 --> 00:27:16.240
How do we actually know that, you know, it wasn't just slop code that the agent produced and tried to find an issue that didn't actually exist?

s319
00:27:16.240 --> 00:27:20.680
Um and this is where the world of agent experimentation comes into play.

s320
00:27:20.680 --> 00:27:23.120
And this is a very, very rich and deep world.

s321
00:27:23.120 --> 00:27:27.960
Um so, I won't spend, you know, time talking about all the different frameworks and providers that exist out there.

s322
00:27:27.960 --> 00:27:33.360
Phoenix team did a good job earlier today talking a little bit about the harbor integration they have in Phoenix.

s323
00:27:33.360 --> 00:27:40.360
Uh but really what we're concerned about is prove that the fix works and prove we didn't break anything else, right?

s324
00:27:40.360 --> 00:27:52.080
Um and there are many cases where, you know, in this non-deterministic world, if you make a small change to a system prompt, it can then have ramifications on other tools that you didn't think were going to break.

s325
00:27:52.080 --> 00:27:56.200
Uh and, you know, kind of um redundancy issues there.

s326
00:27:56.200 --> 00:28:04.200
Um and so, making sure that we can prove that this fix actually worked and that nothing else broke is really, really important.

s327
00:28:04.200 --> 00:28:06.240
And that's kind of the experimentation stage.

s328
00:28:06.240 --> 00:28:11.480
Um so, I will talk a little bit about this through the flow as well.

s329
00:28:11.480 --> 00:28:14.800
So, in the Rise, we support a bunch of different types of experiments.

s330
00:28:14.800 --> 00:28:26.920
Um this is the category code enum fix experiment that I just ran.

s331
00:28:27.080 --> 00:28:32.680
And so, you'll see here that we have actual evals running on this experiment.

s332
00:28:32.680 --> 00:28:36.840
So, what we're essentially doing is we have this living fixture of failure cases.

s333
00:28:36.840 --> 00:28:38.600
We have this proposed fix.

s334
00:28:38.600 --> 00:28:43.880
We actually want to rerun the agent on these failure cases and score the results.

s335
00:28:43.880 --> 00:28:49.880
And if they meet a certain um threshold, that's kind of up to you to define.

s336
00:28:49.880 --> 00:28:52.800
In my case, 88.9% I thought was pretty good.

s337
00:28:52.800 --> 00:28:54.480
So, I was like, ship it.

s338
00:28:54.480 --> 00:29:00.120
Um if they meet that threshold, you actually have results to prove that fix from the agent, right?

s339
00:29:00.120 --> 00:29:04.160
And the agent can actually run these tests for you through the AX CLI as well.

s340
00:29:04.160 --> 00:29:17.720
So, as Signal is going and finding these failure cases, and it has access to the agent code, it can actually pipe those failure cases through and replay them through the agent with the version two fix,

s341
00:29:17.720 --> 00:29:19.640
and then score those results themselves, right?

s342
00:29:19.640 --> 00:29:30.040
So, you come back in the morning, you weren't working up at 4:00 a.m. You not only have a PR that has a proposed fix, you have some validation results that say, "Hey,

s343
00:29:30.040 --> 00:29:42.040
I proved this fix because I actually took the test cases that we failed on, and I re-measured latency or tool call retries, and I saw that we were no longer retrying tools, and latency was lower.

s344
00:29:42.040 --> 00:29:46.240
And hey, by the way, you had three other evals that you don't want to fail on, either.

s345
00:29:46.240 --> 00:29:51.080
So, I made sure to check that those evals are not regressing, right?

s346
00:29:51.080 --> 00:29:57.760
And so, as that living fixture grows over time, you have new cases, these experiments become more and more robust, right?

s347
00:29:57.760 --> 00:30:03.760
Um and in the platform, obviously, you can go view those experiment results, see what the output is, see what the evals are.

s348
00:30:03.760 --> 00:30:11.000
If you disagree with the agent, you can annotate that output, and that becomes another piece of information to the context for the next run, right?

s349
00:30:11.000 --> 00:30:23.680
So, everyone kind of clear on this experiment flow at a high level, like how you can run some of these experiments to take a test case, take an eval that you've created as a benchmark, and actually

s350
00:30:23.680 --> 00:30:27.240
validate your fix before it goes out, before it goes to prod.

s351
00:30:27.240 --> 00:30:31.720
Um and you can shove this into the CI/CD process as well.

s352
00:30:31.720 --> 00:30:33.800
Um up to you on how you want to configure it.

s353
00:30:33.800 --> 00:30:37.400
Obviously, running the the agent may be expensive.

s354
00:30:37.400 --> 00:30:40.360
Um so, your discretion on how you want to run that.

s355
00:30:40.360 --> 00:30:41.440
Evals as well, right?

s356
00:30:41.440 --> 00:30:50.320
If you're using an LM as a judge eval, you want to be careful about which model you're picking, when you're running those evals, what sampling rate you're running those evals at,

s357
00:30:50.320 --> 00:30:59.160
to make sure you're not just ballooning costs and you know, just sending a ton of spend on validating these fixes on every push.

s358
00:31:00.000 --> 00:31:00.600
Cool.

s359
00:31:00.600 --> 00:31:10.280
So, kind of continuing on, we approve the fix, we look at the task completion rates.

s360
00:31:10.280 --> 00:31:21.360
Um we validate those results and while you kind of step away from this process, what keeps running is that the eval that you created runs on every new trace

s361
00:31:21.360 --> 00:31:24.440
and becomes additional context to the next signal run.

s362
00:31:24.440 --> 00:31:35.360
The task system that we have in arise can also automatically add failed eval results to new databases or data sets so that that fixture grows over time.

s363
00:31:35.360 --> 00:31:50.960
And signal keeps rerunning every 6 hours or off of an alert or off of a triggered monitor and make sure that new failures are surfaced, new evals are created and that data set grows with those new examples over time.

s364
00:31:50.960 --> 00:32:00.600
We also have, you know, weekly digests where you can kind of consume these in a little bit more of an easy-to-read result where you just have a summary of what's going on with the agent.

s365
00:32:00.600 --> 00:32:05.520
What is waiting for you is confirming that these issues are real, right?

s366
00:32:05.520 --> 00:32:08.960
You can't just trust every single issue that comes out of this agent.

s367
00:32:08.960 --> 00:32:16.880
I would say use it as a starting point, validate these fixes and use your judgment on when those issues are actually real.

s368
00:32:16.920 --> 00:32:20.760
Writing down what the right answer looks like, so this kind of goes in the annotation flow, right?

s369
00:32:20.760 --> 00:32:32.000
You run the experiment, you get a new output, maybe you don't think this is the perfect answer or the perfect trajectory for the agent and you can actually use the annotations product to kind of codify that taste.

s370
00:32:32.000 --> 00:32:34.360
Merging the fix, this is up to you.

s371
00:32:34.360 --> 00:32:36.720
You can auto merge these PRs.

s372
00:32:36.720 --> 00:32:38.800
I will leave that to your discretion.

s373
00:32:38.800 --> 00:32:41.400
Uh and then actually deploying and promoting the new version, right?

s374
00:32:41.400 --> 00:32:49.880
So, I have V2, I validated it, I think the issue is real, I created the eval, I looked at the score, I validated the fix, I like the code, I reviewed the code.

s375
00:32:49.880 --> 00:32:53.400
Now it's time for me to actually upgrade the agent.

s376
00:32:53.840 --> 00:33:08.840
Um I think the the other thing that's really interesting about this is like you can codify the confidence scores, especially with these decision one models that actually return an explicit confidence score,

s377
00:33:08.840 --> 00:33:12.360
and have those auto route to human beings uh in annotation queues, right?

s378
00:33:12.360 --> 00:33:23.240
So, um not only are you, you know, responsible for reviewing the outputs, but you actually get a ready-made queue for you that is based on all the low confidence results, and

s379
00:33:23.240 --> 00:33:28.240
um you know, in some of our benchmarking uh we've seen these confidence scores to be pretty good.

s380
00:33:28.240 --> 00:33:32.520
It depends on your use case and uh specifics in the industry, for sure.

s381
00:33:32.520 --> 00:33:47.120
Uh so, I wouldn't say that an 80% confidence score is the right threshold for everyone, uh but setting those thresholds is important, and then once you have those thresholds defined, you can kind of auto route those low confidence cases to a person

s382
00:33:47.120 --> 00:33:48.560
to go annotate.

s383
00:33:48.560 --> 00:34:01.760
Um So, I mean, there's, you know, in the spam test that we saw, 4.6 % of cases scored between 0.3 and 0.7 confidence to people, left the rest at 99.5%

s384
00:34:01.760 --> 00:34:02.400
accuracy, right?

s385
00:34:02.400 --> 00:34:11.520
So, that is one example of an industry where, you know, specifically for spam detection, where those confidence thresholds might be more defined.

s386
00:34:11.520 --> 00:34:14.760
It It depends on the use case, for sure.

s387
00:34:14.760 --> 00:34:23.440
Um So, I kind of talked about some of the stuff we've built so far with signal manage agents and traces, right?

s388
00:34:23.440 --> 00:34:33.879
So, the ability to actually read those traces at scale, the ability to add things to data sets, uh the ability to run those experiments with the help of a coding agent,

s389
00:34:33.879 --> 00:34:35.960
um, and actually review those results in line.

s390
00:34:35.960 --> 00:34:37.600
You can also interrupt sessions.

s391
00:34:37.600 --> 00:34:39.600
You can look through the model's reasoning.

s392
00:34:39.600 --> 00:34:47.200
You can kind of continue a session, kick it off to a local agent, uh, so that you can continue development on your own side if you don't want this all managed by a rise.

s393
00:34:47.200 --> 00:34:56.879
But, I think our vision for the future is this running this full end-to-end loop with almost zero human involvement until you need to go test something, right?

s394
00:34:56.879 --> 00:35:00.160
Until you need to go click around in a UI and validate something.

s395
00:35:00.160 --> 00:35:05.560
Um, and so there are a lot of providers out there for that experimentation setup, right?

s396
00:35:05.560 --> 00:35:06.920
There's the harbors of the world.

s397
00:35:06.920 --> 00:35:09.880
There's many, many different ways that you can test the agent.

s398
00:35:09.880 --> 00:35:18.720
Um, I think most of it, and and this is where a lot of the discussion is happening, is how many dependencies do you have to run to run this agent, right?

s399
00:35:18.720 --> 00:35:23.280
Um, running an agent in the experimental phase, does that mean you connect it to a mock DB?

s400
00:35:23.280 --> 00:35:25.280
Do you stop some tools?

s401
00:35:25.280 --> 00:35:29.040
Um, do you actually run the entire thing in a docker image?

s402
00:35:29.040 --> 00:35:31.080
Can you even do that safely?

s403
00:35:31.080 --> 00:35:38.000
Um, those are all sort of questions that I would encourage you guys to think about as you start thinking about this experimentation flow.

s404
00:35:38.000 --> 00:35:40.640
With this agent, you know, it's very simple.

s405
00:35:40.640 --> 00:35:42.880
We just rerun the agent in its entirety.

s406
00:35:42.880 --> 00:35:44.440
There are no dependencies.

s407
00:35:44.440 --> 00:35:48.360
It spins up its own DB, and then we run requests through that.

s408
00:35:48.360 --> 00:35:50.360
So, that might be a very simple use case.

s409
00:35:50.360 --> 00:35:56.760
But, when you have external dependencies that really matter, your use case become can become bit more convoluted.

s410
00:35:56.760 --> 00:36:02.000
Um, so our goal for this is really enabling the one click from issue to experiment.

s411
00:36:02.000 --> 00:36:03.760
Hey, I have this proposed fix.

s412
00:36:03.760 --> 00:36:04.600
Go test it.

s413
00:36:04.600 --> 00:36:07.680
Spin up all the infrastructure necessary to go test it.

s414
00:36:07.680 --> 00:36:15.400
Um, in addition to this, and we've we've worked on the CLI around this, is connecting signal where you work as well.

s415
00:36:15.400 --> 00:36:30.320
So, whether that's in Slack, so tagging uh, from Slack to communicate with signal, whether that's your local agents, which we have through the API, CLI, and skills, where you can actually ask your local agent to go do a digest of what signal is finding and and check those.

s416
00:36:30.320 --> 00:36:40.960
Um making sure that we have the fix drafted straight from the issue and its traces and that's part of what we've done already and actually having the evidence in line.

s417
00:36:40.960 --> 00:36:47.080
So, here are the actual five traces that failed in the PR description so you can actually go validate those.

s418
00:36:47.080 --> 00:36:58.800
Um we have signal now proposing evals as well for you so it'll actually check your evals see which ones are healthy as in, you know, if this eval is passing 99.999%

s419
00:36:58.800 --> 00:37:04.880
of the time maybe we don't want to spend the tokens on this eval and so we can retire this eval going forward.

s420
00:37:04.880 --> 00:37:06.680
Uh we feel pretty confident on it.

s421
00:37:06.680 --> 00:37:12.360
Or, you know, vice versa if any eval is in poor health, making that eval more robust.

s422
00:37:12.360 --> 00:37:22.920
And we have a whole process on eval alignment which is how do I make sure the judge is actually aligned to what I think and isn't just outputting scores that I don't believe in.

s423
00:37:22.920 --> 00:37:25.600
Um so there's a whole process for doing that.

s424
00:37:25.600 --> 00:37:32.560
And then, again, the experimentation flow, running your agent against running these data sets against your agent wherever it lives.

s425
00:37:32.560 --> 00:37:39.800
Uh whether it's your laptop, in my case for this very simple agent, or whether it's on more complicated infrastructure.

s426
00:37:40.040 --> 00:37:40.640
Cool.

s427
00:37:40.640 --> 00:37:45.560
So, I think it would be really great to get started with some of these.

s428
00:37:45.560 --> 00:37:48.520
Uh so I have the workshop repo URL.

s429
00:37:48.520 --> 00:37:53.840
Um we have docs pages as well for agent experimentation.

s430
00:37:53.840 --> 00:37:59.040
And um we have a few other talks throughout this week as well.

s431
00:37:59.040 --> 00:38:07.440
Um So, if you has one tomorrow at 3:20 about um agent fixes and their run-off um implications.

s432
00:38:07.440 --> 00:38:12.440
So, you know, you fix something somewhere and that eventually has impact downstream.

s433
00:38:12.440 --> 00:38:18.120
Um and I have a talk tomorrow at 3:45 talking about Alex which is our in-house agent.

s434
00:38:18.120 --> 00:38:25.160
Um and we use Alex to build Alex so it's all kind of the perfect dog fooding tool for us in the agent space.

s435
00:38:25.160 --> 00:38:33.840
So, I'll talk a little bit about how we designed that agent, you know, how we design context, memory, and some of the decisions and lessons we learned along the way.

s436
00:38:33.840 --> 00:38:37.720
Really, I would encourage you guys to try to get some traces from the agent.

s437
00:38:37.720 --> 00:38:45.760
We have this built out for you, but if you have an your own example and you haven't traced your agent yet, it's very, very simple to install our skills and CLI,

s438
00:38:45.760 --> 00:38:50.000
have it instrument your agent for you, and send those traces to Arize.

s439
00:38:50.000 --> 00:39:01.840
Um once you get those traces in, it's one click to enable signal, and then you can start to see the clustering of issues and patterns across your traces as well.

s440
00:39:01.840 --> 00:39:03.320
Um so, yeah.

s441
00:39:03.320 --> 00:39:14.240
I would say the things I want to leave you guys with are the hard work of reading a ton of data should definitely be just be done by an agent.

s442
00:39:14.240 --> 00:39:21.040
Understanding where to add the highest leverage human points is kind of the theme of this presentation.

s443
00:39:21.240 --> 00:39:29.040
Just because you can automate something doesn't mean it should be automated, but you should use that automation to really inform the reviewer.

s444
00:39:29.040 --> 00:39:35.000
So, things like auto routing low confidence threshold scores to that reviewer queue are really important.

s445
00:39:35.000 --> 00:39:49.320
Um and making sure that all the context is fed back to the same system, and obviously we're here with Arize, and you know, we're telling you how great Arize is, but really I you want to make sure that this context is recorded somewhere, right?

s446
00:39:49.320 --> 00:40:00.560
And that's the only way you build the self-improvement loop is when the human judgment, the eval scores, and the traces, and the the trace evidence are all held together in the same place

s447
00:40:00.560 --> 00:40:06.240
feeding that context for the next run, so you're catching new issues instead of just fighting to stay afloat.

s448
00:40:06.240 --> 00:40:08.440
Um and that's what I have for you guys.

s449
00:40:08.440 --> 00:40:09.400
Thank you so much.

s450
00:40:09.400 --> 00:40:13.880
Um I'm happy to walk around and kind of help you guys get started on some of these.

s451
00:40:13.880 --> 00:40:16.800
Um you can check out the repo, clone down the agent.

s452
00:40:16.800 --> 00:40:23.840
If not, uh we you can install the skills, instrument your own agent, and yeah, I can take questions as well if you would like.

s453
00:40:23.840 --> 00:40:27.160
I would say there's like two lenses to take how we're actually evaluating.

s454
00:40:27.160 --> 00:40:34.320
There's how do we evaluate signal as a product line, and that one of the key factors there would be, okay, how often is this right about our PRs, right?

s455
00:40:34.320 --> 00:40:36.680
That's sort of like an internal evaluation.

s456
00:40:36.680 --> 00:40:44.680
Do you mean more of that how we evaluate our own product or how you can evaluate an an agent use case, generally speaking?

s457
00:40:44.680 --> 00:40:57.960
Uh we will do a whole talk about evals for Alex tomorrow, but to give you the long and short of it, um eval design is you know, as an as an AIPM, I think that's probably my primary job is like

s458
00:40:57.960 --> 00:41:01.960
how do I actually design good evals that actually measure the right things?

s459
00:41:01.960 --> 00:41:10.320
The shorter answer to that question is I think codifying qualitative decisions is very hard.

s460
00:41:10.320 --> 00:41:20.400
Codifying things like latency, retry count, like these are more classical issues that, for example, the issue signal found, I think, is relatively simple, like you found an inconsistency in casing,

s461
00:41:20.400 --> 00:41:24.840
you saw evidence in retries, make the fix, measure the latency.

s462
00:41:24.840 --> 00:41:26.640
That's really easy to do, right?

s463
00:41:26.640 --> 00:41:33.000
Now, qualitative outcomes, I think, are a little bit harder, and I would say there's there's a couple things to index on there.

s464
00:41:33.000 --> 00:41:45.080
One is the the actual like definition of what is good is only emergent from as much signal as you can give it, and I use signal as like the word signal.

s465
00:41:45.080 --> 00:41:52.040
Um so what I mean by that is, okay, I can look at one example and say it's right or wrong.

s466
00:41:52.040 --> 00:41:59.320
I can't get an agent to learn from one example because then it's just going to overfit to that one example, or that example might not generalize properly.

s467
00:41:59.320 --> 00:42:04.800
Um and so part of what I was talking about with the living fixtures and the annotations is a lot of what we do.

s468
00:42:04.800 --> 00:42:08.840
So I'll give you an example, Alex generating um filter queries, right?

s469
00:42:08.840 --> 00:42:11.240
So we have Alex in in platform.

s470
00:42:11.240 --> 00:42:22.200
Uh we have a flow where, okay, if you want to search agents or search traces, you can obviously type in a filter, or you can ask Alex to filter with AI.

s471
00:42:22.200 --> 00:42:24.280
How do we actually measure the performance of that?

s472
00:42:24.280 --> 00:42:35.400
There's no right or wrong answer, cuz it could return a filter that succeeds, and it doesn't, you know, parse this JSON, and it will give you some results, but how do you know that's actually a good result?

s473
00:42:35.400 --> 00:42:36.040
So, there's a few things.

s474
00:42:36.040 --> 00:42:39.240
One, we look at the entire session of users, right?

s475
00:42:39.240 --> 00:42:42.920
So, we're not just saying, "Okay, for this turn, was this a good response?"

s476
00:42:42.920 --> 00:42:44.800
But, did that user ask again?

s477
00:42:44.800 --> 00:42:46.040
Did it Was it happy with the results?

s478
00:42:46.040 --> 00:42:49.040
How many additional filters did they create, right?

s479
00:42:49.040 --> 00:42:53.000
So, that can be one thing to as a signal for you to evaluate on.

s480
00:42:53.000 --> 00:42:57.760
The second is, I mean, annotations like reading through, okay, what was the query?

s481
00:42:57.760 --> 00:42:59.560
Does this look like a right filter?

s482
00:42:59.560 --> 00:43:01.160
Did it return the right eval results?

s483
00:43:01.160 --> 00:43:05.560
And actually annotating that by hand as a PM who knows what to expect.

s484
00:43:05.560 --> 00:43:14.280
Um, and then you combine both of those with new use cases, and that rich context becomes the eval template, right?

s485
00:43:14.280 --> 00:43:23.920
So, you can actually use another coding agent to feed that context in and write a really good prompt template, and then it becomes like a a prompt optimization essentially, right?

s486
00:43:23.920 --> 00:43:29.800
So, like, how good can I get this eval template on the lowest cost model so I can run this at scale?

s487
00:43:29.800 --> 00:43:31.200
Um, so, that's kind of how we think about it.

s488
00:43:31.200 --> 00:43:40.680
We have some more specifics to talk tomorrow where I'll talk through a little bit of which models we choose, and um, how we think about which use cases are worth running an eval on or not.

s489
00:43:40.680 --> 00:43:42.840
Um, but yeah, that's the short answer.

s490
00:43:42.840 --> 00:43:55.520
So, it is kind of like a, you know, post hoc algo propter hoc, like, where can we make sure like what we define as healthy is actually healthy, and where should we define where it breaks?

s491
00:43:55.520 --> 00:43:59.680
So, So, what I mean by that is, okay, it's right 99.9% of the time.

s492
00:43:59.680 --> 00:44:06.080
If that 0.1% is catastrophic, and we need to catch all of those, it's a very different discussion, right?

s493
00:44:06.080 --> 00:44:10.680
And then evals, when I when I say like Evals there's kind of two lenses.

s494
00:44:10.680 --> 00:44:17.240
There's the Evals that you can run in the development flow as CICD checks, as regression checks before you deploy code.

s495
00:44:17.240 --> 00:44:23.880
There's Evals that you're running on live traces, which that becomes a cost-benefit ratio question, right?

s496
00:44:23.880 --> 00:44:35.280
If I'm running this on 100% of live traces is the cost of running this model on all my traces worth it compared to the cost of this 0.1% failure, right?

s497
00:44:35.280 --> 00:44:38.240
So I think a lot of it boils down to ROI.

s498
00:44:38.240 --> 00:44:44.640
I think general guidelines are whatever you can catch in CICD you should, and I understand it's a challenge to run some of these.

s499
00:44:44.640 --> 00:44:53.400
So like and and convince teams to run some of these in CICD I would probably make some of these non-optional checks in the CICD process, right?

s500
00:44:53.400 --> 00:45:03.760
So you have a data set of fixtures that shows where this Eval has actually failed on the 0.1% of cases and you run you rerun those with an Eval score

s501
00:45:03.760 --> 00:45:07.120
every single time you push or before a PR is merged.

s502
00:45:07.120 --> 00:45:11.880
So you're at least you at least have some confidence that that set of cases is not going to fail.

s503
00:45:11.880 --> 00:45:15.960
And then you run an Eval that has a sampling rate on your prod traces, right?

s504
00:45:15.960 --> 00:45:21.880
Um so instead of saying, "Okay, we're going to ignore the development process, evaluate everything that comes in production."

s505
00:45:21.880 --> 00:45:33.280
That can be a very high cost thing to do versus "Okay, we ignore prod completely and only test in development and try to capture every case, which you probably can't do

s506
00:45:33.280 --> 00:45:37.440
um in a data set with coverage, and then just let it happen in prod."

s507
00:45:37.440 --> 00:45:40.880
I would say both of those are extremes, and the answer is somewhere in the middle.

s508
00:45:40.880 --> 00:45:44.400
And it depends a bit on the ROI of the Eval, right?

s509
00:45:44.400 --> 00:46:00.560
Um and what model you decide to use has an implication on cost, etc., etc. It's a very good question, and it gets very meta, and uh I I hope I hope you haven't noticed how meta this conversation has gotten so far where I'm using signal in multiple different ways.

s510
00:46:00.560 --> 00:46:05.880
Um the short answer is you can do that entire loop again, right?

s511
00:46:05.880 --> 00:46:08.800
So, we have a project for eval traces.

s512
00:46:08.800 --> 00:46:09.680
This is a Rise created.

s513
00:46:09.680 --> 00:46:12.360
These are all the evals, and those evals are traced, right?

s514
00:46:12.360 --> 00:46:17.200
And you you can have an agent reading the eval traces and making sure the evals are good.

s515
00:46:17.200 --> 00:46:22.400
And then, each of those agents actually produces their own traces, right?

s516
00:46:22.400 --> 00:46:25.400
So, when you're running Signal, these are Signal's traces.

s517
00:46:25.400 --> 00:46:30.120
So, you'll see here you are Signal, an automated reviewer for Rise.

s518
00:46:30.120 --> 00:46:32.000
These are these are steps you have to do.

s519
00:46:32.000 --> 00:46:32.880
These are the instructions.

s520
00:46:32.880 --> 00:46:34.480
This is what Signal actually called under the hood.

s521
00:46:34.480 --> 00:46:36.600
So, made a few bash calls.

s522
00:46:36.600 --> 00:46:42.520
It called our MC internal MCP to go do this investigation and then mits it, writes it as an issue, right?

s523
00:46:42.520 --> 00:46:49.120
And then, you can run Signal on top of Signal as well to make sure that And that's kind of what how we evaluate internally, right?

s524
00:46:49.120 --> 00:46:52.520
Uh, it's kind of what I was alluding to as the the ultimate dog fooding tool.

s525
00:46:52.520 --> 00:46:57.000
Um, now I I assume your question is more of like a when does this break down?

s526
00:46:57.000 --> 00:46:58.680
When is this useful to do?

s527
00:46:58.680 --> 00:47:09.760
Um, I think the harness changes are useful up to a certain point, and I'll talk a little bit about this in in my talk tomorrow on like how we evaluate Alex.

s528
00:47:09.760 --> 00:47:22.320
So, like there are certain use cases where you just a- after a certain point, you don't need to continue evaluating, uh, and there's just diminishing returns on cost from there.

s529
00:47:22.320 --> 00:47:29.240
Um, where we've seen it is, you know, we monitor Signal and we have, you know, traces coming in from Signal.

s530
00:47:29.240 --> 00:47:37.640
We have some evaluation score criteria, but there is a certain point at which we actually don't know what is good for our customer beyond what the customer knows themselves.

s531
00:47:37.640 --> 00:47:53.160
Um, so, I mean, you can turn on Signal and it can propose the PR, and we can use some proxy metrics like merge rate on the PR, or the session transcript or whether session was handed off or whether data set was created or eval was created as feedback mechanisms,

s532
00:47:53.160 --> 00:47:59.240
but beyond that and beyond that and the merge rate of the PR, we can't really tell that much more.

s533
00:47:59.240 --> 00:48:04.080
So, it becomes, you know, diminishing returns to continue to evaluate from there.

s534
00:48:04.080 --> 00:48:08.840
Um cuz then you're just kind of relying on an agent transcript and you have no ground truth, right?

s535
00:48:08.840 --> 00:48:14.360
You're just vibing with the agent's output and pretending you're increasing shareholder value, right?

s536
00:48:14.360 --> 00:48:16.800
So, um I don't think that's useful.

s537
00:48:16.800 --> 00:48:22.040
Uh but I think we have, to answer your question, we have used signal to evaluate signal as well.

s538
00:48:22.040 --> 00:48:31.800
I think in execution time checks, because such low latencies like probably the most exciting, like where you can actually route a request during execution time.

s539
00:48:31.800 --> 00:48:39.120
So, these these evals that you're seeing are are, you know, post talk where the the agent has already run and we're evaluating it from that.

s540
00:48:39.120 --> 00:48:47.240
So, due to the low latency of decision wall models, I think that's something I'm actually genuinely excited for uh in the execution loop.

s541
00:48:47.240 --> 00:48:58.720
Um Like selfishly, or maybe not even selfishly, but like for my personal enjoyment, I like that people are much more likely to use evals now because they're cheaper.

s542
00:48:58.720 --> 00:49:03.000
Um so, that's like just, you know, purely good for the ecosystem health.

s543
00:49:03.000 --> 00:49:07.200
Uh like more people thinking about evals, more people thinking about actually classifying their traces.

s544
00:49:07.200 --> 00:49:10.360
I think it'll unlock that to a large degree just because it's cheaper.

s545
00:49:10.360 --> 00:49:25.680
Um And then, maybe not related to to this talk, but like just my my personal opinion is I think like um because these models are so small that there's some really interesting stuff you can do with on-device stuff um that I'm really, really excited for.

s546
00:49:25.680 --> 00:49:30.360
Um we're also like kind of pioneering it for some multimodal evals as well.

s547
00:49:30.360 --> 00:49:35.440
Um because I think multimodal evals can be really complicated because of latency.

s548
00:49:35.440 --> 00:49:38.080
Um and they can also be super, super qualitative.

s549
00:49:38.080 --> 00:49:49.240
So, in the multimodal eval space, like trusting an agent's taste on an image that was generated, for example, or how it interpreted a click box can be very hard, but

s550
00:49:49.240 --> 00:49:56.440
getting confidence scores allows you to at least route this to human a lot faster, so I think that's also a space where decision models will help a lot.

s551
00:49:56.440 --> 00:49:57.680
Yes, 100%.

s552
00:49:57.680 --> 00:50:04.680
So, I kind of briefly alluded to this when I talked about skills, but you can actually create your own agents.

s553
00:50:04.680 --> 00:50:10.760
Um And we have the out-of-the-box signal agent, obviously.

s554
00:50:10.760 --> 00:50:15.840
Um Let me do this one.

s555
00:50:18.520 --> 00:50:24.960
So, you'll see here we have a DataDog skill as well where you can pull up metrics, logs, and APM traces around the alert.

s556
00:50:24.960 --> 00:50:29.040
Um we have a bunch of providers available for you to do that as well.

s557
00:50:29.040 --> 00:50:39.400
You can also write your own skills, so the the agent is doesn't just have access to the agent traces, but downstream system application metrics as well.

s558
00:50:39.400 --> 00:50:42.440
Um so, you can connect it to basically any data source.

s559
00:50:42.440 --> 00:50:51.680
Um Obviously, there's like an element of taste to the data source like connect the ones that you need to connect, make sure it has the right access, make sure, you know, the scope key,

s560
00:50:51.680 --> 00:50:57.440
make sure that you're not over flooding context because more isn't always better to actually triage the alert.

s561
00:50:57.440 --> 00:50:59.840
Um but yeah, we have the ability to do all that as well.

s562
00:50:59.840 --> 00:51:03.720
The the shitty answer that everyone always gives is it depends, but it it does kind of depend.

s563
00:51:03.720 --> 00:51:06.560
So, for example, like what harness are you using?

s564
00:51:06.560 --> 00:51:08.960
Um what underlying providers are you using?

s565
00:51:08.960 --> 00:51:10.200
How often are you running this?

s566
00:51:10.200 --> 00:51:12.920
And we expose those as configurations.

s567
00:51:12.920 --> 00:51:18.680
We allow you to kind of monitor the cost, too, so you'll see here that this is the signal traces.

s568
00:51:18.680 --> 00:51:28.240
This is how much it's Oh, I guess this is still flowing in right now, but the same way that you can view kind of your cost for like Wonder Toys, you can view the cost of signal.

s569
00:51:28.240 --> 00:51:38.320
Um so, we make it easy for you to understand the ROI like, okay, if this shopping assistant agent cost me $8, and it cost me 20 bucks to improve

s570
00:51:38.320 --> 00:51:41.320
with the self-improvement loop, is that really worth doing or not?

s571
00:51:41.320 --> 00:51:50.640
Um Now, like I don't have like a hard number as like 5% of your agent budget should be allocated to the self-improvement loop or something like that.

s572
00:51:50.640 --> 00:51:52.440
I think it really just depends.

s573
00:51:52.440 --> 00:52:05.240
Um but I will say like that is exposed to you and available to you and there many ways in which you can kind of pull the levers to make that a little bit cheaper or more expensive depending on how you're seeing results.

s574
00:52:05.240 --> 00:52:08.600
Um so one is the amount of context you're providing, right?

s575
00:52:08.600 --> 00:52:16.800
So which skills are you including up to a certain point without like just spending more money to give it more information that doesn't actually help.

s576
00:52:16.800 --> 00:52:28.360
Um and then the other is like the eval kind of design question, how often you're running these evals, sampling rate, etc. Um And then the other thing is like you I mean you can use this

s577
00:52:28.360 --> 00:52:31.640
part of the platform, but all these are consumable via API as well.

s578
00:52:31.640 --> 00:52:40.760
So if you want to shunt this to coding agents, which you get a subsidized cost for or something like that, you can just export that to a coding agent and have it do many many of the similar things.

s579
00:52:40.760 --> 00:52:42.040
Thank you for the wonderful questions.

s580
00:52:42.040 --> 00:52:44.520
I'll be around over the next 3 days.

s581
00:52:44.520 --> 00:52:46.520
We have the Rise booth as well.

s582
00:52:46.520 --> 00:52:48.240
Um feel free to sign up.

s583
00:52:48.240 --> 00:52:49.880
Try this loop out for yourself.

s584
00:52:49.880 --> 00:52:51.320
Completely free to sign up.

s585
00:52:51.320 --> 00:52:52.920
Completely free to use.

s586
00:52:52.920 --> 00:52:56.520
Um and yeah, happy debugging.

s587
00:52:57.441 --> 00:52:59.441
[applause]