WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=jHMiYtjoJfA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/jHMiYtjoJfA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.480 --> 00:00:15.480
Uh thank you all for coming, first of all.

s3
00:00:15.480 --> 00:00:20.560
And um I want to talk today about uh raising the floor.

s4
00:00:20.560 --> 00:00:23.120
So, it's this kind of a term we use a lot.

s5
00:00:23.120 --> 00:00:35.600
Um mainly I want to talk about very, very practical, like what do I actually see, what do we actually see working in the real world, um how do people how are people making their agents better?

s6
00:00:35.600 --> 00:00:39.400
So, the first thing I want to say is like um I could just I could just say a bunch of stuff.

s7
00:00:39.400 --> 00:00:43.760
Like I think, you know, um the title of this track is like continual learning.

s8
00:00:43.760 --> 00:00:49.160
I think it's like notable that in the real world, there's really not that much continual learning, right?

s9
00:00:49.160 --> 00:00:55.480
Uh if you look at like the labs, if you look at like products that are in the real world, you really don't see a lot of continual learning.

s10
00:00:55.480 --> 00:01:02.960
So, um I think it's very easy to like I could, you know, spend 20 minutes just talking about like, "Hey, here's a bunch of frameworks.

s11
00:01:02.960 --> 00:01:04.559
Here's a bunch of like really nice terms."

s12
00:01:04.559 --> 00:01:07.920
But what I'd rather do is actually turn this a little bit into a dialogue.

s13
00:01:07.920 --> 00:01:19.480
Um this is not just because I procrastinated making a bunch of slides and because Fable was delayed and I was counting on that to make the slides, but also because like the reality is that um there aren't really good standards for these things, right?

s14
00:01:19.480 --> 00:01:22.960
Like there there's not some one-size-fits-all uh solution.

s15
00:01:22.960 --> 00:01:31.760
And so, I'm going to I do have slides, believe it or not, but what we're also going to do is sort of like I'd like to hear from you guys, people actually building agents.

s16
00:01:31.760 --> 00:01:37.480
Like where where have the, you know, Twitter, you know, eval discourse, where has that failed you, right?

s17
00:01:37.480 --> 00:01:39.160
Where where is it not working?

s18
00:01:39.160 --> 00:01:40.520
What are the things you're actually hitting in real life?

s19
00:01:40.520 --> 00:01:42.400
I'd like to talk about that.

s20
00:01:42.400 --> 00:01:49.040
So, please like right now start thinking about your questions, start thinking about the annoying parts of your flow um when you're building agents.

s21
00:01:49.040 --> 00:01:52.160
And I'd like to keep a lot of time for Q&A.

s22
00:01:52.160 --> 00:01:57.040
Um so, the reality is like a year ago, agents barely existed.

s23
00:01:57.040 --> 00:02:05.800
Like I remember being at like a speaker dinner here like a year ago and we're like, yeah, do you think like agents will like, you know, what you know, will they like keep getting better?

s24
00:02:05.800 --> 00:02:06.800
Will they not?

s25
00:02:06.800 --> 00:02:14.760
Um, and I think the crazy thing is like, you know, we're at we were at this point in time a year ago where everything was like a chatbot mostly, right?

s26
00:02:14.760 --> 00:02:19.120
Um, I think people here, people in this room probably, you know, were a little further ahead.

s27
00:02:19.120 --> 00:02:22.440
Um, it was it was a lot simpler then.

s28
00:02:22.440 --> 00:02:29.920
Uh, if you remember Evals, the Eval discourse for like a year or two ago, it would be like, oh, what is the, you know, capital of the United States?

s29
00:02:29.920 --> 00:02:35.760
And you're like, yeah, and you want to make sure that it returns Washington, D.C. and um, that was an easier time, right?

s30
00:02:35.760 --> 00:02:47.880
It was like chatbots were so much more limited in their sort of flexibility that it was kind of easy to like uh, to do a bunch of like, you know, fact-checking

s31
00:02:47.880 --> 00:02:48.120
things.

s32
00:02:48.120 --> 00:02:53.600
Like you knew the answer to most questions your users would ask almost is is another way of saying that.

s33
00:02:53.600 --> 00:02:56.959
At least like a like 80% of them or 90% of them.

s34
00:02:56.959 --> 00:03:01.400
Um, but yeah, we have agents being deployed in finance, healthcare, defense.

s35
00:03:01.400 --> 00:03:03.920
And um, I was actually really against the word.

s36
00:03:03.920 --> 00:03:07.760
One of my like worst takes is like uh, early agents.

s37
00:03:07.760 --> 00:03:09.160
I was like, ah, I hate the word agent.

s38
00:03:09.160 --> 00:03:10.800
Um, I think there were a lot of people that felt the same.

s39
00:03:10.800 --> 00:03:12.840
It was like, come on, it's an LLM, it's a whatever.

s40
00:03:12.840 --> 00:03:18.600
But I I actually think it's it's valuable because I think we're seeing that agents are this like almost self-aware entity, right?

s41
00:03:18.600 --> 00:03:23.280
They kind of like run around their environment, they have these tools they're using.

s42
00:03:23.280 --> 00:03:27.239
Um, when they hit, you know, roadblocks, they start getting really creative, right?

s43
00:03:27.239 --> 00:03:40.440
And that's what makes agents really powerful, but like that's also what makes them like catastrophic oh, well, I'll just, you know, I'll just like uh, decompile this and I'll just, you know, like do this thing that you had no idea that you could have never imagined.

s44
00:03:40.440 --> 00:03:42.440
Um, sometimes those solutions are are helpful, right?

s45
00:03:42.440 --> 00:03:44.400
Sometimes they're they're actually pretty harmful.

s46
00:03:44.400 --> 00:03:49.800
Um, but yeah, what's very certain is we've come a very long way from like next token prediction.

s47
00:03:49.800 --> 00:03:55.040
Like if you think about chatbots, it was literally just like, oh, it was like you know it's going to what is the next likely word?

s48
00:03:55.040 --> 00:03:56.880
It was very easy to reason about.

s49
00:03:56.880 --> 00:04:09.280
Um and I think the thing that you know on the evaluation front, I think the reality is like most of the things you'd read uh online about evals um are really still like stuck in this chatbot era.

s50
00:04:09.280 --> 00:04:15.840
It's very like well like come up with your like 1,000, you know, eval data set and it's like the reality is like nobody's doing that.

s51
00:04:15.840 --> 00:04:18.560
Um very few people anyway, sorry if you are.

s52
00:04:18.560 --> 00:04:31.640
Um and uh you know, I think what teams have seen over and over again is like yeah, you can do that uh but those evals like break as soon as you have a new model, as soon as you like switch harnesses, like you have a bunch of like tools that you're like oh yeah, I'm going to make

s53
00:04:31.640 --> 00:04:42.960
sure that I'm going to write an eval where it has to like call this tool if I ask it this question and it's like oh then you switch to like Claude code CLI and now 80% of your evals suck and it's like okay, you can keep doing that but the reality is like the one thing I could

s54
00:04:42.960 --> 00:04:45.360
promise you is that things are going to keep changing.

s55
00:04:45.360 --> 00:04:53.960
Like we're not done and so I'd be very careful about you know, uh investing you know, months in some sort of eval set that's going to you know, slow you down, right?

s56
00:04:53.960 --> 00:05:02.760
I think the whole thing here is like you want more safety but you you don't want theater and I think that again, I think the evals as has been sort of

s57
00:05:02.760 --> 00:05:06.080
uh prescribed by uh what I call like big eval.

s58
00:05:06.080 --> 00:05:18.440
Um I I think that there's this reality where it's kind of like oh you should really should eval but then like do you actually delay, you know, uh including a new you know, upgrading to the new model on your product in your products?

s59
00:05:18.440 --> 00:05:22.000
Do you actually delay it 2 weeks to update your evals or not, right?

s60
00:05:22.000 --> 00:05:29.160
I I think most people would say no. Um so uh I'm the CTO and co-founder of this company called Raindrop.

s61
00:05:29.160 --> 00:05:40.600
Um very quickly is like we find critical issues in production agents, we verify those fixes actually work without unexpected side effects, and we also simulate changes before they land in production

s62
00:05:40.600 --> 00:05:42.760
um based on past behavior.

s63
00:05:42.760 --> 00:05:48.560
Uh we're used by the best AI companies in the world and Fortune 100s, a lot of logos I'm not allowed to put on here yet.

s64
00:05:48.560 --> 00:05:59.920
Um but companies like Vercel, Speak, Framer, um and uh I think what it means is we get this like amazing peak into like again, what is actually working in the real world.

s65
00:05:59.920 --> 00:06:06.400
I think one of our like tenants as a company is that things are changing constantly and we have to change what we're doing constantly.

s66
00:06:06.400 --> 00:06:14.480
And so I think uh if we we try to be very very honest with our both ourselves and our customers like what works and what does not work.

s67
00:06:14.480 --> 00:06:16.680
We try not to sell things that don't work.

s68
00:06:16.680 --> 00:06:19.080
Um just again, two two things.

s69
00:06:19.080 --> 00:06:22.760
We have this like uh open-source tool that like thousands of people use, maybe you use it yourself.

s70
00:06:22.760 --> 00:06:23.800
It's called Workshop.

s71
00:06:23.800 --> 00:06:24.919
Um so that's made by us.

s72
00:06:24.919 --> 00:06:26.800
It's like an open-source tracing tool.

s73
00:06:26.800 --> 00:06:30.520
It's really really cool if you're trying to experiment with like self-healing loops.

s74
00:06:30.520 --> 00:06:36.960
I think it's the best way to do that because if there's anything it can't do, uh your agent can just like add it, which is pretty cool.

s75
00:06:36.960 --> 00:06:38.960
Um and so highly highly recommend it.

s76
00:06:38.960 --> 00:06:43.880
Like again, I know thousands of people use it like people I anyway, I bump into people all the time that use it.

s77
00:06:43.880 --> 00:06:46.560
Raindrop is like our kind of hosted offering that does issue detection.

s78
00:06:46.560 --> 00:06:50.280
Think of it like Sentry, but detects issues for agents.

s79
00:06:50.280 --> 00:06:51.560
Sorry.

s80
00:06:51.560 --> 00:06:54.160
Um and we also make howtoeval.com.

s81
00:06:54.160 --> 00:06:57.280
And so I think it's one of the most popular resources on how to evaluate AI agents.

s82
00:06:57.280 --> 00:06:59.480
It is again, the link is literally in the name.

s83
00:06:59.480 --> 00:07:00.880
It's how to eval.com.

s84
00:07:00.880 --> 00:07:07.320
Um and I it's it's uh uh our attempt at a very very no-bullshit guide at at what actually works.

s85
00:07:07.320 --> 00:07:10.000
And I'll be talking a little bit about it today.

s86
00:07:10.000 --> 00:07:18.520
Um I think the like root question that we're trying to figure out today together is how do you make your your agent better, right?

s87
00:07:18.520 --> 00:07:21.400
It's not even like what issues does your agent have?

s88
00:07:21.400 --> 00:07:29.000
Uh it is actually how to make your agent better because your agent will have issues that potentially you can't solve or not exactly worth solving, right?

s89
00:07:29.000 --> 00:07:40.720
Like um I think that we saw this uh over and over again where it's like um you know, you can imagine that um th- there are some things that you're like better off waiting for like, you know, we know Fable exists now.

s90
00:07:40.720 --> 00:07:43.720
Maybe Fable, you know, like should you train your own like Fable level model?

s91
00:07:43.720 --> 00:07:44.840
It's like probably not, right?

s92
00:07:44.840 --> 00:07:48.760
Um, and there'll be benefits uh when you can just incorporate that into your product.

s93
00:07:48.760 --> 00:07:55.840
Um, and so there's this actual balance, like how do how do I actually make my agent better uh with the tools that I have?

s94
00:07:56.160 --> 00:08:04.360
When the way that we start thinking about it with customers is something like this, which is like, are you a benchmark maxer or a floor raiser?

s95
00:08:04.360 --> 00:08:12.640
Um, I think that one of the problems when we talk about evaluating agents is that the terms are really confused.

s96
00:08:12.640 --> 00:08:16.960
Like you hear OpenAI has a new like, you know, uh eval benchmark.

s97
00:08:16.960 --> 00:08:19.840
And um, you know, they have evals, they run evals.

s98
00:08:19.840 --> 00:08:21.760
And then you hear like, oh, well, companies have evals.

s99
00:08:21.760 --> 00:08:22.600
There's like online evals.

s100
00:08:22.600 --> 00:08:27.440
And like, it it like the word eval is like uh more or less a meaningless word.

s101
00:08:27.440 --> 00:08:29.960
It literally is just like you're evaluating something, right?

s102
00:08:29.960 --> 00:08:31.480
It's like a test in some cases.

s103
00:08:31.480 --> 00:08:32.840
It's a So, it's a little confusing.

s104
00:08:32.840 --> 00:08:46.480
Um, I think it's helpful like I think what it means is that like uh companies start borrowing like the language that like labs are using and like even copying similar benchmarks, but they're doing completely different things, right?

s105
00:08:46.480 --> 00:08:48.880
Like they have completely different tools at their disposal.

s106
00:08:48.880 --> 00:08:56.920
Like what companies are, you know, like the companies that are downstream of models, uh they just have very, very different responsibilities than labs.

s107
00:08:56.920 --> 00:08:59.600
Um, like labs are trying to make these super general purpose things.

s108
00:08:59.600 --> 00:09:06.680
When they fail, at least on like an API level, my my say if they get something wrong, it's like it it's just different.

s109
00:09:06.680 --> 00:09:13.240
Um, companies are trying to like imbue all this like company specific uh domain knowledge.

s110
00:09:13.240 --> 00:09:18.680
Like, oh, here's the shape of the data and here's what all this data means and here's like how to access it.

s111
00:09:18.680 --> 00:09:21.440
Um, and so it's very, very different.

s112
00:09:21.440 --> 00:09:26.200
Um, we have this like funny quiz on on the how to eval site.

s113
00:09:26.200 --> 00:09:28.120
And um, it's interesting, right?

s114
00:09:28.120 --> 00:09:36.440
Like it's kind of like one one of the questions we we think about is like, oh, are your engineers like or sorry, are your users like domain experts in the thing they're doing?

s115
00:09:36.440 --> 00:09:40.800
It's again, is it almost like replacing someone or is it augmenting them?

s116
00:09:40.800 --> 00:09:47.080
Because if you think about like co-pilot, you know, auto-complete style or cursor what I you know, tab complete now.

s117
00:09:47.080 --> 00:09:49.800
Uh if it gets something wrong like you can just delete it, right?

s118
00:09:49.800 --> 00:09:55.640
Even like Claude code CLI or like codex if you're an engineer, it's like it does do things wrong all the time.

s119
00:09:55.640 --> 00:09:59.680
Um but then when you think about products like Devin, it's actually gets more interesting, right?

s120
00:09:59.680 --> 00:10:06.360
Like the if if something messes up on on the Claude code side, it could be that like you don't have something installed correctly on your computer.

s121
00:10:06.360 --> 00:10:08.040
There's a lot more like user error.

s122
00:10:08.040 --> 00:10:12.040
There's a lot you leave a lot more up to the the users to get correctly.

s123
00:10:12.040 --> 00:10:23.440
Um and I think when you start thinking about things like AI doctors, for example, it's like uh is a very very different shape of responsibility as far as like how much responsibility a user has in actually getting things correctly.

s124
00:10:23.440 --> 00:10:25.160
Um so, anyways, it's kind of funny.

s125
00:10:25.160 --> 00:10:27.680
Um just kind of breaking this down.

s126
00:10:27.680 --> 00:10:35.240
So, we think about the ceiling as like what is the best thing like craziest capability, emergent capability that your your product or agent is capable of.

s127
00:10:35.240 --> 00:10:37.840
Like things that people would just not expect that it could do.

s128
00:10:37.840 --> 00:10:40.680
And then the floor is like what is the worst thing your agent can do?

s129
00:10:40.680 --> 00:10:51.120
Like recommend a competitor or like delete a bunch of data or like accidentally send a you know, AI slop email to a customer because it like technically had access to like your email or something.

s130
00:10:51.120 --> 00:11:01.200
Um and again, I think that like the floor is very interesting because I think that that is the thing that like breaks user trust.

s131
00:11:01.200 --> 00:11:15.320
This is the thing that like the reason why people uh like if you think about the worst things that could start happening in society, whether that's and and things we've already seen um whether that's the uh you know, like 400 kind of psychophancy

s132
00:11:15.320 --> 00:11:21.600
um or uh things in that vein, a lot of it is more on like the floor side rather than the capability side.

s133
00:11:21.600 --> 00:11:24.720
Um and anyways, I could talk about that for a long time.

s134
00:11:24.720 --> 00:11:27.960
Um I'll kind of skip this.

s135
00:11:27.960 --> 00:11:33.240
So, the talk obviously is going to be about floor raising.

s136
00:11:33.240 --> 00:11:36.960
And um the first thing that we're going to talk about is like offline evals.

s137
00:11:36.960 --> 00:11:38.720
Again, we keep it very simple.

s138
00:11:38.720 --> 00:11:44.320
Um I think that like we said before, things have changed a lot sort of since this chatbot era.

s139
00:11:44.320 --> 00:11:52.400
The sort of like, "Oh, you just uh you know, like look at, you know, string contains, you know, on the uh uh on like the text output or something."

s140
00:11:52.400 --> 00:11:57.480
Or even like the style of eval tools that have like a prompt playground, like this sort of thing.

s141
00:11:57.480 --> 00:12:03.080
Like I actually don't know many companies that use some sort of like managed prompt like in the cloud anymore.

s142
00:12:03.080 --> 00:12:05.600
There's like one or two I can think of.

s143
00:12:05.600 --> 00:12:10.680
Um and the reality is just like the prompt is actually like the whole thing now.

s144
00:12:10.680 --> 00:12:11.640
It's like all the code.

s145
00:12:11.640 --> 00:12:12.720
It's all It's your whole harness.

s146
00:12:12.720 --> 00:12:17.520
It's like everything you're connect like It's not just like some string where you tell your the agent what to do.

s147
00:12:17.520 --> 00:12:22.920
Um and so what I think that means is the evals themselves actually should look a lot more like code.

s148
00:12:22.920 --> 00:12:25.120
In other words, like a lot more like tests.

s149
00:12:25.120 --> 00:12:27.760
Whether that's unit tests, whether that's end-to-end tests.

s150
00:12:27.760 --> 00:12:29.720
Um they should look a lot more like tests.

s151
00:12:29.720 --> 00:12:35.280
Um uh Century has this uh a really cool package called like Vitest evals.

s152
00:12:35.280 --> 00:12:39.400
It's literally just like Vitest with like some syntactic sugar on top.

s153
00:12:39.400 --> 00:12:43.400
Um OpenAI calls this like macro evals.

s154
00:12:43.400 --> 00:12:52.200
And um again, I don't think it really matters what you call it, but like essentially run tests on your agent uh locally uh is is is the advice.

s155
00:12:52.200 --> 00:12:54.240
And keep these evals as code.

s156
00:12:54.240 --> 00:13:07.880
Um and again, as much as possible, like the I I don't see a lot of companies using the sort of like prompt playground stuff anymore because of how the shape of agents has really changed.

s157
00:13:08.320 --> 00:13:12.640
Uh When when we think about raising the floor, we think about really like three things.

s158
00:13:12.640 --> 00:13:19.000
One is that like discovering all these like unknown issues that you have in your app, like things you're just like not seeing.

s159
00:13:19.000 --> 00:13:20.680
That's one.

s160
00:13:20.680 --> 00:13:23.600
Two is that for each issue, you really need to know two things.

s161
00:13:23.600 --> 00:13:27.520
You need to know when it actually started, and you need to know how many people it affects.

s162
00:13:27.520 --> 00:13:36.560
It sounds like obvious, but I promise you that like in the day-to-day of actually like having an agent, uh you're going to get like, you know, you already get thousands of people like, "Oh, I saw this weird thing.

s163
00:13:36.560 --> 00:13:37.120
I saw this weird thing."

s164
00:13:37.120 --> 00:13:39.400
So, again, the first thing is like, "Is this new?"

s165
00:13:39.400 --> 00:13:43.080
Cuz if it's not new, like I probably like I'm going to care about it less.

s166
00:13:43.080 --> 00:13:46.400
If I if if if I tell you like, "Hey, look, this issue started yesterday."

s167
00:13:46.400 --> 00:13:51.640
or this issue started like three or four days ago, suddenly like your mind starts turning and you're like, "Oh, what did I do?

s168
00:13:51.640 --> 00:13:53.120
Like, what what what what changed, right?

s169
00:13:53.120 --> 00:13:53.880
Did we change model?

s170
00:13:53.880 --> 00:13:58.120
Did we change you know, so something else um downstream?"

s171
00:13:58.120 --> 00:13:59.760
And again, the second one is like percent of users.

s172
00:13:59.760 --> 00:14:06.440
Like, if I'm uh knowing that it happened to three users versus 100,000 users just is uh critical.

s173
00:14:06.440 --> 00:14:09.560
Because again, I think agents will have an infinite number of problems.

s174
00:14:09.560 --> 00:14:12.120
That's sort of like the the great and terrible thing about them.

s175
00:14:12.120 --> 00:14:16.560
It's like by they're like these little stochastic, you know, crazy things exploring everywhere.

s176
00:14:16.560 --> 00:14:21.920
And so, you just uh in order to even start making things better, you you really need to need to know these two things.

s177
00:14:21.920 --> 00:14:24.520
When it started and percent of users.

s178
00:14:24.520 --> 00:14:32.480
Um and I think also another question we get a lot is like around like, "Oh, like I, you know, what should I be doing?"

s179
00:14:32.480 --> 00:14:37.520
And like, the first question I always ask people is like, "How many users do you have?"

s180
00:14:37.520 --> 00:14:42.720
Like, we have customers with millions of users and we have customers with like five.

s181
00:14:42.720 --> 00:14:54.560
And the real And like, to be clear, like uh customers with five users like especially if let's say it's like an internal um app in an enterprise context where it's like, you know, giving like very critical information.

s182
00:14:54.560 --> 00:15:03.560
Like, it could be very very important to get well uh or sorry, to get correctly, but it does mean you just like should be have to have a radically different uh approach.

s183
00:15:03.560 --> 00:15:12.320
Like, so for example, on the like, you know, uh let's say like 10, 20, 100 million, you know, messages a day side of things, like experiments become extremely valuable.

s184
00:15:12.320 --> 00:15:17.720
Uh if you have a free tier, you can like uh run experiments on a very small sample of your free tier.

s185
00:15:17.720 --> 00:15:19.960
Um and that can just be extremely extremely useful.

s186
00:15:19.960 --> 00:15:31.400
Um obviously, if you have five or 10 users, like uh, I would not recommend, you know, experiments or AB tests, etc. Um, so, uh, this is one of those things that, like, uh, uh, really, really depends on the person.

s187
00:15:31.400 --> 00:15:44.720
Um, what I want to talk about now, before we get into Q&A, are like three very, very, very, very tactical lessons on the sort of like issue, uh, discovery and analysis side.

s188
00:15:44.720 --> 00:15:54.080
These are like three things that I I've never heard anyone talk about, like, three things that we've sort of just discovered from first principles as we do stuff at Raindrop.

s189
00:15:54.080 --> 00:15:57.040
So, any competitors in the audience, please pay attention.

s190
00:15:57.040 --> 00:15:58.931
This is very important.

s191
00:15:58.931 --> 00:15:59.040
[snorts]

s192
00:15:59.040 --> 00:16:04.160
Um, the first one is that clusters are not issues.

s193
00:16:04.160 --> 00:16:15.320
So, um, the sort of like naive approach that we've seen either customers or also, sometimes competitors, uh, take is like, well, you just take all the traces and you just cluster it, right?

s194
00:16:15.320 --> 00:16:16.800
And you get these like clusters.

s195
00:16:16.800 --> 00:16:25.200
It could be like useful from like an analysis, uh, you know, like, how we do things like error analysis, like, the the the this, you know, finding these clusters of things.

s196
00:16:25.200 --> 00:16:27.520
It could be useful to see like, well, what's going on in your data, right?

s197
00:16:27.520 --> 00:16:30.839
Going from like a bunch of logs to like something.

s198
00:16:30.839 --> 00:16:38.760
Um, the problem is that, like, and and again, I I have here, it's useful for one-off analysis, but it just doesn't really scale well.

s199
00:16:38.760 --> 00:16:47.360
Um, and there's like a very good reason why we also, you know, if you think about like normal telemetry, we we we try to think a lot about like normal telemetry, what are the analogies?

s200
00:16:47.360 --> 00:16:52.600
There's a reason why you don't sort of like take all of your, you know, normal logs and just like start clustering it, right?

s201
00:16:52.600 --> 00:17:01.360
Because when you're building software, you you need to know like when something started, um, you need to know how much it's grown.

s202
00:17:01.360 --> 00:17:03.080
Um, those things really matter.

s203
00:17:03.080 --> 00:17:08.120
So, again, with clusters, it's very, very hard to reliably, uh, track over time.

s204
00:17:08.120 --> 00:17:17.079
Um, like uh, like again, it's like called like temporal clustering and there's like research in this, but like, it's pretty hard, um, to do reliably.

s205
00:17:17.160 --> 00:17:21.000
You also just like don't have control of boundaries.

s206
00:17:21.000 --> 00:17:24.360
Um and this also changes a lot depending on your product.

s207
00:17:24.360 --> 00:17:34.040
Like um what you consider to be like, you know, the same issue or not um is actually very, very unique to every company.

s208
00:17:34.040 --> 00:17:38.160
And so you sort of will get these like kind of weird clusters.

s209
00:17:38.160 --> 00:17:43.960
Like, you know, you can imagine each of these is like, "Oh, wrong price quoted and wrong wrong refund calculated."

s210
00:17:43.960 --> 00:17:51.240
Are like actually like you'll get a cluster like, you know, uh you know, price issues or something.

s211
00:17:51.240 --> 00:17:53.160
And it's like, yeah, sort of.

s212
00:17:53.160 --> 00:17:56.400
But like actually these could have like extremely different root causes, right?

s213
00:17:56.400 --> 00:18:03.920
So so price issues is or like, you know, uh issues calculating is like, you know, as a cluster it's not really that useful.

s214
00:18:03.920 --> 00:18:09.360
Um and it also again doesn't doesn't really tell you the things that, you know, we talked about needing.

s215
00:18:09.360 --> 00:18:13.280
Um So uh yes.

s216
00:18:13.280 --> 00:18:16.880
Uh Last one here is going to be uh code mode actually really scales.

s217
00:18:16.880 --> 00:18:20.080
Like you've heard about code mode in the context of MCPs.

s218
00:18:20.080 --> 00:18:24.960
Um I highly recommend just uh trying to apply this to traces.

s219
00:18:24.960 --> 00:18:31.560
Like you can just write uh these classifiers and you can write them and you can run them in a sandbox and you can run them at production scale.

s220
00:18:31.560 --> 00:18:35.720
Um we have a you know, feature that makes this easier, but like you can do this.

s221
00:18:35.720 --> 00:18:38.240
So I highly recommend it.

s222
00:18:38.240 --> 00:18:41.280
Last lesson here is that agents are very, very bad at anomaly detection.

s223
00:18:41.280 --> 00:18:44.120
So don't ask your agent to find anomalies.

s224
00:18:44.120 --> 00:18:47.120
Uh ask it to investigate anomalies you've already found.

s225
00:18:47.120 --> 00:18:53.680
So uh what I mean is like pull out as many deterministic things as you can like keyword frequency, right?

s226
00:18:53.680 --> 00:19:06.000
So if you see a spike in like a keyword it doesn't necessarily mean that there's an issue, but it does mean that you can uh it's like something more tangible tractable that you can have an agent actually investigate.

s227
00:19:06.000 --> 00:19:11.800
Um And I'm going to skip through the rest because we're tight on time.

s228
00:19:11.800 --> 00:19:16.040
And I lied to you uh which is that we're not going to have enough time for Q&A because I only have a minute left.

s229
00:19:16.040 --> 00:19:20.000
But, what I'd love if you could do is, uh, find me after.

s230
00:19:20.000 --> 00:19:22.920
Um, I'll be around for the next hour and, uh, let's just talk.

s231
00:19:22.920 --> 00:19:26.360
It's probably a better format than standing up here and it'll be hard to hear your questions anyway.

s232
00:19:26.360 --> 00:19:30.120
So, uh, yeah, thank you guys so much.