WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=hfEczxdNyvU

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/hfEczxdNyvU.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:13.320 --> 00:00:15.040
Hey everybody.

s2
00:00:15.040 --> 00:00:16.400
My name is Doug Guthrie.

s3
00:00:16.400 --> 00:00:19.760
I'm a solutions engineer at Braintrust.

s4
00:00:19.760 --> 00:00:27.320
Um I have a few colleagues, uh a couple colleagues in the room that uh can help if you get stuck anywhere uh within the workshop today.

s5
00:00:27.320 --> 00:00:35.760
Also, um I get interrupted all day long during calls, so if you want to stop me at any point and ask a question, I'm up for that as well.

s6
00:00:35.760 --> 00:00:38.600
Uh hopefully you've had a chance to grab the QR code by now.

s7
00:00:38.600 --> 00:00:40.800
It just takes you out to this repo.

s8
00:00:40.800 --> 00:00:51.240
This is what we're going to be us- using today uh as part of the workshop and going through understanding how you can do observability within Braintrust and sort of using all of the different features

s9
00:00:51.240 --> 00:00:53.120
within there to build better agents.

s10
00:00:53.120 --> 00:00:54.240
That's really why we're all here.

s11
00:00:54.240 --> 00:00:57.680
Like, how do we actually create good quality agents?

s12
00:00:57.680 --> 00:01:03.440
Um I'll leave this up for uh 10 more seconds or so.

s13
00:01:21.720 --> 00:01:25.360
I'm going to have this again later in the the presentation when we actually get into the workshop.

s14
00:01:25.360 --> 00:01:35.960
So, if you if you don't get it, um you can also Google um dpguthrie on GitHub and search that re- uh repository on on GitHub and you'll be able to find it.

s15
00:01:36.480 --> 00:01:36.640
Cool.

s16
00:01:36.640 --> 00:01:44.800
So, um we're all here to master AI observability, uh and I'm going to give you what that looks like in the context of using Braintrust.

s17
00:01:44.800 --> 00:01:46.040
Uh quick agenda.

s18
00:01:46.040 --> 00:01:54.720
Uh I do have some slides that I want to sort of set the foundation for doing observability, and I think um what we've learned of how to how to do it really well.

s19
00:01:54.720 --> 00:01:57.600
So, I'll sort of set the foundation for what that looks like.

s20
00:01:57.600 --> 00:02:01.880
Uh give you a sense for how do we generate signal in this massive amount of noise?

s21
00:02:01.880 --> 00:02:09.240
Um you all have probably seen a trace or you can understand that the agents that you're building are generating a massive amount of data.

s22
00:02:09.240 --> 00:02:14.959
And being able to parse through all of that and actually derive some type of signal or insight is challenging.

s23
00:02:14.959 --> 00:02:16.360
That's where the challenge resides.

s24
00:02:16.360 --> 00:02:22.959
It's there's one side of actually being able to trace and then have a sort of platform that allows for observability.

s25
00:02:22.959 --> 00:02:27.200
Then there's the other side of how do we actually do something with it and derive some type of insight.

s26
00:02:27.200 --> 00:02:34.040
So I'll show you what that looks like within Braintrust and then we'll sort of walk through all of what what I what I described or what I've shown on

s27
00:02:34.040 --> 00:02:36.080
on these slides in the workshop.

s28
00:02:36.080 --> 00:02:39.320
Um as I said, uh if you have a question, feel free to jump in.

s29
00:02:39.320 --> 00:02:40.520
I do not mind.

s30
00:02:40.520 --> 00:02:45.959
Uh I have sort of at least I'll try to save some time at the end for some Q&A as well.

s31
00:02:45.959 --> 00:02:49.640
So feel free to save your question to the end.

s32
00:02:50.120 --> 00:02:53.800
So just a quick Braintrust intro if you're not uh familiar.

s33
00:02:53.800 --> 00:02:58.240
Uh we are an AI and we are an Evals and observability uh platform.

s34
00:02:58.240 --> 00:03:02.239
Our sole goal is to help our customers build better agents.

s35
00:03:02.239 --> 00:03:11.120
Uh we we sort of uh help enable our customers understand what quality looks like and then how to improve it in a real measurable way.

s36
00:03:11.120 --> 00:03:18.680
Um one of like the the more fun parts about my job is that I actually get to go speak to some of these customers that you see up there in the top right.

s37
00:03:18.680 --> 00:03:21.560
I get to understand how they are building their agents.

s38
00:03:21.560 --> 00:03:25.040
And a lot of those learnings actually make make their way into our product.

s39
00:03:25.040 --> 00:03:28.840
Make their way into my other engagements with customers.

s40
00:03:28.840 --> 00:03:40.680
So I get to go actually talk to the engineers of Cloudflare and I get to talk to the engineers of Dropbox and I get to understand what they're doing, the the types of scores they're creating, how they think about our new feature topics,

s41
00:03:40.680 --> 00:03:44.440
what that sort of creates, the the types of automations they're building on top of it.

s42
00:03:44.440 --> 00:03:47.440
Uh I hope to give you some sense of that today.

s43
00:03:47.440 --> 00:03:48.800
Uh mentioned some of this, right?

s44
00:03:48.800 --> 00:03:51.400
We are an Evals and observability platform.

s45
00:03:51.400 --> 00:03:58.080
Um you think about like evals and observability, I think they're like two diff two sides of the same coin.

s46
00:03:58.080 --> 00:04:03.440
Uh taken together, it's really what's going to allow you to create better quality agents.

s47
00:04:03.440 --> 00:04:12.800
Uh if you if you hang around and I mean like if you if you aren't a Braintrust customer, if you're not using Braintrust today, would highly recommend coming back and looking at Braintrust in

s48
00:04:12.800 --> 00:04:14.280
the next month, 2 months.

s49
00:04:14.280 --> 00:04:21.600
Uh where I think we see the the industry at large today is is a more like very passive way of observing data.

s50
00:04:21.600 --> 00:04:24.840
Uh sort of what I alluded to of like being able to derive insight.

s51
00:04:24.840 --> 00:04:29.120
Uh Braintrust is moving, I think, a little bit more towards what we call active observability.

s52
00:04:29.120 --> 00:04:43.000
It's It's actually giving you the primitives, uh the foundations, to give you the right insights where you don't actually have to pull from your platform and understand what that means in the context of your agent and and what you need to go do to go and improve it.

s53
00:04:43.000 --> 00:04:51.120
Uh so, uh active observability is the is the way in which we can start to bring those insights to you so that you can build better quality agents.

s54
00:04:51.120 --> 00:04:53.520
So, quick quick Braintrust stuff there.

s55
00:04:53.520 --> 00:04:58.400
Um I'll plug this again at the end, but we are We do have a booth at the expo.

s56
00:04:58.400 --> 00:04:59.680
Uh I think it opens around 5:00.

s57
00:04:59.680 --> 00:05:00.240
Come and find.

s58
00:05:00.240 --> 00:05:01.600
I'll be there tonight.

s59
00:05:01.600 --> 00:05:03.640
Come and find me, ask some more questions.

s60
00:05:03.640 --> 00:05:05.880
Uh we'll be here all week as well.

s61
00:05:05.880 --> 00:05:13.720
So, the the foundation to all of this, uh the foundation to observability in general, like I'm guessing most of you already know here, is tracing.

s62
00:05:13.720 --> 00:05:17.840
You have to be able to understand all of the different steps that your agent takes, right?

s63
00:05:17.840 --> 00:05:21.680
What are the the the things that it's doing to get to its final output.

s64
00:05:21.680 --> 00:05:24.880
And not only not only that, but what are the inputs to those tools?

s65
00:05:24.880 --> 00:05:26.240
What are the outputs of those?

s66
00:05:26.240 --> 00:05:32.800
And if you if you start to think about like all of the different steps that your agent can take, this can balloon um pretty quickly.

s67
00:05:32.800 --> 00:05:37.360
And being able to sort of parse through this, this is the challenge that we are trying to solve.

s68
00:05:37.360 --> 00:05:38.919
But without that tracing, right?

s69
00:05:38.919 --> 00:05:44.919
We're not actually able to understand what's going on, where quality is actually deteriorating.

s70
00:05:44.919 --> 00:05:53.920
Uh and so, being able to uh one, trace all of this data and then derive some type of insight again is where where Branch is going to provide some value, but

s71
00:05:53.920 --> 00:06:04.320
it just in in general, if you're thinking about observability, if you don't have some part of your stack in in a trace that looks like this, then it's it's a black box.

s72
00:06:04.320 --> 00:06:12.760
It's going to be very challenging for you to actually surface those quality issues or, you know, point Codex or cloud code or whatever you're using out there to like understand what's going on.

s73
00:06:12.760 --> 00:06:20.280
If that's that that data is not there, you can't actually surface those those places where the agent is falling down, where it could be improved.

s74
00:06:20.280 --> 00:06:22.440
So, tracing is the foundation.

s75
00:06:22.440 --> 00:06:35.240
I think if you look at the eval side and of course talking today the observability, it all sort of centers around us being able to capture all of the different things that our agent is doing.

s76
00:06:35.640 --> 00:06:38.520
So, just another quick call out on the Branch Trust side.

s77
00:06:38.520 --> 00:06:39.880
We we have a lot of different ways.

s78
00:06:39.880 --> 00:06:42.720
So, I'll skip ahead just a little bit.

s79
00:06:42.720 --> 00:06:47.280
The workshop that we're doing is built on it's a you know, dummy sort of support agent.

s80
00:06:47.280 --> 00:06:53.080
The the sort of agent isn't the thing that's important, but it's built with the Open AI agents SDK.

s81
00:06:53.080 --> 00:06:57.400
We have lots of dinner different integrations with all of the different agent frameworks.

s82
00:06:57.400 --> 00:07:00.800
We have integrations with all of the AI providers.

s83
00:07:00.800 --> 00:07:02.400
We have different programming languages.

s84
00:07:02.400 --> 00:07:10.720
So, depending on how you are building your agent, whether it's the example today is in Python but it could be TypeScript, it could be Java, Ruby, Go.

s85
00:07:10.720 --> 00:07:22.160
So, we are pretty flexible in that we allow you to sort of bring whatever it is you're doing today from a programming language perspective or how you are building your agent, whether it's with a framework, whether you're using Open Telemetry,

s86
00:07:22.160 --> 00:07:28.840
whether you're sort of hand rolling your own orchestration, whatever it is, Branch Trust has a way to hook into that pretty seamlessly.

s87
00:07:28.840 --> 00:07:40.880
And I think if you're looking out across the market, right, if you're trying to understand how do I how do I trace this this agent, you're going to have to have the ability to go across a lot of different things generally.

s88
00:07:40.880 --> 00:07:46.800
And you can see over here there's a lot of different ways in which you can integrate your stack into Braintrust.

s89
00:07:46.800 --> 00:07:55.480
So again, I just have the dummy agent today is using the open AI open AI agents SDK, but that's certainly not what we're confined to.

s90
00:07:57.080 --> 00:08:05.000
This is something that I sort of I say the word flywheel like, I don't know, 100 times a day it seems like.

s91
00:08:05.000 --> 00:08:14.800
Um The whole idea and where I think we've gotten a lot of feedback from our customers is the the way in which that the way in which they are building better agents today

s92
00:08:14.800 --> 00:08:16.960
is they are creating something like this, right?

s93
00:08:16.960 --> 00:08:21.080
And what what I mean by that is that production should inform development.

s94
00:08:21.080 --> 00:08:33.000
You should have a really simple and easy to create flywheel that allows you to develop some type of change, fix something within your agent, test whether or not that change

s95
00:08:33.000 --> 00:08:36.240
regress the agent in some way or improved it.

s96
00:08:36.240 --> 00:08:40.120
You should be able to have the confidence to go and deploy that agent to production.

s97
00:08:40.120 --> 00:08:41.000
When you deploy, right?

s98
00:08:41.000 --> 00:08:42.880
We need to be able to observe it.

s99
00:08:42.880 --> 00:08:51.960
That observation though, we need to like tack something on top of that which maybe you don't see necessarily called out specifically within this flywheel, but you need to be able to derive

s100
00:08:51.960 --> 00:08:54.280
some type of insight at scale.

s101
00:08:54.280 --> 00:09:04.440
Most of the customers, especially on the slide that I showed earlier, they have a massive amount of scale and being able to derive inside or having a human comb through that data would just be incredibly challenging.

s102
00:09:04.440 --> 00:09:15.160
And so we need to layer intelligence on top of that and that's one of the things that that Braintrust will offer, but I think just if you're thinking about the the process of building a better agent in general,

s103
00:09:15.160 --> 00:09:16.640
you need a flywheel like this.

s104
00:09:16.640 --> 00:09:22.880
You need to create something where production in some way informs how you go and change your agent.

s105
00:09:22.880 --> 00:09:31.800
And so this is what I'll show you a little bit today when we go through the workshop is how we can actually surface some of those insights in an automated way from our

s106
00:09:31.800 --> 00:09:38.600
support agent, bring those insights back to our development process, make some type of change, run an eval, right?

s107
00:09:38.600 --> 00:09:43.720
That that sort of loop uh becomes sort of uh it's just incredibly important.

s108
00:09:43.720 --> 00:09:51.480
And and hopefully it becomes a very trivial thing that most companies and most developers can just very easily plug into.

s109
00:09:53.880 --> 00:10:02.400
So, when I say signal, uh I just in this context I mean two different things that we will sort of wire up also during the workshop.

s110
00:10:02.400 --> 00:10:04.440
Uh I mean uh scores.

s111
00:10:04.440 --> 00:10:07.120
So, this could be uh you know, evaluators.

s112
00:10:07.120 --> 00:10:11.040
This is just a way to measure the output of your agent in some way.

s113
00:10:11.040 --> 00:10:20.920
Uh you have some particular failure mode, you have something that you're trying to understand about the agent when it um when it received some input.

s114
00:10:20.920 --> 00:10:24.080
Uh we we can create a lot of different types of scores.

s115
00:10:24.080 --> 00:10:26.640
Uh one of them you see over on the far left is a code-based score.

s116
00:10:26.640 --> 00:10:31.520
These are generally uh very fast, very cheap, uh and very easy to run.

s117
00:10:31.520 --> 00:10:34.000
Uh you can imagine like I'm trying to validate a schema.

s118
00:10:34.000 --> 00:10:38.000
I am trying to understand was tool A called before tool B?

s119
00:10:38.000 --> 00:10:40.000
Was it called in the right order?

s120
00:10:40.000 --> 00:10:46.120
Uh things that can be produced very very quickly uh and without sort of having to go into an LLM.

s121
00:10:46.120 --> 00:10:48.440
Uh the more subjective, though, is that LLM judge.

s122
00:10:48.440 --> 00:10:54.200
We want to bring some sort of like human uh understanding to that that that trace, right?

s123
00:10:54.200 --> 00:10:59.520
to that interaction that just happened with that eval case or even uh online as well.

s124
00:10:59.520 --> 00:11:09.240
And so, you could actually, you know, give the LLM some criteria of how you want to assess the output or assess that conversation, and it should produce some score.

s125
00:11:09.240 --> 00:11:19.240
Um And then there's there's a probably actually a fourth one here, but the third one that you see over on the right is an LLM LLM judge aligned with human review.

s126
00:11:19.240 --> 00:11:31.400
What's also important, you could probably imagine, is if I have an LLM judge and it's producing results and we are sort of making uh decisions based on the those results, we should probably understand whether or not those results are good or not.

s127
00:11:31.400 --> 00:11:33.560
And so, there should be some element of human review.

s128
00:11:33.560 --> 00:11:37.760
I think especially if you are using these judges in an online sense, right?

s129
00:11:37.760 --> 00:11:48.320
You are scoring actual interactions of users with your agent, you should be able to provide some type of interface to users to provide that review to calibrate those those judges, right?

s130
00:11:48.320 --> 00:11:50.640
This isn't a set it and forget it type thing.

s131
00:11:50.640 --> 00:11:53.520
Uh your agent will evolve, so too will your scores.

s132
00:11:53.520 --> 00:12:01.960
The the fourth one that I actually have an example of in the uh the work the workshop is a sort of code-based score, but it invokes an LLM under the hood.

s133
00:12:01.960 --> 00:12:12.320
Uh you can imagine like there is branching logic or there's just uh more advanced or complex things that you need to do uh based on some type of logic that you want to encode

s134
00:12:12.320 --> 00:12:13.600
in your code-based score.

s135
00:12:13.600 --> 00:12:16.560
Uh so, you could do that as well.

s136
00:12:16.560 --> 00:12:21.280
Uh these are some of the things that I think I've heard in in different conversations with with our customers.

s137
00:12:21.280 --> 00:12:24.480
Uh I think again, it's like take it with a grain of salt to some degree.

s138
00:12:24.480 --> 00:12:28.000
Like it's not always going to fit 100% of the scenarios every single time.

s139
00:12:28.000 --> 00:12:31.720
It's It's where I've seen a lot of success across our customer base.

s140
00:12:31.720 --> 00:12:40.680
Um just being able to like understand the reasoning that the LLM gave for producing a score, incredibly helpful especially on that human review side where it's like, "Hey, I can understand

s141
00:12:40.680 --> 00:12:41.680
why it produced that."

s142
00:12:41.680 --> 00:12:46.000
That will then inform how I can go change my scores in some way.

s143
00:12:46.000 --> 00:12:57.839
Um binary scoring, I think uh especially at the outset when we have sort of an agent that we are putting out into production, binary scoring I think is is better than, you know, A B C D or or so on.

s144
00:12:57.839 --> 00:12:59.680
And you should be incredibly harsh.

s145
00:12:59.680 --> 00:13:02.800
Uh you should produce really hard examples.

s146
00:13:02.800 --> 00:13:14.280
You should You should if you get 100% on every single one of your evals and and we probably will today because it's uh sort of dummy use case, but if you get 100% of your uh 100% on your evals,

s147
00:13:14.280 --> 00:13:19.320
yeah, one, you probably don't have uh strong enough test cases in your data set.

s148
00:13:19.320 --> 00:13:21.440
Your scores probably aren't harsh enough.

s149
00:13:21.440 --> 00:13:26.360
Uh so, you should sort of look at that and and and sort of calibrate accordingly.

s150
00:13:27.600 --> 00:13:36.800
Um another one that I've seen, you know, again, grain of salt including some examples in in your prompts tends to tends to help and then perhaps using a stronger model

s151
00:13:36.800 --> 00:13:38.840
for scoring than for generation, right?

s152
00:13:38.840 --> 00:13:43.800
Let's bring a little bit closer to human subjectivity as possible to this.

s153
00:13:43.800 --> 00:13:47.680
Obviously, this can change based on the the the type of traffic, right?

s154
00:13:47.680 --> 00:13:58.680
If we start to talk about online scoring and things like that, but um to that point, there are obviously two different types of scores, at least the way that we see that here at BrainTrust.

s155
00:13:58.680 --> 00:13:59.880
There is the offline, right?

s156
00:13:59.880 --> 00:14:01.360
This is our with our e-vals.

s157
00:14:01.360 --> 00:14:06.080
We are sort of trying to derive some signal of whether or not we should push this thing to production.

s158
00:14:06.080 --> 00:14:10.480
This is going to give us indication of did it improve or did it regress?

s159
00:14:10.480 --> 00:14:16.920
Um this is going to be uh you can imagine wiring this up to Excuse me.

s160
00:14:16.920 --> 00:14:19.120
It's my daughter.

s161
00:14:25.600 --> 00:14:28.840
Um yeah, so you can imagine like wiring this up, right?

s162
00:14:28.840 --> 00:14:29.680
To CI.

s163
00:14:29.680 --> 00:14:33.280
You can imagine wiring it up to some sort of automated pipeline.

s164
00:14:33.280 --> 00:14:43.080
Again, where you're running it offline and then on the flip side, the the whole idea or like the thing that we're talking about here is how do we how do we derive signal from things that's actually happening in production?

s165
00:14:43.080 --> 00:14:50.880
And one of those things that you can do is use those online scores and and score that traffic as it's incoming into BrainTrust.

s166
00:14:50.880 --> 00:15:01.839
And again, the idea is to attach some type of signal where then you as a user can actually go in and be like, "Hey, my routing accuracy score is zero in these examples.

s167
00:15:01.839 --> 00:15:05.800
Is there some sort of pattern that I should be aware of based on this feedback, right?

s168
00:15:05.800 --> 00:15:12.360
So I need to go from like this massive amount of data into something that is much smaller and easier for me to consume."

s169
00:15:12.720 --> 00:15:17.960
The second signal is a is a feature that we have in BrainTrust called topics.

s170
00:15:17.960 --> 00:15:21.560
The idea here is that you want to like understand patterns within your data.

s171
00:15:21.560 --> 00:15:26.320
You want to go a little bit further beyond the uh the known unknowns, right?

s172
00:15:26.320 --> 00:15:32.760
So, like your scores that you've already created uh in your offline evals, they are failure modes that perhaps you've already recognized.

s173
00:15:32.760 --> 00:15:41.000
There are things that you already are testing for, but that's certainly not the the scope in which your agent can operate or the failure modes that it can encounter.

s174
00:15:41.000 --> 00:15:46.360
And so, we need to understand beyond that, and Topics is one of those things that will help enable that.

s175
00:15:46.360 --> 00:15:50.200
We should be able to um understand silent failures.

s176
00:15:50.200 --> 00:15:55.000
So, you can see there like these very sort of traditional scores like factuality and and moderation.

s177
00:15:55.000 --> 00:16:03.520
They're They're going to catch a very small subset of errors, and and probably you have a little bit more uh bespoke scores that that you're using.

s178
00:16:03.520 --> 00:16:13.120
And then to that point, like the evals that you're running again are using those those subset of scores to understand improvement or regression, but there is a sort of wide swath of

s179
00:16:13.120 --> 00:16:24.680
use cases that your test scenarios are not capturing, that your users are actually uh probing your agent with and trying to like get some type of output, and perhaps maybe it's not the right one.

s180
00:16:24.680 --> 00:16:26.480
Um so, we want to capture these edge cases.

s181
00:16:26.480 --> 00:16:30.760
We want to capture the tail end, and we want to structure uh then the flywheel, right?

s182
00:16:30.760 --> 00:16:33.040
We want to bring these examples back to our data sets.

s183
00:16:33.040 --> 00:16:33.920
We want to run evals.

s184
00:16:33.920 --> 00:16:37.880
We want to perhaps create new scores based on that feedback.

s185
00:16:37.880 --> 00:16:39.640
So, so what is what is Topics?

s186
00:16:39.640 --> 00:16:46.440
Uh Topics uh in general or at a high level is a way for us to understand patterns within your data.

s187
00:16:46.440 --> 00:16:52.960
Um we can create clusters of those patterns and give you a sense of frequency with which those things are occurring.

s188
00:16:52.960 --> 00:16:57.280
Uh also important to call out here is that there are both built-in and custom facets.

s189
00:16:57.280 --> 00:17:05.360
And And what what I mean by that is when you go into your BrainTrust account, you can enable a feature called Topics, and out of the box you can see task,

s190
00:17:05.360 --> 00:17:07.000
issues, and sentiment.

s191
00:17:07.000 --> 00:17:10.600
So, I can understand, "Hey, what are What is the common user intent?

s192
00:17:10.600 --> 00:17:13.079
How are users feeling when they interact with my agent?"

s193
00:17:13.079 --> 00:17:14.640
And what are the common agent issues?

s194
00:17:14.640 --> 00:17:20.400
So, I can and that that sort of taxonomy, where I'm not necessarily providing that taxonomy up uh up front.

s195
00:17:20.400 --> 00:17:24.680
The LLM uh creates that uh for me as part of this pipeline.

s196
00:17:24.680 --> 00:17:26.520
But, I can also bring my own custom facets.

s197
00:17:26.520 --> 00:17:33.960
I can I can sort of build my own prompt, and I can build my own pipeline to create these patterns within my data, right?

s198
00:17:33.960 --> 00:17:36.160
With respect to the things that I care about.

s199
00:17:36.160 --> 00:17:38.920
Uh go go back to this.

s200
00:17:38.920 --> 00:17:45.720
Um this is you know, just an example of of of an illustration that you might see with the topics that were generated.

s201
00:17:45.720 --> 00:17:46.960
Uh but, why why would you use it?

s202
00:17:46.960 --> 00:17:51.520
I think I sort of alluded to this a little bit, but you want to be able to not have blind spots.

s203
00:17:51.520 --> 00:18:01.200
Uh you want to be able to understand all of the different ways in which your users are interacting with your agents, and then allow that to inform what you go do as a developer, right?

s204
00:18:01.200 --> 00:18:04.680
This this doesn't even necessarily have to be an issue or failure mode.

s205
00:18:04.680 --> 00:18:12.520
This could be like potentially um feature requests, new things that your agent could do based on the way users are actually interacting with your agent.

s206
00:18:12.520 --> 00:18:23.720
Um so, this is like I think one of the ways in which I've seen a lot of our users start to use this uh is actually create this or or sort of understand what's going on and inform what a road map should look like for the agent, right?

s207
00:18:23.720 --> 00:18:31.000
It It starts to uncover things that were not necessarily aware of with respect to these agents running in production.

s208
00:18:31.800 --> 00:18:35.400
Behind the scenes it is a really complex pipeline.

s209
00:18:35.400 --> 00:18:38.640
Um you see over there on the left a preprocessor.

s210
00:18:38.640 --> 00:18:46.720
So, if you think back to my my first slide with the that trace, and that's like a very very small trace uh relative to what I've seen our customers generate.

s211
00:18:46.720 --> 00:18:52.400
What we need to do is preprocess that data into something that is meaningful to give to the LLM.

s212
00:18:52.400 --> 00:18:56.800
Um so, we give you a what we call a default preprocessor.

s213
00:18:56.800 --> 00:19:02.840
We essentially take what what you'll see in our thread view, and you'll see that in just a little bit, but it's the back and forth, right?

s214
00:19:02.840 --> 00:19:05.280
It's the user, it's the assistant, it's the tool calls.

s215
00:19:05.280 --> 00:19:10.080
We preprocess that massive trace into something that is meaningful is much smaller.

s216
00:19:10.080 --> 00:19:12.560
Uh we then pass that to a facet, right?

s217
00:19:12.560 --> 00:19:18.840
The facet is the thing that that I just described, issues, tasks, sentiment, your own built-in facets.

s218
00:19:18.840 --> 00:19:23.120
What we're going to do here that that pre-process data is going to be passed to that facet.

s219
00:19:23.120 --> 00:19:25.320
We are going to then summarize it.

s220
00:19:25.320 --> 00:19:30.440
That summary, we then create a vector embedding of that summary within our back end.

s221
00:19:30.440 --> 00:19:36.120
Uh, on top of that, once we have that vector embedding, we can now go create what we call a topic map.

s222
00:19:36.120 --> 00:19:40.160
This is sort of what you saw on the previous slide of the the scatter plot.

s223
00:19:40.160 --> 00:19:45.040
Uh, this is what now allows us to label our traces and classify them.

s224
00:19:45.040 --> 00:19:57.400
And if I have those labels and those classifiers, it becomes incredibly trivial for me as a user, and also incredibly trivial for me as a user um, using a coding agent on top of Braintrust

s225
00:19:57.400 --> 00:20:01.640
to now start to filter through that noise using all of that really rich data.

s226
00:20:01.640 --> 00:20:10.000
Uh, important callout, the pipeline that you're seeing, the facets and that embedding step is actually using a a hosted model that we have.

s227
00:20:10.000 --> 00:20:18.400
So, we have just done a lot of work behind the scenes to create um, not not like a feature that you would just like show in a demo.

s228
00:20:18.400 --> 00:20:22.560
Obviously, like we'll we'll see some stuff here today, but this should work at scale.

s229
00:20:22.560 --> 00:20:26.080
You should be able to run this on 100% of your traces.

s230
00:20:26.080 --> 00:20:29.040
That's that's the sort of goal of our topics pipeline.

s231
00:20:29.040 --> 00:20:37.960
It is uh, just to get like maybe more detailed than I should, 6 cents per um, million tokens on the input side and 40 cents on the output side.

s232
00:20:37.960 --> 00:20:44.960
Uh, compare that to like other models that are that you might be using uh, on the LLM judge to create that same type of signal.

s233
00:20:44.960 --> 00:20:55.640
You can create a pretty compelling use case, um, especially if you start to see value in the the sort of topics and the labels and classifiers that we generate as a result of it.

s234
00:20:57.160 --> 00:21:01.480
Um, I also alluded to this a little bit, like why did we build this?

s235
00:21:01.480 --> 00:21:06.520
Uh, we've met we have customers that that generate uh, massive amounts of data.

s236
00:21:06.520 --> 00:21:15.880
And one of the things that I think we solved for very early on was being able to ingest that data and then being able to allowing our customers to query over that data

s237
00:21:15.880 --> 00:21:18.480
in in sort of sub-second speeds.

s238
00:21:18.480 --> 00:21:22.600
What we need What we didn't necessarily have at the outset was the ability to go do something with it, right?

s239
00:21:22.600 --> 00:21:24.760
And this is where that active observability comes into play.

s240
00:21:24.760 --> 00:21:25.779
Like

s241
00:21:25.779 --> 00:21:26.040
[snorts]

s242
00:21:26.040 --> 00:21:30.160
how do we now go derive insight from all of this data, especially this data at scale?

s243
00:21:30.160 --> 00:21:32.520
This is where that the challenge comes into play.

s244
00:21:32.520 --> 00:21:43.520
And so if you have this really complex pipeline that has been architected to both run at scale, um and I mean that from a technical perspective as well as a cost one, uh like I alluded to earlier,

s245
00:21:43.520 --> 00:21:53.200
then we can actually provide a I think a really interesting primitive and foundation that our customers can build on to inform that flywheel.

s246
00:21:53.200 --> 00:22:03.040
Um at the end of the day, the the the thing that we're trying to do is enable our customers to build that flywheel and to build it in a way that actually

s247
00:22:03.040 --> 00:22:08.720
um improves their agent quality and they can do it very, very quickly.

s248
00:22:09.920 --> 00:22:10.160
All right.

s249
00:22:10.160 --> 00:22:15.480
Hopefully I I didn't lose anybody there in the in the presentation and you all are ready for workshop.

s250
00:22:15.480 --> 00:22:23.520
Um quick show of hands, anybody still need the QR code or the URL for the the GitHub repo?

s251
00:22:23.520 --> 00:22:25.520
Yeah.

s252
00:22:26.280 --> 00:22:30.760
Yeah, I'll keep it up here for 30 seconds or so.

s253
00:22:54.080 --> 00:22:57.240
Uh my team uses Lightstep.

s254
00:22:57.360 --> 00:22:59.680
I think Lightstep kind of sucks.

s255
00:22:59.680 --> 00:23:04.520
Can you give me a list of things that my team says, "Well, why should we use BrainTrust instead?"

s256
00:23:04.520 --> 00:23:08.560
That I can go to them and say, "These are the things that BrainTrust gives you in the likes of the town.

s257
00:23:08.560 --> 00:23:11.440
Oh, man, you're going like straight competitor here, huh?

s258
00:23:11.440 --> 00:23:12.880
Wow.

s259
00:23:12.880 --> 00:23:15.040
Was not prepared for for that.

s260
00:23:15.040 --> 00:23:16.720
Uh I am I'm certainly not the guy.

s261
00:23:16.720 --> 00:23:20.120
Like I will not be up here bashing anybody.

s262
00:23:20.120 --> 00:23:30.000
Um I I do think that we we have a a stronger offering and I think we have a stronger offering in a in a lot of different regards.

s263
00:23:30.000 --> 00:23:32.400
Um certainly not in any particular order.

s264
00:23:32.400 --> 00:23:36.080
One of them is the the flexibility that I showed to you on the other slide.

s265
00:23:36.080 --> 00:23:42.200
There is the ability to go across multiple languages, multiple programming, excuse me, multiple programming languages, frameworks.

s266
00:23:42.200 --> 00:23:46.120
Um It's a very agnostic way to build your agent.

s267
00:23:46.120 --> 00:23:54.040
Uh LangSmith, I think, has some of that as well, but they are definitely geared towards LangGraph and uh LangChain and deep agents and and those types of things.

s268
00:23:54.040 --> 00:23:57.960
And I think they have features that are only sort of for that.

s269
00:23:57.960 --> 00:24:07.400
Um the other thing that I think if you go back, actually I used to have a slide on this, but BrainTrust built their own database in late 2024.

s270
00:24:07.400 --> 00:24:17.400
And and the reason that we built this database is is Ankur, our CEO, recognized very early on the trajectory that we were going in from a agent perspective.

s271
00:24:17.400 --> 00:24:30.600
And so he we were using ClickHouse behind the scenes at the time and recognized very quickly when we onboarded the likes of, you know, the the customers that you saw on there, Netflix and Microsoft and Dropbox,

s272
00:24:30.600 --> 00:24:33.920
that they ClickHouse just very quickly fell down at scale.

s273
00:24:33.920 --> 00:24:35.400
And we had to go do something different.

s274
00:24:35.400 --> 00:24:38.600
And so this is where the engineering team and Ankur built BrainStore.

s275
00:24:38.600 --> 00:24:39.960
This is our own purpose-built back end.

s276
00:24:39.960 --> 00:24:42.200
I think at the time people thought we were crazy.

s277
00:24:42.200 --> 00:24:45.280
They thought like, "Hey, there are all of these like open-source alternatives.

s278
00:24:45.280 --> 00:24:45.920
There's ClickHouse.

s279
00:24:45.920 --> 00:24:51.560
Like there are these things that actually, um you know, allow us to do this and do this well."

s280
00:24:51.560 --> 00:24:56.760
What you now see, and especially like you probably see see on the LangSmith side, they've just built their own database.

s281
00:24:56.760 --> 00:25:00.080
Um Arise, they just built their own database like 6 months ago.

s282
00:25:00.080 --> 00:25:08.960
I think people are starting to recognize uh that this is actually a problem that's not solved for with out-of-the-box solutions, and you need something custom.

s283
00:25:08.960 --> 00:25:14.000
And so, in that regard, I think we're like a year uh ahead of some of these these competitors.

s284
00:25:14.000 --> 00:25:22.320
And where that shows up is is in topics, is in being able to generate real insights on data at scale.

s285
00:25:22.320 --> 00:25:24.120
Um there there's probably some other other ones.

s286
00:25:24.120 --> 00:25:38.600
If you want to, you know, come find me afterwards, we can we can get into it, but um I'd say that that last one, like the having our own back end, and the things that we were able to build on top of that, has really what is really enabled us, I think, to separate ourselves from other players

s287
00:25:38.600 --> 00:25:40.720
in the space.

s288
00:25:41.160 --> 00:25:43.160
Yeah.

s289
00:25:44.040 --> 00:25:46.160
Yeah, you.

s290
00:26:13.328 --> 00:26:15.328
[snorts]

s291
00:26:51.880 --> 00:27:02.600
Yeah, I mean, like during the workshop sorry if if anybody couldn't hear that the question was just like examples of topics and a little bit more insight into how into perhaps the the process and

s292
00:27:02.600 --> 00:27:09.240
are we actually generating anything that is meaningfully different because the traces are relatively the same.

s293
00:27:09.240 --> 00:27:11.560
They are going to be semantically similar.

s294
00:27:11.560 --> 00:27:15.960
Um I think you'll see this in the workshop, hopefully.

s295
00:27:15.960 --> 00:27:19.880
Uh I think the the customers that are using this in production would would probably disagree with you.

s296
00:27:19.880 --> 00:27:32.200
That's why they are using it is because they are they are seeing those those differences and they are seeing those show up as different classifiers, as things that they they weren't yet aware of before actually running that pipeline.

s297
00:27:32.200 --> 00:27:37.920
Um I I think part of it is just like is getting your hands dirty a little bit and actually putting in your actual traces.

s298
00:27:37.920 --> 00:27:45.320
Like I have dummy demo data where I've, you know, used a script to hopefully create some differences that we create this.

s299
00:27:45.320 --> 00:27:50.360
I think you naturally see even more of that when you have actual users interact with the agent.

s300
00:27:50.360 --> 00:27:54.360
Um the traces look probably somewhat similar, right?

s301
00:27:54.360 --> 00:27:55.760
We're invoking the same tools.

s302
00:27:55.760 --> 00:27:59.400
It's going through the same uh workflow.

s303
00:27:59.400 --> 00:28:11.080
But I think, you know, depending on what your agent is doing and what your users are doing with your agent and I think maybe to back up a little bit, I think a lot of us are building agents that are increasing in complexity

s304
00:28:11.080 --> 00:28:14.320
or they're building systems that are increasing in complexity.

s305
00:28:14.320 --> 00:28:19.200
And it creates, I think, some some differences that that will show up on on screen.

s306
00:28:19.200 --> 00:28:24.040
But you you'll see when you create an account, you actually have you have a free account.

s307
00:28:24.040 --> 00:28:26.680
You can send data up to a certain point.

s308
00:28:26.680 --> 00:28:31.400
You can even toggle on topics to see if it is actually generating some of these things.

s309
00:28:31.400 --> 00:28:35.240
So, we'd just encourage you even after this workshop to go play with it.

s310
00:28:35.240 --> 00:28:41.160
Um and and see if it does generate some interesting uh topics for you.

s311
00:28:41.160 --> 00:28:43.680
Yeah, go for it.

s312
00:28:44.040 --> 00:28:51.040
bring your own traces as you are not showing to your customer using your own data.

s313
00:28:51.040 --> 00:28:53.040
Yeah.

s314
00:28:55.720 --> 00:29:04.400
Yeah, the the question is like uh can we can we use topics perhaps on like uh our own sessions with Cloud Code and Codex and and so on?

s315
00:29:04.400 --> 00:29:05.040
Absolutely.

s316
00:29:05.040 --> 00:29:13.640
One of the things that you'll see if you go to our integrations page is an integration with Cloud Code, integration with Codex, uh Cursor, Open Code.

s317
00:29:13.640 --> 00:29:21.720
And so you can I'm actually working with um multiple customers right now where they are trying to do exactly that and layer topics on top of that.

s318
00:29:21.720 --> 00:29:32.280
I I think a lot of I don't know exactly where the question's coming from, but I think what I've heard is that most engineers today, most teams of engineers are using those tools

s319
00:29:32.280 --> 00:29:42.120
and there there needs to now be I think insight into the efficacy or how well these things are actually improving the workflow and and and those types of things.

s320
00:29:42.120 --> 00:29:50.920
And I and I think there's there's probably some machinery that still needs to be put in place, but if you're just looking to hey, understand what people are doing in those those different tools,

s321
00:29:50.920 --> 00:29:55.040
I think topics is a really good way to start to understand that.

s322
00:29:55.040 --> 00:29:57.040
Yeah.

s323
00:30:01.360 --> 00:30:03.880
So we have uh we have a few different hosting options.

s324
00:30:03.880 --> 00:30:07.480
One is just full SaaS, that's what you're all going to be using today.

s325
00:30:07.480 --> 00:30:11.040
Uh we have both a BYOC and a hybrid model.

s326
00:30:11.040 --> 00:30:19.480
Um and so maybe to like back up even further than this or than the question is like BrainTrust was actually architected from day one to support a hybrid uh deployment.

s327
00:30:19.480 --> 00:30:26.920
And so by that I mean uh the control plane, the web app, right, the the metadata that we store is always going to be hosted by us.

s328
00:30:26.920 --> 00:30:34.960
The data plane, so your traces, your data sets, your experiments can actually be hosted within your VPC, within your infrastructure.

s329
00:30:34.960 --> 00:30:38.200
Uh we deploy across all of the major cloud providers.

s330
00:30:38.200 --> 00:30:46.800
Um but yeah, you actually we have, you know, those customers that you saw up there, I would wager that 90% of them are uh self-hosted ones.

s331
00:30:47.280 --> 00:30:47.720
Yeah.

s332
00:30:47.720 --> 00:30:48.160
Yeah, sorry.

s333
00:30:48.160 --> 00:30:50.360
Green shirt.

s334
00:31:04.120 --> 00:31:08.280
Yeah, so the the question is like how do you handle long-running conversations with topics?

s335
00:31:08.280 --> 00:31:21.880
Uh some of it is I can I'll be show you this during the workshop, but there are different settings that you can configure that operate on uh they can operate on different traces, that can operate or can only be invoked

s336
00:31:21.880 --> 00:31:24.080
after a certain time period.

s337
00:31:24.080 --> 00:31:33.480
Uh so depending on like generally speaking, I think there's an understanding of how long conversations tend to last that tends to inform the settings that you create.

s338
00:31:33.480 --> 00:31:37.480
But there's I don't know getting into the workshop probably answers that a little bit better.

s339
00:31:37.480 --> 00:31:40.120
Uh but I think we can answer that one.

s340
00:31:40.120 --> 00:31:40.440
Cool.

s341
00:31:40.440 --> 00:31:43.440
How about one more and then we'll jump in.

s342
00:31:44.920 --> 00:31:48.880
Yeah, this this the preprocessor, we give you a default one.

s343
00:31:48.880 --> 00:31:51.520
You can bring your own custom preprocessor.

s344
00:31:51.520 --> 00:31:59.760
Uh you have data that exists outside of uh what we can parse uh from input and output or what what's going to and from the LLM.

s345
00:31:59.760 --> 00:32:01.440
You have stuff that's in metadata.

s346
00:32:01.440 --> 00:32:10.680
You have stuff that's within some other span that maybe we're not sort of pre processing as part of that default preprocessor, you can absolutely create your own.

s347
00:32:10.680 --> 00:32:13.280
Uh the other one again like I think I mentioned was the facets.

s348
00:32:13.280 --> 00:32:14.560
You can bring your own facet.

s349
00:32:14.560 --> 00:32:19.320
You can build your own uh thing that you're trying to understand or to derive patterns of within BrainTrust.

s350
00:32:19.320 --> 00:32:26.680
But yeah, this I think one of the the other really cool things about this pipeline is it's completely hackable uh in the best sense of the word.

s351
00:32:28.640 --> 00:32:30.000
Cool.

s352
00:32:30.000 --> 00:32:31.160
I see a 30-minute timer.

s353
00:32:31.160 --> 00:32:35.400
I think we we have till we have till 11:00 though.

s354
00:32:37.200 --> 00:32:46.520
All right, so I'm going to assume everybody has the the workshop or the the QR code here or at least the the access to the repo.

s355
00:32:55.800 --> 00:32:59.040
Is that we zoom in a little bit?

s356
00:33:04.160 --> 00:33:11.120
Okay, this could be a little bit different based on you having a BrainTrust org or not.

s357
00:33:11.120 --> 00:33:16.520
Like I said, Noah is back here and can help if you if you run into anything.

s358
00:33:16.520 --> 00:33:20.920
Also, you know, if you want to like just yell at me for a little bit that's that's fine, too.

s359
00:33:20.920 --> 00:33:30.880
So obviously the first thing that will need here is a is a BrainTrust organization and it's all I'll sort of walk walk through this this guide with you all and and we'll start to understand

s360
00:33:30.880 --> 00:33:32.120
exactly what I just described there.

s361
00:33:32.120 --> 00:33:39.280
We want to create that flywheel and I think we'll we'll focus a little bit more on the online side, but just know that there is the the flip side

s362
00:33:39.280 --> 00:33:44.400
which is the e-vals and I do have an e-val as part of this that we can also chat about.

s363
00:33:47.280 --> 00:33:55.400
All right, so a little bit of a a teaser there that was not meant to do that, but we will create a new organization.

s364
00:34:04.240 --> 00:34:09.800
Um if it if it's your first organization that you've created, uh you're going to navigate to that that link.

s365
00:34:09.800 --> 00:34:12.560
You may have some different steps at the outset.

s366
00:34:12.560 --> 00:34:19.960
It may actually uh ask you to provide some AI providers and a uh generate an API key.

s367
00:34:19.960 --> 00:34:20.960
Please go through that.

s368
00:34:20.960 --> 00:34:23.000
I also have steps for those as well.

s369
00:34:23.000 --> 00:34:29.080
Uh you just see a slightly different uh flow because I already have organizations created here on the BrainTrust side.

s370
00:34:29.080 --> 00:34:32.240
If you see if you see a screen like I do now?

s371
00:34:32.240 --> 00:34:35.399
Quick raise Raise your hands.

s372
00:34:35.480 --> 00:34:40.679
Do most people Are most people just creating a new Braintrust org for the very first time?

s373
00:34:40.679 --> 00:34:41.280
Okay, cool.

s374
00:34:41.280 --> 00:34:41.600
Awesome.

s375
00:34:41.600 --> 00:34:42.919
I love to see that.

s376
00:34:42.919 --> 00:34:45.080
Um we created the org.

s377
00:34:45.080 --> 00:34:48.679
Is the next step to create a project?

s378
00:34:50.800 --> 00:34:52.800
Sorry?

s379
00:34:54.639 --> 00:34:56.040
Oh.

s380
00:34:56.040 --> 00:34:57.120
Internet issues.

s381
00:34:57.120 --> 00:34:58.360
Love that.

s382
00:34:58.360 --> 00:35:01.400
Okay, so nobody can participate.

s383
00:35:04.080 --> 00:35:05.400
Okay, bummer.

s384
00:35:05.400 --> 00:35:06.480
I don't know if anybody was here last year.

s385
00:35:06.480 --> 00:35:09.800
We had the exact same exact same problem.

s386
00:35:09.800 --> 00:35:24.680
Um well, we'll just go through this though, and I will try not to go too fast knowing that nobody else is is doing this here, but So, you'll see that you we we created an organization.

s387
00:35:24.680 --> 00:35:27.960
Uh you'll also see where we automatically created a my project.

s388
00:35:27.960 --> 00:35:30.400
Uh you can you can delete this if if you'd like.

s389
00:35:30.400 --> 00:35:34.960
Uh we're going to create our own project for uh this workshop.

s390
00:35:34.960 --> 00:35:38.720
Uh I would label it this, AIE-workshop.

s391
00:35:38.720 --> 00:35:43.080
All of the sort of code in the repo uh assumes that you have done this.

s392
00:35:43.080 --> 00:35:50.040
Now, you could certainly uh not do this and then change some other things in the repo, but I just wanted to call that out if you are following along.

s393
00:35:50.040 --> 00:35:58.840
Uh just know that you'll need to uh make sure that the project that you've defined here is also matching what you have within the repo.

s394
00:36:00.360 --> 00:36:08.200
So, the the other two things that we'll need to sort of complete this is uh one, we will need to generate or create an AI provider.

s395
00:36:08.200 --> 00:36:17.760
Uh so, if you go over to Braintrust and into your settings, you'll see a uh a place where you can actually configure your AI providers.

s396
00:36:17.760 --> 00:36:20.280
Uh I'll click this one up here.

s397
00:36:20.280 --> 00:36:30.120
There there should be a lot of different uh places for you to add some type of API key or uh what we just released very recently is workload identity federation for

s398
00:36:30.120 --> 00:36:32.360
some of our major AI providers.

s399
00:36:32.360 --> 00:36:40.360
But you can also our organizations tend to also do or where I see them also build is like they'll have their own gateway.

s400
00:36:40.360 --> 00:36:43.040
And they want to connect their their gateway to BrainTrust.

s401
00:36:43.040 --> 00:36:46.080
They could do that using a custom provider.

s402
00:36:46.080 --> 00:36:54.720
You sort of give the underlying spec that that gateway sort of conforms to how we are authenticating to that and some other sort of URL authentication type.

s403
00:36:54.720 --> 00:36:59.640
So generally speaking you can connect to a lot of different AI providers in a lot of ways.

s404
00:36:59.640 --> 00:37:04.440
I haven't seen anybody not be able to connect to anything from from BrainTrust.

s405
00:37:04.440 --> 00:37:16.160
The the other thing maybe just to if if you are sort of following along and you are using a gateway but that gateway has perhaps some firewalls you'd probably see some some errors on this side.

s406
00:37:16.160 --> 00:37:27.240
Just know that you know there are ways in which you can sort of allow certain IP addresses to those different places that perhaps only allow certain IP addresses like a gateway.

s407
00:37:27.240 --> 00:37:34.480
Um I'm going to stop sharing just for a second so you don't see my API key.

s408
00:38:07.280 --> 00:38:11.360
All right, one more I'm going to generate an API key also.

s409
00:38:16.520 --> 00:38:18.520
Okay.

s410
00:38:33.480 --> 00:38:34.720
All right.

s411
00:38:34.720 --> 00:38:40.400
So, just to recap what I just did here, I went over to AI providers.

s412
00:38:40.400 --> 00:38:44.880
I added both OpenAI and Anthropic as API keys.

s413
00:38:44.880 --> 00:38:53.440
Uh the reason that you'll need this is if you do configure online scoring and you're using judges as part of that, uh you will need to choose some type of model.

s414
00:38:53.440 --> 00:39:00.920
So, another call out here is that when you actually invoke those those scores, you are using your own models to do so.

s415
00:39:00.920 --> 00:39:10.320
The only caveat to do that is that Topics pipeline, which uses our BrainTrust built-in models um for that complex pipeline.

s416
00:39:13.920 --> 00:39:14.080
All right.

s417
00:39:14.080 --> 00:39:16.840
So, we essentially just did uh sort of step zero here.

s418
00:39:16.840 --> 00:39:18.440
We uh created a new org.

s419
00:39:18.440 --> 00:39:21.320
We have a project called AI e-workshop.

s420
00:39:21.320 --> 00:39:25.360
We have an AI provider, and then we have an API key.

s421
00:39:25.360 --> 00:39:27.160
Uh if you haven't done this, right?

s422
00:39:27.160 --> 00:39:30.520
So, we can now clone this repo.

s423
00:39:55.120 --> 00:40:01.600
Looks like I did not update something in the code.

s424
00:40:06.960 --> 00:40:07.200
All right.

s425
00:40:07.200 --> 00:40:09.480
One more time.

s426
00:40:27.160 --> 00:40:32.440
All right, anybody thumbs up, thumbs down, is this okay from a you need to zoom in?

s427
00:40:32.680 --> 00:40:33.840
Cool.

s428
00:40:33.840 --> 00:40:35.960
Okay, so let's let's start going through this.

s429
00:40:35.960 --> 00:40:40.960
Um obviously we're going to be using some some different tooling here to uh to power this.

s430
00:40:40.960 --> 00:40:48.560
I am using UV as the the sort of package manager to install all of my uh Python dependencies.

s431
00:40:48.560 --> 00:40:52.400
So, if you don't have UV already installed, you would use this command.

s432
00:40:52.400 --> 00:40:59.240
Uh I already do, so I'm just going to run uh UV sync and then add some dev dependencies as well.

s433
00:40:59.240 --> 00:41:08.200
Uh so, that should create a virtual environment within your directory and then should install all of the different uh things that are required to run this workshop.

s434
00:41:08.240 --> 00:41:15.520
The next thing that we will do is uh create our .env file and we already have one as a example.

s435
00:41:15.520 --> 00:41:17.960
So, you can copy that one over.

s436
00:41:17.960 --> 00:41:26.080
And all you're doing in this one is you are adding an API key that you generated on the Braintrust side and then you are adding a default model.

s437
00:41:26.080 --> 00:41:32.280
Um this is really only needed if you wanted to spin up the the sort of chat UI uh or then run I think evals as well.

s438
00:41:32.280 --> 00:41:40.760
But these are the the two things that you would sort of create or modify within that .env that you just copied over uh to to power and run this workshop.

s439
00:41:40.760 --> 00:41:53.440
There there's some other ones that you know, there's some instructions in that .env file of what you could also uh change like if you wanted to follow maybe those best practices that we talked about earlier of maybe creating or having a better or higher

s440
00:41:53.440 --> 00:41:56.800
uh more powerful judge model, you could do that as well.

s441
00:41:56.800 --> 00:42:11.000
Just if you're if you're not aware what this should be there is the the um this is the the .env that we just copied over.

s442
00:42:11.000 --> 00:42:23.280
Again, I'm just modifying or added my Braintrust API key and then I added a a default model and this default model obviously is going to be something that is accessible via the providers that I that I configured on the Braintrust side.

s443
00:42:23.280 --> 00:42:27.880
So, if I did open AI, I'm going to do a GPT model, Anthropic uh Sonnet, and so on.

s444
00:42:27.880 --> 00:42:31.800
So, just make sure that these those two things uh mirror one another.

s445
00:42:31.800 --> 00:42:38.440
The The other thing that's going to be really important uh for this workshop is to install the Braintrust CLI.

s446
00:42:38.440 --> 00:42:46.560
Uh this is going to allow you to run arbitrary SQL, to run evals, to view your logs, to create data sets.

s447
00:42:46.560 --> 00:42:59.440
You think about that flywheel that I just described, a lot of those things are are part of that flywheel and it's something that we can plug into our agent to actually start to do itself, which you'll see at the end of the uh at the end of the workshop.

s448
00:42:59.440 --> 00:43:04.080
So, you'll see over here, I already have the uh the Braintrust CLI installed.

s449
00:43:04.080 --> 00:43:08.960
Uh I'm also going to do a BT setup skills and I'm going to use Codex.

s450
00:43:08.960 --> 00:43:14.440
Uh you might be using Cloud Code or Open Code or Cursor or Quen or Gemini.

s451
00:43:14.440 --> 00:43:22.640
That's supported also, but the idea here is that we are going to give our coding agent access to some skills to understand how to create a data set, how to

s452
00:43:22.640 --> 00:43:26.880
run evals, um how to write SQL, and so on.

s453
00:43:33.240 --> 00:43:35.240
Yeah.

s454
00:43:37.560 --> 00:43:40.000
Switch to what?

s455
00:43:40.640 --> 00:43:42.920
Oh, yeah.

s456
00:43:43.640 --> 00:43:44.560
That's a really good idea.

s457
00:43:44.560 --> 00:43:48.545
Where I don't I don't think I've ever actually done that in here.

s458
00:43:48.545 --> 00:43:50.545
[laughter]

s459
00:44:00.680 --> 00:44:02.880
Any idea?

s460
00:44:17.960 --> 00:44:18.320
What is it?

s461
00:44:18.320 --> 00:44:22.520
Control B?

s462
00:44:50.760 --> 00:44:53.040
Let's do this.

s463
00:45:43.200 --> 00:45:45.200
Sorry?

s464
00:45:51.160 --> 00:45:53.600
Command shift B?

s465
00:45:54.400 --> 00:45:56.640
Type in what?

s466
00:46:10.280 --> 00:46:11.991
All right.

s467
00:46:11.991 --> 00:46:13.991
[laughter]

s468
00:46:14.640 --> 00:46:22.480
I'm guessing you all all were hoping to see me navigate around a terminal and not have any idea what I'm doing.

s469
00:46:23.280 --> 00:46:24.040
So we're we're all good.

s470
00:46:24.040 --> 00:46:25.440
This is better for everybody.

s471
00:46:25.440 --> 00:46:26.080
Awesome.

s472
00:46:26.080 --> 00:46:28.400
Sorry about that.

s473
00:46:31.080 --> 00:46:31.520
Okay, cool.

s474
00:46:31.520 --> 00:46:36.960
The The next thing that we will do is we will initialize uh the the CLI.

s475
00:46:36.960 --> 00:46:46.880
Um I'm going to do something slightly different because I have uh already initialized uh an ending to OAuth into Brain Trust to access a different organization.

s476
00:46:46.880 --> 00:46:50.800
So I'm going to do Oop.

s477
00:46:56.760 --> 00:47:05.000
So if again, if you already have the Brain Trust CLI and you created a new org, uh you'll go through that, and then you'll just uh select that org that you just created.

s478
00:47:05.000 --> 00:47:09.120
So for me it was ABC DPG.

s479
00:47:14.520 --> 00:47:15.480
Cool.

s480
00:47:15.480 --> 00:47:20.480
And then I will run that BT init, and I'll look for ABC.

s481
00:47:20.480 --> 00:47:29.960
And so this is just going to create a uh config.json file in the directory that when BT runs, it knows what org and project to to access.

s482
00:47:29.960 --> 00:47:33.440
So you can imagine uh checking this into your your repo.

s483
00:47:33.440 --> 00:47:44.000
All of your engineers now sort of like have access to um you know, BT has an easier way to understand what org and project that should uh run an eval and query data from.

s484
00:47:44.480 --> 00:47:44.720
Cool.

s485
00:47:44.720 --> 00:47:51.840
Let's uh now go through uh actually setting up some of the different things that will need as part of uh the workshop.

s486
00:47:51.840 --> 00:47:55.080
Uh I'm just going to seed a database.

s487
00:47:55.080 --> 00:47:59.800
Uh I'm going to skip over this smoke command, but I'm going to run make ready.

s488
00:47:59.800 --> 00:48:07.840
Um, if you don't have uh, make or access to to make, there are the actual commands right below it that you can also run.

s489
00:48:07.840 --> 00:48:12.160
Uh, what this is doing right now is just just trying to understand, "Hey, did we set up everything correctly?

s490
00:48:12.160 --> 00:48:14.000
Do we have the the Braintrust CLI?

s491
00:48:14.000 --> 00:48:16.200
Do we have a Braintrust API key?

s492
00:48:16.200 --> 00:48:17.240
Uh, do we have a default model?

s493
00:48:17.240 --> 00:48:18.000
Those types of things."

s494
00:48:18.000 --> 00:48:20.680
So, uh, I think we're all good to go forward now.

s495
00:48:20.680 --> 00:48:27.440
Uh, then just to kind of give you a sense for maybe the at least in this case the the trivial use case that we are that we are powering.

s496
00:48:27.440 --> 00:48:33.280
Uh, very uh, simple chatbot, but really what I want to understand is is it sort of wired up in the right way?

s497
00:48:33.280 --> 00:48:36.960
Are we seeing logs flow into Braintrust?

s498
00:48:36.960 --> 00:48:39.200
And so, I just clicked the button in our chat.

s499
00:48:39.200 --> 00:48:45.000
We should now see a log show up over on the Braintrust side with the uh, the question that we just asked.

s500
00:48:45.000 --> 00:48:56.000
So, here is an example trace that was just created uh, based on the, you know, the the integration that we have with OpenAI Agents SDK.

s501
00:48:56.000 --> 00:49:04.520
Um, I think maybe to that to that point, uh, just to show you like what sort of integration looks like uh, within Braintrust.

s502
00:49:04.520 --> 00:49:07.920
This is obviously a a single example.

s503
00:49:07.920 --> 00:49:12.960
Um, but all we're all we're really doing is is uh, is this.

s504
00:49:12.960 --> 00:49:21.800
This is what we're pulling in from the the Braintrust SDK and we are setting a trace processor to a Braintrust tracing processor.

s505
00:49:21.800 --> 00:49:23.960
Uh, there is also an easier way to do this.

s506
00:49:23.960 --> 00:49:27.840
There is an auto instrument that is a single um, line change.

s507
00:49:27.840 --> 00:49:33.760
The the the reason that I did this is because I'm also using our gateway under the hood.

s508
00:49:33.760 --> 00:49:36.360
Um, maybe I buried that a little bit.

s509
00:49:36.360 --> 00:49:38.920
Braintrust also has a gateway.

s510
00:49:38.920 --> 00:49:51.360
So, all of those AI providers that you have configured within Braintrust, we can now access uh, from a single interface, uh, which is what we're doing here, but we have our sort of uh, tracing set up from our SDK,

s511
00:49:51.360 --> 00:49:54.160
right directly for OpenAI agents.

s512
00:49:54.160 --> 00:49:58.840
We are setting that trace processor uh to a Braintrust tracing processor.

s513
00:49:58.840 --> 00:50:06.880
And again, this means that all of those traces that are emitted, all those spans that are emitted, are now uh going to Braintrust.

s514
00:50:06.880 --> 00:50:12.240
Uh maybe we'll just do it Actually, no, we won't do it from here because that's not light mode.

s515
00:50:12.920 --> 00:50:13.160
Okay.

s516
00:50:13.160 --> 00:50:19.080
So, I also have uh a script in here or I have uh code in here to run evals.

s517
00:50:19.080 --> 00:50:22.920
And obviously, this isn't like the the necessarily the point of the the workshop, right?

s518
00:50:22.920 --> 00:50:25.200
We're talking a little bit about observability.

s519
00:50:25.200 --> 00:50:29.240
But but I think uh in the context of observability, you need to think about the flywheel.

s520
00:50:29.240 --> 00:50:32.280
And and part of that flywheel is evals.

s521
00:50:32.280 --> 00:50:38.760
Um the things that we are able to understand and derive about what happened in production should make their way back to our evals.

s522
00:50:38.760 --> 00:50:44.480
Uh and so, there's also within the uh the repo that you have a uh sort of dummy data set, right?

s523
00:50:44.480 --> 00:50:48.440
It has a few uh test cases that we want to now run through our eval.

s524
00:50:48.440 --> 00:50:51.960
So, what we've just done again is used our Braintrust CLI.

s525
00:50:51.960 --> 00:50:56.360
We've created a data set, and we've actually loaded in a series of rows via that.

s526
00:50:56.360 --> 00:50:58.080
Um I'll now run the eval.

s527
00:50:58.080 --> 00:51:01.520
Again, this will go uh run through that that data set.

s528
00:51:01.520 --> 00:51:06.040
It'll sort of pass each row within that data set and invoke our agent.

s529
00:51:06.040 --> 00:51:13.920
And now, we will store all of that sort of um uh all that telemetry and the the sort of results of that of that eval uh directly within Braintrust.

s530
00:51:13.920 --> 00:51:22.080
But you could also imagine this is going to be incredibly informative when we do go run that flywheel, when we do go start to under- stand uh and we go make a change.

s531
00:51:22.080 --> 00:51:23.920
Generally, you want to compare it against something.

s532
00:51:23.920 --> 00:51:29.680
You want to be able to understand did I improve or regress from the last time that I actually ran that eval.

s533
00:51:29.680 --> 00:51:32.840
Uh so, we have Actually, I'll just come out to Braintrust.

s534
00:51:32.840 --> 00:51:37.640
Uh so, we should now see over here in our experiments page our eval that we have run.

s535
00:51:37.640 --> 00:51:42.240
You see the the series of scores that we've defined for this particular use case.

s536
00:51:42.240 --> 00:51:47.360
Uh you could also see and and maybe I'll come out to this in just a a bit, but uh we have different scores, right?

s537
00:51:47.360 --> 00:51:51.440
Just like I described earlier, there are there are scores that are um more more code-based.

s538
00:51:51.440 --> 00:51:54.920
So, required tools called, "Hey, I I have this question.

s539
00:51:54.920 --> 00:51:56.520
This is what I expect."

s540
00:51:56.520 --> 00:51:59.000
Uh then I have maybe a little bit more of that LLM judge.

s541
00:51:59.000 --> 00:52:02.360
So, support resolution and the tool use quality.

s542
00:52:02.360 --> 00:52:12.160
So, sort of a series or a um mix of scores that can provide the right signal for understanding what's going on with our agent.

s543
00:52:12.800 --> 00:52:15.560
Okay, let's let's get sort of to the the meat of this, though.

s544
00:52:15.560 --> 00:52:22.360
The The idea here is that we can now uh create some type of signal within our uh within our logs.

s545
00:52:22.360 --> 00:52:27.880
So, obviously, we have a single log right now, uh and we will need to to generate some data and put it into Braintrust.

s546
00:52:27.880 --> 00:52:37.640
But, before we do that, let's um use those series of scores and push those into Braintrust so that we can now configure an automation so that they can run

s547
00:52:37.640 --> 00:52:39.600
when logs come into Braintrust.

s548
00:52:39.600 --> 00:52:42.920
So, again, I'm going to use this uh make push scores command.

s549
00:52:42.920 --> 00:52:50.600
This is actually going to look for uh the series of scores that we've created and push them into Braintrust.

s550
00:52:50.600 --> 00:52:53.800
Uh so, there is a way in which you can define these in the UI.

s551
00:52:53.800 --> 00:52:59.680
You could also, like you you you see here, define your scores in code and push them into Braintrust.

s552
00:52:59.680 --> 00:53:12.160
Uh with the idea being that you'd want to run them as part of an online automation, uh or perhaps uh we also have a playground and you could give that to another user of the playground and they can use that same score, but have it version controlled

s553
00:53:12.160 --> 00:53:13.480
within your repo.

s554
00:53:13.480 --> 00:53:21.160
Um again, here are the series of tools that we just uh Excuse me, the series of scores that we just loaded into Braintrust.

s555
00:53:21.160 --> 00:53:27.200
Uh you should now see, if you go over to your scores tab, the different scores that we created.

s556
00:53:27.200 --> 00:53:30.880
So, I have, let's, you know, support resolution as an example, right?

s557
00:53:30.880 --> 00:53:33.880
This is going to be an LLM judge.

s558
00:53:33.880 --> 00:53:45.200
Uh and and what you're doing here, again, is defining that criteria in which you want to judge something with the with with with respect to the um the the the user's interaction with that agent.

s559
00:53:45.200 --> 00:53:52.560
Uh one thing I'd call out here is that there are different sort of variables you can access and pass to an LLM judge.

s560
00:53:52.560 --> 00:53:55.760
Uh this thread is is sort of what I described earlier.

s561
00:53:55.760 --> 00:54:07.960
Um it's it's what you would see if you come over to our logs page and you click on the thread, which is this icon here, but it's it's really just going to be that back and forth of that agent, right?

s562
00:54:07.960 --> 00:54:16.720
This is really helpful, especially if you have a multi- turn conversation and you want to sort of pass the entirety of that conversation to the LLM, you can use a variable called thread.

s563
00:54:16.720 --> 00:54:26.880
There there there's some other ones uh that that you could also use like user messages or assistant messages or simply input and output from a particular span because you want to create

s564
00:54:26.880 --> 00:54:31.480
a score that understands and scores look up policy as an example.

s565
00:54:31.480 --> 00:54:39.880
But, there's a few different ways in which you can start to think about how do I start to assess the quality of my agent across a lot of different dimensions.

s566
00:54:41.480 --> 00:54:51.280
So, now that we have our scores loaded into Braintrust, the the thing that we need to do now is actually wire them up to run when traces come into Braintrust.

s567
00:54:51.280 --> 00:54:55.680
Uh what what that looks like on the Braintrust side is an automation.

s568
00:54:55.680 --> 00:55:01.240
Uh if you look in your logs page, you'll see up here in the top right uh automations.

s569
00:55:01.240 --> 00:55:06.240
This is where you can start to create rules that your scores can be invoked on.

s570
00:55:06.240 --> 00:55:14.320
Um there are if you if you look at here at the scope, I think this is is really important, especially you start to think about like the different use cases

s571
00:55:14.320 --> 00:55:15.640
that you have.

s572
00:55:15.640 --> 00:55:24.040
Um you could think I have a multi-turn conversation, but my uh turns are actually across different traces.

s573
00:55:24.040 --> 00:55:29.440
I don't see them nested under each um I don't see each turn nested under a single trace.

s574
00:55:29.440 --> 00:55:39.200
So, what I might want to do in that case is select this group scope and then provide some type of ID, some type of way to understand how things are related.

s575
00:55:39.200 --> 00:55:41.560
In my case, it's a conversation ID.

s576
00:55:41.560 --> 00:55:48.120
Um the other thing that you might want to do is create a score that is invoked for a particular span.

s577
00:55:48.120 --> 00:55:54.840
Again, you want to assess some type of intermediate output or intermediate step, and you want to target a particular span.

s578
00:55:54.840 --> 00:55:58.320
When you do this, when you click span, it'll default to the root.

s579
00:55:58.320 --> 00:56:07.760
Um you could also add any type of filter for I want to recognize, let's say, my lookup policy, and I want to say in this case something like that.

s580
00:56:07.760 --> 00:56:15.080
So, I want to understand if this is is root or the span attribute name is in lookup policy.

s581
00:56:15.080 --> 00:56:19.200
Uh there's also, again, like the trace, which is what we will use for this demo.

s582
00:56:19.200 --> 00:56:27.160
Uh this is going to allow us to uh essentially access all of the different spans within that trace to produce some score.

s583
00:56:27.160 --> 00:56:34.680
Uh so, in this case, I'm going to pull out my uh communication quality, and my support resolution and tool use quality.

s584
00:56:34.680 --> 00:56:37.440
These are all LLM judges.

s585
00:56:37.440 --> 00:56:50.120
So, if you're following along, um just know that depending on the sampling rate that you uh that you put here, you're going to incur actual model compute uh when you go and and load data into Braintrust via these judges.

s586
00:56:50.120 --> 00:56:51.760
So, just wanted to highlight that.

s587
00:56:51.760 --> 00:56:56.520
Again, if you're following along and and perhaps maybe you want to uh lower this a little bit.

s588
00:56:56.520 --> 00:57:03.680
Uh I'm just going to go to 100% just so you kind of get a a really good signal or a really good idea of what this could start to look like.

s589
00:57:04.400 --> 00:57:14.520
So, what what this what what I've just done now is I've said, "Hey, any new any new logs that have come into Braintrust, I want to uh run those scores against those, and I want to run against

s590
00:57:14.520 --> 00:57:17.320
the out the the trace."

s591
00:57:20.680 --> 00:57:21.280
Okay.

s592
00:57:21.280 --> 00:57:23.280
Let's uh let's let's jump into topics now.

s593
00:57:23.280 --> 00:57:26.600
So, there's there's one side of our signal that we that we can create.

s594
00:57:26.600 --> 00:57:30.440
This is the more scoped down way of understanding a failure mode, right?

s595
00:57:30.440 --> 00:57:33.720
This is perhaps something that we have We're obviously eval-ing for.

s596
00:57:33.720 --> 00:57:41.240
We have recognized to be something that um you know, we need to understand if we are regressing anything as we go and change something within our agent.

s597
00:57:41.240 --> 00:57:44.720
Uh topics again is is more for those unknown unknowns.

s598
00:57:44.720 --> 00:57:49.400
Those those areas that we are not yet testing for or that we're not yet aware of.

s599
00:57:49.400 --> 00:57:54.400
Uh you'll also notice right next to our automations is an enable topics.

s600
00:57:54.400 --> 00:57:55.840
Uh and actually let me come back this.

s601
00:57:55.840 --> 00:58:00.600
I think the the first thing that we'll do here, enable topics.

s602
00:58:00.600 --> 00:58:03.000
So, you'll see you'll see a screen that looks like this.

s603
00:58:03.000 --> 00:58:05.000
Um and you'll see those built-ins.

s604
00:58:05.000 --> 00:58:07.080
So, task, sentiment, and issues.

s605
00:58:07.080 --> 00:58:15.880
So, sort of what is what I described, what are people or users doing with my agent, what is their sentiment, are they happy, are they sad, and then what are sort of like a broad range of issues

s606
00:58:15.880 --> 00:58:19.960
that they are experiencing uh with respect to that interaction of that agent.

s607
00:58:19.960 --> 00:58:21.640
So, I'll enable this.

s608
00:58:21.640 --> 00:58:23.960
Um this will apply to existing traces.

s609
00:58:23.960 --> 00:58:29.200
Obviously, not a lot of data here, so um we'll eventually see some data show up here.

s610
00:58:29.200 --> 00:58:39.120
The thing that I again allows you to bring your own way of understanding patterns within your data is where you can actually create your own facet.

s611
00:58:39.120 --> 00:58:44.440
Um you'll also notice here, this is where you can also define your own preprocessor.

s612
00:58:44.440 --> 00:58:52.560
Uh we're just going to use the default thread one here today, but just know that if you wanted to bring your own function to BrainTrust to parse something within that trace,

s613
00:58:52.560 --> 00:58:55.480
uh you could certainly do that.

s614
00:58:56.880 --> 00:59:02.680
So, if you look, there's also a file within that repo called support workflow issue facet.

s615
00:59:02.680 --> 00:59:04.120
It's a markdown file.

s616
00:59:04.120 --> 00:59:08.840
Uh this is the prompt that we will be using for our custom facet.

s617
00:59:08.840 --> 00:59:18.360
And so, back in uh BrainTrust, I'm going to grab the uh support workflow issue name.

s618
00:59:19.280 --> 00:59:21.760
That will be our custom facet.

s619
00:59:21.760 --> 00:59:25.720
And then I want to grab the the prompt here.

s620
00:59:29.600 --> 00:59:35.520
Um the other thing that you'll most likely want to do is to exclude certain facet outputs.

s621
00:59:35.520 --> 00:59:41.960
So, you can see here I've instructed the LLM to output none when the following criteria has been met.

s622
00:59:41.960 --> 00:59:49.280
Uh what I don't necessarily want to see is um facets that are that are generated where it's like, "Hey, this there's no problem here."

s623
00:59:49.280 --> 00:59:52.640
Uh I'm I'm really only interested in the things that are that are going wrong.

s624
00:59:52.640 --> 00:59:57.240
So, I will grab um this regex also at the top.

s625
00:59:59.360 --> 01:00:01.000
And I'll paste it here.

s626
01:00:01.000 --> 01:00:04.360
Uh you can also choose to apply to existing traces.

s627
01:00:04.360 --> 01:00:08.800
Um not super important in this case because we have uh we don't have very many.

s628
01:00:08.800 --> 01:00:18.520
So, on on the topic side, that's that's really all you need to need to do to um start to understand whether or not you can see some patterns within your your data.

s629
01:00:18.520 --> 01:00:26.560
Um if you come back over to our our repo here and you look for a perhaps a different way.

s630
01:00:26.560 --> 01:00:34.840
Uh I'm not going to go through this, but I just wanted to highlight again the the ability to utilize the BrainTrust CLI to do a lot of um interesting things.

s631
01:00:34.840 --> 01:00:40.760
So, that's sort of workflow that I just went through in the UI, I can also do here within uh BT.

s632
01:00:40.760 --> 01:00:43.560
I can actually configure these topics to to run.

s633
01:00:43.560 --> 01:00:46.200
I can configure those those automation settings.

s634
01:00:46.200 --> 01:00:51.200
Um so, here I've defined my task sentiment issues and my support workflow issues.

s635
01:00:51.200 --> 01:00:55.560
Uh the other thing that I did, you can also see some different configurations down there.

s636
01:00:55.560 --> 01:00:58.840
There is different ways to configure the settings, right?

s637
01:00:58.840 --> 01:01:06.160
So, that pipeline that runs in real time, you can control how it runs or what it runs on.

s638
01:01:06.160 --> 01:01:08.600
And so, by that I mean you might want to add a filter.

s639
01:01:08.600 --> 01:01:13.960
You might want to say like, "I only want to run my topic pipeline on traces that meet this criteria."

s640
01:01:13.960 --> 01:01:24.120
Uh you may again want to apply some type of sampling rate based on uh some understood metric you have about traffic and uh cost this could potentially incur, so on.

s641
01:01:24.120 --> 01:01:27.160
You also want to understand the the topic window.

s642
01:01:27.160 --> 01:01:34.200
Like when I generate my topic map, what is the sort of range of traces that I should use based on some window?

s643
01:01:34.200 --> 01:01:44.880
Uh this one I'm going to This is I think maybe the question you had earlier in the green shirt around like how do I how do I maybe configure this when I have long-running conversations?

s644
01:01:44.880 --> 01:01:53.800
Um part of that is that group by scope is being able to pull in perhaps other traces that other turns that exist across other traces.

s645
01:01:53.800 --> 01:02:00.040
Uh this also is another setting you can configure where you want to perhaps, you know, this could be a day, this could be 2 days.

s646
01:02:00.040 --> 01:02:08.040
Like you all understand that the the conversations that your agents are having or users are having with your agents stretch a very um long time period.

s647
01:02:08.040 --> 01:02:13.320
And so you can configure sort of when the topics pipeline gets gets invoked.

s648
01:02:13.320 --> 01:02:15.800
Uh I'm going to do something really short here.

s649
01:02:15.800 --> 01:02:22.400
Um but again, you can configure this based on your understanding of the users' interaction with your uh with your data.

s650
01:02:22.400 --> 01:02:25.280
Excuse me, users' interaction with your agent.

s651
01:02:25.280 --> 01:02:29.400
Uh then again, there's the same sort of scope that that I highlighted with the scores.

s652
01:02:29.400 --> 01:02:31.520
Uh you can have this run at the individual trace.

s653
01:02:31.520 --> 01:02:40.680
Or again, if you have that multi-turn conversation that exists across multiple traces, you would use this group and then use that sort of common um identifier, right?

s654
01:02:40.680 --> 01:02:45.720
In my case, uh I have a conversation ID that's being logged out as metadata.

s655
01:02:46.400 --> 01:02:48.880
So I'll click save.

s656
01:02:51.040 --> 01:02:54.600
So now we have sort of like the the topics configured.

s657
01:02:54.600 --> 01:03:07.920
We've enabled our uh out-of-the-box ones, and we have also created our own custom facet, and then we have configured our automation settings to uh to be run when data comes into BrainCert.

s658
01:03:07.920 --> 01:03:12.120
Uh on top of that, so now we can actually start to see what some of this looks like.

s659
01:03:12.120 --> 01:03:19.120
There is a command here where you can run um essentially the I I have some trace data stored out in an S3 bucket.

s660
01:03:19.120 --> 01:03:21.080
Uh we are going to just import it.

s661
01:03:21.080 --> 01:03:25.200
Again, under the hood here, what you're going to see is BT sync.

s662
01:03:25.200 --> 01:03:28.920
So, we can actually take that data and push it into Braintrust.

s663
01:03:28.920 --> 01:03:40.240
Um maybe to your to your question in the blue shirt about LangSmith earlier, I've actually helped a lot of customers import traces from other providers because they wanted to like maintain

s664
01:03:40.240 --> 01:03:42.480
or still have some of that that data.

s665
01:03:42.480 --> 01:03:48.520
But, just highlighting like this is a really great way to uh to do this at scale.

s666
01:03:49.720 --> 01:03:54.680
So, you can see here we just we just loaded in uh 968 traces.

s667
01:03:54.680 --> 01:03:59.200
Uh 1,052 spans, or that's the the rate, excuse me.

s668
01:03:59.200 --> 01:04:01.240
Uh 5,260 spans.

s669
01:04:01.240 --> 01:04:04.120
So, pretty pretty quick way to get data into Braintrust.

s670
01:04:04.120 --> 01:04:12.720
But, again, like the whole idea here is for us to start to generate some signal and start to understand where we can make some improvements within our agent.

s671
01:04:12.720 --> 01:04:19.040
Uh so, you'll start to see like here's uh just some uh some examples of traces.

s672
01:04:19.040 --> 01:04:23.160
Uh you'll also see that this is the this is actually two things you'll see here.

s673
01:04:23.160 --> 01:04:27.520
Um each turn is actually represented uh as its own trace.

s674
01:04:27.520 --> 01:04:32.440
So, this is the the newest user question and here is the the output for that question.

s675
01:04:32.440 --> 01:04:36.240
Uh you can also notice that there are 11 previous messages.

s676
01:04:36.240 --> 01:04:42.360
And so, I might want to do something where I I pull in my conversation ID.

s677
01:04:42.360 --> 01:04:47.960
And so, now I can start to pull in other traces that are that have that sort of same metadata.

s678
01:04:47.960 --> 01:04:54.600
But, the the sort of OpenAI agents gives you somewhat of a cheat code where it actually just attaches all of those previous messages.

s679
01:04:54.600 --> 01:05:07.560
So, we don't necessarily have to go through this step, but just wanted to highlight another feature within Braintrust that allows you to group traces by some arbitrary metadata that you've defined within those that they they they share across those traces,

s680
01:05:07.560 --> 01:05:12.680
and you can pull all of those relevant traces into the tree here.

s681
01:05:13.000 --> 01:05:14.200
Um really quick, right?

s682
01:05:14.200 --> 01:05:19.800
The thread view, I think is a really great way to understand sort of like the back and forth uh of your agent.

s683
01:05:19.800 --> 01:05:30.400
Uh there's also um the timeline view, which is a great way to understand like perhaps where bottlenecks exist, or you could even see there's really interesting information for, you know, cash hit

s684
01:05:30.400 --> 01:05:32.200
uh across the different LLM calls.

s685
01:05:32.200 --> 01:05:37.440
So, this information, I think, can be useful in a lot of different types of of workflows.

s686
01:05:37.440 --> 01:05:44.160
The the other one I'd highlight, and I haven't really talked about this yet, um this little blue icon in the bottom right is our agent loop.

s687
01:05:44.160 --> 01:05:45.920
Um everybody has an agent, right?

s688
01:05:45.920 --> 01:05:50.800
Uh so, you can actually use loop to do a lot of different things across the platform.

s689
01:05:50.800 --> 01:05:56.400
Uh you can use loop, in fact, to create your own UI on top of that trace data.

s690
01:05:56.400 --> 01:06:04.040
And so, this is, I think, a really great way when I was talking about the other scoring method of I want to uh allow humans to go and review

s691
01:06:04.040 --> 01:06:08.720
output of whether it's the agent or the LLM judges themselves and provide some calibration.

s692
01:06:08.720 --> 01:06:14.480
This is actually something you can prompt loop with, and it'll vibe code uh a UI on top of that trace data.

s693
01:06:14.480 --> 01:06:20.920
So, just a different way to bring in perhaps another persona to sort of inform that flywheel.

s694
01:06:20.920 --> 01:06:26.400
Um you you'll also see as part of this, like we're starting to see some some automation start to be triggered.

s695
01:06:26.400 --> 01:06:32.280
Uh so, we have my uh judge automation, so communication quality, support resolution, and so on.

s696
01:06:32.280 --> 01:06:35.960
And what you should also see is some topics starting to be generated as well.

s697
01:06:35.960 --> 01:06:37.000
So, in this case, right?

s698
01:06:37.000 --> 01:06:42.160
We have a user who wants to check the status of a returned item refund, and so on.

s699
01:06:42.160 --> 01:06:43.960
Um we have a negative sentiment.

s700
01:06:43.960 --> 01:06:48.800
This user expressed frustration and dissatisfaction with the status of a pending transaction.

s701
01:06:48.800 --> 01:06:52.320
Here's my support workflow, my custom facet here.

s702
01:06:52.320 --> 01:06:54.480
There's an order record retrieval failure.

s703
01:06:54.480 --> 01:06:59.240
Both specific order ID lookups and broader customer searches failed to locate the order.

s704
01:06:59.240 --> 01:07:02.240
So, just wanted to you know, two different things here, right?

s705
01:07:02.240 --> 01:07:10.640
We loaded traces into Braintrust, and based on the sort of automations and topics we've configured, we are now starting to get that signal in real time.

s706
01:07:10.640 --> 01:07:10.800
Right?

s707
01:07:10.800 --> 01:07:16.000
So, that complex pipeline that generates patterns within your data it shows up immediately.

s708
01:07:16.000 --> 01:07:16.160
Right?

s709
01:07:16.160 --> 01:07:20.200
It shows up um based on how you've configured that pipeline, obviously.

s710
01:07:20.200 --> 01:07:23.160
But, this is the now the signal that we're going to be able to use.

s711
01:07:23.160 --> 01:07:32.040
Um while this while this starts to uh populate, I'm going to go out to another project where I where I already ran this.

s712
01:07:32.040 --> 01:07:38.440
Um so, I showed you some illustrations earlier of what this uh perhaps starts to look like um within Braintrust.

s713
01:07:38.440 --> 01:07:54.880
So, at the top here uh in this page, what you start to see is a uh snapshot in time of your of your topics and the frequency with which uh the traces that we have recognized some pattern in patterns in, where they are being grouped.

s714
01:07:54.880 --> 01:07:58.840
And so, in the first example, I'll expand this.

s715
01:07:58.840 --> 01:08:02.680
Uh I can understand what users are trying to do with my agent.

s716
01:08:02.680 --> 01:08:03.000
Right?

s717
01:08:03.000 --> 01:08:11.640
I can see at the top uh damage and transit resolution seems to be uh the most frequent thing that again users are uh using for my agent.

s718
01:08:11.640 --> 01:08:20.080
And this is I think going back to what I was describing earlier when I was in the slides is I can actually use some of this information to understand like perhaps

s719
01:08:20.080 --> 01:08:27.359
where my agent can actually go um further or the thing that I don't yet have built within my agent that users are trying to do.

s720
01:08:27.359 --> 01:08:32.759
Um all of this taken together will will be used to sort of inform that flywheel.

s721
01:08:32.759 --> 01:08:36.640
Um a little bit further down, I have my sentiment topic.

s722
01:08:36.640 --> 01:08:39.040
Uh my issues did not get generated.

s723
01:08:39.040 --> 01:08:47.040
The The reason this didn't get generated is because we we look for at least 100 matches or 100 facets that were generated as part of that pipeline.

s724
01:08:47.040 --> 01:09:01.400
Um and this sort of agent that I built and the out-of-the-box prompt that we have for issues don't exactly jive or match and we don't really generate um we don't we don't really generate facets as a result of that, but I I that this is a sort of a good thing.

s725
01:09:01.400 --> 01:09:03.640
Like, we don't want to just generate bad data.

s726
01:09:03.640 --> 01:09:07.319
We don't want to just generate signal that you can't actually go do something with.

s727
01:09:07.319 --> 01:09:11.240
And so this is where I've created my support workflow issues.

s728
01:09:11.240 --> 01:09:15.040
And in this case, I'm starting to understand, hey, there's a record lookup failure.

s729
01:09:15.040 --> 01:09:25.920
There's something going on with my agent that we haven't again recognized with our evals, but we are understanding that there is a lookup failure as part of that agent going through its process.

s730
01:09:25.920 --> 01:09:28.960
Right, this is the kind of information sort of all taken together.

s731
01:09:28.960 --> 01:09:36.960
This with the the context of our online scores that actually allow me to go do something with it and and improve my agent in some way.

s732
01:09:38.880 --> 01:09:41.000
Any questions?

s733
01:09:47.920 --> 01:09:49.920
Cool.

s734
01:09:52.080 --> 01:09:52.720
Okay.

s735
01:09:52.720 --> 01:10:02.440
I want to show you sort of like a a workflow that that I think, you know, I certainly did when I when I first started at BrainTrust and I think a lot of lot of folks

s736
01:10:02.440 --> 01:10:05.000
um they did as well, right?

s737
01:10:05.000 --> 01:10:06.160
We've generated that signal.

s738
01:10:06.160 --> 01:10:07.400
This is great.

s739
01:10:07.400 --> 01:10:10.480
Um now I need to be able to go do something with that.

s740
01:10:10.480 --> 01:10:20.920
And so I think, you know, obviously within the the BrainTrust UI here, there's a lot of different things that I can utilize to start to understand or again filter through that noise.

s741
01:10:20.920 --> 01:10:28.960
Um I might want to look at, you know, the the different um classifiers or the different labels that were placed on top of these traces, right?

s742
01:10:28.960 --> 01:10:32.080
That map to those topics that you saw in that other illustration.

s743
01:10:32.080 --> 01:10:33.080
This I can use, right?

s744
01:10:33.080 --> 01:10:34.600
To filter this down.

s745
01:10:34.600 --> 01:10:38.400
Um probably increase this.

s746
01:10:38.640 --> 01:10:45.800
No. Uh this I can use theoretically to filter this down in some way, right?

s747
01:10:45.800 --> 01:10:51.840
Based on the the the feedback that we are getting uh and understand the thing that I can do from there, right?

s748
01:10:51.840 --> 01:10:54.240
So like I filtered my result set down.

s749
01:10:54.240 --> 01:11:02.320
I'm drilling into a trace and I'm recognizing by kind of parsing through the trace and the spans within there what went wrong.

s750
01:11:02.320 --> 01:11:08.280
Um maybe I'm again using my online scores to filter this result set down again, perhaps even further.

s751
01:11:08.280 --> 01:11:17.000
But I but I think like that the workflow that you might go through is you'd use this UI and you'd start to understand all the different things, like all what's going wrong.

s752
01:11:17.000 --> 01:11:24.360
I'm going to take my task X and combine it with this sentiment and I want to see what sort of insight I can derive from it.

s753
01:11:24.360 --> 01:11:26.520
Um obviously generating the signal, really great.

s754
01:11:26.520 --> 01:11:34.480
We have a lot of data now that we can actually go make some of these informed decisions, but I think you all can probably recognize the the problem with that approach as well.

s755
01:11:34.480 --> 01:11:36.680
It's like I am very reactive.

s756
01:11:36.680 --> 01:11:43.720
The the onus is on me as a user of Brain Trust to like go through and click buttons and filter and then parse through these traces.

s757
01:11:43.720 --> 01:11:45.840
Like there's got to be a better way.

s758
01:11:45.840 --> 01:11:48.120
Um I might want to use Loop.

s759
01:11:48.120 --> 01:11:54.880
Loop is a right again, the agent that is embedded here within the platform that can do different things.

s760
01:11:54.880 --> 01:12:01.440
Um I asked it in this case, help me understand what I should improve in my support agent based on recent logs.

s761
01:12:01.440 --> 01:12:12.680
So one of the things that you'll start to see that's I think again, really interesting from a uh flywheel perspective, is how we can start to pull in relevant data to inform that flywheel.

s762
01:12:12.680 --> 01:12:22.800
Um another thing that I think is again, really interesting about Brain Trust and specifically Brain Store, the back end that powers it, is that you can write SQL directly against it.

s763
01:12:22.800 --> 01:12:27.440
All of that trace data, all of those spans can be accessed by writing SQL.

s764
01:12:27.440 --> 01:12:33.120
Um you may not want to write that SQL, but your coding agent can very very easily write that SQL.

s765
01:12:33.120 --> 01:12:35.880
And so what you're seeing happen here is Loop.

s766
01:12:35.880 --> 01:12:38.280
Again, Loop is our our agent in the platform.

s767
01:12:38.280 --> 01:12:42.920
I asked it a question and what Loop did is it started firing off a bunch of SQL queries.

s768
01:12:42.920 --> 01:12:47.920
It's pulling in context that is relevant for the question that I asked it.

s769
01:12:47.920 --> 01:12:57.280
What you're probably not able to do and again, thinking at scale, you're not going to pull in every single trace searching for signal, right, using your agents, your coding agents.

s770
01:12:57.280 --> 01:13:02.040
You need that signal attached to your traces in some way, and you need that signal to be real.

s771
01:13:02.040 --> 01:13:10.240
And then you need a very flexible way to start to query across all of that data, so you can only pull in that relevant bit of information.

s772
01:13:10.240 --> 01:13:18.840
Um the the last thing that you you don't see in this one, but you'll see a little bit later, is once I find those interesting examples based on the signal that was generated,

s773
01:13:18.840 --> 01:13:20.480
I now want to go to this.

s774
01:13:20.480 --> 01:13:22.280
I want to investigate that trace.

s775
01:13:22.280 --> 01:13:24.320
I want to click into that span.

s776
01:13:24.320 --> 01:13:25.480
I want to look at the input.

s777
01:13:25.480 --> 01:13:26.960
I want to look at the output.

s778
01:13:26.960 --> 01:13:32.280
Right, this is the kind of analysis that I would probably do, but I'm not able to do at scale.

s779
01:13:32.280 --> 01:13:44.200
But if I can offload some of that again to loop to inform what I might go change or might might go fix or might go add, or I could offload to my coding agent, this is the kind of thing that actually I think drives agent quality

s780
01:13:44.200 --> 01:13:47.080
in a really efficient way.

s781
01:13:52.080 --> 01:14:03.800
Um so, I'm going to skip through some of this, but sort of what I described here is using topics and scores to understand and and to like actually parse through that that data based on the signal.

s782
01:14:03.800 --> 01:14:09.240
Um what you might also do now is you might actually use that that filtered set of data.

s783
01:14:09.240 --> 01:14:13.440
You actually found an interesting example, and you want to go add that to a data set, right?

s784
01:14:13.440 --> 01:14:16.240
The thing that allows us to now go improve that agent.

s785
01:14:16.240 --> 01:14:20.120
I want to go bring that example back to my offline evals.

s786
01:14:20.120 --> 01:14:21.640
Right, I could do that here from the UI.

s787
01:14:21.640 --> 01:14:23.400
I'm going to add this to my data set.

s788
01:14:23.400 --> 01:14:30.920
Now I have this sort of edge case, this thing that I need to go and sort of improve upon or a new feature I need to add based on what I've seen

s789
01:14:30.920 --> 01:14:33.880
with this user's interaction of my with my agent.

s790
01:14:33.880 --> 01:14:35.400
I can go add that back to my data set.

s791
01:14:35.400 --> 01:14:45.360
But again, like the thing that becomes important here is adding the right thing back to the data set based on the signal that was created within within our logs.

s792
01:14:48.400 --> 01:14:53.480
And then of course, you'd go and you you guys don't want me to see me code and and figure out if I can improve the agent.

s793
01:14:53.480 --> 01:14:56.440
But once I once I did, I'd go run my eval again.

s794
01:14:56.440 --> 01:14:59.400
I'd go compare that to the last time I ran the eval.

s795
01:14:59.400 --> 01:15:00.040
Did I improve?

s796
01:15:00.040 --> 01:15:00.680
Did I regress?

s797
01:15:00.680 --> 01:15:04.600
That new example that I pulled in, did I make it better?

s798
01:15:04.600 --> 01:15:08.360
Did I actually improve those scores that I ran in an online sense?

s799
01:15:08.360 --> 01:15:14.720
The other thing that you might want to do is like you actually recognized these new edge cases, you probably want to go create a score.

s800
01:15:14.720 --> 01:15:14.880
Right?

s801
01:15:14.880 --> 01:15:16.680
I have a record lookup failure.

s802
01:15:16.680 --> 01:15:20.320
That that seemed to be the the problem that I encountered the most.

s803
01:15:20.320 --> 01:15:27.360
I probably want to go create a score that allows me to capture that before go before pushing my my agent to production.

s804
01:15:27.360 --> 01:15:29.240
So, there are all of these steps, right?

s805
01:15:29.240 --> 01:15:33.280
These things that you would probably want to do to create that flywheel.

s806
01:15:33.280 --> 01:15:38.560
And what I think becomes really powerful here is when you can run this in an automated sense.

s807
01:15:38.560 --> 01:15:47.640
When I can plug a skill or I can plug some type of workflow using all of the primitives that I just described to automate all of that.

s808
01:15:47.640 --> 01:15:55.840
And so if you look here at at our workshop markdown file, there is a GitHub repo for skills that that we have out there.

s809
01:15:55.840 --> 01:15:58.520
There's actually a single skill at the moment.

s810
01:15:58.520 --> 01:16:04.600
The idea is to give your coding agents the ability to run that automated flywheel.

s811
01:16:04.600 --> 01:16:04.840
Right?

s812
01:16:04.840 --> 01:16:07.080
We want to actually go through that exact same thing.

s813
01:16:07.080 --> 01:16:11.160
I want I want Codex or Cloud Code or whatever to write SQL.

s814
01:16:11.160 --> 01:16:12.680
I want it to view traces.

s815
01:16:12.680 --> 01:16:14.040
I want it to view spans.

s816
01:16:14.040 --> 01:16:22.680
I want it to create new scores and run evals and modify something within my agent and I want to do that, you know, watching my my screen.

s817
01:16:22.680 --> 01:16:25.160
Yeah, go for it.

s818
01:16:34.320 --> 01:16:57.840
Yeah, the the So, if you look at my If you look at my eval that I've created So, this is this is sort of what an eval looks like in in Braintrust.

s819
01:16:57.840 --> 01:17:09.440
Um your data set here is actually part of like the the commands that we ran at the outset was to load our sample data, like our series of test cases into Braintrust.

s820
01:17:09.440 --> 01:17:16.520
But but you could also imagine like this is at the outset, this is like my sort of scope down view of the world in which my agent operates, right?

s821
01:17:16.520 --> 01:17:23.000
These are the test cases that I want to give to it to understand, "Hey, should I go push this thing to production?"

s822
01:17:23.000 --> 01:17:33.520
But the the sort of like operation of the thing that I'm describing here is that the where you start to improve the quality the most is is when you can curate

s823
01:17:33.520 --> 01:17:39.760
traces, curate examples from production, and bring those back to your your data set.

s824
01:17:39.760 --> 01:17:46.480
So, this right here, like this sort of like very sim- simplistic workflow, is what I was describing.

s825
01:17:46.480 --> 01:17:52.840
This pin this um this record here, this row, this trace, it perhaps is indicative of some failure mode.

s826
01:17:52.840 --> 01:18:00.760
And I want to bring this back to either an existing data set or a new data set that I want to run an an eval against.

s827
01:18:02.600 --> 01:18:04.600
Correct.

s828
01:18:10.600 --> 01:18:20.320
Um so, I think like the sort of culmination of of all of this is uh how we can uh use all of those those sort of primitives that I described earlier.

s829
01:18:20.320 --> 01:18:26.600
I think the topics pipeline uh running behind the scenes is is kind of the the star of the show.

s830
01:18:26.600 --> 01:18:35.800
Um the the the importance of adding to the the data set though is your your evals should evolve with your agent.

s831
01:18:35.800 --> 01:18:39.360
And when when I think about evals, I think about a data set.

s832
01:18:39.360 --> 01:18:41.240
I think about my agent, right?

s833
01:18:41.240 --> 01:18:43.520
So, it's like function that invokes my agent.

s834
01:18:43.520 --> 01:18:52.360
And then I think of the series of scores that allow me to understand good or bad with respect to those test cases being passed to my agent.

s835
01:18:52.360 --> 01:18:59.200
But, when I create my eval set, it's a very sort of like it's a way for me to establish a baseline.

s836
01:18:59.200 --> 01:19:03.600
It is certainly not representative entirely of what users can do with my agent.

s837
01:19:03.600 --> 01:19:18.040
And so, by me adding new records to that data set, it allows me to capture different perhaps failure modes over time and go make changes within my agent so that, right?

s838
01:19:18.040 --> 01:19:21.920
We don't we don't continually experience those those issues in the future.

s839
01:19:21.920 --> 01:19:30.800
But, it's it's more about like the the evolution of your agent and ensuring that all of those sort of like things that come along with your evals evolve with it.

s840
01:19:30.800 --> 01:19:42.640
Um so, that you can one capture new things, but also like as you go change your code, you're not regressing based you're not regressing your agent based on previous interactions that you sort of curated into that data set.

s841
01:19:42.640 --> 01:19:47.000
Um there's there's a few different ways to uh install this.

s842
01:19:47.000 --> 01:19:49.415
Uh I'm going to choose just the top one.

s843
01:19:49.415 --> 01:19:50.720
[clears throat]

s844
01:19:50.720 --> 01:19:55.840
Uh so, if you have the the GitHub CLI installed, you can use that.

s845
01:19:55.840 --> 01:19:58.120
Uh there's also uh NPX.

s846
01:19:58.120 --> 01:20:07.080
And there's just cloning the the repo locally and and following the different install so, like with respect to the agents that you're using.

s847
01:20:16.880 --> 01:20:17.800
Okay.

s848
01:20:17.800 --> 01:20:19.440
So, we we have the the skill file.

s849
01:20:19.440 --> 01:20:21.120
We've installed that.

s850
01:20:21.120 --> 01:20:24.000
Um so, now I might want to just like run this this flywheel.

s851
01:20:24.000 --> 01:20:25.440
I want to see this in action.

s852
01:20:25.440 --> 01:20:28.160
So, I'm going to I'm going to do two different things here.

s853
01:20:28.160 --> 01:20:30.920
I'm going to run this.

s854
01:20:30.920 --> 01:20:37.160
And then I'll actually show you um a previous example so you're not just sitting here watching Codex run.

s855
01:20:37.160 --> 01:20:39.720
So, we'll do that.

s856
01:20:39.720 --> 01:20:40.760
We'll let that run.

s857
01:20:40.760 --> 01:20:51.480
I'm going to go out to another repo.

s858
01:20:53.840 --> 01:20:54.480
Okay.

s859
01:20:54.480 --> 01:20:56.640
So, same same prompt.

s860
01:20:56.640 --> 01:21:03.160
Um, and actually let me, I think I sort of skipped over one thing here that I think is probably important to highlight.

s861
01:21:03.160 --> 01:21:08.880
Right, the actual, uh, logs here that were generated or like the topics that were that were generated.

s862
01:21:08.880 --> 01:21:11.200
So, we should now we have all this signal, right?

s863
01:21:11.200 --> 01:21:15.680
This is the thing that I just, um, executed Codex against this project.

s864
01:21:15.680 --> 01:21:18.760
It should now be able to pull back all of these different things, right?

s865
01:21:18.760 --> 01:21:23.120
these tasks, these sentiments, uh, this support workflow issue.

s866
01:21:23.120 --> 01:21:29.320
All of this and these scores are attached to those traces and are now part of this workflow.

s867
01:21:29.880 --> 01:21:33.920
So, we'll let this kind of go, uh, in in this in this terminal over here.

s868
01:21:33.920 --> 01:21:36.600
All right, we'll we'll come to this one as it's running.

s869
01:21:36.600 --> 01:21:46.360
So, sort of like what you saw in in the loop session, uh, what you'll start to see here is, uh, Codex writing SQL commands.

s870
01:21:46.360 --> 01:21:54.360
So, BT SQL and we're going to pull back all of that information or the relevant information into the session.

s871
01:21:54.360 --> 01:22:04.560
And so, we can flexibly query across this project now and across these traces and we should be able to also recognize and understand or Codex should be able to recognize and understand

s872
01:22:04.560 --> 01:22:08.920
those topics and those, uh, those scores that were run.

s873
01:22:08.920 --> 01:22:14.600
Um, so we have, you know, you could see here we have 968 support traces, that's exactly what we loaded in.

s874
01:22:14.600 --> 01:22:20.400
There are scored failures, a visible cluster of escalate to human tool errors.

s875
01:22:20.400 --> 01:22:26.440
Um, I'm going to inspect the classifier outputs and representative failed traces and turn that into a concrete summary.

s876
01:22:26.440 --> 01:22:32.760
So again, the ability to flexibly query with that SQL tool uh becomes pretty impressive.

s877
01:22:32.760 --> 01:22:36.680
Uh looks like it was able to recognize something with our scores.

s878
01:22:36.680 --> 01:22:38.800
Uh and then is, you know, continually querying.

s879
01:22:38.800 --> 01:22:43.760
And then this is what I described earlier of actually being able to drill into the individual trace.

s880
01:22:43.760 --> 01:22:44.600
So, it found something.

s881
01:22:44.600 --> 01:22:51.200
It found some example that would allow us to um improve our agent again in some measurable way.

s882
01:22:51.200 --> 01:23:01.320
We want to view that trace, write some more SQL, um but this is really what allows us to be very comprehensive and and actually pulling in all of that really rich data,

s883
01:23:01.320 --> 01:23:08.840
uh but only pulling in those things that are actually going to drive uh some some change that we can make within our agent.

s884
01:23:10.440 --> 01:23:12.760
Okay, so let's see what what else we just did here.

s885
01:23:12.760 --> 01:23:15.560
Um looks like we're looking at how to run evals.

s886
01:23:15.560 --> 01:23:17.840
We are viewing existing data sets.

s887
01:23:17.840 --> 01:23:20.320
Uh we just created a a new data set.

s888
01:23:20.320 --> 01:23:21.880
Actually, we're in the help command.

s889
01:23:21.880 --> 01:23:30.160
Um but but hopefully what you're able to see here is this.

s890
01:23:32.080 --> 01:23:34.440
This is what we're trying to create.

s891
01:23:34.440 --> 01:23:38.920
This is what the This is what like the the primitives in uh in Braintrust.

s892
01:23:38.920 --> 01:23:48.840
Our topics, our Braintrust CLI, the backend allowing you to write SQL, uh having a platform that does both evals and observability.

s893
01:23:48.840 --> 01:23:57.760
Um all of these things sort of taken together allow you to again create that that flywheel and then do so in a in a really meaningful way.

s894
01:23:57.760 --> 01:24:01.000
Um more than happy to take some questions right now.

s895
01:24:01.000 --> 01:24:10.640
What's it's really just waiting a little bit on Codex to to cook, but curious if if anybody has Yeah, go for it.

s896
01:24:15.200 --> 01:24:16.960
Yeah, we have a hybrid deployment.

s897
01:24:16.960 --> 01:24:31.840
So, it means the data plane, so where all of the data resides, so your traces, your experiments, your data sets, they would remain in your infrastructure, but the control plane, right, just like the web app that you're looking at, uh, would stay with us.

s898
01:24:37.240 --> 01:24:40.480
The The hybrid deployment is only available for the enterprise plan.

s899
01:24:40.480 --> 01:24:41.120
Yeah.

s900
01:24:41.120 --> 01:24:44.640
Um, to that point, just if you're curious, like there's two different flavors of that.

s901
01:24:44.640 --> 01:24:51.200
There is a BYOC one, where, again, data plane is deployed in your infrastructure, but we manage it for you.

s902
01:24:51.200 --> 01:24:55.840
Then there is more hybrid, deployed in your data plane, your team manages it.

s903
01:24:55.840 --> 01:25:03.760
We have sort of a a Terraform provider or Helm charts based on, uh, the the the cloud that you're on.

s904
01:25:03.760 --> 01:25:04.640
Back of the room?

s905
01:25:04.640 --> 01:25:07.040
You don't have to yell.

s906
01:25:26.760 --> 01:25:36.840
Um, I I think for the for the most part, and I think this is changing a little bit, uh, folks are using more of the, uh, the models from the the big model providers.

s907
01:25:36.840 --> 01:25:38.880
Uh, that's generally what I see.

s908
01:25:38.880 --> 01:25:42.080
Um, I I think that's that's starting to shift a little bit.

s909
01:25:42.080 --> 01:25:49.160
Um, I know like at the outset, it it was very very heavily skewed towards the the major model providers.

s910
01:25:49.160 --> 01:26:01.120
Um, I don't know like specifically, I haven't seen small models necessarily, but I have seen folks start to, um, perhaps fine-tune their own models and and see if it can't drive,

s911
01:26:01.120 --> 01:26:13.040
you know, the whole idea, again, like beyond behind it, like evals, as as one example, is to be able to understand if I can swap in and out models without sort of introducing regressions or, uh,

s912
01:26:13.040 --> 01:26:15.400
deteriorating quality of the agent.

s913
01:26:15.400 --> 01:26:22.440
But, yeah, I'd say largely it's it's the the major model providers, perhaps maybe shifting a little bit.

s914
01:26:35.720 --> 01:26:38.920
Sorry, is this like for an LLM judge?

s915
01:26:40.680 --> 01:26:42.680
Oh.

s916
01:26:48.360 --> 01:27:04.040
Yeah, so uh important callout, um could actually see it probably back at at this screen when you're looking at the uh topics automation settings and advanced.

s917
01:27:04.200 --> 01:27:11.320
The The The only option you have for both the summarization and vector embedding step are Braintrust hosted models.

s918
01:27:11.320 --> 01:27:13.760
Um the there's a couple reasons for that.

s919
01:27:13.760 --> 01:27:18.440
One is that we we've trained trained these models to do specifically this.

s920
01:27:18.440 --> 01:27:31.040
And and two, like the the complexity of the pipeline sort of um makes it so that you can't just necessarily plug in any model and and and hope to achieve similar results.

s921
01:27:31.040 --> 01:27:41.040
Uh we started off by allowing different models to be plugged into that, but the the topics pipeline or the topics that were generated as a result of that just were not

s922
01:27:41.040 --> 01:27:41.640
were not good.

s923
01:27:41.640 --> 01:27:48.120
And so, we have enforced it so it's only Braintrust hosted models for this step.

s924
01:27:56.600 --> 01:27:58.080
Yes, you can.

s925
01:27:58.080 --> 01:28:16.640
Um I can go out to Uh so, because all of that uh all of that data exists as metadata essentially on top of the trace.

s926
01:28:16.640 --> 01:28:21.080
You can pull it out for different uh visualizations.

s927
01:28:21.320 --> 01:28:24.760
So, I think in this case, I have a saved view.

s928
01:28:24.760 --> 01:28:28.160
Um and just, you know, some some brain trust stuff here.

s929
01:28:28.160 --> 01:28:34.920
Uh if you wanted to create your own sort of dashboard or your own view, uh this is how you would do that.

s930
01:28:34.920 --> 01:28:42.600
Uh all of these views are like uh you can save it, you can create your own custom charts, but uh up here at the top, you can see sort of sentiment over time,

s931
01:28:42.600 --> 01:28:50.600
tasks over time, top sentiments, top tasks, like the things that you've built custom uh can be pulled out in here.

s932
01:28:50.600 --> 01:28:57.400
Uh you can see that like in this case, I'm looking for classifications.sentiment is not null, and I'm grouping by that label.

s933
01:28:57.400 --> 01:29:00.000
So, these these become pretty simple to to set up.

s934
01:29:00.000 --> 01:29:02.320
Yeah, I think so this again, grain of salt.

s935
01:29:02.320 --> 01:29:10.560
Uh generally speaking, what I've seen though is like uh newer agents or newer things that are just being pushed to production have a higher sampling rate.

s936
01:29:10.560 --> 01:29:20.560
And over time, those sort of like known unknowns or the things that your scores tend to surface become less and less, and the signal that you derive from it will will also become less.

s937
01:29:20.560 --> 01:29:23.800
And so, you start to scale back that sampling rate.

s938
01:29:23.800 --> 01:29:32.960
Uh I I think there's also if you're doing things with an LLM judge that is a little bit more on the classification side or on the taxonomy side, you can look to

s939
01:29:32.960 --> 01:29:36.440
alter or you can look to swap that out with topics.

s940
01:29:36.440 --> 01:29:50.720
Um Dropbox is a customer a customer that I work with, and they have this like very complex series of classification scores that they run to understand some taxonomy uh with respect to their traces.

s941
01:29:50.720 --> 01:29:54.760
And they have experimented with uh using topics to do that.

s942
01:29:54.760 --> 01:30:06.520
Um one because actually the most important reason for that is is it's a much more cost-effective way to run across the majority or 100% of their traces, where the LLM judge just was not able to scale

s943
01:30:06.520 --> 01:30:06.960
to that.

s944
01:30:06.960 --> 01:30:08.520
You're you're able to generate as much signal.

s945
01:30:08.520 --> 01:30:15.600
So, I think there is to maybe to summarize, there is a maturity that I think informs what the sampling rate should look like.

s946
01:30:15.600 --> 01:30:19.560
It tends to scale back as the maturity of that agent increases.

s947
01:30:19.560 --> 01:30:24.400
And then there is the types of things that the that your LLM judge might be producing.

s948
01:30:24.400 --> 01:30:31.320
If it's a little bit more on that classification or taxonomy side, you can look to swap that out with with topics.

s949
01:30:31.320 --> 01:30:35.160
Yeah, so not directly within the UI, but you could create a code base score.

s950
01:30:35.160 --> 01:30:35.400
Right?

s951
01:30:35.400 --> 01:30:39.080
You could imagine like I invoke score one.

s952
01:30:39.080 --> 01:30:42.600
Based on score one's output, I go do something and I invoke score two.

s953
01:30:42.600 --> 01:30:47.440
The return of that function is an array of scores.

s954
01:30:47.440 --> 01:30:47.600
Right?

s955
01:30:47.600 --> 01:30:49.040
It's an array of score objects.

s956
01:30:49.040 --> 01:30:51.960
But it's yeah, it's certainly possible, but you define it within code.

s957
01:30:51.960 --> 01:31:00.720
There's nothing within Braintrust that allows you to say, "Hey, I want to run score A based on the result of that, I want to wire it into score B or C."

s958
01:31:00.720 --> 01:31:04.760
But it is certainly possible just to create it within code.

s959
01:31:12.440 --> 01:31:12.920
Oh, yeah.

s960
01:31:12.920 --> 01:31:23.080
You could you could also just um your filter could look specifically for the presence of that score span and then only be invoked when when it when it's there.

s961
01:31:23.080 --> 01:31:25.080
Yeah.

s962
01:31:41.000 --> 01:31:46.920
Um I forget the the customer, but this was surfaced the other day and it was more on the on the task side.

s963
01:31:46.920 --> 01:31:54.720
Um I don't remember specifically what it was, but it was actually something that they didn't realize that users were doing with their agent.

s964
01:31:54.720 --> 01:32:06.240
So, it was more of the use case of like uh feature request or um you know, unknown unknowns, but with respect to what users were doing with their agent, not necessarily a failure mode.

s965
01:32:06.240 --> 01:32:13.760
And I think that's been the perhaps the most surprising thing for me is that people are using it to uncover those types of things.

s966
01:32:13.760 --> 01:32:25.000
I think when I was starting to like play around with the feature and talk to customers, it was you know, the the majority of it was was for me at least focused around the types of issues that you would be able to uncover,

s967
01:32:25.000 --> 01:32:35.160
but I think there's just like such a broad range of patterns that you're now able to sort of uh attach to your traces and then use those sort of patterns together

s968
01:32:35.160 --> 01:32:39.200
to inform something about like the the world that you didn't know about your agent.

s969
01:32:39.200 --> 01:32:50.920
Um so, yeah, it's I don't have the exact uh thing that the uh topics uncover, but it was much more around like, "Hey, I didn't realize this uh these users were using our agents or trying to use

s970
01:32:50.920 --> 01:32:53.440
my agent for this."

s971
01:32:58.320 --> 01:33:05.680
Um right now, I would say they're they're like use case driven or you could even imagine uh agent specific.

s972
01:33:05.680 --> 01:33:14.920
So, in my case, I have this support agent and in my project, I am writing my logs for my support agent into my support agent project.

s973
01:33:14.920 --> 01:33:19.720
I'm also storing the data sets that I'm running my evals for my support agent.

s974
01:33:19.720 --> 01:33:23.560
I'm also storing the scores for that support agent in that project.

s975
01:33:23.560 --> 01:33:32.800
So, they tend to be a uh a a container for relevant resources for a use case use case or an agent.

s976
01:33:32.800 --> 01:33:42.920
Uh we are introducing very soon a a sort of uh first-class agent object into the project and so, you could imagine maybe there is a multi-agent system and you want

s977
01:33:42.920 --> 01:33:52.840
all of the agents' logs to go into the same project, but still have some delineation between agent A, B, and C. Uh so, you could imagine doing that as well.

s978
01:34:03.800 --> 01:34:10.000
Um yeah, so the the sort of monitor page that I showed earlier is uh project scoped.

s979
01:34:10.000 --> 01:34:18.600
Um if you wanted to create a more like organization scoped view, you'd have to uh create queries and pull data out of BrainTrust.

s980
01:34:18.600 --> 01:34:20.520
Um that'll eventually change.

s981
01:34:20.520 --> 01:34:25.600
Uh it's more How do we How do we allow our customers to do this?

s982
01:34:25.600 --> 01:34:26.480
at scale.

s983
01:34:26.480 --> 01:34:35.680
Um you could imagine trying to query across however many different projects and pulling that data into a dashboard uh becomes a little bit challenging.

s984
01:34:35.680 --> 01:34:42.960
So, yeah, at the moment we are uh project scoped, but you could imagine that that that's starting to change over time.

s985
01:34:42.960 --> 01:34:44.920
I mean, I think that's probably fair.

s986
01:34:44.920 --> 01:34:55.560
Um I I think maybe what you'd potentially see is your your evals changing um as often as you are changing the agent.

s987
01:34:55.560 --> 01:35:06.440
Like and it's probably perhaps it isn't even that frequent, um but I do think it does have the prospect to change in some way based on what you're able to infer from users interacting with that agent, right?

s988
01:35:06.440 --> 01:35:08.520
Like this could be This could be a prompt.

s989
01:35:08.520 --> 01:35:09.520
This could be a tool.

s990
01:35:09.520 --> 01:35:10.480
This could be a new tool.

s991
01:35:10.480 --> 01:35:16.200
This could be um something within the runtime or the architecture that that needs to change.

s992
01:35:16.200 --> 01:35:30.160
And to me, like what I've seen more frequently is that your evals will generally change as a result of something that you are change Like obviously if it's a simple prompt change, like you're probably not changing a lot within your evals outside of maybe

s993
01:35:30.160 --> 01:35:33.200
curating some new examples into that data set.

s994
01:35:33.200 --> 01:35:46.360
Um but but I do think what I've seen is that agents tend to evolve um somewhat frequently, and I think if you are being informed by actual interactions with uh with users in a real way,

s995
01:35:46.360 --> 01:35:49.640
maybe they're actually changing a little bit more than you'd expect.

s996
01:35:49.640 --> 01:35:56.280
Um but yeah, I I do think it's a fair, sort of, uh piece of feedback where uh the agent isn't very complex.

s997
01:35:56.280 --> 01:36:03.720
It doesn't have, or it's not, you know, it doesn't necessitate a lot of changes based on the scope of of what the agent is doing.

s998
01:36:03.720 --> 01:36:07.640
And so, the evals may not change too much as a result of it.

s999
01:36:07.640 --> 01:36:15.640
Uh I do still think there is value in being able to generate that signal in production of of what's going on.

s1000
01:36:15.640 --> 01:36:23.640
Um especially if you can do it in a cost-effective way, then then, yeah, it becomes like that part of it I still think there's there's value to.

s1001
01:36:23.640 --> 01:36:25.920
Yeah, like I I have customers that do both.

s1002
01:36:25.920 --> 01:36:29.040
Um I I I think some of it's preference.

s1003
01:36:29.040 --> 01:36:33.160
Some of it um sometimes is just not understanding what Braintrust can do.

s1004
01:36:33.160 --> 01:36:46.240
Um allowing So, like the the codebase scores, I think, is a really good example of this, where you may have a really complex codebase score that uh uses different Python dependencies or TypeScript dependencies,

s1005
01:36:46.240 --> 01:36:54.560
and you need to run that, or you think you need to run that in your own infrastructure, because you have all of these, sort of, like, complex dependencies that are powering it.

s1006
01:36:54.560 --> 01:37:04.160
What you can do within Braintrust, when you when you go push a score, and even when you push a codebase score, you can bundle dependencies within that push.

s1007
01:37:04.160 --> 01:37:08.480
And so, the things that that score relies upon can still use those.

s1008
01:37:08.480 --> 01:37:12.920
And I know some customers that I've worked with just didn't understand that they could they could do that.

s1009
01:37:12.920 --> 01:37:25.720
Uh to me, the value is, like, not having some of that, like, asynchronous code within your own codebase that um looks for those triggers, or is like it looks for, "Hey, this trace was

s1010
01:37:25.720 --> 01:37:30.720
generated, and I need to wait 60 seconds from the last event to go, sort of, be invoked."

s1011
01:37:30.720 --> 01:37:35.360
And so, to me, the value is, like, offloading some of that complexity into Braintrust.

s1012
01:37:35.360 --> 01:37:37.520
Though, you still can do that.

s1013
01:37:37.520 --> 01:37:48.920
Um one of the, I think, unique things about, like, the Braintrust data model is that you can actually update spans in in So, if that span already exists within BrainTrust,

s1014
01:37:48.920 --> 01:37:49.840
it is mutable.

s1015
01:37:49.840 --> 01:37:56.200
And so, you could add you can you know, change the metadata, attach scores, attach different metrics.

s1016
01:37:56.200 --> 01:38:06.320
And so, if you wanted to do that in code, you had sort of like the um you know, the machinery in place on your side, it it becomes pretty trivial to go and like, "Hey, I have this span ID.

s1017
01:38:06.320 --> 01:38:09.640
I want to go attach these different scores and metrics to it."

s1018
01:38:09.640 --> 01:38:12.120
So, I think some of it is just like it's preference.

s1019
01:38:12.120 --> 01:38:16.320
Um some people just want to have a little bit more control, perfectly fine.

s1020
01:38:16.320 --> 01:38:17.840
Uh you can certainly do that.

s1021
01:38:17.840 --> 01:38:23.040
So, to me it's a one it's our back probably.

s1022
01:38:23.040 --> 01:38:29.200
Like you you still want to like be able to write some of that trace data and then lock down who can see it.

s1023
01:38:29.200 --> 01:38:38.680
I think that workflow that I just described is is a like that flywheel, that automated flywheel, is actually a great way to do that because I can surface insights to people.

s1024
01:38:38.680 --> 01:38:44.480
I can generate a PR based off of all of those insights, but I don't give them access to the underlying traces.

s1025
01:38:44.480 --> 01:38:47.720
I give them access to the analysis of those traces.

s1026
01:38:47.720 --> 01:38:49.680
Here are the SQL queries that I ran.

s1027
01:38:49.680 --> 01:38:52.280
Here is the aggregate data that I pulled back.

s1028
01:38:52.280 --> 01:39:02.360
Um if you have just like no engine engineers are able to actually view that data, write it to a project that nobody has access to, um lock it down from that perspective,

s1029
01:39:02.360 --> 01:39:11.720
but then create sort of a service account that is actually able to like go run that sort of flywheel on top of it, so you don't lose access to all of those insights.

s1030
01:39:12.000 --> 01:39:14.000
Yep.

s1031
01:39:19.640 --> 01:39:21.560
Most underrated features.

s1032
01:39:21.560 --> 01:39:28.840
Um My guess is the BrainTrust CLI is very properly rated.

s1033
01:39:28.840 --> 01:39:35.160
I I think I I I'd tout that uh any chance that I can get or I uh tell any engineer.

s1034
01:39:35.160 --> 01:39:39.600
The reason I say that is because I think we we released it in late February.

s1035
01:39:39.600 --> 01:39:41.480
We had a user conference.

s1036
01:39:41.480 --> 01:39:47.000
Um and I still think like I'm going out to customers today and I'm asking like are you using the Braintrust CLI CLI?

s1037
01:39:47.000 --> 01:39:48.000
Are you using it?

s1038
01:39:48.000 --> 01:39:49.400
And some of them still aren't.

s1039
01:39:49.400 --> 01:39:55.560
Um, but to me, most like I'm I would wager 100% of you are using some coding agent, right?

s1040
01:39:55.560 --> 01:39:57.680
Like this is how you are coding today.

s1041
01:39:57.680 --> 01:40:06.240
Um, having the CLI as part of that, like I I hope you were able to see some of the like types of workflows that you were able to create.

s1042
01:40:06.240 --> 01:40:09.520
Um, to me it just is it becomes so incredibly compelling.

s1043
01:40:09.520 --> 01:40:13.280
You're already in a terminal, you're already in your IDE, whatever it is.

s1044
01:40:13.280 --> 01:40:19.240
Uh, use the CLI, attach that or or give your coding agent the skills to understand how to use that.

s1045
01:40:19.240 --> 01:40:21.280
And then use that to augment your workflow.

s1046
01:40:21.280 --> 01:40:35.520
Um, the other one that the other one that could be like somewhat compelling, it depends I think a little bit on your your organization and the different folks contributing to that flywheel.

s1047
01:40:35.520 --> 01:40:38.080
Uh, we have a feature called remote eval.

s1048
01:40:38.080 --> 01:40:41.880
It's a way for you to expose that eval to a playground.

s1049
01:40:41.880 --> 01:40:51.520
And this is more like how can I perhaps uh, like in this scenario, there are perhaps subject matter experts, there are product managers, there are folks sort of like

s1050
01:40:51.520 --> 01:40:53.840
contributing to that flywheel in some way.

s1051
01:40:53.840 --> 01:40:57.320
Like they are modifying prompts or they are doing some of that hill climbing.

s1052
01:40:57.320 --> 01:41:05.920
And we sort of crafted like all of this um, you know, the these applications for them to go plug into in a low code low code type of way.

s1053
01:41:05.920 --> 01:41:15.480
The remote eval allows you to expose that eval that I showed to you earlier that like eval with a task and a data set, you can expose that to a playground

s1054
01:41:15.480 --> 01:41:22.960
with parameters so that the user of the playground can actually go and start changing the system prompt or a tool description or something like that.

s1055
01:41:22.960 --> 01:41:37.080
Like why that might be interesting for an AI engineer is that you don't necessarily have to be sort of like the bottleneck in producing applications or uh, producing something for a product manager to go plug into and allow them to

s1056
01:41:37.080 --> 01:41:39.200
modify a prompt or contribute to that flywheel.

s1057
01:41:39.200 --> 01:41:45.800
So, maybe not as not as impactful as the CLI, but potentially given your organizational structure could be.

s1058
01:41:45.800 --> 01:41:47.160
I think I got most of that.

s1059
01:41:47.160 --> 01:41:49.160
Tell me where where I'm off.

s1060
01:41:49.160 --> 01:41:55.120
I I think like the the thing that I that I started doing here is exactly that.

s1061
01:41:55.120 --> 01:41:59.960
Uh like Braintrust itself, at least at the moment, doesn't have access to your code base, right?

s1062
01:41:59.960 --> 01:42:10.040
It doesn't understand like what you've written and based on the information it has, it can't go recommend a change necessarily to specific functions or agent architecture.

s1063
01:42:10.040 --> 01:42:21.280
But when you bring the um Braintrust CLI into your sort of coding agent, and then attach that obviously like I'm I'm within my repo here, I'm able to now go through let's see if we've been able to

s1064
01:42:21.280 --> 01:42:22.120
like get something here.

s1065
01:42:22.120 --> 01:42:30.480
So, in my session, uh based on what I've been able to sort of understand, we are creating new new scores.

s1066
01:42:30.480 --> 01:42:32.640
So, there's a score change.

s1067
01:42:32.640 --> 01:42:36.080
Um there's we can skip the tests.

s1068
01:42:37.160 --> 01:42:45.160
There is uh looks like newer tools that we were able to add to our agent again based on some of that insight.

s1069
01:42:46.080 --> 01:42:51.280
So, I I think like the the answer to your question is like this is absolutely possible.

s1070
01:42:51.280 --> 01:43:04.880
What what I've just sort of demonstrated here is that flywheel of like, "Hey, figure out what's going on in production, use my scores or use my topics because I have sort of attached

s1071
01:43:04.880 --> 01:43:18.160
information of you know, issues or sentiment or whatever it is, this becomes incredibly compelling now or interesting for that agent to pull in, and then go modify your agent in some way.

s1072
01:43:18.160 --> 01:43:25.680
Um I think eventually as you get further down here, see now we have new examples to our data set.

s1073
01:43:25.680 --> 01:43:30.120
We have uh looks like we've created a new data set with those examples.

s1074
01:43:30.120 --> 01:43:34.560
We're now going to go run our eval probably a little bit further down here.

s1075
01:43:34.560 --> 01:43:39.840
Uh you can also see like we are querying directly from that experiment that was run.

s1076
01:43:39.840 --> 01:43:47.400
So, that whole sort of flywheel is captured here within um within this agent improvement skill.

s1077
01:43:47.400 --> 01:43:49.680
And I think hopefully I answered your question to some degree.

s1078
01:43:49.680 --> 01:43:50.600
Cool.

s1079
01:43:50.600 --> 01:43:54.600
Um this to me is like one of the most compelling parts of Braintrust right now.

s1080
01:43:54.600 --> 01:43:59.440
Uh we're all sort of developing our agents such to some degree with the help of a coding agent.

s1081
01:43:59.440 --> 01:44:09.120
If I can bring the right context, if I can have access to the right primitives that attach the right data to my traces, then this this flywheel, this like automated

s1082
01:44:09.120 --> 01:44:12.960
automated flywheel, becomes pretty interesting.

s1083
01:44:30.600 --> 01:44:33.640
So, I I I I absolutely think it's feasible.

s1084
01:44:33.640 --> 01:44:38.760
Um if I could show you I'll show you this example.

s1085
01:44:38.760 --> 01:44:47.320
Um actually, if I come back here So, within the platform right now, there isn't like a button that you hit that says like, "Hey, go do the the flywheel."

s1086
01:44:47.320 --> 01:44:53.480
But what I showed you is that you can very easily plug into this using all of the tools that you already have.

s1087
01:44:53.480 --> 01:44:56.280
The things that you're now able to to do on top of this, right?

s1088
01:44:56.280 --> 01:45:05.240
You could imagine like maybe plugging this into a GitHub action that runs at some particular cadence that that gets triggered based on something and we open a PR.

s1089
01:45:05.240 --> 01:45:08.760
Uh you could also imagine it's like triggered off of a Slack message, right?

s1090
01:45:08.760 --> 01:45:12.080
There are lots of different ways in which I think this could be triggered.

s1091
01:45:12.080 --> 01:45:17.080
Under the hood, all of the things are available for you to go sort of opt into this.

s1092
01:45:17.080 --> 01:45:25.680
Um one example Here's a here's an example of uh a GitHub action that that I that I created.

s1093
01:45:25.680 --> 01:45:30.760
So, under the if you look, there's a a YAML file that actually runs through uh Claude code.

s1094
01:45:30.760 --> 01:45:32.240
It pulls in the Braintrust CLI.

s1095
01:45:32.240 --> 01:45:33.880
It pulls in the skill.

s1096
01:45:33.880 --> 01:45:40.600
But the the sort of again, like the thing that we we need still here is we need to understand in this case like what changed.

s1097
01:45:40.600 --> 01:45:44.640
And this is like my very very trivial supervisor agent uh example.

s1098
01:45:44.640 --> 01:45:46.120
What changed?

s1099
01:45:46.120 --> 01:45:47.480
Why these things changed?

s1100
01:45:47.480 --> 01:45:51.880
Why did Why did uh in this case Claude code recommend these changes?

s1101
01:45:51.880 --> 01:45:53.840
Here is the the actual impact, right?

s1102
01:45:53.840 --> 01:45:55.720
Based on the evals that that I ran.

s1103
01:45:55.720 --> 01:46:00.280
Here's some links out to Braintrust where you can actually go inspect those.

s1104
01:46:00.280 --> 01:46:04.720
Here are these regret regressions that I pulled in based on that analysis.

s1105
01:46:04.720 --> 01:46:06.640
And then here is your PR.

s1106
01:46:06.640 --> 01:46:07.680
You You're the reviewer.

s1107
01:46:07.680 --> 01:46:12.520
Go figure out if this is like a meaningful thing that we should go and and and produce.

s1108
01:46:12.520 --> 01:46:16.440
Um so, I absolutely think this is something that that you can opt into today.

s1109
01:46:16.440 --> 01:46:19.280
Like we have customers that are absolutely doing this.

s1110
01:46:19.280 --> 01:46:27.520
Um one of them that was on a that the first slide that I showed is using this to generate five to 10 more PRs per day than they were before.

s1111
01:46:27.520 --> 01:46:40.000
Um if you have again all of that information, if you have like the uh that really intelligent Codex Claude code or whatever querying very flexibly over your production data and pulling in the things that are that are interesting,

s1112
01:46:40.000 --> 01:46:44.240
I think it becomes pretty compelling from a lot of different use cases.

s1113
01:46:48.000 --> 01:46:49.040
Yeah.

s1114
01:46:49.040 --> 01:46:52.080
Yeah, that that Braintrust skills repo.

s1115
01:46:52.080 --> 01:46:54.560
This is open source.

s1116
01:46:55.440 --> 01:47:02.480
There's a There's a YAML file in here as well that like that went through the actual um like GitHub like it did it within a GitHub action.

s1117
01:47:02.480 --> 01:47:03.720
Open source as well.

s1118
01:47:03.720 --> 01:47:05.720
Yeah.

s1119
01:47:13.320 --> 01:47:15.320
Yeah.

s1120
01:47:22.200 --> 01:47:40.440
Yeah, so what it looks like is Um, so I I I showed earlier how we write evals or one of the ways in which you can write evals with BrainTrust is by using

s1121
01:47:40.440 --> 01:47:42.400
that eval code.

s1122
01:47:42.400 --> 01:47:47.200
So, here's my eval for a different project that I'm taking you outside of.

s1123
01:47:47.200 --> 01:47:52.880
The the one sort of thing that I call out here that's different that you didn't see in the previous code is the parameters.

s1124
01:47:52.880 --> 01:48:00.080
Um, under the hood here I am essentially uh I have exposed different parameters that that touch my sort of agentic system.

s1125
01:48:00.080 --> 01:48:03.160
And in this case it's, you know, something somewhat trivial, right?

s1126
01:48:03.160 --> 01:48:10.640
It's the supervisor agent, but if you look over at the playground itself, um, come over here.

s1127
01:48:10.840 --> 01:48:14.120
So, you have different things that you can pull into BrainTrust, right?

s1128
01:48:14.120 --> 01:48:23.120
The I think, you know, I don't think a lot of people use it very much for prompts anymore outside of like the the very basic stuff, but that remote eval,

s1129
01:48:23.120 --> 01:48:29.480
I can spin up a dev server, so I can be local, I could write um eval my file name {dash} {dash} dev.

s1130
01:48:29.480 --> 01:48:34.160
It spins up a server and it um automatically looks for it on localhost 8300.

s1131
01:48:34.160 --> 01:48:37.280
Um, you could also expose this server within your own infrastructure.

s1132
01:48:37.280 --> 01:48:44.920
You have sort of um created that eval and you have exposed it and then you've also exposed different parameters.

s1133
01:48:44.920 --> 01:48:49.920
So, in this case I've exposed a system prompt and the model for that system prompt.

s1134
01:48:49.920 --> 01:48:55.520
And if I scroll down a little bit further, there's also math agent prompt, model, research agent prompt, and model.

s1135
01:48:55.520 --> 01:49:02.720
But, I can sort of interact with all of that complexity, all of the tools that each one of those agents have directly from the playground.

s1136
01:49:02.720 --> 01:49:14.040
I can simply click run, it'll go through, and all of the code execution remains on the server and then the results of that eval are streamed back into the playground.

s1137
01:49:27.840 --> 01:49:35.240
Yes, Brain Trust will initiate a post request to an eval's endpoint that you have exposed on that server.

s1138
01:49:35.240 --> 01:49:40.080
So there's there's like very I can show you another example here.

s1139
01:49:40.080 --> 01:49:50.640
Um where you can there's a sort of like create app function that you can use or you can sort of strip out the things that you would need to spin up the server.

s1140
01:49:50.640 --> 01:49:57.200
But that that remote eval server right now is is stood up on modal.

s1141
01:49:57.200 --> 01:49:59.240
I'm using this create app.

s1142
01:49:59.240 --> 01:50:05.480
It exposes all of the evals, at least in the way that I've configured it, in my evals directory.

s1143
01:50:05.480 --> 01:50:09.440
And now those are exposed to the playground for users to go and play with.

s1144
01:50:09.440 --> 01:50:17.560
One of the things that we are doing internally right now is we are and it's really driven a lot by interactions that we have with our customers.

s1145
01:50:17.560 --> 01:50:29.360
I think I mentioned this at the outset like people are thinking very heavily right now about how well their their coding agent is actually how well how efficient they are with their coding agents.

s1146
01:50:29.360 --> 01:50:33.680
Um and part of that is actually it's running evals on the skills that they're using.

s1147
01:50:33.680 --> 01:50:38.600
It's understanding the sessions that they have with cloud code and codex and so on.

s1148
01:50:38.600 --> 01:50:52.120
I think they're still I say that with um I say that because we are putting a lot of effort behind the scenes into creating some of the different primitives and machinery that our customers can go like plug into

s1149
01:50:52.120 --> 01:50:56.320
or use to start to go understand those those types of things.

s1150
01:50:56.320 --> 01:51:08.600
Um yeah, just because you see like two commits and and one star there is not indicative of like where we uh like what we perceive to be as a very very important problem

s1151
01:51:08.600 --> 01:51:11.480
uh for engineers today.

s1152
01:51:12.400 --> 01:51:14.240
Yeah.

s1153
01:51:14.240 --> 01:51:15.960
Cool.

s1154
01:51:15.960 --> 01:51:16.360
Awesome.

s1155
01:51:16.360 --> 01:51:16.960
Thanks, everybody.

s1156
01:51:16.960 --> 01:51:20.000
Appreciate you coming out.

s1157
01:51:34.832 --> 01:51:36.832
[music]