WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=T3SS931wU0I

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/T3SS931wU0I.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.680 --> 00:00:14.040
Everyone, I'm Jess.

s3
00:00:14.040 --> 00:00:15.280
I'm from BrainTrust.

s4
00:00:15.280 --> 00:00:26.120
I'm a dev rel engineer at BrainTrust, and in this talk today, I'm going to be teaching you guys what evals are, why they're important, and then I'm going to be talking through a relatively

s5
00:00:26.120 --> 00:00:31.640
complex real-life eval that I did to try to solidify some of the concepts I'm about to teach you.

s6
00:00:31.640 --> 00:00:39.160
Uh before we get started, has Can you just raise your hand if you've heard of evals before or like would confidently say that you know what they are?

s7
00:00:39.160 --> 00:00:40.680
Okay, cool.

s8
00:00:40.680 --> 00:00:41.600
Pretty much everyone here.

s9
00:00:41.600 --> 00:00:42.600
Makes sense.

s10
00:00:42.600 --> 00:00:44.080
Um cool.

s11
00:00:44.080 --> 00:00:48.160
So, I'm just going to start off with a basis of what evals are, why they're important.

s12
00:00:48.160 --> 00:01:02.080
Um I've been on um calls with customers where they've said that they've shipped features because either the engineering team told them that it was ready or maybe a PM said that they tried out a couple of prompts, and they said it looked good.

s13
00:01:02.080 --> 00:01:09.440
Uh and this is what you don't want to hear cuz essentially, these teams are making ship decisions based off of vibes, uh which is not good.

s14
00:01:09.440 --> 00:01:19.960
Uh and essentially, what you want to hear is more something like, "Oh, I ran um 200 different test test cases, and 94% of them passed, and that's why we're shipping this feature."

s15
00:01:19.960 --> 00:01:29.960
Or be able to say nuanced things like, um you know, "I shipped this feature that increased accuracy by 5%, but actually decreased tone by 5%."

s16
00:01:30.160 --> 00:01:38.200
So, when people ask me what evals are, I think the best way to answer it is that it's just a way to answer questions about your AI system.

s17
00:01:38.200 --> 00:01:41.760
So, for example, uh which LLM is the best choice for our needs?

s18
00:01:41.760 --> 00:01:48.640
Especially when there's new models dropping all the time, it's good to be able to have data to back up your decisions.

s19
00:01:48.640 --> 00:01:52.640
Uh how does AI perform across diverse real-world scenarios?

s20
00:01:52.640 --> 00:02:02.440
So, for example, your system could do really well with English outputs, but not Japanese or English inputs but not Japanese or types uh do really well with Python but not TypeScript.

s21
00:02:02.440 --> 00:02:04.600
Um a big one is cost efficiency.

s22
00:02:04.600 --> 00:02:08.280
So, how can we achieve high performance without excessive costs?

s23
00:02:08.280 --> 00:02:10.960
This is definitely becoming more topical now.

s24
00:02:10.960 --> 00:02:12.240
Uh brand consistency.

s25
00:02:12.240 --> 00:02:15.959
So, does AI reliably affect our company's voice and standards?

s26
00:02:15.959 --> 00:02:16.680
Uh feedback.

s27
00:02:16.680 --> 00:02:19.600
Are we learning from users and improving iteratively?

s28
00:02:19.600 --> 00:02:25.280
And then one of the biggest ones is how do we know when something breaks or gets worse?

s29
00:02:25.280 --> 00:02:36.240
So, a real-world scenario that I like to use a lot is uh about a year ago, so April of 2025, um OpenAI actually shipped a model change and model update

s30
00:02:36.240 --> 00:02:45.800
um that essentially made their model more was supposed to make it more helpful but actually made it sycophantic and therefore uh too agreeable and therefore less truthful.

s31
00:02:45.800 --> 00:03:01.800
So, this is a really good example of where evals can help you catch that um especially because uh AI systems are so complex and undeterministic that it's really important to have these eval systems in place at your company or whatever you're trying to build.

s32
00:03:01.800 --> 00:03:06.640
So, we're going to get down into more of the specifics of what evals are comprised of.

s33
00:03:06.640 --> 00:03:10.000
So, evals comprise of four main things.

s34
00:03:10.000 --> 00:03:12.280
Uh the first thing is a data set.

s35
00:03:12.280 --> 00:03:22.760
So, this is the data that you're passing into your AI system and it usually includes things like uh your golden standard uh cases, your edge cases, your failure modes.

s36
00:03:22.760 --> 00:03:26.880
Um then once you have your data set, you want to write a task.

s37
00:03:26.880 --> 00:03:34.880
So, a task defines how you want your AI system to behave when it takes in the input and produces an output.

s38
00:03:34.880 --> 00:03:42.959
So, concretely this usually includes a system prompt and then the model that you're picking to execute this task.

s39
00:03:43.000 --> 00:03:51.120
Then the next thing you need to do is you need to write a scoring system that will tell you whether the AI's output is good or bad.

s40
00:03:51.120 --> 00:03:58.920
Um and so usually it's uh maybe the PM or engineer that is writing the criteria that comprises that goodness and badness.

s41
00:03:58.920 --> 00:04:03.680
Uh, and you might have heard of terms like uh deterministic scoring, LLM as a judge, human in review.

s42
00:04:03.680 --> 00:04:06.959
Those are different types of scoring um frameworks.

s43
00:04:06.959 --> 00:04:09.640
Uh, but yeah, that's so you write your scoring system.

s44
00:04:09.640 --> 00:04:14.120
And then the last thing you do, which is the most fun part of writing evals, is you experiment.

s45
00:04:14.120 --> 00:04:22.560
So, every configuration of a specific data set, a specific task, and a specific scoring system equals to one experiment.

s46
00:04:22.560 --> 00:04:35.240
And as you are testing out different hypotheses for how to make your system better, you're tweaking either your data set, your task, or your scoring system, and these all lead to different experiments.

s47
00:04:35.520 --> 00:04:38.760
So, once you have your different experiments, then you start to compare them.

s48
00:04:38.760 --> 00:04:44.880
So, you might run experiment one and experiment two, and you can see whether there's been improvements or regressions.

s49
00:04:44.880 --> 00:04:50.800
And you can also uh click into specific rows to see exactly which test cases have changed.

s50
00:04:50.800 --> 00:04:54.360
Uh, this screenshot is from the Braintrust UI.

s51
00:04:54.360 --> 00:05:01.320
Another thing that you can do is you can use uh natural language to query across your traces and experiments.

s52
00:05:01.320 --> 00:05:08.560
Um, so for example, you can ask it things like summarize my experiments for me, um highlight the problems in these experiments.

s53
00:05:08.560 --> 00:05:20.960
Um, I find this especially useful when you're running experiment or you're running like um tra- you see traces with like long multi-turn conversations, it's good for um helping you catch things like hallucination

s54
00:05:20.960 --> 00:05:23.720
or drift that's kind of hard to catch manually.

s55
00:05:23.720 --> 00:05:32.680
Um, you can also if you're running different experiments, you can ask it like which performance or which experiments performed the best off of out of all of these experiments.

s56
00:05:32.680 --> 00:05:37.800
Um So, this is another way you can like find patterns in your traces.

s57
00:05:37.800 --> 00:05:46.760
Um, we also this is using like the UI in Braintrust, but we also have a CLI and MCP server if you prefer to do it that way as well.

s58
00:05:46.760 --> 00:05:50.760
Um, and then the last pillar of Braintrust is just like regular observability.

s59
00:05:50.760 --> 00:06:06.880
So, you can basically use the Braintrust SDK to wrap your code um and then uh have your production logs show up in Braintrust and essentially do traditional observability where you can monitor your production logs and create dashboards and things like that.

s60
00:06:06.880 --> 00:06:15.920
Um, so kind of to tie this together, I'm going to walk you through kind of a flow of how you might use observability and evals to improve your AI system.

s61
00:06:15.920 --> 00:06:20.280
So, we're going to start on the top left corner where it says app.

s62
00:06:20.280 --> 00:06:23.080
So, imagine you have some sort of AI application.

s63
00:06:23.080 --> 00:06:34.240
Uh in this scenario, I'll say you have a a new docs page that you just shipped with like an AI chatbot built in so that users can ask questions to your AI chat chatbot about your documentation.

s64
00:06:34.240 --> 00:06:37.000
So, let's say you shipped that yesterday.

s65
00:06:37.000 --> 00:06:40.560
And then you have customers start to use your AI chatbot.

s66
00:06:40.560 --> 00:06:46.440
So, when they start to hit your AI chatbot, you're going to start to get real production logs coming into your system.

s67
00:06:46.440 --> 00:06:55.080
Once you get those production logs, you can uh sample a subset of them, maybe like 10 to 20 rows or something, and create a kind of data set.

s68
00:06:55.080 --> 00:06:58.320
Uh once you get that data set, then we can run evals on that, right?

s69
00:06:58.320 --> 00:07:05.120
So, you can you have your data set, you can write your task, and you can write your scoring system, and you can run uh evals on them.

s70
00:07:05.120 --> 00:07:12.000
And you can start to tweak your data set, you can start to tweak your prompts or you tweak your score and run multiple different evals on that.

s71
00:07:12.000 --> 00:07:26.760
Once you have your evals like run a couple different evals um and start to see which eval does the best, you might learn something about your system, and then you're going to go ahead and make your code change or make a change to whatever AI system that you're using

s72
00:07:26.760 --> 00:07:36.440
uh and then deploy that back into your application, and then the whole flow repeats again because uh those changes will show up in your production logs and the whole system happens again.

s73
00:07:36.440 --> 00:07:40.440
So, that's kind of how the flow works as a whole.

s74
00:07:40.440 --> 00:07:50.600
Um, the last slide I'll show before I get into a concrete example is I love this slide because uh conceptually I I want to hammer into people that evals are a team sport.

s75
00:07:50.600 --> 00:07:59.240
Uh, there's so much that goes into creating evals uh from getting real-world data into your platform, labeling it, developing hypotheses, things like that.

s76
00:07:59.240 --> 00:08:05.880
So, the AI engineer kind of works on obviously making the code changes, but also getting the data into the platform.

s77
00:08:05.880 --> 00:08:07.960
Uh, the product manager is very important.

s78
00:08:07.960 --> 00:08:18.919
I think they're they're usually spearhead evals at a company, but they're the ones that develop the hypothesis of what success looks like, um and help uh tweak tweak the prompts and things like that.

s79
00:08:18.919 --> 00:08:28.280
Uh, subject matter experts are really important, especially in like more niche uh areas like um insurance, medicine, law, and things like that.

s80
00:08:28.280 --> 00:08:34.640
Um, they can help uh label the data or provide a ground truth for what what excellence looks like.

s81
00:08:34.640 --> 00:08:37.400
Uh, and then data analysts obviously to analyze the data.

s82
00:08:37.400 --> 00:08:41.840
So, I always think that this slide is is really useful to look at.

s83
00:08:41.840 --> 00:08:42.360
Okay.

s84
00:08:42.360 --> 00:08:48.520
So, let's go into a like interesting real-world example of how you might use an eval.

s85
00:08:48.520 --> 00:08:52.480
Um, in this one I'm comparing agentic search to vector search.

s86
00:08:52.480 --> 00:09:02.920
And the story behind this specific eval is um is that my CEO Ankur sometimes goes on Twitter and sends me articles of things he wants me to eval.

s87
00:09:02.920 --> 00:09:14.320
Uh, so he sent me this uh tweet from Cursor where Cursor talks about how they used semantic search, also known as agentic search, um to significantly improve their coding agent uh performance.

s88
00:09:14.320 --> 00:09:16.640
And he was like, "Jess, can you eval this for me?"

s89
00:09:16.640 --> 00:09:17.920
And I was like, "Okay.

s90
00:09:17.920 --> 00:09:19.280
I'm busy, but okay."

s91
00:09:19.280 --> 00:09:28.920
Um, and so I kind of went on Twitter and I saw that there was a lot of uh discourse or discussion around this topic, so I thought it might be interesting to eval.

s92
00:09:28.920 --> 00:09:36.600
So, uh we're going to eval a vector search and agentic search, so I just want to give a quick uh summary of what each one is just so everyone's on the same page.

s93
00:09:36.600 --> 00:09:49.840
So, vector search essentially is the process of taking uh a data like uh code or text and turning them into embeddings where embeddings basically represent the semantic meaning of that chunk of text or code.

s94
00:09:49.840 --> 00:09:59.200
So if you look at the graph here, it's a very good visual example where um the embeddings uh wolf, dog, and cat are very close to each other cuz semantically

s95
00:09:59.200 --> 00:10:00.760
they all are animals.

s96
00:10:00.760 --> 00:10:10.080
And then uh the embeddings banana and apple are also close to each other cuz they're fruit, but then uh they're far apart from each other because fruit and animals are different things.

s97
00:10:10.080 --> 00:10:19.640
Um And so what happens is in, you know, um in a real vector search implementation, you would take these embeddings and store them in a vector database.

s98
00:10:19.640 --> 00:10:23.200
So something like Qdrant or Pinecone or something like that.

s99
00:10:23.200 --> 00:10:29.800
And if you come up with a query like, uh for example, where does the database logic in my code base live?

s100
00:10:29.800 --> 00:10:40.040
Uh what would happen is the vector database would return back the chunks of um embeddings that most closely semantically match the words in your query.

s101
00:10:40.040 --> 00:10:44.800
So the the chunks that are closest to things like database or logic.

s102
00:10:44.800 --> 00:10:46.080
So that's how vector search works.

s103
00:10:46.080 --> 00:10:47.960
Agentic search is slightly different.

s104
00:10:47.960 --> 00:10:50.920
Agentic search works more like a human would.

s105
00:10:50.920 --> 00:10:58.880
So essentially with agentic search uh you give an LM tools like uh grep, find, ls, cat, um bash commands.

s106
00:10:58.880 --> 00:11:02.680
And it basically explores explores your code base like a human would.

s107
00:11:02.680 --> 00:11:14.400
So it would uh look up a function name, open a file, read the file, follow that function call to another file, then open that file and read that file, and then continue until it's found what it's what it needs to.

s108
00:11:14.400 --> 00:11:23.520
So this is uh much more similar to how a human would kind of go through a code base and and look up look for certain pieces of logic.

s109
00:11:23.520 --> 00:11:26.040
Okay, so then how did we eval this?

s110
00:11:26.040 --> 00:11:29.360
So the first part of creating eval is creating our data set, right?

s111
00:11:29.360 --> 00:11:36.040
So to do this, we basically used uh Microsoft's TypeScript go repo, uh which is open source.

s112
00:11:36.040 --> 00:11:42.880
And what we did is we basically found all the PRs, the merged PRs with the title fix in them.

s113
00:11:42.880 --> 00:11:45.880
And we basically checked out the parent commit.

s114
00:11:45.880 --> 00:11:50.800
So, essentially, we checked out the commit to the point where it still has the bug in it.

s115
00:11:50.800 --> 00:11:57.640
And then we used Claude to generate a like a bug task or like a linear task based on the diff.

s116
00:11:57.640 --> 00:12:01.160
So, based on the diff between the buggy code and the fixed code.

s117
00:12:01.160 --> 00:12:12.360
And then we fed that into an LLM and had that LLM either use rag search or agentic search to find the buggy code, to locate where the logic, the buggy logic lives.

s118
00:12:12.360 --> 00:12:13.880
So, that's how we created our data set.

s119
00:12:13.880 --> 00:12:21.680
We had like maybe we we found like 20 different PRs and and ran the eval on that.

s120
00:12:21.680 --> 00:12:27.640
In terms of our task, we actually had to implement vector search and agentic search, obviously.

s121
00:12:27.640 --> 00:12:33.040
Implementing agentic search was really easy because Claude code uses agentic search by default.

s122
00:12:33.040 --> 00:12:36.839
So, we just set Claude code on these tasks.

s123
00:12:36.839 --> 00:12:47.400
Implementing vector search was slightly more complicated because we still wanted to use the Claude code harness to execute the vector search, just to keep the experiment consistent.

s124
00:12:47.400 --> 00:12:52.400
But we had to basically prevent Claude code from defaulting to agentic search.

s125
00:12:52.400 --> 00:12:54.040
So, we did two things.

s126
00:12:54.040 --> 00:12:59.280
The first thing is that we explicitly said in the prompt instructions not to use agentic search.

s127
00:12:59.280 --> 00:13:02.720
But if you've ever done this before, you know that prompting is never enough.

s128
00:13:02.720 --> 00:13:12.320
So, what we also had to do is use the disallowed tools flag to prevent Claude code from calling certain things that it would normally use in agentic search.

s129
00:13:12.320 --> 00:13:18.720
So, that's how we implemented vector search and agentic search, both using the Claude code harness.

s130
00:13:19.040 --> 00:13:33.920
Another kind of interesting thing that we ran into is when we were trace running the traces, we ran the Claude code runs as sub processes, but it essentially mean meant that the traces were orphaned.

s131
00:13:33.920 --> 00:13:40.000
Um, and uh, our solution was basically passing the parent span IDs as environment variables.

s132
00:13:40.000 --> 00:13:43.800
So, if that doesn't make sense, that's fine because I'll show you via pictures.

s133
00:13:43.800 --> 00:13:52.040
So, this picture that you see here is what the trace looks like when it's running as a orphaned sub process.

s134
00:13:52.040 --> 00:13:56.760
So, you can see in the highlighted unlike the left panel where it says run Claude agent.

s135
00:13:56.760 --> 00:14:01.480
Um, that's basically in the trace saying, "Okay, it's running the Claude sub process."

s136
00:14:01.480 --> 00:14:07.040
But, you can't you don't actually get any visibility into what happens when it's running that Claude process.

s137
00:14:07.040 --> 00:14:10.840
With the fix that we did, uh, this is what you see now in the trace.

s138
00:14:10.840 --> 00:14:17.760
So, you can see that it's calling the Claude code sub process, but you can also see everything that's being run in that sub process.

s139
00:14:17.760 --> 00:14:26.080
So, you can see all the turns, all the LLM calls, every single terminal command that's being called like grep, bash, ls, find, things like that.

s140
00:14:26.080 --> 00:14:30.120
Uh, and it gives you visibility into what's happening in this actual run.

s141
00:14:30.120 --> 00:14:40.800
Uh, and implementing this fix was really important because being able to have transparency into what's happening, um, is is important for debugging, uh, later on.

s142
00:14:40.800 --> 00:14:42.280
And scoring was really simple.

s143
00:14:42.280 --> 00:14:57.160
Uh, for our scoring we basically said, uh, "Because we're using Microsoft Microsoft's TypeScript Go repo, we basically said if the test uh, passed the uh, uh, Microsoft TypeScript Go test suite, then it gets 100% and if it fails it gets a 0%."

s144
00:14:57.160 --> 00:15:00.600
So, it was very binary scoring here.

s145
00:15:00.600 --> 00:15:01.040
Uh, okay.

s146
00:15:01.040 --> 00:15:15.360
So, in terms of our results, uh, the summary is basically vector search and agentic search both got the same level of accuracy, but essentially vector search cost, uh, four times more than agentic search.

s147
00:15:15.360 --> 00:15:23.440
So, when we looked at the actual traces and uh, kind of manually like went through to try to understand what happened, we had a couple of learnings.

s148
00:15:23.440 --> 00:15:34.440
So, the first thing is that vector search returns back chunks of code, but these chunks of code often miss things like imports or tool calls or the calling code above it.

s149
00:15:34.440 --> 00:15:41.120
And so, as a result, it's not enough it doesn't give enough context for how the agent should actually solve the bug.

s150
00:15:41.120 --> 00:15:52.080
So, one example is in one of the rows we saw that the vector agent made 26 searches and still couldn't piece together how three functions interacted across the file.

s151
00:15:52.080 --> 00:15:57.480
In contrast, agentic search was actually able to like as I said, work more like a human would.

s152
00:15:57.480 --> 00:16:07.080
So, grep the function name, read the full function, spot the flaw in the logic, and then follow the calling calling logic into another file.

s153
00:16:07.080 --> 00:16:17.000
And the way that I kind of like to think about it when I was just going through the traces, is it felt like vector search gave a lot of proximity to the correct code,

s154
00:16:17.000 --> 00:16:23.400
but agentic search had more of the connective tissue to connect the actual logic of those chunks of code.

s155
00:16:23.400 --> 00:16:35.960
And then tying that into the third learning I had is the reason why vector search was so much more expensive than agentic search is because it made more searches because it was

s156
00:16:35.960 --> 00:16:41.160
constantly searching like returning a chunk of code and then searching again, returning a chunk of code, searching again.

s157
00:16:41.160 --> 00:16:48.800
And as a result, all of those LLM calls piled up and made it a lot more expensive to execute.

s158
00:16:48.839 --> 00:16:56.760
And this is the last slide, but essentially you might be sitting here thinking oh there's so many holes in this eval, which is very fair.

s159
00:16:56.760 --> 00:16:59.960
Like there's lots of things I could have done.

s160
00:16:59.960 --> 00:17:13.360
You know, there's lots of best practices when it comes to running evals like running multiple trials per task, especially because LLMs are so non-deterministic it's good to run multiple trials to be able to trust the score that you're getting back a lot better.

s161
00:17:13.360 --> 00:17:16.520
My vector search implementation was relatively weak.

s162
00:17:16.520 --> 00:17:18.560
Uh could have done a better implementation.

s163
00:17:18.560 --> 00:17:20.520
Um also expanding the data set.

s164
00:17:20.520 --> 00:17:23.760
Um I think we only ran this on like 20 different rows.

s165
00:17:23.760 --> 00:17:29.880
So, it would have been better to expand this uh to even different um different repos or things like that.

s166
00:17:29.880 --> 00:17:42.400
Uh but the the point of this talk is not to create the best uh off like the best eval ever, but it's more to illustrate to you the concept of the kind of how to create an eval end to end.

s167
00:17:42.400 --> 00:17:44.000
So, hopefully I've done that.

s168
00:17:44.000 --> 00:17:46.440
Um and that is the end of my talk.

s169
00:17:46.440 --> 00:17:53.480
Um so, hopefully you learned a little bit of what evals are, why they're important, and how you might do one in real life.

s170
00:17:53.480 --> 00:17:55.520
Um and yeah, I think that's it.

s171
00:17:55.520 --> 00:17:59.160
So, thank you for listening.