Why Building an Eval Platform Is Harder Than It Looks — Braintrust

AI Engineer · 17 min · 166 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Thank you for joining this session on why building agent quality platforms is hard.
00:17My name's Hussein.
00:19I lead the solution engineering organization on the West for Braintrust.
00:23Um, I spent 15 years in solutions between Salesforce and Databricks.
00:28I've been a lifelong technologist.
00:30I've built computers since I was 5 years old.
00:32So, I have this like insatiable appetite for technology, which finds me kind of on the bleeding edge of AI here at Braintrust.
00:42For those of you, I know some of you were in the room earlier when Jess did her session.
00:45I saw some hands about evals.
00:47How many of you know what Braintrust is and what we do as a company?
00:51Just show of hands.
00:53Perfect.
00:53I see one or two.
00:54So, at a glance, Braintrust as a platform is focused around agent quality.
01:03And the core idea is helping teams build and maintain confidence with the AI features or the AI agents that they're shipping.
01:12And there are two key pillars of agent quality.
01:17The first is evals, which we talked about earlier.
01:20Evals is what you do before your agent reaches production.
01:25This is when teams experiment, they test the behavior, and they build the confidence that they want to have before they ship to production.
01:34The second pillar is observability.
01:37Observability is once your agent is in production and is interfacing with real-life users.
01:43It's creating interactions.
01:46And the goal is to take that hypothesis that you built in your offline development process and then reinforce that in your production scenario through continuous monitoring.
01:59Evals and observability are closely related to the same problem.
02:05One happens in production, the other one happens in development, and both is the understanding to improve agent quality.
02:13Now, I won't spend much more time here.
02:17Let me close that.
02:19Let's talk about why we're here, which is why are evals important?
02:27You know, evals matter because LLMs by nature are non-deterministic.
02:32They are highly variable.
02:35That variability is exactly what makes them so powerful.
02:39They can reason across many different domains.
02:42They can solve different kinds of problems, and they can handle a wide range of user needs.
02:48But that same flexibility also creates risk.
02:53Agents use LLM as their brain.
02:56And agentic experiences are starting to become the primary way in which users will interact with companies.
03:04So teams need confidence in their agents that they will behave reliably and produce the outcomes that they expect.
03:14And without evals, companies face real risks.
03:18Brand Brands become a risk if there's inconsistent behaviors.
03:25Compliance if the agent says or does the wrong thing.
03:30And there's a cost and maintenance risk if the systems are hard or unreliable to debug or control.
03:38The goal of evals is to reduce that uncertainty before launch so that customers have a good experience and the agents behave the way that you expect them to.
03:51So you might be thinking, "Okay, well, Evals is just a UI on a spreadsheet, right?"
03:58And I guess I should ask the audience.
04:01Just show of hands, how many of you have run Evals and logged the results into a spreadsheet?
04:06Has that happened?
04:07Okay, I see one or two.
04:08So, there's no shame in that, by the way.
04:10It's actually the a good first step.
04:14The important thing is that teams acknowledge that this is a real problem and they need a way to understand how agents behave based on different inputs.
04:24At the simplest level, an eval system tests against or it has three different kind of criteria.
04:33It has a way to execute agents against test inputs.
04:37Second, a way to view the outputs or the scores, even if those are being logged in a spreadsheet.
04:44And third, a set of test inputs or examples that can invoke the agent.
04:50An input example is whatever information is needed to start the agent run.
04:56It could be a prompt, a request, or context that causes the agent to act.
05:03Now, spreadsheet-based Evals are not wrong.
05:07However, they're often the first step for a useful version of eval workflow.
05:13But, of course, as you can imagine, there's more to the iceberg than just what we just talked about.
05:21And if Evals were only, you know, run the agent, view the output, and then score it in a spreadsheet, this talk would be over right now.
05:30But, there's a lot more that happens behind the scenes or underneath the iceberg.
05:36There are many supporting teams that eventually need to build better data sets, scoring systems, review process, debugging tools, and the way to connect pre-production testing with production behavior.
05:54We'll touch on some of these today, and anything I don't get to, you feel free to come up afterwards and we can chat about.
06:01And this is where things start to get complicated.
06:05The underlying technology is complex.
06:07LLMs are not simply deterministic.
06:10Agent quality is a multi-persona problem.
06:13It's not just software engineers and AI engineers.
06:16There are PMs who used to build PRDs that are now running evals.
06:20There are SMEs who have the domain expertise of the product that you're building.
06:27And evals themselves are just one part of the development and operating workflow.
06:34And gone are the days where you build once with unit tests and regression tests, you ship to production, and you don't think about it.
06:41Evals are a way for you to continue hill climbing against one target, which is agent quality.
06:49So, let's talk about the different stages of the the eval platforms that we see.
06:56And before we do that, this is the North Star for many teams.
07:00They want to build the improvement loop, which is I have a feature or application that's in production.
07:07I can observe the failure modes that happen based on dimensions that I define.
07:12I want to be able to grab those failure modes and create test cases so that I can iterate offline, so I can improve my agent quality without introducing new regressions.
07:25And then you continue iterating over time, because just like in classical ML, drift becomes a real thing.
07:31The way users interact with your agents will change over time.
07:35The way you build your agents will change over time.
07:39So, what What phase one look like?
07:41There were some hands that were raised around spreadsheets and it is not There's nothing to be shameful about.
07:47This is where many people start.
07:51And as the basic setup is pretty simple.
07:54It's a four loop with a set of input examples and a way to execute the agent.
08:01And with that, you can run the same examples that you tweak and see how the agent responds with the outputs over time.
08:09The big biggest advantage is accessibility and time.
08:13You can start this with basically zero barrier to entry.
08:17But the the returns diminish pretty quickly.
08:20At this stage, you're pretty much only documenting.
08:23You can't really do true experimentation.
08:26And let alone do any kind of type of analysis over time.
08:32Common limitations are analytics are hard, they're manual, human scoring is valuable but difficult to do, collaboration is very weak, and domain experts are usually boxed out of these type of results.
08:48So, again, spreadsheets are a great starting place but difficult to evolve.
08:54Then someone, usually a product engineer, you know, they see this problem and they think, "Okay, I can build a nice UI, a bespoke UI to help serve to solve this problem."
09:07And while it might be nice at a glance, um it helps you bring in the different personas that you would otherwise not have access to.
09:17This allows PMs to now kind of get into the loop of doing evals.
09:24But the limitation that you might have thought about, it continues.
09:28That the documentation, the review process, though it might visually look nicer, still collaboration becomes difficult.
09:36And though iteration cycles become quicker, it is still not easy for users to persist that data and do long horizon analytics on.
09:48Still just a reporting tool.
09:52Then the next evolution becomes hey, I want my non-technical users to have an interface where not that they just log the results, but they should be able to experiment.
10:03I want to be able to tweak a system prompt.
10:06I want to change the underlying model.
10:08I want to iterate on some parameter values.
10:12So, being able to do this type of side-by-side comparison becomes the next evolution of the eval cycle that we see.
10:21But, there's still one primary gap.
10:23And that is that the test cases in which you use to do these kind of comparisons are still a curated set that arise in your offline evals.
10:34But, what happens in production?
10:36I've done my iterations here.
10:38I have a good idea of what I can expect to happen in prod, but once I ship it to production, I'm operating in the blind.
10:45So, I have no idea what steps my agent is taking and when those failures are actually happening in my live environment.
10:55And when we think kind of going back to the flywheel, when we think about what are the best systems that enable this, well, you should be able to starting at the top
11:04observe the failure modes, log every input, output, every step of your trace execution that your agent takes, analyze, understand what went wrong.
11:14You create the dimensions of failure and success because you know the outcomes.
11:18Then grabbing those failure modes, those different scenarios where your users had poor interactions, and creating evals in them in your offline process, iterating on them so you can create the improvements
11:32without developing new regressions.
11:34And then shipping to production and hill climbing against this.
11:40And when you think about the final outcome of this, well, you get teams that are building the flywheel.
11:48You get that entire development cycle inside of your platform, but then the problem becomes you have to have you have to maintain it.
11:56You own this this product that you've built.
12:00And especially this is easy to do when it comes to smaller scale or POC and demo environments.
12:07But agent traces are nasty.
12:10They are semi-structured JSON.
12:12In some cases we see teams that are logging a movie's worth of data in each interaction, hundreds of megabytes.
12:20So being able to query that type of data can be difficult.
12:24And your typical cloud data warehouses kind of collapse under this type of volume and load.
12:32And this is the same problem that we ran into.
12:35I won't spend much time talking about Brain Trust, but when you have production traces coming in, you want to be able to query that data in real time as it lands in there.
12:48Because if your users are having poor experiences, you want to be able to know in real time what is happening, why are they having these poor experiences, and how can I remedy that as soon as possible.
13:00You also want to be able to do long-running queries.
13:03If you think about being able to fine-tune your your agent experience or even have humans who are doing alignment by annotating your your judges' outcomes.
13:18Two different workloads.
13:19So we built an abstraction layer called BTQL, which is a another form of complexity because you want an interface for people to be able to query that data.
13:31So, it's not necessarily that it's just a UI or UX problem, but building an eval platform truly becomes a systems problem.
13:44And there are novel set of issues when the chat GPT boom happened a few years ago where I need to do real-time ingest.
13:54I have huge payloads that are tens of megabytes, hundreds of megabytes in size sometimes, compared to traditional heartbeat observability, which are just kilobytes in size.
14:05The structure and the shape of the data is different.
14:09Having deeply nested semi-structured full text is difficult to query, and the read patterns are different as well.
14:18You want to be able to aggregate in large volumes, but also be able to do snapshots of of of in-time data.
14:28And building the right system should allow you to not only empower the AI engineer, but also the PMs, as well as the SMEs.
14:40And more recently, we've seen that agents have become a first-class citizen of these eval platforms where you want to be able to use natural language in a headless experience
14:53where you can tell your coding agent of choice, "Find me all traces in the last 24 hours where the user had a poor experience.
15:02Run the evals for me."
15:04And because these coding agents have access to all of your underlying infrastructure and your code base, they become the mechanism to run the evals and log those results.
15:16And then it becomes a question of like, so so what?
15:18Why is this a Why is this important?
15:22Well, the goal is everything that I've shown you so far, it requires the humans, the the engineer, the PM to be deeply involved in this process and it could be laborious
15:36at times.
15:39I BrainTrust, the way we think about this is we want to help you operate at scale.
15:46Rather than you having to think about these dimensions and of success and failure for your AI agent, BrainTrust can surface these insights automatically.
15:57Because we log your tracing data into our platform, we can run inference on them and we can tell you the unknown unknowns.
16:05Hey, what are what are scenarios and when my agent is failing silently or what are scenarios that my users are experiencing frustration by prompting my agent for the same request over and over again.
16:20And it's a lot of things that we didn't talk about.
16:22Things like the underlying database that powers it or having to build our back into the system to manage controls and permissions or data masking.
16:32You know, each one of these features proves that this is not just a UI and a spreadsheet, but rather it is a a systems problem that powers it.
16:43And as we think about the evolution of of the improvement loop, I said this earlier where humans were doing the improvement loop and now we see coding agents where they could iteratively
16:54make changes and suggest what kind of improvements you should be making to your application.
17:01And ultimately, the humans responsibility is to review the outcome.
17:06If I have different iterations of these e-vals that are run by my coding agent, I can look at the outcome and make the decision of which version of it is the best for what I want to ship to production.
17:20So, with that, that is the end of my talk.
17:23I appreciate you all coming out and learning about why it's hard to build eval systems.
17:27Thank you.
17:44[music]