WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=mUQoVz7THu0 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/mUQoVz7THu0.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.440 --> 00:00:17.880 Thank you for joining this session on why building agent quality platforms is hard. s3 00:00:17.880 --> 00:00:19.000 My name's Hussein. s4 00:00:19.000 --> 00:00:23.320 I lead the solution engineering organization on the West for Braintrust. s5 00:00:23.320 --> 00:00:28.520 Um, I spent 15 years in solutions between Salesforce and Databricks. s6 00:00:28.520 --> 00:00:30.120 I've been a lifelong technologist. s7 00:00:30.120 --> 00:00:32.439 I've built computers since I was 5 years old. s8 00:00:32.439 --> 00:00:41.480 So, I have this like insatiable appetite for technology, which finds me kind of on the bleeding edge of AI here at Braintrust. s9 00:00:42.440 --> 00:00:45.760 For those of you, I know some of you were in the room earlier when Jess did her session. s10 00:00:45.760 --> 00:00:47.880 I saw some hands about evals. s11 00:00:47.880 --> 00:00:51.040 How many of you know what Braintrust is and what we do as a company? s12 00:00:51.040 --> 00:00:53.080 Just show of hands. s13 00:00:53.080 --> 00:00:53.320 Perfect. s14 00:00:53.320 --> 00:00:54.760 I see one or two. s15 00:00:54.760 --> 00:01:03.360 So, at a glance, Braintrust as a platform is focused around agent quality. s16 00:01:03.360 --> 00:01:12.760 And the core idea is helping teams build and maintain confidence with the AI features or the AI agents that they're shipping. s17 00:01:12.760 --> 00:01:17.160 And there are two key pillars of agent quality. s18 00:01:17.160 --> 00:01:20.080 The first is evals, which we talked about earlier. s19 00:01:20.080 --> 00:01:25.480 Evals is what you do before your agent reaches production. s20 00:01:25.480 --> 00:01:34.960 This is when teams experiment, they test the behavior, and they build the confidence that they want to have before they ship to production. s21 00:01:34.960 --> 00:01:37.600 The second pillar is observability. s22 00:01:37.600 --> 00:01:43.800 Observability is once your agent is in production and is interfacing with real-life users. s23 00:01:43.800 --> 00:01:46.240 It's creating interactions. s24 00:01:46.240 --> 00:01:59.760 And the goal is to take that hypothesis that you built in your offline development process and then reinforce that in your production scenario through continuous monitoring. s25 00:01:59.840 --> 00:02:05.520 Evals and observability are closely related to the same problem. s26 00:02:05.520 --> 00:02:13.920 One happens in production, the other one happens in development, and both is the understanding to improve agent quality. s27 00:02:13.920 --> 00:02:17.160 Now, I won't spend much more time here. s28 00:02:17.160 --> 00:02:19.240 Let me close that. s29 00:02:19.240 --> 00:02:27.920 Let's talk about why we're here, which is why are evals important? s30 00:02:27.920 --> 00:02:32.480 You know, evals matter because LLMs by nature are non-deterministic. s31 00:02:32.480 --> 00:02:35.000 They are highly variable. s32 00:02:35.000 --> 00:02:39.520 That variability is exactly what makes them so powerful. s33 00:02:39.520 --> 00:02:42.680 They can reason across many different domains. s34 00:02:42.680 --> 00:02:48.640 They can solve different kinds of problems, and they can handle a wide range of user needs. s35 00:02:48.640 --> 00:02:53.080 But that same flexibility also creates risk. s36 00:02:53.200 --> 00:02:56.840 Agents use LLM as their brain. s37 00:02:56.840 --> 00:03:04.880 And agentic experiences are starting to become the primary way in which users will interact with companies. s38 00:03:04.880 --> 00:03:14.000 So teams need confidence in their agents that they will behave reliably and produce the outcomes that they expect. s39 00:03:14.000 --> 00:03:18.519 And without evals, companies face real risks. s40 00:03:18.519 --> 00:03:25.040 Brand Brands become a risk if there's inconsistent behaviors. s41 00:03:25.040 --> 00:03:30.080 Compliance if the agent says or does the wrong thing. s42 00:03:30.080 --> 00:03:38.560 And there's a cost and maintenance risk if the systems are hard or unreliable to debug or control. s43 00:03:38.560 --> 00:03:50.080 The goal of evals is to reduce that uncertainty before launch so that customers have a good experience and the agents behave the way that you expect them to. s44 00:03:51.320 --> 00:03:58.600 So you might be thinking, "Okay, well, Evals is just a UI on a spreadsheet, right?" s45 00:03:58.600 --> 00:04:01.240 And I guess I should ask the audience. s46 00:04:01.240 --> 00:04:06.440 Just show of hands, how many of you have run Evals and logged the results into a spreadsheet? s47 00:04:06.440 --> 00:04:07.000 Has that happened? s48 00:04:07.000 --> 00:04:08.560 Okay, I see one or two. s49 00:04:08.560 --> 00:04:10.680 So, there's no shame in that, by the way. s50 00:04:10.680 --> 00:04:14.280 It's actually the a good first step. s51 00:04:14.280 --> 00:04:24.480 The important thing is that teams acknowledge that this is a real problem and they need a way to understand how agents behave based on different inputs. s52 00:04:24.600 --> 00:04:33.040 At the simplest level, an eval system tests against or it has three different kind of criteria. s53 00:04:33.040 --> 00:04:37.600 It has a way to execute agents against test inputs. s54 00:04:37.600 --> 00:04:44.400 Second, a way to view the outputs or the scores, even if those are being logged in a spreadsheet. s55 00:04:44.400 --> 00:04:50.240 And third, a set of test inputs or examples that can invoke the agent. s56 00:04:50.240 --> 00:04:56.520 An input example is whatever information is needed to start the agent run. s57 00:04:56.520 --> 00:05:03.280 It could be a prompt, a request, or context that causes the agent to act. s58 00:05:03.280 --> 00:05:07.840 Now, spreadsheet-based Evals are not wrong. s59 00:05:07.840 --> 00:05:13.440 However, they're often the first step for a useful version of eval workflow. s60 00:05:13.440 --> 00:05:21.280 But, of course, as you can imagine, there's more to the iceberg than just what we just talked about. s61 00:05:21.280 --> 00:05:30.160 And if Evals were only, you know, run the agent, view the output, and then score it in a spreadsheet, this talk would be over right now. s62 00:05:30.160 --> 00:05:36.240 But, there's a lot more that happens behind the scenes or underneath the iceberg. s63 00:05:36.320 --> 00:05:54.240 There are many supporting teams that eventually need to build better data sets, scoring systems, review process, debugging tools, and the way to connect pre-production testing with production behavior. s64 00:05:54.240 --> 00:06:01.960 We'll touch on some of these today, and anything I don't get to, you feel free to come up afterwards and we can chat about. s65 00:06:01.960 --> 00:06:05.400 And this is where things start to get complicated. s66 00:06:05.400 --> 00:06:07.920 The underlying technology is complex. s67 00:06:07.920 --> 00:06:10.720 LLMs are not simply deterministic. s68 00:06:10.720 --> 00:06:13.200 Agent quality is a multi-persona problem. s69 00:06:13.200 --> 00:06:16.000 It's not just software engineers and AI engineers. s70 00:06:16.000 --> 00:06:20.240 There are PMs who used to build PRDs that are now running evals. s71 00:06:20.240 --> 00:06:26.680 There are SMEs who have the domain expertise of the product that you're building. s72 00:06:27.120 --> 00:06:33.840 And evals themselves are just one part of the development and operating workflow. s73 00:06:34.880 --> 00:06:41.600 And gone are the days where you build once with unit tests and regression tests, you ship to production, and you don't think about it. s74 00:06:41.600 --> 00:06:48.280 Evals are a way for you to continue hill climbing against one target, which is agent quality. s75 00:06:49.280 --> 00:06:56.240 So, let's talk about the different stages of the the eval platforms that we see. s76 00:06:56.680 --> 00:07:00.919 And before we do that, this is the North Star for many teams. s77 00:07:00.919 --> 00:07:07.760 They want to build the improvement loop, which is I have a feature or application that's in production. s78 00:07:07.760 --> 00:07:12.400 I can observe the failure modes that happen based on dimensions that I define. s79 00:07:12.400 --> 00:07:25.000 I want to be able to grab those failure modes and create test cases so that I can iterate offline, so I can improve my agent quality without introducing new regressions. s80 00:07:25.000 --> 00:07:31.919 And then you continue iterating over time, because just like in classical ML, drift becomes a real thing. s81 00:07:31.919 --> 00:07:35.680 The way users interact with your agents will change over time. s82 00:07:35.680 --> 00:07:39.600 The way you build your agents will change over time. s83 00:07:39.600 --> 00:07:41.800 So, what What phase one look like? s84 00:07:41.800 --> 00:07:47.720 There were some hands that were raised around spreadsheets and it is not There's nothing to be shameful about. s85 00:07:47.720 --> 00:07:51.000 This is where many people start. s86 00:07:51.240 --> 00:07:54.800 And as the basic setup is pretty simple. s87 00:07:54.800 --> 00:08:01.400 It's a four loop with a set of input examples and a way to execute the agent. s88 00:08:01.800 --> 00:08:09.200 And with that, you can run the same examples that you tweak and see how the agent responds with the outputs over time. s89 00:08:09.200 --> 00:08:13.080 The big biggest advantage is accessibility and time. s90 00:08:13.080 --> 00:08:17.480 You can start this with basically zero barrier to entry. s91 00:08:17.480 --> 00:08:20.520 But the the returns diminish pretty quickly. s92 00:08:20.520 --> 00:08:23.760 At this stage, you're pretty much only documenting. s93 00:08:23.760 --> 00:08:26.919 You can't really do true experimentation. s94 00:08:26.960 --> 00:08:32.560 And let alone do any kind of type of analysis over time. s95 00:08:32.560 --> 00:08:47.920 Common limitations are analytics are hard, they're manual, human scoring is valuable but difficult to do, collaboration is very weak, and domain experts are usually boxed out of these type of results. s96 00:08:48.600 --> 00:08:54.920 So, again, spreadsheets are a great starting place but difficult to evolve. s97 00:08:54.920 --> 00:09:07.720 Then someone, usually a product engineer, you know, they see this problem and they think, "Okay, I can build a nice UI, a bespoke UI to help serve to solve this problem." s98 00:09:07.720 --> 00:09:17.040 And while it might be nice at a glance, um it helps you bring in the different personas that you would otherwise not have access to. s99 00:09:17.040 --> 00:09:23.080 This allows PMs to now kind of get into the loop of doing evals. s100 00:09:24.839 --> 00:09:28.760 But the limitation that you might have thought about, it continues. s101 00:09:28.760 --> 00:09:36.839 That the documentation, the review process, though it might visually look nicer, still collaboration becomes difficult. s102 00:09:36.839 --> 00:09:48.240 And though iteration cycles become quicker, it is still not easy for users to persist that data and do long horizon analytics on. s103 00:09:48.320 --> 00:09:52.320 Still just a reporting tool. s104 00:09:52.520 --> 00:10:03.720 Then the next evolution becomes hey, I want my non-technical users to have an interface where not that they just log the results, but they should be able to experiment. s105 00:10:03.720 --> 00:10:06.360 I want to be able to tweak a system prompt. s106 00:10:06.360 --> 00:10:08.560 I want to change the underlying model. s107 00:10:08.560 --> 00:10:12.320 I want to iterate on some parameter values. s108 00:10:12.320 --> 00:10:21.000 So, being able to do this type of side-by-side comparison becomes the next evolution of the eval cycle that we see. s109 00:10:21.000 --> 00:10:23.800 But, there's still one primary gap. s110 00:10:23.800 --> 00:10:34.680 And that is that the test cases in which you use to do these kind of comparisons are still a curated set that arise in your offline evals. s111 00:10:34.680 --> 00:10:36.600 But, what happens in production? s112 00:10:36.600 --> 00:10:38.440 I've done my iterations here. s113 00:10:38.440 --> 00:10:45.880 I have a good idea of what I can expect to happen in prod, but once I ship it to production, I'm operating in the blind. s114 00:10:45.880 --> 00:10:52.640 So, I have no idea what steps my agent is taking and when those failures are actually happening in my live environment. s115 00:10:55.120 --> 00:11:04.440 And when we think kind of going back to the flywheel, when we think about what are the best systems that enable this, well, you should be able to starting at the top s116 00:11:04.440 --> 00:11:14.240 observe the failure modes, log every input, output, every step of your trace execution that your agent takes, analyze, understand what went wrong. s117 00:11:14.240 --> 00:11:18.960 You create the dimensions of failure and success because you know the outcomes. s118 00:11:18.960 --> 00:11:32.160 Then grabbing those failure modes, those different scenarios where your users had poor interactions, and creating evals in them in your offline process, iterating on them so you can create the improvements s119 00:11:32.160 --> 00:11:34.880 without developing new regressions. s120 00:11:34.880 --> 00:11:39.680 And then shipping to production and hill climbing against this. s121 00:11:40.200 --> 00:11:48.400 And when you think about the final outcome of this, well, you get teams that are building the flywheel. s122 00:11:48.400 --> 00:11:56.640 You get that entire development cycle inside of your platform, but then the problem becomes you have to have you have to maintain it. s123 00:11:56.640 --> 00:12:00.000 You own this this product that you've built. s124 00:12:00.000 --> 00:12:07.960 And especially this is easy to do when it comes to smaller scale or POC and demo environments. s125 00:12:07.960 --> 00:12:10.360 But agent traces are nasty. s126 00:12:10.360 --> 00:12:12.800 They are semi-structured JSON. s127 00:12:12.800 --> 00:12:20.920 In some cases we see teams that are logging a movie's worth of data in each interaction, hundreds of megabytes. s128 00:12:20.920 --> 00:12:24.520 So being able to query that type of data can be difficult. s129 00:12:24.520 --> 00:12:31.520 And your typical cloud data warehouses kind of collapse under this type of volume and load. s130 00:12:32.920 --> 00:12:35.760 And this is the same problem that we ran into. s131 00:12:35.760 --> 00:12:48.520 I won't spend much time talking about Brain Trust, but when you have production traces coming in, you want to be able to query that data in real time as it lands in there. s132 00:12:48.520 --> 00:13:00.720 Because if your users are having poor experiences, you want to be able to know in real time what is happening, why are they having these poor experiences, and how can I remedy that as soon as possible. s133 00:13:00.720 --> 00:13:03.520 You also want to be able to do long-running queries. s134 00:13:03.520 --> 00:13:18.160 If you think about being able to fine-tune your your agent experience or even have humans who are doing alignment by annotating your your judges' outcomes. s135 00:13:18.160 --> 00:13:19.920 Two different workloads. s136 00:13:19.920 --> 00:13:30.480 So we built an abstraction layer called BTQL, which is a another form of complexity because you want an interface for people to be able to query that data. s137 00:13:31.360 --> 00:13:42.920 So, it's not necessarily that it's just a UI or UX problem, but building an eval platform truly becomes a systems problem. s138 00:13:44.440 --> 00:13:54.120 And there are novel set of issues when the chat GPT boom happened a few years ago where I need to do real-time ingest. s139 00:13:54.120 --> 00:14:05.720 I have huge payloads that are tens of megabytes, hundreds of megabytes in size sometimes, compared to traditional heartbeat observability, which are just kilobytes in size. s140 00:14:05.720 --> 00:14:09.040 The structure and the shape of the data is different. s141 00:14:09.040 --> 00:14:18.120 Having deeply nested semi-structured full text is difficult to query, and the read patterns are different as well. s142 00:14:18.120 --> 00:14:27.640 You want to be able to aggregate in large volumes, but also be able to do snapshots of of of in-time data. s143 00:14:28.800 --> 00:14:40.080 And building the right system should allow you to not only empower the AI engineer, but also the PMs, as well as the SMEs. s144 00:14:40.080 --> 00:14:53.120 And more recently, we've seen that agents have become a first-class citizen of these eval platforms where you want to be able to use natural language in a headless experience s145 00:14:53.120 --> 00:15:02.440 where you can tell your coding agent of choice, "Find me all traces in the last 24 hours where the user had a poor experience. s146 00:15:02.440 --> 00:15:04.200 Run the evals for me." s147 00:15:04.200 --> 00:15:15.800 And because these coding agents have access to all of your underlying infrastructure and your code base, they become the mechanism to run the evals and log those results. s148 00:15:16.080 --> 00:15:18.720 And then it becomes a question of like, so so what? s149 00:15:18.720 --> 00:15:22.840 Why is this a Why is this important? s150 00:15:22.840 --> 00:15:36.440 Well, the goal is everything that I've shown you so far, it requires the humans, the the engineer, the PM to be deeply involved in this process and it could be laborious s151 00:15:36.440 --> 00:15:38.600 at times. s152 00:15:39.560 --> 00:15:46.560 I BrainTrust, the way we think about this is we want to help you operate at scale. s153 00:15:46.920 --> 00:15:57.640 Rather than you having to think about these dimensions and of success and failure for your AI agent, BrainTrust can surface these insights automatically. s154 00:15:57.640 --> 00:16:05.320 Because we log your tracing data into our platform, we can run inference on them and we can tell you the unknown unknowns. s155 00:16:05.320 --> 00:16:19.120 Hey, what are what are scenarios and when my agent is failing silently or what are scenarios that my users are experiencing frustration by prompting my agent for the same request over and over again. s156 00:16:20.560 --> 00:16:22.880 And it's a lot of things that we didn't talk about. s157 00:16:22.880 --> 00:16:32.080 Things like the underlying database that powers it or having to build our back into the system to manage controls and permissions or data masking. s158 00:16:32.080 --> 00:16:42.960 You know, each one of these features proves that this is not just a UI and a spreadsheet, but rather it is a a systems problem that powers it. s159 00:16:43.400 --> 00:16:54.120 And as we think about the evolution of of the improvement loop, I said this earlier where humans were doing the improvement loop and now we see coding agents where they could iteratively s160 00:16:54.120 --> 00:17:01.760 make changes and suggest what kind of improvements you should be making to your application. s161 00:17:01.760 --> 00:17:06.480 And ultimately, the humans responsibility is to review the outcome. s162 00:17:06.480 --> 00:17:20.240 If I have different iterations of these e-vals that are run by my coding agent, I can look at the outcome and make the decision of which version of it is the best for what I want to ship to production. s163 00:17:20.319 --> 00:17:23.199 So, with that, that is the end of my talk. s164 00:17:23.199 --> 00:17:27.880 I appreciate you all coming out and learning about why it's hard to build eval systems. s165 00:17:27.880 --> 00:17:30.079 Thank you. s166 00:17:44.387 --> 00:17:46.387 [music]