WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=A3FL_zTQKU0 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/A3FL_zTQKU0.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.480 --> 00:00:14.520 My name is Stephanie Jarmak. s3 00:00:14.520 --> 00:00:27.200 I am an AI engineer as of a couple weeks ago at Omni Analytics where we have a platform that has like a governance semantic layer to have a shared understanding s4 00:00:27.200 --> 00:00:31.240 of business data using our beloved agent Bloopy. s5 00:00:31.240 --> 00:00:42.960 Before that I spent a year at Sourcegraph where I was a solutions engineer for Amp, a coding agent, and then I moved into some sort of amorphous type role where I was largely doing their evaluation s6 00:00:42.960 --> 00:00:43.800 platform. s7 00:00:43.800 --> 00:00:51.320 I built a very large public benchmark called Code Scale Bench to understand how agents are working across very complex code bases. s8 00:00:51.320 --> 00:00:53.560 And then prior to that I was an astronomer. s9 00:00:53.560 --> 00:01:04.000 So I come from this from a research scientist sort of point of view where I relentlessly question everything, hence the title of Lies, Damned Lies, and Evals. s10 00:01:04.000 --> 00:01:10.840 It says how to trust your agents, but I think largely this talk is mainly about how to not trust your agents. s11 00:01:11.440 --> 00:01:20.400 So here's one example of a run I had done back at Sourcegraph when I was testing out their code search MCP tool. s12 00:01:20.400 --> 00:01:26.800 This is just to show all of the different sort of numbers that you can have after you run a certain evaluation. s13 00:01:26.800 --> 00:01:34.800 The what I was trying to do was figure out hey, why isn't this coding agent actually using the tool that I'm trying to get it to use? s14 00:01:34.800 --> 00:01:43.520 And so you know it ran, I have all the traces and everything and then I you know looked at some of the primary sort of metrics that you're looking at when you evaluate the overall s15 00:01:43.520 --> 00:01:55.680 effectiveness of a tool and it showed we basically what I had done was I changed the initial prompt given to the agent and then I ran it cuz I wanted it to use the tool. s16 00:01:55.680 --> 00:01:59.120 And what I saw was okay, uh, it was 28% cheaper. s17 00:01:59.120 --> 00:02:00.840 That's that's interesting. s18 00:02:00.840 --> 00:02:10.119 Um, but, you know, that shouldn't really change any of my decisions around what I'm actually trying what I'm interested in measuring and understanding in changes to um, the actual s19 00:02:10.119 --> 00:02:13.840 harness and setup, uh, because it still didn't use the tool. s20 00:02:13.840 --> 00:02:14.200 Right? s21 00:02:14.200 --> 00:02:19.000 So, you need to be able to understand like what it is you're actually setting this up. s22 00:02:19.000 --> 00:02:23.000 The the 28% um, is maybe an interesting avenue to go explore. s23 00:02:23.000 --> 00:02:31.240 Okay, we have some level of like experimental space to be figuring out what's the variance and what's going on with the with the cost measurements. s24 00:02:31.240 --> 00:02:37.240 Um, but we still weren't like seeing it actually use the tool um, that it was supposed to. s25 00:02:37.240 --> 00:02:47.120 And, uh, basically we had this whole like, um, indication that it was receiving certain information and then just trusting that that information wasn't there. s26 00:02:47.120 --> 00:02:50.880 And that was purely because it didn't use that tool. s27 00:02:50.880 --> 00:02:55.280 And I also want to talk a bit about the impact of the harness itself. s28 00:02:55.280 --> 00:03:07.880 Um, so, here's an example, and this was from a third-party source where they had 63 or so tasks, um, on Turbo Bench 4, which is a pretty popular, widely used, um, benchmark for coding agents. s29 00:03:07.880 --> 00:03:13.760 And they, you know, you have like let's say you're trying to pick which model you're going to, um, put into your harness. s30 00:03:13.760 --> 00:03:18.959 Um, so, here's a comparison they did they did they did with, um, Opus 5 then and Astra. s31 00:03:18.959 --> 00:03:24.760 And they saw that, you know, with Open Hands, which is one of the open harnesses, um, Opus 5 beat it out. s32 00:03:24.760 --> 00:03:31.120 But then when they swapped the just the harness, um, itself, then they saw that Astra did a a much better job, right? s33 00:03:31.120 --> 00:03:37.880 So, it was like 30-plus percent sort of difference difference in the, um, effectiveness of these configurations. s34 00:03:37.880 --> 00:03:51.560 So, the takeaway here is you want to be actually evaluating the closest like configuration that you can that's going to reflect the the work and the environment that you're going to be measuring because there are so many different variables involved. s35 00:03:51.560 --> 00:04:00.960 And, you know, you might be uh, you know, you might see all these um, benchmarks and whatnot and see the the fancy results, but it might not do what you expect it to do within your actual s36 00:04:00.960 --> 00:04:02.560 environment. s37 00:04:02.560 --> 00:04:13.440 Um, which leads a bit into what I kind of disambiguate and expand a little bit more on how to interpret um like public benchmarks versus what is an evaluation. s38 00:04:13.440 --> 00:04:21.120 So, an evaluation is you're making measurements to make a decision about your product or process and those changes. s39 00:04:21.120 --> 00:04:30.480 And the the public benchmarks as they exist um are really primarily helpful for you to have a broader understanding of like the larger landscape and get a short list s40 00:04:30.480 --> 00:04:36.040 of what you might be exploring within your own um organization or your own um sort of tooling. s41 00:04:36.040 --> 00:04:45.120 Um, but you don't want to necessarily over index too much on what some of those results are because of the spiky nature of all of the training of these different models and things like that. s42 00:04:45.120 --> 00:04:55.960 So, you know, if you are a law firm, um you don't want to necessarily be looking at like the terminal bench results unless of course you are looking at incorporating coding agents s43 00:04:55.960 --> 00:04:56.880 with your devs, right? s44 00:04:56.880 --> 00:05:02.520 So, you want to make sure that the configurations reflecting um your actual environments. s45 00:05:02.520 --> 00:05:11.440 Um, and then the you can also like think of benchmarks really as just a um standardized sort of reference set for comparison. s46 00:05:11.440 --> 00:05:30.520 So, you have these public benchmarks, but then you can also set up benchmarks within your organization to use as like setting them up as reflective of actual workloads as possible and just have this sort of iterative flywheel process where you are um increasingly sort of hill climbing to to optimize for your s47 00:05:30.520 --> 00:05:32.680 for your configurations. s48 00:05:33.720 --> 00:05:48.000 And uh you want to approach this uh to the best of your ability as a as a scientist kind of when like once you sort of short list what your various configurations might be, you want to look at what are the different variables we're going to think about changing s49 00:05:48.000 --> 00:05:54.240 um so that we're actually going to be like that the results will reflect what our system is, right? s50 00:05:54.240 --> 00:05:56.240 So you can think of workload coverage. s51 00:05:56.240 --> 00:06:01.600 So you want to pick tasks that are going to actually be reflective and you could cluster them into different families and things like that. s52 00:06:01.600 --> 00:06:11.480 Like when I was making the coding benchmark, um didn't want to just like the reason I built out a whole different one was because at the time it was largely like STB bench was just bug s53 00:06:11.480 --> 00:06:18.400 uh fixing, but there's a whole other different swath of tasks that are relevant to developers across the software development life cycle. s54 00:06:18.400 --> 00:06:24.360 And the way that you integrate different tooling into those different like workloads is going to potentially be different. s55 00:06:24.360 --> 00:06:30.560 So like a model might be more effective at code understanding than it is at actually fixing a bug, for example. s56 00:06:30.560 --> 00:06:41.160 And then you want to try to see, okay, how many like different runs do I want to do um because as we all know, these systems are non-deterministic and so you might see some level of variation s57 00:06:41.160 --> 00:06:43.200 um across run to run. s58 00:06:43.200 --> 00:06:53.960 Um and then as I had shown before with like that different configuration of the model and the harness, you want to think about how is how are the different sort of um combinations of potential s59 00:06:53.960 --> 00:06:58.240 changes going to have these um interaction type effects, right? s60 00:06:58.240 --> 00:07:03.960 And how how is that going to potentially be um influencing downstream effects? s61 00:07:05.560 --> 00:07:10.880 And you also want to be able to gather as much observability data as possible. s62 00:07:10.880 --> 00:07:13.680 So, you know, retain all of your traces and everything. s63 00:07:13.680 --> 00:07:21.560 Um so this sort of goes back to one of my the first examples when I was trying to figure out like what is happening, why why is it not s64 00:07:21.560 --> 00:07:23.000 um calling this tool. s65 00:07:23.000 --> 00:07:31.560 So first, you know, you want to sort of break it down into what is the sort of entire uh context available to the to the agent. s66 00:07:31.560 --> 00:07:33.760 So it was that tool even available, right? s67 00:07:33.760 --> 00:07:39.160 Cuz if it wasn't, then you okay, you stop there and you you fix that um particular, you know, scenario. s68 00:07:39.160 --> 00:07:50.560 And then you see was it called, um etc. or like what what were the values returned so you like basically keep going along the the pipeline of available um, information. s69 00:07:50.560 --> 00:08:01.040 Um, and I this one um, the like so as you're going along that pipeline let's say uh, I do this improvement, right? s70 00:08:01.040 --> 00:08:04.360 For this uh, tool it finally uses the much better tool. s71 00:08:04.360 --> 00:08:10.680 Um, but then the actual like outcome that you care about at the end of the task, like that still doesn't change, right? s72 00:08:10.680 --> 00:08:23.400 So there are scenarios too where you can have this beautiful tool that you're like really optimizing for, um, but then the agent the configuration or or or whatnot for whatever reason which then is your job to go in and and figure out and fix, s73 00:08:23.400 --> 00:08:25.760 um, it's not leveraging um, that information. s74 00:08:25.760 --> 00:08:35.599 So you want to keep all of these different um, things in mind like the non-use and the failed attempts um, and everything um, when you're trying to basically be a detective s75 00:08:35.599 --> 00:08:40.960 in understanding what is actually going on um, in your system. s76 00:08:41.200 --> 00:08:49.040 And uh, you also want to uh, really control what the measurement is that you're making. s77 00:08:49.040 --> 00:08:59.560 Um, so in this example, um, when I was doing the uh, tests of um, the comparisons of different sort of uh, search tools, right? s78 00:08:59.560 --> 00:09:06.560 I was interested in optimizing for like a better basically like an F1 score balancing for precision in addition to recall. s79 00:09:06.560 --> 00:09:08.880 Um, so in this example was with like grep, right? s80 00:09:08.880 --> 00:09:13.839 So Claude or whatever were going out and basically pulling in the entire kitchen sink. s81 00:09:13.839 --> 00:09:21.280 Um, so in this example Claude went out and got like 295 files returned when it only really needed six. s82 00:09:21.280 --> 00:09:30.080 So you know, if I was somebody who in this uh, scenario was like, okay, all I really care about is that it pulls in any possibly relevant piece of information, s83 00:09:30.080 --> 00:09:37.680 then maybe a recall of one is sufficient, but in my scenario I was interested in optimizing for a completely different um, type of value. s84 00:09:37.680 --> 00:09:48.280 Um so, this is relevant to for when you're looking at those like single number scores and public benchmarks, like what was the actual validator measuring and why did they consider that to be important? s85 00:09:48.280 --> 00:09:55.000 Um cuz what what you might want is going to be dependent entirely like on your business needs. s86 00:09:55.240 --> 00:10:05.920 And um there's like a interesting sort of overlap that kind of happens when you're thinking about evals um and then sort of quality gates and reviews and things like that. s87 00:10:05.920 --> 00:10:12.000 So, I just sort of wanted to touch on that a little bit here um in terms of like the trust and the safety. s88 00:10:12.000 --> 00:10:27.040 Um so, I have like a software factory that I've been experimenting with having um these different approval uh queues essentially of I want to automate as much as possible with uh for the work to get done, but I also do not trust these systems like at all. s89 00:10:27.040 --> 00:10:29.760 Um so, I'll have these uh regular audits. s90 00:10:29.760 --> 00:10:40.400 Um So, uh as an example, um basically I will have like a rep- like put through the request for some work to get done um and then it it um s91 00:10:40.400 --> 00:10:52.880 has approval and required checks, but uh basically what was happening was the agent noticed things were going wrong, but then there was no mechanism in place for it to actually like enforce s92 00:10:52.880 --> 00:10:55.320 uh a stopping situation. s93 00:10:55.320 --> 00:11:09.360 So, you need to sort of decouple what's happening with um an agent or some other system like flagging something versus um actually enforcing that um like gate within the code itself. s94 00:11:09.360 --> 00:11:13.920 And then how that is eventually potentially escalated to a human. s95 00:11:13.920 --> 00:11:21.040 Um cuz this sort of goes into you need to also think about balancing that human review load, right? s96 00:11:21.040 --> 00:11:31.240 So, if you have for in this this comes from um basically auditing and repairing LLM as a judge um failures that were happening um in a separate system. s97 00:11:31.240 --> 00:11:35.600 Um but they were seeing like a 1% uh, false alarm rate. s98 00:11:35.600 --> 00:11:42.600 And if all of those, like, um, basically the alarms that are raised go to a human, that would result in 10,000 reviews. s99 00:11:42.600 --> 00:11:48.040 And like, does your organization actually have the capacity for domain experts to be going and doing that? s100 00:11:48.040 --> 00:11:59.640 Um, so it's, you know, you need to think about where that bottleneck might be and how you, um, like, change what the sort of alarm rates, um, tend to be. s101 00:11:59.640 --> 00:12:07.720 And, you know, what, how consequential are those, are those, um, alarms going to impact like downstream effects? s102 00:12:09.120 --> 00:12:20.600 And, uh, we want to also consider that, um, you know, the actual agent trace itself can sometimes not be, um, reliable or like trustworthy. s103 00:12:20.600 --> 00:12:33.800 So, in this particular example, um, researchers were doing, um, some checks to basically seed in, um, different sort of like security issues, um, that were provided to the agent. s104 00:12:33.800 --> 00:12:39.800 And, um, they had like a hash check in place to check for like any sort of tampering with that system. s105 00:12:39.800 --> 00:12:45.160 Um, and with, uh, using that setup, basically, uh, everything passed. s106 00:12:45.160 --> 00:12:50.320 It didn't It didn't notice, um, any of those, um, injected security issues. s107 00:12:50.320 --> 00:13:03.440 So, but when they had an independent, um, sort of, uh, record comparison, um, take a look at basically all of the traffic history in the model, um, uh, changes and things like that, then they were able to detect everything. s108 00:13:03.440 --> 00:13:12.520 So, you want to make sure that like, depending on the security needs of your system and how you're actually measuring and and observing, um, what's going on, that you s109 00:13:12.520 --> 00:13:20.760 have sufficient sort of, um, trust in, uh, the the actual record that you're retaining of what happened. s110 00:13:21.839 --> 00:13:34.040 And, so this is basically just like an overview of, um, like, going back to that, original um, example of, you know, how we're trying to think about um evaluating s111 00:13:34.040 --> 00:13:39.880 realistic sort of work um associated with your particular um task loads. s112 00:13:39.880 --> 00:13:44.880 So, you want to identify like a target work comparison. s113 00:13:44.880 --> 00:13:49.920 So, um I'm doing a lot of this right now at um Omni and working on their evaluation harness. s114 00:13:49.920 --> 00:14:01.840 So, we're um looking at our internal production data and seeing how uh our users are interacting with the agents and sort of uh having a classification of the different types of s115 00:14:01.840 --> 00:14:03.360 um things that can be done. s116 00:14:03.360 --> 00:14:08.560 So, in our example, it's um Q&A, which may be easy questions to harder questions. s117 00:14:08.560 --> 00:14:11.280 Or it could be dashboard building and app building and things like that. s118 00:14:11.280 --> 00:14:15.840 So, there's all these different sort of ways of classifying the actual tests. s119 00:14:15.840 --> 00:14:30.600 Um and then you want to um be like identifying um are these changes actually like the putting into the system actually uh adding value or where the failures in downstream? s120 00:14:30.600 --> 00:14:32.760 Um and we're doing all of this, right? s121 00:14:32.760 --> 00:14:40.360 And trying to come up with representative proxies of actual uh workloads, but ultimately like production is truth. s122 00:14:40.360 --> 00:14:52.160 So, after you deploy some of these changes, then you need to also have measurement systems in place to get all of that relevant data in the actual environment that that your system is being deployed in. s123 00:14:52.160 --> 00:14:59.920 And you can do this in shadow and whatnot as well to to gain some of that information if if some of it would be kind of um less trustworthy s124 00:14:59.920 --> 00:15:05.320 to to change and you have a higher blast radius uh radius of of effects and things like that. s125 00:15:05.320 --> 00:15:20.360 Um but ultimately, you want to be like essentially optimizing for as much observability um and realistic environments for where your system is going to be deployed for you to be able to trust some of these um values. s126 00:15:20.839 --> 00:15:34.200 So, you to um identify like what level of um sort of risk you are going to be um willing to accept. s127 00:15:34.200 --> 00:15:53.280 Um so, the way I have it sort of set up within my own um systems um is I within my software factory, like as an example, um for like giving the agent some level of um autonomy in making some of the lower risk um decisions s128 00:15:53.280 --> 00:16:07.240 is um uh basically, they will identify what the next um action would like be, like why it's needed from a human to um be able to weigh in on that, what the actual operating s129 00:16:07.240 --> 00:16:08.800 scope is. s130 00:16:08.800 --> 00:16:20.000 Um and then, you know, what um like potential um suggested sort of human check removed, like given the available um sort of spaces. s131 00:16:20.000 --> 00:16:30.280 But, um you want to be able to come up with those different sort of boundaries of when is it safe to autonomous autonomously deploy versus um stop and escalate s132 00:16:30.280 --> 00:16:31.920 to the human. s133 00:16:31.920 --> 00:16:38.600 Um so, I have a GitHub repo with a bunch of the different um skills um that I've worked on. s134 00:16:38.600 --> 00:16:47.720 Um there's a few in there that have uh direct uh relevance to um evaluation design and harness traceability and reliability procedures. s135 00:16:47.720 --> 00:16:55.360 And then, I also have this like monstrous 300-page archive tome that, you know, you can point your agents at. s136 00:16:55.360 --> 00:16:57.520 Um you can point your agents at all of this, right? s137 00:16:57.520 --> 00:17:02.040 Um to like pull in some of those best practices into your own systems. s138 00:17:02.040 --> 00:17:04.360 So, thank you. s139 00:17:04.431 --> 00:17:06.431 [applause]