WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=aHhB3sjGjkI NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/aHhB3sjGjkI.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.960 --> 00:00:01.720 Hi everyone. s2 00:00:01.720 --> 00:00:05.120 Today we will talk about agents, building agents. s3 00:00:05.120 --> 00:00:12.840 So, how we are leveraging coding assistance and spec driven development to build reliable and secure agents. s4 00:00:12.840 --> 00:00:15.400 First of all, just a couple of notes about myself. s5 00:00:15.400 --> 00:00:16.520 I'm Alfonso. s6 00:00:16.520 --> 00:00:18.320 I'm a tech lead at NearForm. s7 00:00:18.320 --> 00:00:21.040 We are a services company. s8 00:00:21.040 --> 00:00:25.200 I am currently working on some of our AI agentic projects. s9 00:00:25.200 --> 00:00:35.880 I'm supporting multiple teams adopting AI native engineering and I'm also a O'Reilly author of the book Learning AI Native Software Engineering. s10 00:00:35.880 --> 00:00:49.360 So, we are seeing in the in the industry that right now everyone wants AI agents for workflow automation, search, you know, every use case that you might have in mind. s11 00:00:49.360 --> 00:00:54.440 But, AI agents come with a sometimes a very high cost. s12 00:00:54.440 --> 00:00:55.640 They have hallucinations. s13 00:00:55.640 --> 00:00:57.960 They have an entire new set of problems. s14 00:00:57.960 --> 00:01:00.440 Of course, they are non-deterministic. s15 00:01:00.440 --> 00:01:07.680 So, today we will see how we are building and improving iteratively AI agents. s16 00:01:07.680 --> 00:01:12.120 And of course, as you may guess, how can we do that, right? s17 00:01:12.120 --> 00:01:17.160 AI is very powerful and very good at building any type of software. s18 00:01:17.160 --> 00:01:25.960 And given that AI agents is just one type of software, as you may guess, we are using AI to build AI. s19 00:01:25.960 --> 00:01:34.920 Um As I was mentioning, there are a lot of problems, a lot of different problem classes with building AI agents. s20 00:01:34.920 --> 00:01:43.040 Uh non-determinism is one of those, latency, cost, hallucinations, a lot of stuff really. s21 00:01:43.040 --> 00:01:54.800 Uh but today we will see how to solve them at least partially by leveraging agents and the repeatable process that we developed over time on our projects. s22 00:01:54.800 --> 00:02:06.280 Just as a very very quick refresher, um we can say that an AI agent is basically just an LLM, which is like the brain of the agent. s23 00:02:06.280 --> 00:02:10.320 Then the LLM is connected to a bunch of tools. s24 00:02:10.320 --> 00:02:15.680 It has access to a bunch of context and basically lives into an agentic loop. s25 00:02:15.680 --> 00:02:18.320 So, that's the that's the gist of it. s26 00:02:18.320 --> 00:02:22.000 Of course, there is way more into AI agents. s27 00:02:22.000 --> 00:02:25.160 There is everything around observability. s28 00:02:25.160 --> 00:02:30.440 How do how do we ensure that, you know, it's doing what's what's necessary? s29 00:02:30.440 --> 00:02:33.480 So, everything around evals that we will discuss in a minute. s30 00:02:33.480 --> 00:02:46.360 But like just from the first principles, we can say that an agent is an LLM inside a an agentic loop, which is connected to tools and can retrieve context. s31 00:02:46.360 --> 00:02:48.200 That's it. s32 00:02:48.200 --> 00:02:52.760 Now, there are a couple of classes of problems that we will analyze today. s33 00:02:52.760 --> 00:02:59.640 So, the first one is bad performances on the evals, and we will see what the evals are in a minute. s34 00:02:59.640 --> 00:03:12.160 And the second one is bad performances on live data, which sometimes is similar to what we have on the evals, but you know, live data coming from real users s35 00:03:12.160 --> 00:03:20.440 with, you know, real expectations from the system can be a lot wider and sometimes a lot messier as well. s36 00:03:20.440 --> 00:03:24.600 So, let's analyze the first the first failure mode. s37 00:03:24.600 --> 00:03:28.160 So, we have bad performances on evals. s38 00:03:28.160 --> 00:03:35.480 Before we analyze this failure mode, I just want to introduce you to go to the context of a golden data set. s39 00:03:35.480 --> 00:03:46.239 Basically, a golden data set is a file or a set of files that we develop together with the subject matter experts when we are building our AI agents. s40 00:03:46.239 --> 00:03:57.520 And basically, this file defines what is the input that the system should retrieve um, and should get sorry, and what is the expected output that we should get. s41 00:03:57.520 --> 00:04:10.240 Now, in the naive case, the expected output, as we can see here on the right, can just be like impossible, or like a number, or a text, so it can be a lot of things. s42 00:04:10.240 --> 00:04:23.080 In real-world scenarios, the expected output can be, for example, I do expect the system to call this tool, or call this tool with this parameter, or call this tool in this chain, right? s43 00:04:23.080 --> 00:04:35.680 Because maybe I want to do a retrieval and then do an update, you know, so how we define the output is very different in every in every agent. s44 00:04:35.680 --> 00:04:42.880 And the idea is that we want a way, basically, to ensure that our agent is working correctly. s45 00:04:42.880 --> 00:04:45.000 So, that's why we're building the golden dataset. s46 00:04:45.000 --> 00:04:50.480 You can see the golden dataset as a test suite, but in a non-deterministic scenario. s47 00:04:50.480 --> 00:05:02.440 So, when we build the golden dataset, we also have to build a scorer, or a set of scorers, that can basically, go through the golden dataset together with the LLM, s48 00:05:02.440 --> 00:05:09.960 and then can give us, basically, a number in the end, which is saying, "What is the current accuracy of the of the system?" s49 00:05:09.960 --> 00:05:17.320 So, that, you know, we have a baseline, we can look for regressions, and we can further improve the system over time. s50 00:05:17.320 --> 00:05:22.640 Now, let's let's assume that we have a very simple hello world agent. s51 00:05:22.640 --> 00:05:24.560 In this case, I'm using Mastra. s52 00:05:24.560 --> 00:05:26.840 It's a very very simple agent, as we can see. s53 00:05:26.840 --> 00:05:33.680 I've called it mad agent, but in reality, in the golden dataset, we have multiple types of questions, right? s54 00:05:33.680 --> 00:05:36.560 But, let's assume that this is our very simple agent, right? s55 00:05:36.560 --> 00:05:41.720 So, it doesn't aim, it has just couple of instructions, has has a model, of course. s56 00:05:41.720 --> 00:05:45.280 As you can see, it doesn't have any tools yet. s57 00:05:45.280 --> 00:05:48.960 So, it's like the the bare minimum. s58 00:05:48.960 --> 00:05:52.320 Um we also built a naive evaluator. s59 00:05:52.320 --> 00:05:57.680 So, if you remember from the golden data set, we only have inputs and outputs. s60 00:05:57.680 --> 00:06:09.560 And what this very naive evaluator is doing is checking whether the um LLM output contains the ground truth output that we are expecting. s61 00:06:09.560 --> 00:06:13.640 Of course, once we run the eval suite here, the results are very poor. s62 00:06:13.640 --> 00:06:19.360 So, as we can see, um we have a pass rate of 18%, right? s63 00:06:19.360 --> 00:06:26.080 Uh just because a a lot of questions are simple enough, like additions, multiplications, right? s64 00:06:26.080 --> 00:06:41.120 Are simple enough so that the actual um LLM has this knowledge in its training data, and it doesn't need access to any external context or like any particular tool to answer these questions. s65 00:06:41.120 --> 00:06:47.160 So, 18% of the questions can be answered by the weights of the LLM, the rest can't. s66 00:06:47.160 --> 00:06:52.760 So, we need a way to actually improve this pass rate. s67 00:06:53.440 --> 00:06:59.480 Some failure modes that we can expect are well, first of all, the agent is missing the right tools, right? s68 00:06:59.480 --> 00:07:06.160 So, let's assume that I expect from my agent to be able to surf the web, fetch web pages. s69 00:07:06.160 --> 00:07:14.240 If I'm not giving those specific tools to my agent, of course, every eval that is expecting that, it's going to fail miserably. s70 00:07:14.240 --> 00:07:21.200 So, first of all, we are maybe just missing some tools, or uh we don't have the right system prompt. s71 00:07:21.200 --> 00:07:31.080 Because the system prompt basically contains all the instructions, uh all the behaviors that we should expect, and that we shouldn't expect from our agent. s72 00:07:31.080 --> 00:07:41.280 So, in a lot of cases, um a lot of the optimizations will be just to tweak the system prompt and update it so that um you know, our agent has s73 00:07:41.280 --> 00:07:45.120 all the information it needs to work on our domain. s74 00:07:45.120 --> 00:07:52.240 And then, um of course, context retrieval, which is maybe our agent doesn't know where to look for information. s75 00:07:52.240 --> 00:07:58.880 Of course, usually context retrieval is implemented as tools, right? s76 00:07:58.880 --> 00:08:02.920 Um maybe it's, I don't know, an accurate search in our system. s77 00:08:02.920 --> 00:08:06.360 Maybe it's a fetch on the web, you know, all these kind of things. s78 00:08:06.360 --> 00:08:18.480 So, we need to give enough um let's say tools uh to our agent so that it can retrieve all the context it needs, of course, in a secure way. s79 00:08:18.480 --> 00:08:27.040 Now, the question is can an AI agent improve another AI agent autonomously or semi-autonomously? s80 00:08:27.080 --> 00:08:38.960 And um Karpathy, Andrej Karpathy, which is one of the most well-known AI scientist on the planet at the moment, implemented something which is called auto research. s81 00:08:38.960 --> 00:08:52.240 Basically, auto research is a uh loop which um updates the code, updates the hyper parameters, updates the the actual Python code of machine learning algorithms. s82 00:08:52.240 --> 00:09:04.640 And it actually shows that a coding agent tweaking the code of machine learning um of a deep learning algorithm can actually improve the results, right? s83 00:09:04.640 --> 00:09:09.280 So, that's the um that's that's the loss that we can see. s84 00:09:09.280 --> 00:09:21.240 And basically, every uh dot here in this chart is an example is a task that the system is doing um by changing some of the parameters. s85 00:09:21.240 --> 00:09:25.920 And we can see that the baseline has a specific accuracy. s86 00:09:25.920 --> 00:09:31.040 And while we go um while we go with the number of experiments, accuracy is improving. s87 00:09:31.040 --> 00:09:36.000 And therefore, of course, the loss um the the loss is decreasing. s88 00:09:36.000 --> 00:09:43.080 Now, the question is, can we do the same with AI agents and not just like machine learning models? s89 00:09:43.080 --> 00:09:44.480 The answer is yes. s90 00:09:44.480 --> 00:09:56.400 So, I built something which is called AutoAgent, which is basically the same idea, but applied to um actually like AI agents, right? s91 00:09:56.400 --> 00:10:09.240 So, basically this is a loop which is able to run the evals and then um update the code of the system, try new system prompts, create new tools, doing everything autonomously, s92 00:10:09.240 --> 00:10:13.480 and then check again whether things improved or not. s93 00:10:13.480 --> 00:10:16.000 And it actually works quite well. s94 00:10:16.000 --> 00:10:25.400 So, here we have a few iterations, and this is the AI agent This is the naive AI agent stat agent that we have seen earlier. s95 00:10:25.400 --> 00:10:38.440 And in that agent, as you can see here on the bottom left, um we have the baseline accuracy, which was 18%, and we managed to reach up to 83% s96 00:10:38.440 --> 00:10:42.640 um in like something around 10 uh 10 iterations. s97 00:10:42.640 --> 00:10:49.280 Now, of course, that's a um that that's a very naive case, right? s98 00:10:49.280 --> 00:10:53.800 Because we started with something which didn't have any tools and nothing else. s99 00:10:53.800 --> 00:10:58.040 So, it's relatively easy for the system to improve this agent. s100 00:10:58.040 --> 00:11:08.240 But, uh we also improved um some evals by 10% on a production agent that was already humanly optimized. s101 00:11:08.240 --> 00:11:21.320 So, actually the the machine uh the the agents the the coding agent found new ways that humans didn't find um to improve the agent, and we got plus 10% s102 00:11:21.320 --> 00:11:24.400 on some of our internal benchmarks. s103 00:11:24.400 --> 00:11:25.800 Now, how does it works? s104 00:11:25.800 --> 00:11:33.960 Actually, I I gave already like a few insights of how this works, but let's see it in details. s105 00:11:33.960 --> 00:11:42.680 So, basically, the core idea is that we have a coding agent, in this case Claude Code, but it can work with multiple coding agents. s106 00:11:42.680 --> 00:11:55.640 It builds the agent, so it writes the code of our target agent, and then the target agent is giving feedback to our Claude Code or whatever coding tool we're using s107 00:11:55.640 --> 00:11:58.280 as evals, right? s108 00:11:58.280 --> 00:12:06.200 So, it's giving some information as okay, here I got some regressions or these evals are not passing. s109 00:12:06.280 --> 00:12:21.200 Also, Claude Code can read the traces, the thinking traces, or like the full trace of the target agent to see whether something broke or something is not working as expected, so it can do like self-improvement. s110 00:12:21.800 --> 00:12:31.200 Of course, there is the human in the loop, and the human in the loop is usually in structuring the initial agent and giving as much context as we can s111 00:12:31.200 --> 00:12:43.040 to our coding agent about what it can do with other relevant files, what it cannot do, like just to give an example, updating the golden data sets or the scorers s112 00:12:43.040 --> 00:12:52.720 just to let the evals pass is not a good idea, so we want to enforce we want to tell the we want to tell the AI agent to not do that, right? s113 00:12:52.720 --> 00:13:02.240 So, as humans, we can we can steer a little bit our our coding agent so that it can optimize the target agent. s114 00:13:02.240 --> 00:13:03.920 Now, there are a few steps, right? s115 00:13:03.920 --> 00:13:07.080 The first step is to create an optimization job. s116 00:13:07.080 --> 00:13:20.240 Of course, it's very easy, everything is a markdown, we all love markdown now, so we have we can define the objective, we can define target repository, the metrics, and literally everything, right? s117 00:13:20.240 --> 00:13:31.200 So, in this file, we can give as much context as we want to our coding agent so that it can actually run and improve the target agent. s118 00:13:31.200 --> 00:13:33.120 And then the second step is to run the loop. s119 00:13:33.120 --> 00:13:38.480 So basically, we run the evals once, so we we we generate our baseline data. s120 00:13:38.480 --> 00:13:44.760 So we understand what's working, what's not working, you know, the system will generate a first report. s121 00:13:44.760 --> 00:13:47.240 And then we run the loop, the optimization loop. s122 00:13:47.240 --> 00:13:54.640 So we can see that in the first case, we get a regression, so basically the system works by creating an hypothesis. s123 00:13:54.640 --> 00:13:58.240 So it's tackling one class of problems at a time. s124 00:13:58.240 --> 00:14:02.400 It's updating the the agent and it's running the evals again. s125 00:14:02.400 --> 00:14:14.840 And then it's saying whether we got a regression, an improvement, or something else happened and so the agent can decide whether to roll back the change or move on. s126 00:14:14.840 --> 00:14:19.960 So in this case, in the first iteration, we got a regression, so we want to roll back. s127 00:14:19.960 --> 00:14:24.120 Um then we see a 5% improvement, which is great. s128 00:14:24.120 --> 00:14:35.520 So in this case, we are getting we're creating a new hypothesis, we get a 0% improvement, unfortunately, but of course the agent is keeping track of all the hypothesis s129 00:14:35.520 --> 00:14:43.080 and so in this case can go and create a new one and we can see a 12% hypothesis and and so on and so forth, right? s130 00:14:43.080 --> 00:14:47.880 Of course, we define the number of iterations that we want to run. s131 00:14:47.880 --> 00:14:59.480 Um of course, the way we create the baseline data is, you know, we just run the evals once and we generate basically a document and this is super helpful because s132 00:14:59.480 --> 00:15:03.920 it gives the coding agent the information about the current status of the system. s133 00:15:03.920 --> 00:15:10.080 Then this generates a baseline report which contains, you know, all these informations. s134 00:15:10.080 --> 00:15:16.120 It contains what are the cases, it contains summary, what's working, what's not working. s135 00:15:16.120 --> 00:15:23.120 So basically, our coding agent it's going to have a clear picture of what's happening right now. s136 00:15:23.120 --> 00:15:29.320 And so, it will try to understand, okay, how do I improve this system from this moment? s137 00:15:29.320 --> 00:15:32.640 And so, from now on, we can start with the first iteration. s138 00:15:32.640 --> 00:15:35.640 So, every iteration starts with a new branch. s139 00:15:35.640 --> 00:15:37.440 We create an hypothesis. s140 00:15:37.440 --> 00:15:42.040 Um the system changes the agent to implement that hypothesis. s141 00:15:42.040 --> 00:15:43.280 We run the evals. s142 00:15:43.280 --> 00:15:45.480 We run our eval suite. s143 00:15:45.480 --> 00:15:52.080 And we generate a reports.md file, which contains everything that happened after we run the evals. s144 00:15:52.080 --> 00:15:58.320 We update the memory file, uh which is like a global memory file across all the runs. s145 00:15:58.320 --> 00:16:02.480 And if the metrics improved, then we continue from this branch. s146 00:16:02.480 --> 00:16:10.520 Um if the metrics didn't improve or we have a strong regression or something bad happened, uh then we roll back to the previous branch. s147 00:16:10.520 --> 00:16:24.440 Of course, um the generated hypotheses are based on what the agent reads when it starts the investigation, which is the memory file, uh the reports file, so it has access to pretty much everything. s148 00:16:24.440 --> 00:16:27.720 And that's what happens when we run multiple iterations, right? s149 00:16:27.720 --> 00:16:38.480 So, let's assume that in the first case, I create a branch, I create an hypothesis, uh I run the evals, everything goes well, so I get an improvement, so I am fixing one failure mode. s150 00:16:38.480 --> 00:16:41.400 Um therefore, I continue from that branch. s151 00:16:41.400 --> 00:16:45.839 I create a new one so that I can keep track of every change. s152 00:16:45.839 --> 00:16:48.040 Um I create the new that new branch. s153 00:16:48.040 --> 00:16:50.320 I try new hypothesis. s154 00:16:50.320 --> 00:16:57.040 This time, it didn't really improve, so maybe I get a regression, maybe things didn't improve really. s155 00:16:57.040 --> 00:17:04.800 Uh so, what I do is I roll back to the previous branch, so the first branch where we had a real improvement in the system. s156 00:17:04.800 --> 00:17:11.280 And then from here, I create a new branch, and I try a new hypothesis, and so on and so forth. s157 00:17:11.360 --> 00:17:24.480 At the end, what we're building is a full change log of all the changes we have done, um every improvement in improvement or regression as we can see here s158 00:17:24.480 --> 00:17:25.959 on the right. s159 00:17:25.959 --> 00:17:35.840 We are seeing all the hypothesis that we are doing and of course, if we want we can also see all the hypothesis because every hypothesis is building a report. s160 00:17:35.840 --> 00:17:41.920 So, let's assume that something didn't really go well in terms of evals. s161 00:17:41.920 --> 00:17:47.480 Maybe the evals didn't improve after an hypothesis, but maybe that hypothesis was promising, right? s162 00:17:47.480 --> 00:17:55.840 Maybe the agent was onto something, but it just didn't implement the system the change in a correct way, right? s163 00:17:55.840 --> 00:18:07.560 What we can do is as humans, we can just go back into the hypothesis, read, understand what the agent was trying to do, and then maybe steer it in the right direction next time. s164 00:18:07.800 --> 00:18:15.920 Of course, this is the real task on a on a real agent that we did have. s165 00:18:15.920 --> 00:18:33.120 And as we can see here, the baseline accuracy was 67% um but then in something around 10 iterations, we managed to reach 86% in our evals without actually cheating s166 00:18:33.120 --> 00:18:44.480 because it found edge cases, it improved the system prompt, it improved the tool descriptions to catch more edge cases, and it also fixed some tools logic. s167 00:18:44.480 --> 00:18:54.040 So, that's a real agent that is now running in production and actually this system found ways to improve it more and more. s168 00:18:54.040 --> 00:19:05.000 Now, the second class of problems that we are going to tackle today is not the evals data, but bad performances on live data. s169 00:19:05.000 --> 00:19:17.360 By live data, we mean data that we collected by our real users, beta testers, subject matter experts, you know, all the people that are actively testing and using the system s170 00:19:17.360 --> 00:19:19.160 and they're giving us a feedback. s171 00:19:19.160 --> 00:19:23.400 Just as an example, you can see here on the right, was this response helpful? s172 00:19:23.400 --> 00:19:26.280 And you got yes or no, and then you get a note. s173 00:19:26.280 --> 00:19:33.400 And basically, we leverage this data to uh further optimize our agents. s174 00:19:33.400 --> 00:19:35.400 Now, how do we do that? s175 00:19:35.400 --> 00:19:38.960 Uh this is the full flow that we will see in details. s176 00:19:38.960 --> 00:19:44.160 So, first of all, of course, we will need our users to actually use the system. s177 00:19:44.160 --> 00:19:47.000 Uh then, we collect all the traces, right? s178 00:19:47.000 --> 00:19:50.920 So, we collect all the informations from our users. s179 00:19:50.920 --> 00:20:02.640 Our users can either give us a feedback that we've just seen, so thumbs up, thumbs down with the with the comment, or uh our subject matter experts can annotate the trace. s180 00:20:02.640 --> 00:20:12.800 So, basically, they can go on the platform that we are using, they can look at the trace, look at the input, look at the output, uh look at basically like all the tools that have been called, s181 00:20:12.800 --> 00:20:23.560 uh look at the agent behavior, and then annotate the trace by giving us a feedback about how the agent performed and how it should have performed. s182 00:20:23.560 --> 00:20:34.640 Once we collect a a a relatively good number of traces, once we have um enough information, we basically run an agent workflow that we built. s183 00:20:34.640 --> 00:20:43.720 So, basically, we analyze all the traces with both the negative and positive feedback, but we are more interested about the negative feedback here. s184 00:20:43.720 --> 00:20:46.120 So, we analyze all these traces. s185 00:20:46.120 --> 00:20:48.800 We do clustering of the failure modes, right? s186 00:20:48.800 --> 00:21:00.360 So, based on all the feedback that we collected, we try to understand, okay, given all this five feedback, what are like the five, six, seven failure modes that we have here? s187 00:21:00.360 --> 00:21:09.400 And then, we analyze these um we analyze these clusters of failure modes with our subject matter experts. s188 00:21:09.400 --> 00:21:20.200 So, we say, "Okay, based on those 10 traces, we can note that the agent in this specific case performed poorly because of reason X, for example." s189 00:21:20.200 --> 00:21:23.720 We validate all this with our subject matter experts. s190 00:21:23.720 --> 00:21:30.440 And then we do have a fix proposal, which gets generated and implemented by our coding agent. s191 00:21:30.440 --> 00:21:37.160 Then we implement this and then we ship it to production to see whether it actually improved on real data. s192 00:21:37.160 --> 00:21:41.960 And of course, we use the traces that we do have as regressions. s193 00:21:41.960 --> 00:21:46.720 So, we test our fix against our traces first of all. s194 00:21:46.720 --> 00:21:49.120 But let's see that in detail. s195 00:21:49.120 --> 00:21:52.200 So, first of all, the user tests the system. s196 00:21:52.200 --> 00:21:58.360 So, in this case, we have a very simple example, "How can I optimize a React component?" s197 00:21:58.360 --> 00:21:59.840 Here we have the answer. s198 00:21:59.840 --> 00:22:04.200 Here we have, you know, the information about the component, whatever, right? s199 00:22:04.200 --> 00:22:06.360 So, that's the first step. s200 00:22:06.360 --> 00:22:11.960 Second step is we are going to collect all the tracing information. s201 00:22:11.960 --> 00:22:24.960 So, we are collecting everything from user interaction, the tool usage, the how much time the AI took to respond, you know, how many tokens we burned, everything. s202 00:22:25.120 --> 00:22:35.520 The second step then is the user, once we collected the user behavior, we do expect the user to give us some feedback. s203 00:22:35.520 --> 00:22:47.240 As I was mentioning, thumbs up, thumbs down, and then a a comment saying what went well, what didn't, and what's the expected behavior of the system. s204 00:22:47.240 --> 00:22:54.000 And then of course, the other option that we were discussing is that the actual subject matter experts can annotate the trace. s205 00:22:54.000 --> 00:23:05.240 So, in case you don't have a feedback from the real user, I can ask the subject matter experts to take a look at at trace, invest their time, and give us feedback as they were users. s206 00:23:05.240 --> 00:23:14.040 The third step is now to collect all the traces with feedback and download them locally in a JSON or whatever format you're using. s207 00:23:14.040 --> 00:23:21.840 As you can see here in this case, we have 114 traces, and we're using those traces to then do the cluster analysis. s208 00:23:21.840 --> 00:23:23.760 How do we run the analysis? s209 00:23:23.760 --> 00:23:26.760 Well, this is actually a skill that we built. s210 00:23:26.760 --> 00:23:41.080 Basically, the skill it's going to instruct the coding agent properly to go in the go in the JSON file, look at all the traces, uh run some clustering, um then we do an adversarial review of that. s211 00:23:41.080 --> 00:23:52.680 So, basically, um once we have all the clusters, we do also a little bit of root cause analysis because, of course, our coding agent has access to the actual code of the AI agent, s212 00:23:52.680 --> 00:23:54.280 so it can do some root cause analysis. s213 00:23:54.280 --> 00:24:01.240 It can try to understand what happened, uh especially because it has also access to the details of every trace. s214 00:24:01.240 --> 00:24:08.360 So, it can go within the trace, understand why the agent answered that way, you know, trace it down to the root cause. s215 00:24:08.360 --> 00:24:13.680 Um and then what's going to do is going to um propose possible improvements. s216 00:24:13.680 --> 00:24:20.920 And what we get at the end of this process is a full markdown report, which is very helpful for us. s217 00:24:20.920 --> 00:24:30.240 Uh it's going to give us like the positive feedback, the negative feedback, the negative rate, you know, we are just trying to collect as many data as possible. s218 00:24:30.240 --> 00:24:43.720 And so, we get this very complete um executive summary and this very complete report, which tells us what are like all the clusters which are not working as the user was expected. s219 00:24:43.720 --> 00:24:54.880 So, in this case, uh we have that we have a small markdown formatting failure, which is saying URLs are not hyperlinked, and there are some formatting inconsistencies. s220 00:24:54.880 --> 00:24:58.920 So, it's linking some of the traces with the trace ID. s221 00:24:58.920 --> 00:25:01.480 Uh it's mentioning the user feedback. s222 00:25:01.480 --> 00:25:08.200 It's showing like some root possible root causes and it's telling us, "Okay, how do we fix it?" s223 00:25:08.200 --> 00:25:08.800 Right? s224 00:25:08.800 --> 00:25:20.680 Once we have this full report, our work is not done yet because this report will contain all the possible failure clusters reported by users, but now what we have to do is to triage s225 00:25:20.680 --> 00:25:24.600 everything and validate with subject matter experts. s226 00:25:24.600 --> 00:25:35.760 Once we are doing Once we do the triage and the validation, then we prioritize and we understand what we want to fix now, fix later, don't fix. s227 00:25:35.760 --> 00:25:45.680 Um and so, either we fix it with the agent, so we give to the agent, you know, all the context, we give it the failure mode, we give it the s228 00:25:45.680 --> 00:25:49.400 traces, and we ask our agent to fix it. s229 00:25:49.400 --> 00:26:00.040 Or we discard it because sometimes it can be the the failure cluster can be a false positive or can be an intended behavior or maybe the user gave us s230 00:26:00.040 --> 00:26:03.760 um a feedback which is not really useful for us at this moment. s231 00:26:03.760 --> 00:26:07.520 So, you know, there is a little bit of human judgment here. s232 00:26:07.520 --> 00:26:19.440 As I was mentioning earlier, all the failure modes that we are founding during this investigation step, they will become part of the golden dataset that we mentioned earlier and the eval suite is updated s233 00:26:19.440 --> 00:26:21.520 to spot those regressions. s234 00:26:21.520 --> 00:26:32.880 So, in case these failure modes for whatever reason is introduced uh later on in the code, we will spot it very fast because now it lives into our golden dataset and our scorers. s235 00:26:32.880 --> 00:26:38.240 In terms of how often the report gets generated, it really depends on your use case. s236 00:26:38.240 --> 00:26:44.360 Uh we found out that one per sprint is good enough, but really depends on how much data you have, right? s237 00:26:44.360 --> 00:26:57.880 So, if you have 100 traces, it's you might want to generate that at 100 traces, but if you have like 10,000 traces with feedback, might be too much, so you might want to, you know, generate multiple reports. s238 00:26:57.880 --> 00:27:00.000 So, it's really on your use case. s239 00:27:00.000 --> 00:27:03.800 We found out that once per sprint is actually reasonable. s240 00:27:03.800 --> 00:27:16.600 Uh in multiple use cases, what we found is that a coding agent, when instructed, uh has been able to fix an entire suite of issues like the one that we have seen earlier, s241 00:27:16.600 --> 00:27:18.120 uh with just one prompt. s242 00:27:18.120 --> 00:27:32.880 So, once we give enough context and once we give the coding agent the ability to uh test its changes against some regression tests, uh then everything becomes way, way easier for the for the coding agent. s243 00:27:32.880 --> 00:27:47.640 And then, uh in some complex cases, uh we have paired this approach with AutoAgent, uh and so we basically pointed out AutoAgent to some of the failure clusters, um and we let AutoAgent s244 00:27:47.640 --> 00:27:51.720 build some draft PRs for us. s245 00:27:51.800 --> 00:28:01.560 Now, of course, both AutoAgent and this pipeline to improve um agents based on live data, they're possible thanks to Harness Engineering. s246 00:28:01.560 --> 00:28:14.440 So, basically, the core idea is that we are leveraging a set of things so that our coding agent has a powerful enough harness so that it can change the codes, s247 00:28:14.440 --> 00:28:27.560 validate its own changes, which is like super important, uh and then propose new changes if the updates that it did didn't lead to the to the outcome that we were looking for, right? s248 00:28:27.560 --> 00:28:36.600 So, just very, very quickly, Harness Engineering is the idea of building the environment around our coding agent so that they can work reliably. s249 00:28:36.600 --> 00:28:45.040 Uh so, we are giving all the constraints, all all the tasks, the feedback loop, we are giving the governance to our um coding agents. s250 00:28:45.040 --> 00:28:46.960 And we are doing that with a few things, right? s251 00:28:46.960 --> 00:28:48.280 Which we've already mentioned. s252 00:28:48.280 --> 00:28:50.680 The first one is spectrum environment. s253 00:28:50.680 --> 00:28:58.440 So, for example, when we are trying to fix some um failure modes, every failure mode becomes a spec. s254 00:28:58.440 --> 00:29:04.520 Um we are creating a spec for the expected behavior of the agent, and then we are implementing that. s255 00:29:04.520 --> 00:29:09.600 Then we have quality gates uh like linting uh our unit tests. s256 00:29:09.600 --> 00:29:11.600 We have our our evals. s257 00:29:11.600 --> 00:29:16.040 Um we have LLM uh code review, so a bunch of stuff. s258 00:29:16.040 --> 00:29:21.360 Uh everything related to context engineering, so giving the right context to our agent. s259 00:29:21.360 --> 00:29:24.080 And then, of course, observability. s260 00:29:24.080 --> 00:29:27.200 Um because we need to know what's happening, right? s261 00:29:27.200 --> 00:29:32.600 Um if we don't know what's happening when we ship in production, we are basically blind, right? s262 00:29:32.600 --> 00:29:41.720 So, we want to ensure that we know what's happening so that our coding agent can fix any bugs or anything once once it's found. s263 00:29:41.720 --> 00:29:46.120 So, um I hope that this has been interesting or helpful. s264 00:29:46.120 --> 00:29:49.840 I hope you're going to implement some of those techniques in your agents. s265 00:29:49.840 --> 00:29:55.320 And in case you have any questions about this talk, uh please feel free to reach out on LinkedIn. s266 00:29:55.320 --> 00:29:59.280 I would love to have a chat with other people which are building agents. s267 00:29:59.280 --> 00:30:00.680 I know how hard it is. s268 00:30:00.680 --> 00:30:02.560 I know how interesting it is. s269 00:30:02.560 --> 00:30:08.320 So, in case you have like anything that you would like to talk, uh I would love to get a message on LinkedIn. s270 00:30:08.320 --> 00:30:11.840 So, I hope you enjoyed this talk, and wish you a great day. s271 00:30:11.840 --> 00:30:13.840 Cheers.