Agents Building Agents - Alfonso Graziano, Nearform

AI Engineer · 30 min · 271 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00Hi everyone.
00:01Today we will talk about agents, building agents.
00:05So, how we are leveraging coding assistance and spec driven development to build reliable and secure agents.
00:12First of all, just a couple of notes about myself.
00:15I'm Alfonso.
00:16I'm a tech lead at NearForm.
00:18We are a services company.
00:21I am currently working on some of our AI agentic projects.
00:25I'm supporting multiple teams adopting AI native engineering and I'm also a O'Reilly author of the book Learning AI Native Software Engineering.
00:35So, we are seeing in the in the industry that right now everyone wants AI agents for workflow automation, search, you know, every use case that you might have in mind.
00:49But, AI agents come with a sometimes a very high cost.
00:54They have hallucinations.
00:55They have an entire new set of problems.
00:57Of course, they are non-deterministic.
01:00So, today we will see how we are building and improving iteratively AI agents.
01:07And of course, as you may guess, how can we do that, right?
01:12AI is very powerful and very good at building any type of software.
01:17And given that AI agents is just one type of software, as you may guess, we are using AI to build AI.
01:25Um As I was mentioning, there are a lot of problems, a lot of different problem classes with building AI agents.
01:34Uh non-determinism is one of those, latency, cost, hallucinations, a lot of stuff really.
01:43Uh but today we will see how to solve them at least partially by leveraging agents and the repeatable process that we developed over time on our projects.
01:54Just as a very very quick refresher, um we can say that an AI agent is basically just an LLM, which is like the brain of the agent.
02:06Then the LLM is connected to a bunch of tools.
02:10It has access to a bunch of context and basically lives into an agentic loop.
02:15So, that's the that's the gist of it.
02:18Of course, there is way more into AI agents.
02:22There is everything around observability.
02:25How do how do we ensure that, you know, it's doing what's what's necessary?
02:30So, everything around evals that we will discuss in a minute.
02:33But like just from the first principles, we can say that an agent is an LLM inside a an agentic loop, which is connected to tools and can retrieve context.
02:46That's it.
02:48Now, there are a couple of classes of problems that we will analyze today.
02:52So, the first one is bad performances on the evals, and we will see what the evals are in a minute.
02:59And the second one is bad performances on live data, which sometimes is similar to what we have on the evals, but you know, live data coming from real users
03:12with, you know, real expectations from the system can be a lot wider and sometimes a lot messier as well.
03:20So, let's analyze the first the first failure mode.
03:24So, we have bad performances on evals.
03:28Before we analyze this failure mode, I just want to introduce you to go to the context of a golden data set.
03:35Basically, a golden data set is a file or a set of files that we develop together with the subject matter experts when we are building our AI agents.
03:46And basically, this file defines what is the input that the system should retrieve um, and should get sorry, and what is the expected output that we should get.
03:57Now, in the naive case, the expected output, as we can see here on the right, can just be like impossible, or like a number, or a text, so it can be a lot of things.
04:10In real-world scenarios, the expected output can be, for example, I do expect the system to call this tool, or call this tool with this parameter, or call this tool in this chain, right?
04:23Because maybe I want to do a retrieval and then do an update, you know, so how we define the output is very different in every in every agent.
04:35And the idea is that we want a way, basically, to ensure that our agent is working correctly.
04:42So, that's why we're building the golden dataset.
04:45You can see the golden dataset as a test suite, but in a non-deterministic scenario.
04:50So, when we build the golden dataset, we also have to build a scorer, or a set of scorers, that can basically, go through the golden dataset together with the LLM,
05:02and then can give us, basically, a number in the end, which is saying, "What is the current accuracy of the of the system?"
05:09So, that, you know, we have a baseline, we can look for regressions, and we can further improve the system over time.
05:17Now, let's let's assume that we have a very simple hello world agent.
05:22In this case, I'm using Mastra.
05:24It's a very very simple agent, as we can see.
05:26I've called it mad agent, but in reality, in the golden dataset, we have multiple types of questions, right?
05:33But, let's assume that this is our very simple agent, right?
05:36So, it doesn't aim, it has just couple of instructions, has has a model, of course.
05:41As you can see, it doesn't have any tools yet.
05:45So, it's like the the bare minimum.
05:48Um we also built a naive evaluator.
05:52So, if you remember from the golden data set, we only have inputs and outputs.
05:57And what this very naive evaluator is doing is checking whether the um LLM output contains the ground truth output that we are expecting.
06:09Of course, once we run the eval suite here, the results are very poor.
06:13So, as we can see, um we have a pass rate of 18%, right?
06:19Uh just because a a lot of questions are simple enough, like additions, multiplications, right?
06:26Are simple enough so that the actual um LLM has this knowledge in its training data, and it doesn't need access to any external context or like any particular tool to answer these questions.
06:41So, 18% of the questions can be answered by the weights of the LLM, the rest can't.
06:47So, we need a way to actually improve this pass rate.
06:53Some failure modes that we can expect are well, first of all, the agent is missing the right tools, right?
06:59So, let's assume that I expect from my agent to be able to surf the web, fetch web pages.
07:06If I'm not giving those specific tools to my agent, of course, every eval that is expecting that, it's going to fail miserably.
07:14So, first of all, we are maybe just missing some tools, or uh we don't have the right system prompt.
07:21Because the system prompt basically contains all the instructions, uh all the behaviors that we should expect, and that we shouldn't expect from our agent.
07:31So, in a lot of cases, um a lot of the optimizations will be just to tweak the system prompt and update it so that um you know, our agent has
07:41all the information it needs to work on our domain.
07:45And then, um of course, context retrieval, which is maybe our agent doesn't know where to look for information.
07:52Of course, usually context retrieval is implemented as tools, right?
07:58Um maybe it's, I don't know, an accurate search in our system.
08:02Maybe it's a fetch on the web, you know, all these kind of things.
08:06So, we need to give enough um let's say tools uh to our agent so that it can retrieve all the context it needs, of course, in a secure way.
08:18Now, the question is can an AI agent improve another AI agent autonomously or semi-autonomously?
08:27And um Karpathy, Andrej Karpathy, which is one of the most well-known AI scientist on the planet at the moment, implemented something which is called auto research.
08:38Basically, auto research is a uh loop which um updates the code, updates the hyper parameters, updates the the actual Python code of machine learning algorithms.
08:52And it actually shows that a coding agent tweaking the code of machine learning um of a deep learning algorithm can actually improve the results, right?
09:04So, that's the um that's that's the loss that we can see.
09:09And basically, every uh dot here in this chart is an example is a task that the system is doing um by changing some of the parameters.
09:21And we can see that the baseline has a specific accuracy.
09:25And while we go um while we go with the number of experiments, accuracy is improving.
09:31And therefore, of course, the loss um the the loss is decreasing.
09:36Now, the question is, can we do the same with AI agents and not just like machine learning models?
09:43The answer is yes.
09:44So, I built something which is called AutoAgent, which is basically the same idea, but applied to um actually like AI agents, right?
09:56So, basically this is a loop which is able to run the evals and then um update the code of the system, try new system prompts, create new tools, doing everything autonomously,
10:09and then check again whether things improved or not.
10:13And it actually works quite well.
10:16So, here we have a few iterations, and this is the AI agent This is the naive AI agent stat agent that we have seen earlier.
10:25And in that agent, as you can see here on the bottom left, um we have the baseline accuracy, which was 18%, and we managed to reach up to 83%
10:38um in like something around 10 uh 10 iterations.
10:42Now, of course, that's a um that that's a very naive case, right?
10:49Because we started with something which didn't have any tools and nothing else.
10:53So, it's relatively easy for the system to improve this agent.
10:58But, uh we also improved um some evals by 10% on a production agent that was already humanly optimized.
11:08So, actually the the machine uh the the agents the the coding agent found new ways that humans didn't find um to improve the agent, and we got plus 10%
11:21on some of our internal benchmarks.
11:24Now, how does it works?
11:25Actually, I I gave already like a few insights of how this works, but let's see it in details.
11:33So, basically, the core idea is that we have a coding agent, in this case Claude Code, but it can work with multiple coding agents.
11:42It builds the agent, so it writes the code of our target agent, and then the target agent is giving feedback to our Claude Code or whatever coding tool we're using
11:55as evals, right?
11:58So, it's giving some information as okay, here I got some regressions or these evals are not passing.
12:06Also, Claude Code can read the traces, the thinking traces, or like the full trace of the target agent to see whether something broke or something is not working as expected, so it can do like self-improvement.
12:21Of course, there is the human in the loop, and the human in the loop is usually in structuring the initial agent and giving as much context as we can
12:31to our coding agent about what it can do with other relevant files, what it cannot do, like just to give an example, updating the golden data sets or the scorers
12:43just to let the evals pass is not a good idea, so we want to enforce we want to tell the we want to tell the AI agent to not do that, right?
12:52So, as humans, we can we can steer a little bit our our coding agent so that it can optimize the target agent.
13:02Now, there are a few steps, right?
13:03The first step is to create an optimization job.
13:07Of course, it's very easy, everything is a markdown, we all love markdown now, so we have we can define the objective, we can define target repository, the metrics, and literally everything, right?
13:20So, in this file, we can give as much context as we want to our coding agent so that it can actually run and improve the target agent.
13:31And then the second step is to run the loop.
13:33So basically, we run the evals once, so we we we generate our baseline data.
13:38So we understand what's working, what's not working, you know, the system will generate a first report.
13:44And then we run the loop, the optimization loop.
13:47So we can see that in the first case, we get a regression, so basically the system works by creating an hypothesis.
13:54So it's tackling one class of problems at a time.
13:58It's updating the the agent and it's running the evals again.
14:02And then it's saying whether we got a regression, an improvement, or something else happened and so the agent can decide whether to roll back the change or move on.
14:14So in this case, in the first iteration, we got a regression, so we want to roll back.
14:19Um then we see a 5% improvement, which is great.
14:24So in this case, we are getting we're creating a new hypothesis, we get a 0% improvement, unfortunately, but of course the agent is keeping track of all the hypothesis
14:35and so in this case can go and create a new one and we can see a 12% hypothesis and and so on and so forth, right?
14:43Of course, we define the number of iterations that we want to run.
14:47Um of course, the way we create the baseline data is, you know, we just run the evals once and we generate basically a document and this is super helpful because
14:59it gives the coding agent the information about the current status of the system.
15:03Then this generates a baseline report which contains, you know, all these informations.
15:10It contains what are the cases, it contains summary, what's working, what's not working.
15:16So basically, our coding agent it's going to have a clear picture of what's happening right now.
15:23And so, it will try to understand, okay, how do I improve this system from this moment?
15:29And so, from now on, we can start with the first iteration.
15:32So, every iteration starts with a new branch.
15:35We create an hypothesis.
15:37Um the system changes the agent to implement that hypothesis.
15:42We run the evals.
15:43We run our eval suite.
15:45And we generate a reports.md file, which contains everything that happened after we run the evals.
15:52We update the memory file, uh which is like a global memory file across all the runs.
15:58And if the metrics improved, then we continue from this branch.
16:02Um if the metrics didn't improve or we have a strong regression or something bad happened, uh then we roll back to the previous branch.
16:10Of course, um the generated hypotheses are based on what the agent reads when it starts the investigation, which is the memory file, uh the reports file, so it has access to pretty much everything.
16:24And that's what happens when we run multiple iterations, right?
16:27So, let's assume that in the first case, I create a branch, I create an hypothesis, uh I run the evals, everything goes well, so I get an improvement, so I am fixing one failure mode.
16:38Um therefore, I continue from that branch.
16:41I create a new one so that I can keep track of every change.
16:45Um I create the new that new branch.
16:48I try new hypothesis.
16:50This time, it didn't really improve, so maybe I get a regression, maybe things didn't improve really.
16:57Uh so, what I do is I roll back to the previous branch, so the first branch where we had a real improvement in the system.
17:04And then from here, I create a new branch, and I try a new hypothesis, and so on and so forth.
17:11At the end, what we're building is a full change log of all the changes we have done, um every improvement in improvement or regression as we can see here
17:24on the right.
17:25We are seeing all the hypothesis that we are doing and of course, if we want we can also see all the hypothesis because every hypothesis is building a report.
17:35So, let's assume that something didn't really go well in terms of evals.
17:41Maybe the evals didn't improve after an hypothesis, but maybe that hypothesis was promising, right?
17:47Maybe the agent was onto something, but it just didn't implement the system the change in a correct way, right?
17:55What we can do is as humans, we can just go back into the hypothesis, read, understand what the agent was trying to do, and then maybe steer it in the right direction next time.
18:07Of course, this is the real task on a on a real agent that we did have.
18:15And as we can see here, the baseline accuracy was 67% um but then in something around 10 iterations, we managed to reach 86% in our evals without actually cheating
18:33because it found edge cases, it improved the system prompt, it improved the tool descriptions to catch more edge cases, and it also fixed some tools logic.
18:44So, that's a real agent that is now running in production and actually this system found ways to improve it more and more.
18:54Now, the second class of problems that we are going to tackle today is not the evals data, but bad performances on live data.
19:05By live data, we mean data that we collected by our real users, beta testers, subject matter experts, you know, all the people that are actively testing and using the system
19:17and they're giving us a feedback.
19:19Just as an example, you can see here on the right, was this response helpful?
19:23And you got yes or no, and then you get a note.
19:26And basically, we leverage this data to uh further optimize our agents.
19:33Now, how do we do that?
19:35Uh this is the full flow that we will see in details.
19:38So, first of all, of course, we will need our users to actually use the system.
19:44Uh then, we collect all the traces, right?
19:47So, we collect all the informations from our users.
19:50Our users can either give us a feedback that we've just seen, so thumbs up, thumbs down with the with the comment, or uh our subject matter experts can annotate the trace.
20:02So, basically, they can go on the platform that we are using, they can look at the trace, look at the input, look at the output, uh look at basically like all the tools that have been called,
20:12uh look at the agent behavior, and then annotate the trace by giving us a feedback about how the agent performed and how it should have performed.
20:23Once we collect a a a relatively good number of traces, once we have um enough information, we basically run an agent workflow that we built.
20:34So, basically, we analyze all the traces with both the negative and positive feedback, but we are more interested about the negative feedback here.
20:43So, we analyze all these traces.
20:46We do clustering of the failure modes, right?
20:48So, based on all the feedback that we collected, we try to understand, okay, given all this five feedback, what are like the five, six, seven failure modes that we have here?
21:00And then, we analyze these um we analyze these clusters of failure modes with our subject matter experts.
21:09So, we say, "Okay, based on those 10 traces, we can note that the agent in this specific case performed poorly because of reason X, for example."
21:20We validate all this with our subject matter experts.
21:23And then we do have a fix proposal, which gets generated and implemented by our coding agent.
21:30Then we implement this and then we ship it to production to see whether it actually improved on real data.
21:37And of course, we use the traces that we do have as regressions.
21:41So, we test our fix against our traces first of all.
21:46But let's see that in detail.
21:49So, first of all, the user tests the system.
21:52So, in this case, we have a very simple example, "How can I optimize a React component?"
21:58Here we have the answer.
21:59Here we have, you know, the information about the component, whatever, right?
22:04So, that's the first step.
22:06Second step is we are going to collect all the tracing information.
22:11So, we are collecting everything from user interaction, the tool usage, the how much time the AI took to respond, you know, how many tokens we burned, everything.
22:25The second step then is the user, once we collected the user behavior, we do expect the user to give us some feedback.
22:35As I was mentioning, thumbs up, thumbs down, and then a a comment saying what went well, what didn't, and what's the expected behavior of the system.
22:47And then of course, the other option that we were discussing is that the actual subject matter experts can annotate the trace.
22:54So, in case you don't have a feedback from the real user, I can ask the subject matter experts to take a look at at trace, invest their time, and give us feedback as they were users.
23:05The third step is now to collect all the traces with feedback and download them locally in a JSON or whatever format you're using.
23:14As you can see here in this case, we have 114 traces, and we're using those traces to then do the cluster analysis.
23:21How do we run the analysis?
23:23Well, this is actually a skill that we built.
23:26Basically, the skill it's going to instruct the coding agent properly to go in the go in the JSON file, look at all the traces, uh run some clustering, um then we do an adversarial review of that.
23:41So, basically, um once we have all the clusters, we do also a little bit of root cause analysis because, of course, our coding agent has access to the actual code of the AI agent,
23:52so it can do some root cause analysis.
23:54It can try to understand what happened, uh especially because it has also access to the details of every trace.
24:01So, it can go within the trace, understand why the agent answered that way, you know, trace it down to the root cause.
24:08Um and then what's going to do is going to um propose possible improvements.
24:13And what we get at the end of this process is a full markdown report, which is very helpful for us.
24:20Uh it's going to give us like the positive feedback, the negative feedback, the negative rate, you know, we are just trying to collect as many data as possible.
24:30And so, we get this very complete um executive summary and this very complete report, which tells us what are like all the clusters which are not working as the user was expected.
24:43So, in this case, uh we have that we have a small markdown formatting failure, which is saying URLs are not hyperlinked, and there are some formatting inconsistencies.
24:54So, it's linking some of the traces with the trace ID.
24:58Uh it's mentioning the user feedback.
25:01It's showing like some root possible root causes and it's telling us, "Okay, how do we fix it?"
25:08Right?
25:08Once we have this full report, our work is not done yet because this report will contain all the possible failure clusters reported by users, but now what we have to do is to triage
25:20everything and validate with subject matter experts.
25:24Once we are doing Once we do the triage and the validation, then we prioritize and we understand what we want to fix now, fix later, don't fix.
25:35Um and so, either we fix it with the agent, so we give to the agent, you know, all the context, we give it the failure mode, we give it the
25:45traces, and we ask our agent to fix it.
25:49Or we discard it because sometimes it can be the the failure cluster can be a false positive or can be an intended behavior or maybe the user gave us
26:00um a feedback which is not really useful for us at this moment.
26:03So, you know, there is a little bit of human judgment here.
26:07As I was mentioning earlier, all the failure modes that we are founding during this investigation step, they will become part of the golden dataset that we mentioned earlier and the eval suite is updated
26:19to spot those regressions.
26:21So, in case these failure modes for whatever reason is introduced uh later on in the code, we will spot it very fast because now it lives into our golden dataset and our scorers.
26:32In terms of how often the report gets generated, it really depends on your use case.
26:38Uh we found out that one per sprint is good enough, but really depends on how much data you have, right?
26:44So, if you have 100 traces, it's you might want to generate that at 100 traces, but if you have like 10,000 traces with feedback, might be too much, so you might want to, you know, generate multiple reports.
26:57So, it's really on your use case.
27:00We found out that once per sprint is actually reasonable.
27:03Uh in multiple use cases, what we found is that a coding agent, when instructed, uh has been able to fix an entire suite of issues like the one that we have seen earlier,
27:16uh with just one prompt.
27:18So, once we give enough context and once we give the coding agent the ability to uh test its changes against some regression tests, uh then everything becomes way, way easier for the for the coding agent.
27:32And then, uh in some complex cases, uh we have paired this approach with AutoAgent, uh and so we basically pointed out AutoAgent to some of the failure clusters, um and we let AutoAgent
27:47build some draft PRs for us.
27:51Now, of course, both AutoAgent and this pipeline to improve um agents based on live data, they're possible thanks to Harness Engineering.
28:01So, basically, the core idea is that we are leveraging a set of things so that our coding agent has a powerful enough harness so that it can change the codes,
28:14validate its own changes, which is like super important, uh and then propose new changes if the updates that it did didn't lead to the to the outcome that we were looking for, right?
28:27So, just very, very quickly, Harness Engineering is the idea of building the environment around our coding agent so that they can work reliably.
28:36Uh so, we are giving all the constraints, all all the tasks, the feedback loop, we are giving the governance to our um coding agents.
28:45And we are doing that with a few things, right?
28:46Which we've already mentioned.
28:48The first one is spectrum environment.
28:50So, for example, when we are trying to fix some um failure modes, every failure mode becomes a spec.
28:58Um we are creating a spec for the expected behavior of the agent, and then we are implementing that.
29:04Then we have quality gates uh like linting uh our unit tests.
29:09We have our our evals.
29:11Um we have LLM uh code review, so a bunch of stuff.
29:16Uh everything related to context engineering, so giving the right context to our agent.
29:21And then, of course, observability.
29:24Um because we need to know what's happening, right?
29:27Um if we don't know what's happening when we ship in production, we are basically blind, right?
29:32So, we want to ensure that we know what's happening so that our coding agent can fix any bugs or anything once once it's found.
29:41So, um I hope that this has been interesting or helpful.
29:46I hope you're going to implement some of those techniques in your agents.
29:49And in case you have any questions about this talk, uh please feel free to reach out on LinkedIn.
29:55I would love to have a chat with other people which are building agents.
29:59I know how hard it is.
30:00I know how interesting it is.
30:02So, in case you have like anything that you would like to talk, uh I would love to get a message on LinkedIn.
30:08So, I hope you enjoyed this talk, and wish you a great day.
30:11Cheers.