WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=pSto5YaNGUo NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/pSto5YaNGUo.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.400 --> 00:00:03.280 Hi everybody. s2 00:00:03.280 --> 00:00:06.280 Welcome to our talk the Agentic AI engineer. s3 00:00:06.280 --> 00:00:07.520 I'm Bene. s4 00:00:07.520 --> 00:00:12.760 CEO and co-founder of Mutagent and I'm here with my colleague. s5 00:00:13.720 --> 00:00:14.720 Hi, I'm Burak. s6 00:00:14.720 --> 00:00:24.920 I'm the CTO of Mutagent and today we're basically going to talk about loops and how the Agentic AI engineer works. s7 00:00:28.120 --> 00:00:35.440 So, as you all are aware of now, loops is the hot topic how you build software in an Agentic loop. s8 00:00:35.440 --> 00:00:40.400 And we apply the same loop to the building of AI agents. s9 00:00:40.400 --> 00:00:44.160 And as you all aware, there is uh two concepts here. s10 00:00:44.160 --> 00:00:54.120 One is the offline loop where while you build, you iterate um on your agent, you test it, you evaluate it, you improve it, and you go on. s11 00:00:54.120 --> 00:01:05.400 And then you have a second loop which is we call the online loop where once the agent is deployed to production, you monitor its traces, you diagnose it, and then you feed it back into your optimization s12 00:01:05.400 --> 00:01:11.760 loop uh yeah, to iterate and have multiple versions of your agents. s13 00:01:13.400 --> 00:01:20.280 Yeah, up to until now um what we did was uh just doing this loop manually. s14 00:01:20.280 --> 00:01:21.760 It's quite slow. s15 00:01:21.760 --> 00:01:28.480 Um the life cycle is basically um you have an issue, um you want to change something to your agent. s16 00:01:28.480 --> 00:01:30.880 Um yeah, you implement the change. s17 00:01:30.880 --> 00:01:31.045 Um s18 00:01:31.045 --> 00:01:31.280 [snorts] s19 00:01:31.280 --> 00:01:35.240 you maybe vibe implement the change if you use coding agents for it. s20 00:01:35.240 --> 00:01:39.720 Um yeah, you generate some samples for this new feature or issue to test it. s21 00:01:39.720 --> 00:01:45.120 Um yeah, then you look at the result, you look through the traces, how does the outcome look like. s22 00:01:45.120 --> 00:01:51.520 Then you maybe ship it, you do AB testing, and all your feedback is kind of manually, it takes very long. s23 00:01:51.520 --> 00:02:01.720 Um yeah, and uh the the bottleneck basically becomes the human review and the human yeah, building time. s24 00:02:01.720 --> 00:02:22.680 And yeah, that you can't scale especially if in your organization you're not planning to roll out hundreds of agents etc. Yeah, and uh yeah, this is why we think the Agentic AI engineer is the natural next step to build agents. s25 00:02:22.680 --> 00:02:33.040 And I'll have Burak deep dive into how we improve timing and the road to production reliability with the Agentic AI engineer. s26 00:02:33.040 --> 00:02:49.480 Uh so, yeah, the key thing here is basically once you reach a certain number of agents or AI-based features, the human performing this loop again cannot really scale in enough time. s27 00:02:49.480 --> 00:03:00.880 So, this is why doing this agentically is the key to increasing the throughput because then you can fit many more cycles into the same time window. s28 00:03:00.880 --> 00:03:09.080 And now how that loop works is basically we have a few stages. s29 00:03:09.080 --> 00:03:23.239 So, this is when you are starting from scratch like the current software development practices, you first create a spec for your agent or your skill in this case. s30 00:03:23.239 --> 00:03:36.120 And here you need to define all the responsibilities and the functions that agents needs to handle, the decisions that it has to make on certain conditions. s31 00:03:36.120 --> 00:03:40.280 And here again, this is only the definition stage. s32 00:03:40.280 --> 00:03:46.519 Once you defined your agent's requirements, then you can finally go on to the build. s33 00:03:46.519 --> 00:04:01.240 And build is where you then realize that spec in a specific harness or uh agent framework or in these days you could even build it as a cloud code or a codex agent. s34 00:04:01.240 --> 00:04:03.280 Then comes the next step. s35 00:04:03.280 --> 00:04:17.560 This is where you define clear evaluations to evaluate your agent's performance because these are the key metrics then where you can say, "Hey, my agent is functional or not." s36 00:04:17.560 --> 00:04:23.120 Uh can think of essentially the equivalent to unit tests for coding. s37 00:04:23.120 --> 00:04:27.480 This is how you verify your agent works. s38 00:04:27.480 --> 00:04:37.600 Then after evaluation, if everything looks fine, you usually have the ship basically where you deploy this agent to production. s39 00:04:37.600 --> 00:04:40.640 Uh again, this can be a code update. s40 00:04:40.640 --> 00:04:48.800 This can be a direct update on a any agent platform or again your local harness agents. s41 00:04:48.800 --> 00:04:51.000 Then comes the online part. s42 00:04:51.000 --> 00:05:01.600 This is where then the agent is continuously monitored for issues and based on certain trigger conditions, then you can start automatic diagnostics. s43 00:05:01.600 --> 00:05:10.680 Again, this can be based on the volume of traces that your agent generates or, you know, weekly or daily jobs. s44 00:05:10.680 --> 00:05:13.680 Uh then we go into diagnosis stage. s45 00:05:13.680 --> 00:05:25.160 This is where you collect all the failures for your agents and do structured root cause analysis to then understand where the failures are coming from. s46 00:05:25.160 --> 00:05:31.720 Once you understand and categorize the failures, then you can finally go on to the optimization stage. s47 00:05:31.720 --> 00:05:43.680 This is where then you create, let's say, very specific changes or mutations for your agents and to deal with the found failure modes. s48 00:05:43.680 --> 00:05:46.919 And then the whole cycle repeats again. s49 00:05:46.919 --> 00:05:52.480 You evaluate and if everything looks good, then you can deploy again. s50 00:05:52.480 --> 00:05:59.760 Now, we will maybe do a deep dive on each stage, what that entails. s51 00:06:00.080 --> 00:06:01.080 Then s52 00:06:01.080 --> 00:06:07.360 So, before we continue, Burak, um we have two passes here. s53 00:06:07.360 --> 00:06:11.000 Like one is the cold start path, and one is basically existing features. s54 00:06:11.000 --> 00:06:18.520 Um should we dive deeper for like a half a minute on what differences here, what we see, and why this is important. s55 00:06:18.520 --> 00:06:19.800 Yeah. s56 00:06:19.800 --> 00:06:27.560 Yeah, so today, if you again sit down to build an agent, there are, you know, two options. s57 00:06:27.560 --> 00:06:36.120 Either you already have an agent, it's built, it's already running somewhere with um, you know, certain accuracy. s58 00:06:36.120 --> 00:06:43.240 However, the other option is again you are creating from scratch. s59 00:06:43.240 --> 00:06:50.240 So, when you create from scratch, obviously you design with the spec and the conceptualization stage. s60 00:06:50.240 --> 00:07:00.840 If you have an existing feature, the most likely again the agent is there, but then you are optimizing over something that already exists. s61 00:07:01.040 --> 00:07:02.440 Okay, cool. s62 00:07:02.440 --> 00:07:02.479 And s63 00:07:02.479 --> 00:07:04.120 [snorts] s64 00:07:04.120 --> 00:07:06.160 Yeah, let's dive deeper into each phase. s65 00:07:06.160 --> 00:07:14.800 I mean, you just mentioned the specking of new building agents, so this seems to be an important artifact, so let's dive into it. s66 00:07:15.560 --> 00:07:16.960 Right. s67 00:07:16.960 --> 00:07:36.680 So, with the spec-driven development, also prevalent for building software artifacts these days, or any kind of coding agent workflows, basically the goal is to capture the requirements for the agent, and especially s68 00:07:36.680 --> 00:07:39.480 the success criteria. s69 00:07:39.760 --> 00:07:56.880 The reason being, apart from coding agents, the agents in other domains they handle specific processes and this differs from company to company, so it's very specialized to the environment s70 00:07:56.880 --> 00:07:59.160 where the agent is in. s71 00:07:59.160 --> 00:08:13.560 And then the key point here is to clearly define which context requirements that the agent has, again which integrations and tools it then needs to have. s72 00:08:13.560 --> 00:08:21.760 What are the jobs to be done or the responsibilities that the agent will handle and what it will not. s73 00:08:21.760 --> 00:08:32.000 And then finally again the constraints and in general the boundaries for the agent. s74 00:08:32.520 --> 00:08:32.800 Okay. s75 00:08:32.800 --> 00:08:46.440 Now once we can define a clear spec as I said it becomes like a blueprint for any future development which then the implementation is held against. s76 00:08:46.600 --> 00:08:47.360 Okay. s77 00:08:47.360 --> 00:08:53.560 And let's dive into how we would then build an agent from the spec, right? s78 00:08:53.560 --> 00:08:54.760 Right. s79 00:08:54.760 --> 00:09:10.720 So basically then the spec tells your coding agent what to build and then here the target platform choice is entirely yours. s80 00:09:10.720 --> 00:09:19.320 Cuz as I mentioned agent space is very changing very rapidly these days. s81 00:09:19.320 --> 00:09:26.720 So the framework you are using today, you might want to change, you know, in a year or so. s82 00:09:26.720 --> 00:09:35.320 So essentially because of that spec is isolated from the implementation details. s83 00:09:35.320 --> 00:09:41.160 So once you decide to you can pick any target. s84 00:09:41.160 --> 00:09:53.960 Here then your coding agent of your choice will take that spec and give you a, you know, initial version of that agent, which is then basically customized to run on any platform s85 00:09:53.960 --> 00:09:55.720 that you see fit. s86 00:09:55.720 --> 00:10:00.560 Why would I want to change the platform in a year down the line? s87 00:10:00.560 --> 00:10:04.640 So, what what what did we learn out of experience here? s88 00:10:04.640 --> 00:10:05.520 Um s89 00:10:05.520 --> 00:10:18.120 Uh again building agents for the last 3 years, sometimes the agent framework or the harness does not always have the capabilities. s90 00:10:18.120 --> 00:10:34.440 So, occasionally you hit um like a bottleneck or a roadblock, and then you have to rely on the underlying framework to kind of get rid of that, and this can sometimes take a while. s91 00:10:34.440 --> 00:10:49.960 The key here is to be flexible because yeah, essentially again, it you want to pick the best harness or the framework that can, you know, fulfill your requirements. s92 00:10:49.960 --> 00:10:51.600 Mhm. s93 00:10:51.600 --> 00:11:05.760 I mean, we've all seen it in the in the last months, we how new harnesses shipped and yeah, how everything went from like a agents building code towards yeah, defined as like a agent loop run time. s94 00:11:05.760 --> 00:11:10.360 I mean, we've seen Hermes coming up, deep agents, and all of these frameworks, right? s95 00:11:10.360 --> 00:11:12.160 Yeah. s96 00:11:12.160 --> 00:11:14.080 Okay, let's continue. s97 00:11:14.080 --> 00:11:16.280 What happens after build? s98 00:11:16.280 --> 00:11:29.160 Now, after you build for agents, you essentially go into the eval-driven development loop, which I would call. s99 00:11:29.160 --> 00:11:40.920 And this is kind of equivalent to test-driven development for building software with agents because then the agents needs uh termination condition, right? s100 00:11:40.920 --> 00:11:46.840 So, when is an AI feature or an agent is good enough? s101 00:11:46.840 --> 00:11:59.760 So, here again there are two ways to kind of create your eval suite, which is composed of the evals like the metrics and the criteria you evaluate. s102 00:11:59.760 --> 00:12:06.880 And the data sets themselves, which usually contain the cases you have to satisfy. s103 00:12:06.880 --> 00:12:34.760 Now, in the beginning, the original option is that you can sit with your domain experts and then try to write down eval metrics and criteria that would cover the feature or the agent you want to build, but most teams working on that will already know that this is a bit difficult s104 00:12:34.760 --> 00:12:41.600 as in you cannot pre-guess the entire evaluation suite from the beginning. s105 00:12:41.600 --> 00:12:57.160 And secondly, you can always start with historical data or synthesized data from a known you know, sample of the data that you would like agent to be tested on. s106 00:12:57.160 --> 00:13:04.600 But essentially, the real and the complete eval suite is a product of discovery. s107 00:13:04.600 --> 00:13:26.600 What that means is over time, from user feedback, from production failures, you collect the metrics and criteria plus the additional data set cases, uh which is often representative for edge cases or you know, hard cases that the agent needs to deal with. s108 00:13:26.600 --> 00:13:38.680 And with that, then you finally have an evaluation suit where you can run the agent against an exactly know where it fails. s109 00:13:38.920 --> 00:13:44.000 Okay, before we continue, why does the Agentic AI engineer help us here so much? s110 00:13:44.000 --> 00:13:47.520 So, um obviously we're talking about the evaluator agent. s111 00:13:47.520 --> 00:13:56.040 So, why is it so why is it so good to use this kind of concept or thought process in the building? s112 00:13:56.040 --> 00:14:04.840 Well, I mean one issue is again, imagine you have a data set item of 200. s113 00:14:04.840 --> 00:14:14.920 Here, without automated evals, running this and evaluating by like human eyes takes quite a while. s114 00:14:14.920 --> 00:14:29.640 Um you would have to again scroll through an observability dashboard and logs and this in turn increases your loop time per eval state. s115 00:14:29.640 --> 00:14:41.600 So, as soon as you have a lot of features that you need to evaluate and experiment on, then it suddenly becomes impossible to, you know, do this quickly or in parallel. s116 00:14:41.600 --> 00:14:45.400 Then the human essentially becomes the bottleneck. s117 00:14:45.400 --> 00:14:45.920 Okay. s118 00:14:45.920 --> 00:14:50.640 So, we just have an agent do this work in sifting through traces. s119 00:14:50.640 --> 00:15:04.680 Yes, as the current era says a bit, you know, um you design loops for your agent so then they can autonomously work as many of these things in the background. s120 00:15:04.680 --> 00:15:12.240 Uh and then your job becomes designing these loops with a clear eval or termination gate. s121 00:15:12.240 --> 00:15:13.000 Okay. s122 00:15:13.000 --> 00:15:13.720 Understood. s123 00:15:13.720 --> 00:15:23.040 So, let's look at the how good eval uh supposed to be constructed and what it kind of evaluates. s124 00:15:23.040 --> 00:15:24.040 Right. s125 00:15:24.040 --> 00:15:42.640 So, in the context of agents in general, the mainly the trajectory is important because agents receive input or let's say intent or tasks and then they have a specific s126 00:15:42.640 --> 00:15:48.920 system prompt or like a decision tree that they kind of operate over. s127 00:15:48.920 --> 00:15:56.800 And when doing agent evaluations, again in general, we check, "Hey, was the context complete?" s128 00:15:56.800 --> 00:16:03.800 As in, did the agent have all the required context to perform the task end to end? s129 00:16:03.800 --> 00:16:19.440 Then this includes also chain checking every, you know, tool output in the trajectory because every wrong tool output in session can in the end lead to our wrong output s130 00:16:19.440 --> 00:16:21.880 as the final output. s131 00:16:21.880 --> 00:16:29.760 And apart from that, again, there are different things you can evaluate on. s132 00:16:29.760 --> 00:16:40.440 So, these days also the harness that the agent is operating on has quite drastic effects on the agent behavior. s133 00:16:40.440 --> 00:16:44.640 And this is again another vector of optimization. s134 00:16:44.640 --> 00:16:54.920 But in the end, when you evaluate an agent, you would like to evaluate all of these things and not something in just isolation. s135 00:16:54.920 --> 00:16:56.920 Mhm. s136 00:16:58.000 --> 00:16:59.720 Right. s137 00:16:59.720 --> 00:17:06.280 And in general, here, what makes an eval useful? s138 00:17:06.280 --> 00:17:22.680 So, you can always have you know, metrics or evals that are working in a LLM as a judge fashion and give you some score, but here, in order to make an eval, s139 00:17:22.680 --> 00:17:27.760 you know, useful, it has to provide actionable feedback. s140 00:17:27.760 --> 00:17:42.640 Uh, as in when you use score-based evals, unless your rubric is very well defined, then this does not exactly tell you what to fix. s141 00:17:42.640 --> 00:17:55.480 Uh, in such cases, using uh, binary type of evals or criteria, is preferred because there you have a kind of a call to action. s142 00:17:55.480 --> 00:18:05.120 If an eval or a criteria fails, then you know exactly what happened and how to kind of deal with that problem. s143 00:18:05.120 --> 00:18:23.440 Then, uh, another point is your LLM as a judge solution should be calibrated so that you don't have the, uh, scoring noise between judges because since LLM LLMs are undeterministic, s144 00:18:23.440 --> 00:18:33.200 what you will mostly encounter is the same judge can, uh, evaluate a problem different ways on each run. s145 00:18:33.200 --> 00:18:40.720 And then here, you have to make sure your LLM as a judge solution deals with this variance problem. s146 00:18:40.720 --> 00:18:54.120 Otherwise, it's hard to run experiments and conclusively say, "Hey, my initial or my improved version is better than my initial version of my agent." s147 00:18:54.120 --> 00:18:56.120 Mhm. s148 00:18:57.200 --> 00:18:58.800 Right. s149 00:18:58.800 --> 00:19:20.800 Uh, after then evaluation, basically, we go live and this is where then, uh, collecting, let's say, learned failures, failure or primary signals from the execution takes place. s150 00:19:20.800 --> 00:19:32.360 Um here the idea is again when an agent encounters a problem over time, there will most likely be multiple occurrences of these. s151 00:19:32.360 --> 00:19:50.640 So, it starts by identifying what failure mode uh that the agent has encountered and then grouping essentially these failure modes by the root causes and where they originate from. s152 00:19:50.640 --> 00:19:53.680 Again, this could be a section in the agent prompt. s153 00:19:53.680 --> 00:20:00.920 This could be missing tools or, let's say, malfunctioning tools. s154 00:20:00.920 --> 00:20:10.440 But, after your diagnostics results are categorized, then you can finally generate new evaluations. s155 00:20:10.440 --> 00:20:13.800 First, to detect these problems. s156 00:20:13.800 --> 00:20:21.000 And then, second, you can generate improvements and remedies based on these problems. s157 00:20:24.280 --> 00:20:33.800 And yeah, with that, over time, you have um build-up of these learned failure modes. s158 00:20:33.800 --> 00:20:44.840 So, then every agent over time uh gathers these historical data that it can always check against. s159 00:20:44.840 --> 00:20:58.960 When you uh start diagnosing, there's a bit of a upfront cost in the beginning because you often need to deep read the LLM traces to find out what's going on. s160 00:20:58.960 --> 00:21:06.720 Over time, you can collect code-checkable indicators per failure mode. s161 00:21:06.720 --> 00:21:22.000 What that means is there are maybe specific pieces of content or there's a specific tool called sequence where you know that the agent will encounter an issue and then this later on helps you to diagnose s162 00:21:22.000 --> 00:21:28.480 problems in your traces without actually having to read through all your traces. s163 00:21:28.480 --> 00:21:30.600 Now, why is that a problem? s164 00:21:30.600 --> 00:21:44.640 Because if you have let's say millions of agent traces and to read all of these uh it actually costs more than the execution itself. s165 00:21:44.640 --> 00:21:50.960 So, it's not the most efficient way of diagnosing the problems. s166 00:21:50.960 --> 00:22:12.280 Uh what you want to aim for is again try to uh pick a representative sample from your um whole traces by you know intelligent segmentation strategies and using again as I said the learned indicators. s167 00:22:13.600 --> 00:22:14.680 Okay, cool. s168 00:22:14.680 --> 00:22:36.720 And then with that you can finally start building the autonomous optimization loop because here then given that you have a eval suit, you can you know vary your feature again update certain s169 00:22:36.720 --> 00:22:47.720 or even run auto research style experiments to see that whether you can reach the desired or the target scores for your evals. s170 00:22:47.720 --> 00:22:52.880 As long as you can reach that then it's automatically shipped to production. s171 00:22:52.880 --> 00:22:58.080 Then from production you have your outer loop. s172 00:22:58.080 --> 00:23:16.480 Again, every issue that's found then on diagnostics gives you a improvement or a remedy which then you can optimize on and then as long as the eval suite is green then this again gets deployed to production. s173 00:23:16.480 --> 00:23:19.840 Let's continue now with the whole life cycle. s174 00:23:19.840 --> 00:23:25.680 Um yeah, as we learned now the details of each everything starts with a spec. s175 00:23:25.680 --> 00:23:27.920 Um you define and design it. s176 00:23:27.920 --> 00:23:32.160 Uh you build your agent and all of this can be done obviously agentic. s177 00:23:32.160 --> 00:23:34.960 So you define agents that do this work. s178 00:23:34.960 --> 00:23:38.880 And then they all together become uh the agentic AI engineer. s179 00:23:38.880 --> 00:23:51.320 Um you go through the offline loop where once you build your evaluation system uh yeah, you evaluate, you optimize, you test, you improve your accuracy on your test data set. s180 00:23:51.320 --> 00:23:54.720 Once you feel ready, you deploy it to production. s181 00:23:54.720 --> 00:23:57.080 And that's when the online loop starts. s182 00:23:57.080 --> 00:23:59.560 Here you get real feedback from users. s183 00:23:59.560 --> 00:24:13.960 Um yeah, uh you review the test cases, you continuously monitor them, you look at your live traces, you diagnose them as we learned before with the diagnose agent you can do root cause analysis, you can cluster your failure modes. s184 00:24:13.960 --> 00:24:16.640 Uh you'll derive new eval criteria from it. s185 00:24:16.640 --> 00:24:18.720 They become part of the spec. s186 00:24:18.720 --> 00:24:21.040 Uh they become part of the agent. s187 00:24:21.040 --> 00:24:26.920 So they they continuously grow with you while you use your agent in production. s188 00:24:26.920 --> 00:24:29.720 And uh yeah, this becomes the online loop. s189 00:24:29.720 --> 00:24:38.280 And the more use cases you collect, the more is production data you see, the better your agent becomes and the better scoring your agent becomes. s190 00:24:38.280 --> 00:24:44.560 And this all together now um becomes one end-to-end loop that you can run agentic. s191 00:24:44.560 --> 00:24:53.920 And uh yeah, from here on uh yeah, you can now see the combination to software-driven development with coding agents. s192 00:24:53.920 --> 00:24:58.960 You can transport this concepts also for AI engineering and building AI agents. s193 00:24:58.960 --> 00:25:06.400 And uh now, as we at Mutagen work on this, we're going to show you now how this looks like as a product. s194 00:25:06.400 --> 00:25:13.440 Um This is kind of how it looks like. s195 00:25:13.440 --> 00:25:16.960 Um for now, everything runs in your environment. s196 00:25:16.960 --> 00:25:23.160 And we it as cloud and local, um we'll offer a managed service down the line. s197 00:25:23.160 --> 00:25:26.280 Um so, it's a set of agents. s198 00:25:26.280 --> 00:25:27.920 We have two in research preview. s199 00:25:27.920 --> 00:25:30.840 The ones we talked deep about this in our talk. s200 00:25:30.840 --> 00:25:39.800 First one is the evaluator agent that helps you build an eval set, a good data set, because this is the core of the optimization, the eval driven development loop. s201 00:25:39.800 --> 00:25:44.800 The second one we have is the diagnose the diagnostics agent. s202 00:25:44.800 --> 00:26:00.800 It does you It helps you diagnose your traces you already have in productions, because we learned from our users that reading through these traces took them a lot of time, and having kind of they can just spin off from their coding environment to analyze the traces s203 00:26:00.800 --> 00:26:03.800 is very helpful to them. s204 00:26:03.800 --> 00:26:12.200 And as you can see here, um both of these agents um are connected through an orchestrator, which runs in your coding environment. s205 00:26:12.200 --> 00:26:20.360 And what you need basically is connectors to sources where you basically have all your traces, where you get your incidents from. s206 00:26:20.360 --> 00:26:27.000 This could also be like your ticketing system, um Slack where people report failures of your agents. s207 00:26:27.000 --> 00:26:28.880 Then you connect them. s208 00:26:28.880 --> 00:26:33.080 And then, obviously, as we mentioned before, there's different target platforms. s209 00:26:33.080 --> 00:26:37.200 Um this could be come a PR that automatically gets raised in GitHub. s210 00:26:37.200 --> 00:26:43.520 Um this could be just the adjustment of your agents in uh in MD files. s211 00:26:43.520 --> 00:26:45.920 Uh yeah, different frameworks we target. s212 00:26:45.920 --> 00:26:48.920 And or being deployed to managed services. s213 00:26:48.920 --> 00:26:52.880 Um yeah, kind of this is how our platform works. s214 00:26:52.880 --> 00:26:54.360 We spin up different agents. s215 00:26:54.360 --> 00:27:04.880 They are connected to an orchestra and I guess Burak, let's show uh our first agent in research preview to the audience. s216 00:27:05.120 --> 00:27:18.760 Uh in practice, as I said, you have um orchestrator which kind of handles the dispatch for all the other sub agents or stages we have. s217 00:27:18.760 --> 00:27:26.360 And then you can use it in any coding harness or agent you want. s218 00:27:26.360 --> 00:27:29.120 In this case, I'm using code code. s219 00:27:29.120 --> 00:27:39.280 But then what it will do is when you first boot up, it will show you kind of like a dashboard of which stages you have and also kind of the existing s220 00:27:39.280 --> 00:27:46.440 things in uh let's say your code base, your agents, and the prior configurations. s221 00:27:46.440 --> 00:27:50.040 Now here I have some start commands. s222 00:27:50.040 --> 00:27:57.440 These are basically, let's say, trigger commands for the workflows or specific stages. s223 00:27:57.440 --> 00:28:07.400 And if I type diagnose here, this will basically start the diagnostic stage for the root cause analysis. s224 00:28:07.400 --> 00:28:13.920 And here again, you can point the diagnostics into a specific scope. s225 00:28:13.920 --> 00:28:15.480 This can be an agent. s226 00:28:15.480 --> 00:28:17.840 This can be a skill as well. s227 00:28:17.840 --> 00:28:24.320 So you could basically diagnose all invocations of a particular skill. s228 00:28:24.320 --> 00:28:36.160 Here uh when you start the diagnostics, it will uh retrieve traces from your configured source platform. s229 00:28:36.160 --> 00:28:39.800 In this case, again, it can be something like LangFuse. s230 00:28:39.800 --> 00:28:53.240 It can be your local cloud transcripts or another you know, like a JSON L format that's like exported from one of the observability platforms. s231 00:28:53.240 --> 00:29:08.840 Now, the diagnostics run itself takes quite a while to finish, so I will show you instead a pre-generated kind of version of what it does and how that looks like in the end. s232 00:29:08.840 --> 00:29:12.080 Okay, so what do we see here now? s233 00:29:12.840 --> 00:29:32.320 So, basically, once you run the diagnostics agent, as I said, on your agents, you will get a um, generated HTML artifact, which basically shows you, you know, the details of your agent, s234 00:29:32.320 --> 00:29:35.240 which tools you have. s235 00:29:35.240 --> 00:29:46.440 Uh, then in general, if you if you have code access, it will also tell you which harness or the framework that the agent is using. s236 00:29:46.440 --> 00:30:05.880 And then here, in general, it will basically go through your traces and then select primary signals, or let's say failure modes, depending on how frequent they occur in the trace samples. s237 00:30:05.880 --> 00:30:14.960 Now, as I said, most of the time you don't want to read all of your traces because this is not cost-efficient. s238 00:30:14.960 --> 00:30:35.800 So, there the diagnostics use like a a multi-tier filtering or segmentation to kind of pick a representative sample, so uh, originally, a few, you know, pieces of the traces, or let's say, s239 00:30:35.800 --> 00:30:44.280 portion will be read by the LLM to see that if there any detectable obvious problems. s240 00:30:44.280 --> 00:30:54.920 And based on that, then diagnostics can decide or to focus on a particular failure mode or a signal. s241 00:30:54.920 --> 00:31:08.880 Uh the other option is then trigger the diagnostics, you can also specify an issue that you have seen yourself or issue that you're looking for. s242 00:31:08.880 --> 00:31:12.880 This could be something that the users reported. s243 00:31:12.880 --> 00:31:26.960 Uh in this case, it's again more of a guided search, meaning hey, if you tell the diagnostics agent you have a particular problem you're interested in, then it will try to find all s244 00:31:26.960 --> 00:31:32.520 incidents or occurrences regarding the same problem. s245 00:31:32.520 --> 00:31:33.480 Okay. s246 00:31:33.480 --> 00:31:38.920 And what do we see here in uh in yeah, in full here? s247 00:31:38.920 --> 00:31:56.000 Uh so, this is in general the overview, which tells you again which issues kind of were detected and kind of shows you like the frequency giving the given time frame. s248 00:31:56.000 --> 00:32:04.720 Then for each tab, you will have a failure mode and kind of like an explanation of the problem. s249 00:32:04.720 --> 00:32:12.080 And then here, you can kind of see where that problem comes from or why it happens. s250 00:32:12.080 --> 00:32:25.680 The agent will provide kind of like um uh recursive why chain, let's say, to then tell you, "Hey, this is where the issue is originating from." s251 00:32:25.680 --> 00:32:33.360 Now, in addition, it will offer you kind of remedies or fixes to deal with that. s252 00:32:33.360 --> 00:32:36.800 And then here, you will see an assumptions block. s253 00:32:36.800 --> 00:32:53.280 Uh here, this is important because uh when you are reading traces, sometimes you don't always have access to the code, so this helps us detect uh, LLM or let's say the diagnostics s254 00:32:53.280 --> 00:33:00.800 agent makes certain assumptions that are also not correct and we can see and correct them here. s255 00:33:00.800 --> 00:33:10.720 Uh, but in the end you will be presented by certain corrections or remedies, which then kind of can fix your problem. s256 00:33:10.720 --> 00:33:18.200 You can, you know, pick either the recommended or, you know, as many as you need as it's multi-choice. s257 00:33:18.200 --> 00:33:36.320 And once you basically go through all your problems, uh, at the end you kind of get to a decisions page and here again for those using, you know, text-to-speech s258 00:33:36.320 --> 00:33:46.960 or speech-to-text AI features like Whisper flow, there's always a general feedback box so that you can always talk into your mic. s259 00:33:46.960 --> 00:33:59.520 And finally, when you are happy with all the decisions, you get a markdown, let's say, um, task definition for your coding agent. s260 00:33:59.520 --> 00:34:10.399 So, all the fixes or the remedies can then be directly applied to once you go back to your terminal and to your coding agent. s261 00:34:11.080 --> 00:34:13.000 Thanks for the demo, Burak. s262 00:34:13.000 --> 00:34:18.879 So, last words on our product. s263 00:34:20.399 --> 00:34:24.560 Yeah, so, uh, thanks for listening into our talk. s264 00:34:24.560 --> 00:34:32.960 Um, yeah, I hope we could show you some, uh, good insights on how we envision the future of agent building with the Agent AI engineer. s265 00:34:32.960 --> 00:34:36.159 Um yeah, stop uh, the debugging. s266 00:34:36.159 --> 00:34:42.240 Um have agents do the tedious work and reach out to us if you have further questions. s267 00:34:42.240 --> 00:34:46.800 And yeah, have a great conference.