WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=J17o5r5PKmw NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/J17o5r5PKmw.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:13.519 --> 00:00:15.040 Hello everybody. s2 00:00:15.040 --> 00:00:16.640 Uh we're going to get started. s3 00:00:16.640 --> 00:00:18.320 Um my name is Wllm. s4 00:00:18.320 --> 00:00:22.080 I am the co-founder and uh CTO of Cleric. s5 00:00:22.080 --> 00:00:25.760 So we build agents that keep your systems online. s6 00:00:25.760 --> 00:00:28.880 Today we're going to be talking about how things fail in production. s7 00:00:28.880 --> 00:00:35.920 I think everyone here is really familiar with shipping code, but writing code is not the same as like running it in production. s8 00:00:35.920 --> 00:00:40.320 Um, so let's dive into some of the ways things go wrong. s9 00:00:41.680 --> 00:00:46.239 So I think we're all familiar with uh like prod failures or issues in the production environment. s10 00:00:46.239 --> 00:00:52.399 And I think the modus that people get into today is like applying agents to solving these problems. s11 00:00:52.399 --> 00:00:55.360 You're running claude code to diagnose some alert. s12 00:00:55.360 --> 00:01:00.640 Um unlike with a coding use case, what we find is in a lot of cases, there's a lot of back and forth. s13 00:01:00.640 --> 00:01:06.159 So if you deploy an application, you get maybe like a 500 um errors are spiking. s14 00:01:06.159 --> 00:01:08.960 Um the agent comes back very confidently with an answer. s15 00:01:08.960 --> 00:01:17.280 It says um you know there's a memory leak that you just deployed and it requires you as a human still to go and look and see did this actually happen. s16 00:01:17.280 --> 00:01:26.000 And often this results in this back and forth where you have to second guess the agent because unlike in the coding case, the production environment is a lot harder. s17 00:01:26.000 --> 00:01:29.119 And so you go back and forth, it claims it's the readiness probe. s18 00:01:29.119 --> 00:01:33.439 You say, well, you look at the the event and you see that it's not the case. s19 00:01:33.439 --> 00:01:39.360 And eventually you realize or through iteration with the agent what the problem actually was. s20 00:01:39.360 --> 00:01:40.960 So this is a frustrating experience. s21 00:01:40.960 --> 00:01:48.640 Even though the agent's doing a lot of work for you, you still are in the hook on on the hook to iterate with it. s22 00:01:48.720 --> 00:01:50.960 So we're going to talk about how do we improve this? s23 00:01:50.960 --> 00:01:54.640 How do we get this to a point where there's higher agency and autonomy with agents. s24 00:01:54.640 --> 00:02:05.040 So we'll talk about what are the failure classes that we see and how we as cleric have developed some approaches that you can apply on your own to improving the accuracy of your agent in the production environment s25 00:02:05.040 --> 00:02:06.240 as with coding. s26 00:02:06.240 --> 00:02:10.160 Um and then how you measure that and and roll that out. s27 00:02:10.160 --> 00:02:14.640 So one of the key differences with the production environment is that there's no verification. s28 00:02:14.640 --> 00:02:27.040 So if you take dev or if you take even CI, there's tests, uh there's linting, there's very focused signal that comes back from all the changes that your agent is making that makes iterations very fast. s29 00:02:27.040 --> 00:02:31.840 Um and the responses that the agent gives you are very accurate in most cases. s30 00:02:31.840 --> 00:02:34.319 Prod on the other hand is diffuse. s31 00:02:34.319 --> 00:02:35.840 You have a little bit of logs here. s32 00:02:35.840 --> 00:02:37.200 You have noise over there. s33 00:02:37.200 --> 00:02:38.640 People are deploying things. s34 00:02:38.640 --> 00:02:41.280 there's no focused signal that's coming back to you. s35 00:02:41.280 --> 00:02:49.200 So the real question is what do you have on the prod side of things where the the time you can sync there could be in the order of hours. s36 00:02:49.200 --> 00:02:52.720 Um like you need some kind of signal coming back to your agent. s37 00:02:52.720 --> 00:02:54.080 So that's really what this talk is about. s38 00:02:54.080 --> 00:02:56.879 How do you build that signal? s39 00:02:57.360 --> 00:03:02.560 Let's talk about a little bit about why prod and large language models are really hard compared to code. s40 00:03:02.560 --> 00:03:05.280 In the case of code, you have a very bounded environment. s41 00:03:05.280 --> 00:03:06.400 It's a single codebase. s42 00:03:06.400 --> 00:03:07.519 There's a line around that. s43 00:03:07.519 --> 00:03:08.720 it's the repo. s44 00:03:08.720 --> 00:03:15.280 But unlike code, a production environment is often an amorphous uh environment where the boundaries aren't clear. s45 00:03:15.280 --> 00:03:25.519 Sure, you have like one GCP or AWS account, but there are multiple teams, there are multiple clusters, and for an agent, it's unclear how wide or narrow to go. s46 00:03:25.519 --> 00:03:27.599 Additionally, prod is also moving. s47 00:03:27.599 --> 00:03:30.159 So, there's state that's constantly changing. s48 00:03:30.159 --> 00:03:32.159 Um, people are deploying things. s49 00:03:32.159 --> 00:03:35.760 Systems are not sitting still unlike a codebase that's static. s50 00:03:35.760 --> 00:03:39.519 And I think one of the most pernicious parts is prod is also transient. s51 00:03:39.519 --> 00:03:42.799 A lot of the failures you get can't be reproduced in a codebase. s52 00:03:42.799 --> 00:03:45.440 If the agent runs into a problem, it can reset and try again. s53 00:03:45.440 --> 00:03:50.799 But in prod, sometimes a failure is only seen for a specific moment in time. s54 00:03:50.799 --> 00:03:52.000 But that's not the only challenge. s55 00:03:52.000 --> 00:03:54.480 The other challenge is that models are not really trained for this. s56 00:03:54.480 --> 00:04:00.640 In fact, large language models are trained to almost be the opposite for what you need for a production debugger. s57 00:04:00.640 --> 00:04:07.439 So what you need for production is doubt and uncertainty and trying many different theories and eliminating those theories. s58 00:04:07.439 --> 00:04:14.640 But these large language models are really trained to be decisive uh confident and to give you an answer as quickly as possible. s59 00:04:14.640 --> 00:04:30.720 And in fact like one of the uh system cards or the reports that OpenAI released showed that the confidence and accuracy deviated in the RLHF process and this is partly to do with the the post training being optimized for preferences of humans. s60 00:04:30.720 --> 00:04:40.000 So it's a part of the the models and it's something that you have to counteract and so how you build the counteraction to that is a part of the challenge here. s61 00:04:40.800 --> 00:04:43.440 So let's talk about some of the failures that we see in production. s62 00:04:43.440 --> 00:04:46.720 We work with many different uh companies and enterprises. s63 00:04:46.720 --> 00:04:56.639 Uh we see how agents fail in debugging uh production systems and and the failures they um uh are trying to diagnose. s64 00:04:56.639 --> 00:05:05.680 The most common failure we see and this is for claude code for our product pretty much across the board is uh mistaking a cause and a symptom. s65 00:05:05.680 --> 00:05:16.000 Often when you have an alert, the agent will run to that service where the fire or the smoke is and it'll find an error log right there and it'll come back very confidently and say this is it. s66 00:05:16.000 --> 00:05:20.080 I found the root cause instead of going deeper into the stack. s67 00:05:20.080 --> 00:05:31.120 This is a very hard one to debug because these agents or these models are very good at uh finding a needle in a hstack, but they don't know the breadth of the search space that they should be looking at. s68 00:05:31.280 --> 00:05:35.680 The second one we see a lot is not knowing what's normal and what is anomalous. s69 00:05:35.680 --> 00:05:46.160 So if you don't have a baseline for production, what happens is that the first thing that they see, they will think it's anomalous and they won't won't try and continue to search for something else. s70 00:05:46.400 --> 00:05:50.160 The third one which is a new class of problem is the abstraction problem. s71 00:05:50.160 --> 00:05:53.440 If you look at most modern production environments, they have a lot of information. s72 00:05:53.440 --> 00:05:56.800 So you need classes of agents to process that. s73 00:05:56.800 --> 00:06:04.560 But these agents now don't bubble up all of the uh raw findings to the master agent or the main agent that's making a decision. s74 00:06:04.560 --> 00:06:11.360 And that leads to overconfidence in the main agent making a decision based on partial information. s75 00:06:12.160 --> 00:06:22.639 And the final one that I think is also very common now with memory systems and context being so important in diagnosing issues is using past context or state as part of the investigation s76 00:06:22.639 --> 00:06:25.759 often leads to agents anchoring on that information. s77 00:06:25.759 --> 00:06:38.319 and for example saying okay last week we had this issue this is probably the same thing here's your diagnosis so this is really hard because the agent is very eager to please and give you an answer and you kind of have to destabilize s78 00:06:38.319 --> 00:06:48.240 it a little bit to have it go further and deeper so let's talk about some of the techniques we've applied and and how they've been effective for us at least s79 00:06:48.240 --> 00:07:01.360 so the thing that you need to do is replicate what works in coding in coding you have signal that comes back very quickly it's critical signal So the agent tries a code change and it comes back and says this didn't work, it didn't compile or here's an error message. s80 00:07:01.360 --> 00:07:14.080 So you want something that the agent can ground uh ground on um as part of an investigation or also at the end and that's basically going to give you like a confidence score that the agent has as it's executing. s81 00:07:14.080 --> 00:07:16.639 So what we do is we generate many different theories. s82 00:07:16.639 --> 00:07:25.759 This is a very effective like primary thing that we do to give the agent uh kind of like a breadth a diversity of ideas that it's exploring simultaneously. s83 00:07:25.759 --> 00:07:27.520 This is important for multiple reasons. s84 00:07:27.520 --> 00:07:37.680 One is the diver diversity but also secondly agents are really good at relative ranking and it's they're terrible at absolute scoring um of their theories or ideas. s85 00:07:37.680 --> 00:07:45.280 So generating multiple ideas for why a root why uh what the cause of a failure is is very important. s86 00:07:45.759 --> 00:07:48.319 Secondly, what we do is we compute a baseline. s87 00:07:48.319 --> 00:07:52.080 This is also extremely important, especially in large production environments. s88 00:07:52.080 --> 00:07:57.759 Most prod environments are in a permanent state of kind of disarray and chaos. s89 00:07:57.759 --> 00:07:59.919 Engineers know to ignore certain error logs. s90 00:07:59.919 --> 00:08:04.080 They need know to some metrics are not even used, but an agent doesn't know that. s91 00:08:04.080 --> 00:08:05.680 An LLM doesn't know that. s92 00:08:05.680 --> 00:08:14.960 So what we do like an example of this is if we see logs or if we're uh presenting logs to an agent we'll cluster the logs and tell the agent which logs are s93 00:08:14.960 --> 00:08:18.479 um like reoccurring every day and very common. s94 00:08:18.479 --> 00:08:29.759 So we give it the statistics on those logs not just the raw logs and that makes it a lot more uh effective but this applies as well to metrics or object state or many other things. s95 00:08:30.080 --> 00:08:36.399 What we also do is we use a topology or we build dependency graph of your production environment. s96 00:08:36.399 --> 00:08:41.760 This is very important because the agent will often claim there's some causal connection between two services. s97 00:08:41.760 --> 00:08:45.839 So for example, it'll say like an all service is connected to a cache. s98 00:08:45.839 --> 00:08:56.880 But if you have a graph or if if you know the dependencies between services, you can output uh all the connections that exist and then have another agent critique that. s99 00:08:56.880 --> 00:09:06.560 And this is a very effective way to introduce doubt into the answer and tell the agent to go further if it doesn't uh map to an existing connection. s100 00:09:06.560 --> 00:09:10.880 Another one that we do is or another technique we apply is using time. s101 00:09:10.880 --> 00:09:15.120 So you know that the cause is going to happen before the symptom temporally. s102 00:09:15.120 --> 00:09:21.200 But a lot of people don't even apply temporal um assessment into their uh root causing. s103 00:09:21.200 --> 00:09:28.800 Additionally, you also know that in most cases t uh temporally things will be closer together in time versus further apart. s104 00:09:28.800 --> 00:09:38.399 So you can add a relevant score to any findings that you have um that are closer to um the alerting time stamp. s105 00:09:39.200 --> 00:09:42.080 And then the most obvious one is just propagating sources. s106 00:09:42.080 --> 00:09:46.880 So if you have sub agents or agents handing context over to each other, have them build a case. s107 00:09:46.880 --> 00:09:49.040 Don't just tell them to give you the answer. s108 00:09:49.040 --> 00:09:51.279 And this is also by the way what SRRES do. s109 00:09:51.279 --> 00:09:53.360 They don't just say you know prod is up. s110 00:09:53.360 --> 00:09:58.160 They say I looked at this metric and it was at this point um at this point in time. s111 00:09:58.160 --> 00:09:59.360 They give hard evidence. s112 00:09:59.360 --> 00:10:07.600 So if you don't propagate the evidence um this loses a lot of the fidelity and causes hallucination and overconfidence. s113 00:10:08.959 --> 00:10:14.959 Um and then finally this is these are some of the results that we've had um in our simulations. s114 00:10:14.959 --> 00:10:18.240 Grounding the agent has been extremely effective. s115 00:10:18.240 --> 00:10:29.680 So what we find is that for localized incidents, incidents where um effectively alerting service and the the root cause are in one area, there's a very small delta. s116 00:10:29.680 --> 00:10:31.360 It's almost the same as clawed code. s117 00:10:31.360 --> 00:10:41.120 But what we found is that for distributed problems where the cause and the symptom are not in the same service, uh claude code is not as effective, not even nearly as effective. s118 00:10:41.120 --> 00:10:44.880 So of course you can't use these numbers as gospel because this is a simulation environment. s119 00:10:44.880 --> 00:10:46.000 Every company is different. s120 00:10:46.000 --> 00:10:50.480 If you're optimizing your stack to just be like one monolith, you'll have a good time. s121 00:10:50.480 --> 00:10:52.480 But most companies can't do that. s122 00:10:52.480 --> 00:10:56.800 So, it's much harder and the the problem space is much more diverse. s123 00:10:56.800 --> 00:11:00.079 The there's one downside to doing what we're doing and that is cost. s124 00:11:00.079 --> 00:11:03.040 Our runs are very expensive compared to just running clawed code. s125 00:11:03.040 --> 00:11:04.959 So, if you're running claude code, it's very surgical. s126 00:11:04.959 --> 00:11:08.959 It's one thing, but you have to guide it and steer it or any agent, right? s127 00:11:08.959 --> 00:11:14.079 But if you're running five or six theories concurrently, um, you know, it's going to cost you a lot more money. s128 00:11:14.079 --> 00:11:19.360 you're going to burn tokens, but it's also going to keep the human out of the loop much longer. s129 00:11:21.120 --> 00:11:22.720 But how do we know this actually worked? s130 00:11:22.720 --> 00:11:24.079 I mean, these are all just vibes. s131 00:11:24.079 --> 00:11:26.480 We're just adding more context into the agent. s132 00:11:26.480 --> 00:11:30.720 So, we actually need to do a few more things to know what actually happened. s133 00:11:30.720 --> 00:11:38.160 So, the way we look at the um production incident response flow with agents is there's a detection, a diagnosis, and a fix. s134 00:11:38.160 --> 00:11:41.120 And most folks use agents for diagnosis. s135 00:11:41.120 --> 00:11:45.279 But what we do is we also look at verification and the outcome. s136 00:11:45.279 --> 00:11:50.079 So we triangulate across many different signals what happened when you apply a fix. s137 00:11:50.079 --> 00:12:03.600 So if we give you a suggestion, here's a diagnosis, we monitor you as an engineer and we see if you apply the change like actually merge code that maps to the diagnosis we gave you and then we see if that solved the problem. s138 00:12:03.600 --> 00:12:13.760 So we looked for a return to normal um in the service the metrics the logs and all of the state that um should be there in the baseline. s139 00:12:13.760 --> 00:12:22.560 Now of course if you just mute an alert you're also going to see a return to normal and so we look at multiple signals and ensure that it is actually lining up with our expectations. s140 00:12:22.560 --> 00:12:28.399 Now this is very important because you know why even have all these machinery to track this outcome. s141 00:12:28.399 --> 00:12:33.440 It's important because you want to calibrate your um evaluation at the end of the day. s142 00:12:33.440 --> 00:12:45.440 So what we had earlier earlier was grounding that we added into our agent basically a critic and what you see is that the blue line is the critic and the red line is when you don't have grounding s143 00:12:45.440 --> 00:12:51.200 and so when you verify the outcome what actually happened at the end of the day did the the service return to normal. s144 00:12:51.200 --> 00:12:58.800 You can see that with grounding if you the 45° line is the um confidence versus accuracy. s145 00:12:58.800 --> 00:13:01.839 So you want to be on the 45 degree line. s146 00:13:01.839 --> 00:13:06.480 What happens if you aren't grounded is you become extremely overconfident. s147 00:13:06.480 --> 00:13:12.079 So if the agent is just using vibes effectively, it'll always give you a very confident answer. s148 00:13:12.079 --> 00:13:22.399 And so the effect of this is if you have the grounding and you track verif the the outcomes um you have a signal that's going to come back from your agent to s149 00:13:22.399 --> 00:13:24.720 whatever is going to apply the fix that you can trust. s150 00:13:24.720 --> 00:13:36.160 So if the agent says I'm 90% sure that this is the answer you can trust that signal and so at the end of the day you can complete this loop s151 00:13:36.160 --> 00:13:40.079 um taking the signal all the way from prod back to your agent. s152 00:13:40.079 --> 00:13:43.040 And so this has been our focus for three years. s153 00:13:43.040 --> 00:13:50.639 We've been building effectively ways to extract signal from the production environment and bring it back to our agents. s154 00:13:50.639 --> 00:13:56.000 So this is both for oper systems that are operational already as well as during your development loop. s155 00:13:56.000 --> 00:14:00.480 And where we're going next is we're bringing that signal to the developer environment. s156 00:14:00.480 --> 00:14:11.839 Instead of waiting for an application to be deployed before it fails or or fail after it's deployed, we bring the model, this baseline model, the system graph, the uh the ontology s157 00:14:11.839 --> 00:14:17.279 all the way to your dev environment so you can predict failure before you deploy or make a change. s158 00:14:17.279 --> 00:14:20.959 Um yeah, so this is the direction we're going and I hope that was useful. s159 00:14:20.959 --> 00:14:23.440 This is some of the learnings that we've had along the way. s160 00:14:23.440 --> 00:14:26.720 If anybody has questions, feel free to shoot. s161 00:14:26.720 --> 00:14:29.600 But yep, that's a talk. s162 00:14:30.504 --> 00:14:32.504 [applause] s163 00:14:38.365 --> 00:14:40.365 [music]