WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=bZISsg7H7DA NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/bZISsg7H7DA.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.520 --> 00:00:06.280 Have you ever looked at your agent execution and asked yourself the question, why did it do that? s2 00:00:06.280 --> 00:00:08.760 What if it had done a different thing? s3 00:00:08.760 --> 00:00:10.080 Would it have been cheaper? s4 00:00:10.080 --> 00:00:11.840 Would it have been faster? s5 00:00:11.840 --> 00:00:18.840 Well, you can do all these things if your agents have a save button. s6 00:00:20.480 --> 00:00:24.240 We've had the save button for documents for decades now. s7 00:00:24.240 --> 00:00:34.080 Since the 1980s, people have been used to pressing control S, command S uh or auto saving while you're working to have a persistent state. s8 00:00:34.080 --> 00:00:37.600 But agents, they don't have that today. s9 00:00:37.600 --> 00:00:40.360 The only thing we have which is closest is a trace. s10 00:00:40.360 --> 00:00:48.400 A trace gives you the emitted telemetry data of how an agent calls tools in the input and output of that state. s11 00:00:48.400 --> 00:00:55.560 Now, while this is a good start, it is actually very disconnected from the runtime in which these agents actually execute. s12 00:00:55.560 --> 00:01:05.320 So, all the variables that are in state, all the file system that is in in flight, um the decisions that it makes in the code, uh the actual code itself, s13 00:01:05.320 --> 00:01:16.640 all of that is lost and it is only stamped as a read-only trace by the end, which is sitting in another tool far away from where the actual code is. s14 00:01:16.640 --> 00:01:27.960 And I think this is what's missing today in the industry is that we don't have a clear connection between the observability spans that are emitted with Odel and the execution. s15 00:01:28.480 --> 00:01:32.080 And maybe at this point you might be wondering, but why even bother? s16 00:01:32.080 --> 00:01:35.200 Why do I need to have a save button? s17 00:01:35.200 --> 00:01:37.840 Well, save allows you to replay. s18 00:01:37.840 --> 00:01:41.480 You can go back in history and ask the what if question. s19 00:01:41.480 --> 00:01:43.840 What are the types of questions you might want to ask? s20 00:01:43.840 --> 00:01:45.000 Well, you might want to swap the model. s21 00:01:45.000 --> 00:01:48.560 Maybe you use a an open-source model that is cheaper. s22 00:01:48.560 --> 00:01:52.880 Maybe you mock a tool and you override the what it returns. s23 00:01:52.880 --> 00:01:58.200 Maybe you degrade it intentionally to see what would happen if things are wrong. s24 00:01:58.200 --> 00:02:03.720 And these sorts of questions are only possible if you have that state. s25 00:02:04.440 --> 00:02:09.200 there is a category of the stack emerging which actually allows that. s26 00:02:09.200 --> 00:02:16.000 And this sit on top of the harness sit on top of the frameworks that allow you to create agents. s27 00:02:16.000 --> 00:02:19.920 And put a durable runtime below that. s28 00:02:19.920 --> 00:02:30.600 Um and augments the traces that are emitted with actually the code execution and the things that are around it to actually complete the state of this of the of the system. s29 00:02:31.120 --> 00:02:44.920 And the good news is that once you have such a system in production, you already have the information you need to ask those questions that are relevant to making your system better, s30 00:02:44.920 --> 00:02:46.680 cheaper, and faster. s31 00:02:46.680 --> 00:02:48.720 Production already has the traces. s32 00:02:48.720 --> 00:02:57.800 It already has these state checkpoints ideally from the runtime that can allow you to go back in time and ask those questions. s33 00:02:57.800 --> 00:03:06.519 For example, let's take an agent example which does a customer resolution and refunds after a chargeback dispute. s34 00:03:06.519 --> 00:03:19.480 Well, you can then see if the order status is changing languages or whether the request should have been escalated or maybe a smaller model would have handled it if the runtime is checkpointing each of the state as it goes along. s35 00:03:19.480 --> 00:03:24.640 Almost like an auto save, a command S, a control S in your agent. s36 00:03:24.640 --> 00:03:28.200 And once you have that, you can even close the loop. s37 00:03:28.200 --> 00:03:30.400 Uh this conference is all about loops. s38 00:03:30.400 --> 00:03:32.760 So, this is nothing different. s39 00:03:32.760 --> 00:03:39.560 You have a cohort of runs that you think maybe matter because maybe they're too expensive, they took too long. s40 00:03:39.560 --> 00:03:51.360 You replay a change, you diff it, you see what would have happened when you have the baseline, which you know what happened in the first place, and then you decide and you route and you ship back. s41 00:03:51.360 --> 00:03:54.040 That's closing the loop on your e-vals. s42 00:03:54.040 --> 00:03:59.440 It's It's It's basically evaluating using your production traces. s43 00:04:01.600 --> 00:04:07.960 So, it's basically evaluating using your production checkpoints. s44 00:04:08.360 --> 00:04:11.240 So, checkpoint replay diff decide. s45 00:04:11.240 --> 00:04:18.720 And this is really the methodology that that I've seen and I've seen others do, uh which has really scaled. s46 00:04:18.720 --> 00:04:19.799 For example, DoorDash. s47 00:04:19.799 --> 00:04:28.640 DoorDash has a a blog post on the 1st of June where they talk about having a simulated environment where they replayed customer bots. s48 00:04:28.640 --> 00:04:34.280 And they've done what-if scenarios and seeing how they could have made it better. s49 00:04:34.280 --> 00:04:45.160 And where And where it used to take them hours and hours to to do this, now they've reduced it to 5 minutes uh with hundreds of simulations, have 90% less hallucinations, s50 00:04:45.160 --> 00:04:49.040 and they're still two points within what they've seen in production. s51 00:04:49.040 --> 00:04:53.800 So, the simulations are pretty good because they're grounded in what's already happened. s52 00:04:54.520 --> 00:05:02.240 So, we're going to just walk through this um in an example, and we're going to see how this works. s53 00:05:02.919 --> 00:05:06.880 Uh for this demo, we're going to be using a tool called Kitaru. s54 00:05:06.880 --> 00:05:12.720 Kitaru is a very new tool that is launched by the team at uh ZenML. s55 00:05:12.720 --> 00:05:17.720 ZenML is has been around for many years and is a player in the orchestration space. s56 00:05:17.720 --> 00:05:19.280 I'm one of the co-founders. s57 00:05:19.280 --> 00:05:27.640 And Kitaru is something we've launched recently, which allows you to have a runtime layer below your harness layer, and also connect to your traces. s58 00:05:27.640 --> 00:05:33.040 Um and do all the checkpointing that we were just talking about and then running replay scenarios. s59 00:05:33.040 --> 00:05:44.520 So, you can see here I have all of my I have a support uh agent which looks at my customer requests and escalates it when it needs to to humans. s60 00:05:44.520 --> 00:05:47.240 You can see various things as you might expect. s61 00:05:47.240 --> 00:05:51.600 Um, every tool call uh can see a timeline view if I wanted. s62 00:05:51.600 --> 00:06:04.600 And here the difference is that if I if I click on a particular um um checkpoint like a tool call, I can see the configuration, where it ran, uh the code it it took s63 00:06:04.600 --> 00:06:08.400 um and the artifacts which came in and out of it. s64 00:06:08.400 --> 00:06:17.840 Um and this combination of code and the artifacts that it created and the environment in which it ran in, whether it was a Docker image or a sandbox, those are all snapshotted in state here s65 00:06:17.840 --> 00:06:20.040 between the checkpoints. s66 00:06:20.040 --> 00:06:30.880 And you can see that here in this particular example, there was a few tool calls every time it went to the LLM, and I can actually see um you know, how long it took. s67 00:06:30.880 --> 00:06:33.160 But imagine I wanted to do something different. s68 00:06:33.160 --> 00:06:40.000 Imagine I wanted to change the model if I maybe at this point I wanted to use a cheaper model. s69 00:06:40.000 --> 00:06:42.200 Would it have done the same stuff afterwards? s70 00:06:42.200 --> 00:06:45.760 So, to do this in Kitaro, it's quite easy. s71 00:06:45.760 --> 00:06:50.560 All you have to do Now, to do this in Kitaro is quite easy. s72 00:06:50.560 --> 00:06:58.600 All you have to do is you have to take your execution and replay it at a particular point. s73 00:06:58.600 --> 00:07:04.520 So, I'm just going to copy this over and I'm going to put it in my terminal and see what happens. s74 00:07:04.520 --> 00:07:15.880 So, here what I'm doing is essentially I am saying, "Okay, after this particular tool call, change the model to GPT-5 Nano, which is obviously a bit cheaper." s75 00:07:15.880 --> 00:07:28.919 Um and then what's going to happen here is there's going to be a new execution that's launched from 71 and you can see the first three checkpoints are skipped. s76 00:07:28.919 --> 00:07:35.480 So, they're all skipped because Kitaro already has the state of all the checkpoints before that. s77 00:07:35.480 --> 00:07:41.240 All it needs to do is just change this particular checkpoint and then execute start executing from here. s78 00:07:41.240 --> 00:07:47.560 And this is really cool because now I can see what would have happened in this scenario if there was a cheaper model. s79 00:07:47.560 --> 00:07:53.280 So this looks pretty similar to me, but what if I had wanted to do a even different change? s80 00:07:53.280 --> 00:07:57.760 What if I wanted to mock a tool or or or if I wanted to change a tool call? s81 00:07:57.760 --> 00:08:02.880 Well, in order to do that, you can mock up for example the lookup policy tool. s82 00:08:02.880 --> 00:08:04.920 And here it's also very easy. s83 00:08:04.920 --> 00:08:07.800 I can just go back here. s84 00:08:08.520 --> 00:08:09.400 Replay. s85 00:08:09.400 --> 00:08:14.280 And this time, rather than changing the model, I am changing the lookup policy. s86 00:08:14.280 --> 00:08:20.840 And I'm mocking it with another function in my code base, which returns a different lookup policy. s87 00:08:20.840 --> 00:08:24.800 And I'm just trying to understand if the policy had changed, what would have happened. s88 00:08:24.800 --> 00:08:26.960 This time I'm holding the model constant. s89 00:08:26.960 --> 00:08:40.240 Now, the interesting thing here is that because I have the code, it's very easy for me to do tool calls and to change these particular things and do more experiments than I would have had if I was completely disconnected from the code base. s90 00:08:40.240 --> 00:08:45.520 And here you can see that the code base is a little bit different, so I see my logs here. s91 00:08:45.520 --> 00:08:47.080 They look a little bit different. s92 00:08:47.080 --> 00:08:53.240 And here you can see that I got a slightly different artifact maybe from the tool call, and it published the thing. s93 00:08:53.240 --> 00:08:54.839 So now again, I have three runs. s94 00:08:54.839 --> 00:08:57.160 I have the original run and I have the two replays. s95 00:08:57.160 --> 00:09:00.400 Now, what if I wanted to see them side by side, right? s96 00:09:00.400 --> 00:09:15.320 So Kitaro actually has this very handy diff command that lets me give an original ID of an execution and then allows me on the other side to actually see them side by side. s97 00:09:15.320 --> 00:09:18.320 So I can see what happened. s98 00:09:18.320 --> 00:09:27.080 So there's a few warnings here about some artifacts, but in a second it will go and give me a URL. s99 00:09:27.080 --> 00:09:31.680 I can copy this URL and I can actually put it directly here. s100 00:09:31.680 --> 00:09:39.960 And now I have a very nice um comparison of the original and the two forks. s101 00:09:39.960 --> 00:09:43.560 Um I can see a bunch of things here, but I think what's really interesting is this view. s102 00:09:43.560 --> 00:09:47.440 So, in this view, you can see that the first part of the the baseline is the same, right? s103 00:09:47.440 --> 00:09:48.760 So, these things were skipped. s104 00:09:48.760 --> 00:09:49.840 They were exactly as it is. s105 00:09:49.840 --> 00:09:52.120 The state is exactly where it was. s106 00:09:52.120 --> 00:09:56.240 Um but now, right after that, in the third replay, it's a little bit different, right? s107 00:09:56.240 --> 00:10:01.840 The tool call happened a bit differently because we used a different policy, and then something changed after here. s108 00:10:01.840 --> 00:10:04.400 Um here, it took a little bit longer. s109 00:10:04.400 --> 00:10:07.800 And you can really start looking at the final result. s110 00:10:07.800 --> 00:10:12.320 And if I click on the final result, I can actually see the artifacts side by side. s111 00:10:12.320 --> 00:10:15.920 I can see what decision um it actually ended up making. s112 00:10:15.920 --> 00:10:19.880 So, you can see that it uh here, it restricted the count charge. s113 00:10:19.880 --> 00:10:21.040 Here, it also restricted it. s114 00:10:21.040 --> 00:10:22.720 Here, it also restricted it. s115 00:10:22.720 --> 00:10:25.040 In the first two, it actually needs review. s116 00:10:25.040 --> 00:10:28.880 In the third, because we changed the policy, it's safe to answer. s117 00:10:28.880 --> 00:10:34.080 So, this might or might not be good in your scenario, but if is this is what you expected to see? s118 00:10:34.080 --> 00:10:35.360 Well, maybe. s119 00:10:35.360 --> 00:10:38.040 Because the these two were cheaper at the end of the day. s120 00:10:38.040 --> 00:10:43.000 For for for the number of tokens consumed, uh they were cheaper. s121 00:10:43.000 --> 00:10:43.839 Input and output. s122 00:10:43.839 --> 00:10:52.280 And you can see a bunch of detail here uh in the in the UI, which might be useful for these for these analyses. s123 00:10:52.280 --> 00:10:54.839 Okay, but this is just one point, right? s124 00:10:54.839 --> 00:10:57.600 What if I wanted to do this across a cohort? s125 00:10:57.600 --> 00:11:01.560 What if I wanted to have a bunch of runs, right? s126 00:11:01.560 --> 00:11:06.160 Which actually um maybe sorted by cost. s127 00:11:06.160 --> 00:11:08.200 So, maybe I took all of my expensive ones. s128 00:11:08.200 --> 00:11:11.320 Then I wanted to do one change across the entire cohort. s129 00:11:11.320 --> 00:11:19.560 So, change the change all the tool calls that happened uh that happened for this particular set of configurations across the cohort. s130 00:11:19.560 --> 00:11:25.240 Or change the model itself um and use a cheaper model across the cohort. s131 00:11:25.240 --> 00:11:30.720 So, this gives me a bigger distribution of information and replays that I can use. s132 00:11:30.720 --> 00:11:33.560 Um and now, I can just this. s133 00:11:33.560 --> 00:11:47.120 Um, and the way you would replay it is you can use the replay mini command where you actually um, start from a particular point and you do the same as you did before just across the cohort. s134 00:11:47.120 --> 00:11:50.200 Um, this is going to take a while, so I'm not going to do it now. s135 00:11:50.200 --> 00:12:02.640 But once you execute it, in this particular case, I just emitted it to JSON, so I have a very nice, uh, JSON here, and this JSON gives me a bunch of things. s136 00:12:02.640 --> 00:12:04.400 So, this is a bit hard to do the UI, right? s137 00:12:04.400 --> 00:12:05.880 So, there's a lot of things going on. s138 00:12:05.880 --> 00:12:09.600 You can't just do the many, many, many comparisons here. s139 00:12:09.600 --> 00:12:16.640 Um, but what you could do, and what I love to do personally, is use the Kitaru MCP server. s140 00:12:16.640 --> 00:12:28.920 And here what you do is you can just say, "Hey, read this you read this JSON report and do an analysis on what you think I should be doing across the cohort." s141 00:12:28.920 --> 00:12:40.280 And I think this is what What is really important is to be using agents and LLMs to analyze these cohorts across a plethora of data because at some point s142 00:12:40.280 --> 00:12:43.720 uh, I mean, 10 is probably easy to do, but what if you have thousands? s143 00:12:43.720 --> 00:12:56.120 Um, and doing thousands and thousands is hard, and this is where skills and MCP servers get really relevant and having the runtime be queryable and go into your execution and fetch the artifacts is very important. s144 00:12:56.120 --> 00:13:00.160 So, it's going to be it's going to be uh, doing a lot of things. s145 00:13:00.160 --> 00:13:10.680 So, it's it's it's trying to, you know, um, run an analysis around these decisions and it's going to flag any um, red flags that that happen. s146 00:13:10.680 --> 00:13:12.000 So. s147 00:13:12.000 --> 00:13:17.240 While this is going on, um, maybe we can go back and continue our presentation. s148 00:13:17.240 --> 00:13:23.839 So, um, Here you can see, yeah. s149 00:13:23.839 --> 00:13:28.720 Yes, if you change the model, it can get very cheap. s150 00:13:29.040 --> 00:13:36.760 And of course, you don't want to do it across just one sample, but across many because then you can see a bigger variation. s151 00:13:36.760 --> 00:13:47.320 I think one thing which is important here is and to be very honest is that while we've been doing it this with our users and customers, what I've personally seen s152 00:13:47.320 --> 00:13:52.640 is that having a naive model swap usually or often times doesn't work. s153 00:13:52.640 --> 00:14:05.920 So just changing for example to a cheaper model and just looking at the cost one single dimensionally, obviously it could be that you're you're spending a lot less money, but what happens if your support board is not resolving the requests, right? s154 00:14:05.920 --> 00:14:07.800 So this is a study from BrainTrust. s155 00:14:07.800 --> 00:14:17.760 Excellent study where they actually looked at that and they saw that there could be a false economy if you do a naive model swap because it might look on paper that you're faster and you're cheaper, s156 00:14:17.760 --> 00:14:20.600 but at the end of the day you have to look at the value created, right? s157 00:14:20.600 --> 00:14:26.400 So it's a trade-off between how much money you want to spend and and yeah, the result. s158 00:14:26.400 --> 00:14:36.800 And also, if you look at the towel bench, what we must understand is that a model that passes 60% of the time is only self-consistent about a quarter of the time. s159 00:14:36.800 --> 00:14:47.240 So which basically means that one replay is just an anecdote and having a cohort analysis is way way way better. s160 00:14:47.240 --> 00:14:57.480 Um and because then you can really see across a population and across scale what would have happened not just looking at one estimate. s161 00:14:57.480 --> 00:15:05.640 This can get very expensive of course and this is where you have to be really smart about what you replay and have tooling that really helps you. s162 00:15:05.640 --> 00:15:09.839 And this you can really bake into your production process as well, right? s163 00:15:09.839 --> 00:15:23.160 So again, if you if you take a step back and look at the playbook, you can start from real runs, not synthetic, but real runs, real production uh state. s164 00:15:23.160 --> 00:15:25.200 Build cohorts that matter. s165 00:15:25.200 --> 00:15:29.440 Um maybe take the expensive ones, maybe take the long ones, maybe take the risky ones. s166 00:15:29.440 --> 00:15:33.000 Um never ship anything by just replaying one or two things. s167 00:15:33.000 --> 00:15:39.120 Um and just do this at scale and uh ship, route, and hold, and try to automate that loop as much as possible. s168 00:15:39.120 --> 00:15:41.760 Maybe there's an agent that's doing that for you. s169 00:15:41.760 --> 00:15:43.920 Um that's even better. s170 00:15:43.920 --> 00:15:47.240 But you just have a human in the loop at the end. s171 00:15:50.640 --> 00:15:55.000 Now, let's see where our cohort analysis has gotten us. s172 00:15:55.000 --> 00:15:55.840 Okay, so we're done. s173 00:15:55.840 --> 00:15:57.680 So, the verdict is don't ship. s174 00:15:57.680 --> 00:16:12.760 So, even though it looked like from a single replay that it was cheaper to do and we reached the same result, across a bunch of those support cases, you actually saw that our agent concludes that you shouldn't be using a cheaper model in this particular case for your data. s175 00:16:12.760 --> 00:16:15.720 But this might be different for your data. s176 00:16:16.360 --> 00:16:34.120 In conclusion, um if you want to be replaying your agent executions and answering the questions what if I had done something different while designing this agent or what if the agent could be driven to do something different? s177 00:16:34.120 --> 00:16:49.280 Well, you can do this if you model your agent with your harness in a runtime that can checkpoint state and is able to replay that state from code with different scenarios. s178 00:16:49.280 --> 00:16:55.520 If you want to use Guitar Ru, the the the tool that I showed that allows you to do it, um you can scan the repo. s179 00:16:55.520 --> 00:17:01.440 It's open source, free to use, and uh we'd appreciate the feedback and love. s180 00:17:01.440 --> 00:17:06.040 Thank you so much, and see you guys on the next one.