WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=kZsf_Sfm7RU NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/kZsf_Sfm7RU.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.520 --> 00:00:01.400 All right. s2 00:00:01.400 --> 00:00:06.960 So, you built an agent, you launched it, everything works pretty well in a demo, everyone is happy. s3 00:00:06.960 --> 00:00:09.000 But now let me ask you a few simple questions. s4 00:00:09.000 --> 00:00:12.200 So, how do you know if it's actually working out there? s5 00:00:12.200 --> 00:00:16.720 How do you watch across hundreds or thousands of real conversations every day? s6 00:00:16.720 --> 00:00:20.000 How do you feel or understand the health of the system? s7 00:00:20.000 --> 00:00:21.240 How do you make it better? s8 00:00:21.240 --> 00:00:24.680 How do you find the holes that you don't know are there yet? s9 00:00:24.680 --> 00:00:25.760 And that's the thing, right? s10 00:00:25.760 --> 00:00:29.640 So, most of the talks about the agents end up the moment when you ship. s11 00:00:29.640 --> 00:00:31.920 So, apparently it did work, the end. s12 00:00:31.920 --> 00:00:35.520 But I think the shipping is the moment when the real work begins. s13 00:00:35.520 --> 00:00:38.760 And somehow only a few people are talking about that. s14 00:00:38.760 --> 00:00:40.440 And I'm calling it the missing layer. s15 00:00:40.440 --> 00:00:42.480 So, let's dive into it. s16 00:00:42.480 --> 00:00:44.280 So, that's the world that we're living. s17 00:00:44.280 --> 00:00:49.640 So, you can create the whole product, you can create a whole startup in a couple of days, in a couple of weeks. s18 00:00:49.640 --> 00:00:52.280 You can write 100,000 of lines of code. s19 00:00:52.280 --> 00:00:54.760 You can spend a lot of tokens. s20 00:00:54.760 --> 00:00:58.840 And to be honest, it is the easiest part today with the help of the latest models. s21 00:00:58.840 --> 00:01:02.040 But I think the shipping is the moment when the real work begins. s22 00:01:02.040 --> 00:01:05.120 Because you need to close the loop as soon as possible. s23 00:01:05.120 --> 00:01:09.680 So, after you launch, you need to have some control and understanding of the system. s24 00:01:09.680 --> 00:01:16.960 And uh from my experience, the loop is uh at least as important as the product itself, sometimes even more. s25 00:01:16.960 --> 00:01:22.520 Because the tight feedback is the one that helps you to make product better every single day. s26 00:01:22.520 --> 00:01:26.600 And that's the missing layer, and that's what the rest of this talk is all about. s27 00:01:26.600 --> 00:01:28.360 So, what happens after you launch? s28 00:01:28.360 --> 00:01:30.560 And this is not something surprising. s29 00:01:30.560 --> 00:01:33.480 We had the same questions in the classical old software. s30 00:01:33.480 --> 00:01:41.360 You need to monitor what is happening, you need to understand how it behaves, you need to have some logs to detect the problems and fix them. s31 00:01:41.360 --> 00:01:50.600 Uh and for agentic systems, each one of those are even harder, and sometimes uh they turn into something genuinely new. s32 00:01:50.600 --> 00:01:54.320 So, let's talk about why this is hard and why this is hard now. s33 00:01:54.320 --> 00:01:56.680 So, the agent is not a normal software, right? s34 00:01:56.680 --> 00:01:59.680 You You don't have a few features and several buttons. s35 00:01:59.680 --> 00:02:05.920 You don't have a predefined flow that you can test before you go to the to the live. s36 00:02:05.920 --> 00:02:07.400 Uh and the coverage is endless. s37 00:02:07.400 --> 00:02:16.000 So, think about like cloud code or codex, they can do a giant range of stuff whatever user needs and most of the agents do the same, right? s38 00:02:16.000 --> 00:02:18.760 So, you give the instructions and they can handle it. s39 00:02:18.760 --> 00:02:23.360 And you cannot write all the conversations in advance. s40 00:02:23.360 --> 00:02:26.240 So, this leads to a detailed part of the problem. s41 00:02:26.240 --> 00:02:28.880 The part that keeps me up all night. s42 00:02:28.880 --> 00:02:31.240 Which is you lose the feel for your own system. s43 00:02:31.240 --> 00:02:38.760 So, after you build the product, you need to have some kind of understanding, does it get better or worse? s44 00:02:38.760 --> 00:02:41.640 So, we need to monitor and understand what is happening. s45 00:02:41.640 --> 00:02:46.520 And the problem is that the normal the normal safety nets don't save you here. s46 00:02:46.520 --> 00:02:49.880 Uh and believe me, we tried a few stuff. s47 00:02:49.880 --> 00:02:52.080 We we we build the unit test. s48 00:02:52.080 --> 00:02:55.120 We had some regex, some rule-based checks. s49 00:02:55.120 --> 00:02:59.160 We even created some scripts to simulate the customer conversation. s50 00:02:59.160 --> 00:03:03.080 Uh and yeah, it helps in some ways. s51 00:03:03.080 --> 00:03:08.440 but at the end of the day, it is like only one slice of the whole problem. s52 00:03:08.440 --> 00:03:12.959 Because customers always do something different. s53 00:03:12.959 --> 00:03:14.760 Uh you cannot write it all down. s54 00:03:14.760 --> 00:03:18.200 They are too many and they are all different. s55 00:03:18.200 --> 00:03:20.640 And Harrison for it in the same way. s56 00:03:20.640 --> 00:03:25.280 So, you don't know what your agent will do until it is in the production. s57 00:03:25.280 --> 00:03:27.040 So, why this is a new problem? s58 00:03:27.040 --> 00:03:31.640 So, as you know, LLMs are not deterministic. s59 00:03:31.760 --> 00:03:34.160 The same input can have a different path. s60 00:03:34.160 --> 00:03:39.280 Even slight modification of the input can cause a different trajectory and the coverage is endless. s61 00:03:39.280 --> 00:03:40.760 You cannot list it all down. s62 00:03:40.760 --> 00:03:44.920 You cannot pretest until you go in the production. s63 00:03:44.920 --> 00:03:48.640 The second and the scariest one is the failure height itself. s64 00:03:48.640 --> 00:03:52.760 So, let's think about what happens when the agent is running for a long time. s65 00:03:52.760 --> 00:03:54.880 It struggled in the middle of a task. s66 00:03:54.880 --> 00:03:57.960 Uh it has some problems but it was lucky. s67 00:03:57.960 --> 00:03:59.400 Uh it was recovered. s68 00:03:59.400 --> 00:04:01.480 It finds some workarounds. s69 00:04:01.480 --> 00:04:04.560 Uh try some other tool calls or whatever. s70 00:04:04.560 --> 00:04:08.080 And you don't get any red alerts, any problems on a dashboard. s71 00:04:08.080 --> 00:04:09.760 Everything looks fine. s72 00:04:09.760 --> 00:04:13.000 But you know, this is a early warning for you. s73 00:04:13.000 --> 00:04:18.079 This is a problem that uh that is hidden uh and that lives in your code base. s74 00:04:18.079 --> 00:04:27.520 And uh you need to fix it as soon as possible because if you're talking about uh reliable agents, uh it shouldn't be depend on the luck, right? s75 00:04:27.520 --> 00:04:38.600 And uh also as an Anthropic mentioned in the blog post, sometimes the agent love to make marketing features as complete uh without checking they actually worked. s76 00:04:38.600 --> 00:04:40.760 The next one is uh the tool calling, right? s77 00:04:40.760 --> 00:04:43.200 So, the tool surface is a huge. s78 00:04:43.200 --> 00:04:47.160 So, long-running task needs a hundred of tools and thousands of tools. s79 00:04:47.160 --> 00:04:51.640 Um sometimes it have a uh several summarization in the middle. s80 00:04:51.640 --> 00:04:53.680 It use some sub agent, writes a lot of code. s81 00:04:53.680 --> 00:04:55.680 It use a terminal for sure. s82 00:04:55.680 --> 00:04:59.600 It calls some other company services, third-party libraries. s83 00:04:59.600 --> 00:05:04.720 And uh they they work pretty different way uh all the time. s84 00:05:04.720 --> 00:05:07.520 So, actually you don't know what you're looking for. s85 00:05:07.520 --> 00:05:11.240 So, sometimes finish doesn't mean it is helpful for the user. s86 00:05:11.240 --> 00:05:13.640 Maybe there was any there wasn't any problem. s87 00:05:13.640 --> 00:05:16.360 So, agent finished, everything looks good. s88 00:05:16.360 --> 00:05:18.919 The answer was successful. s89 00:05:18.919 --> 00:05:24.240 But what happens like for example for our use case, sometimes uh user asked to build the itinerary. s90 00:05:24.240 --> 00:05:25.919 Agent run the flow. s91 00:05:25.919 --> 00:05:27.600 It built the trip. s92 00:05:27.600 --> 00:05:29.800 But it took a different service. s93 00:05:29.800 --> 00:05:32.280 It made a lot of mistakes in the calculating the price. s94 00:05:32.280 --> 00:05:33.680 So, user is not happy. s95 00:05:33.680 --> 00:05:37.880 So, technically it's successful but still failing the task. s96 00:05:37.880 --> 00:05:41.240 And as I mentioned, the unit test don't save you here. s97 00:05:41.240 --> 00:05:46.960 And production is the place when you learn what you need to uh what you need to test on the first place. s98 00:05:46.960 --> 00:05:47.320 All right. s99 00:05:47.320 --> 00:05:49.800 so what happens after you launch the product, right? s100 00:05:49.800 --> 00:05:54.480 So you need to monitor and you need to understand and improve the system. s101 00:05:54.480 --> 00:05:57.040 Uh and what's the source of the truth, right? s102 00:05:57.040 --> 00:05:57.760 This is a log. s103 00:05:57.760 --> 00:05:58.800 Everyone has the logs. s104 00:05:58.800 --> 00:06:00.080 Everyone loves the logs. s105 00:06:00.080 --> 00:06:02.520 You have structured information. s106 00:06:02.520 --> 00:06:14.400 And uh you know, those machines are the best to understand and explore the logs much better than any human, much faster, taking rise and scripts to filter the giant of walls. s107 00:06:14.400 --> 00:06:20.080 Um And this is the most obvious way that you can hand it to the agent. s108 00:06:20.080 --> 00:06:29.840 Uh but as soon as you start working on that, maybe you build an agent or skill or whatever, you you quickly understand this is not as easy as you imagine s109 00:06:29.840 --> 00:06:31.840 because it needs a lot of reasoning. s110 00:06:31.840 --> 00:06:43.560 You need to understand the problem to uh differentiate if it's a real bug or a noise, to deep dive into the fresh trajectory, understand if it's a a symptom or a root cause. s111 00:06:43.560 --> 00:06:50.720 So actually, you'll find out that the operate operating agent itself is an agent problem. s112 00:06:50.760 --> 00:06:52.640 So this is a look like end-to-end. s113 00:06:52.640 --> 00:06:57.800 So we have a traces, you have trajectories, you give the agent to access the code base. s114 00:06:57.800 --> 00:07:01.880 It diagnose the problem, understand what is happening, and send the PR. s115 00:07:01.880 --> 00:07:11.040 And then you can have a skill or sub-agent or or whatever that controls the PR uh because you know, most of the agents are pretty eager to send the PR. s116 00:07:11.040 --> 00:07:12.960 They love to fix the problems. s117 00:07:12.960 --> 00:07:17.640 We prefer to have a separate agent which has a like fresh context. s118 00:07:17.640 --> 00:07:27.360 It try to check um the PR on different angle, on different view, run the um the focus test. s119 00:07:27.360 --> 00:07:30.240 And uh it is not biased of the problem itself. s120 00:07:30.240 --> 00:07:32.919 It tries to criticize call the PR. s121 00:07:32.919 --> 00:07:36.560 And uh sometimes it uh requests the changes. s122 00:07:36.560 --> 00:07:42.080 Maybe it close the PR directly and it helps us to to filter those problems. s123 00:07:42.080 --> 00:07:47.280 Uh and uh then you have the human in the loop, sometimes you don't. s124 00:07:47.280 --> 00:07:49.560 Uh and we can talk about this later. s125 00:07:49.560 --> 00:07:57.680 But, you know, this is the most obvious thing that you can hand it to the agent, which is which seems like a pretty obvious for most of the people. s126 00:07:57.680 --> 00:08:07.320 And uh a lot of teams use the same practice, but I think people don't appreciate how important it is, and you you need to spend some time on that. s127 00:08:07.320 --> 00:08:08.720 You need to calibrate. s128 00:08:08.720 --> 00:08:12.080 You need to make it reliable to trust the loop. s129 00:08:12.080 --> 00:08:15.080 And this is the the fastest loop that we ever had. s130 00:08:15.080 --> 00:08:26.720 So, you know, after you build this simple uh simple system, you uh already have the feeling and understanding how behaves, what is happening. s131 00:08:26.720 --> 00:08:37.880 You detect the problem, some local fixes as soon as possible, and you can have the PR in PR ready in half an hour, and you can uh easily understand what is happening. s132 00:08:37.880 --> 00:08:40.719 So, let me walk you through uh how I'm handling this. s133 00:08:40.719 --> 00:08:46.600 Everything is not optimal, but I think this this will help you to to get some point. s134 00:08:46.600 --> 00:08:48.760 So, actually we have a two main flow. s135 00:08:48.760 --> 00:08:56.600 The first one is the the fastest loop that helps you to to detect and uh fix the problems as soon as possible. s136 00:08:56.600 --> 00:09:07.080 And the another one is like more on a zoom out that that helps you to have a on a high-level understanding, and it helps you to have hand on a pulse. s137 00:09:07.080 --> 00:09:10.960 So, the first one is a log monitoring agent, the one that I already mentioned. s138 00:09:10.960 --> 00:09:23.440 So, you have trajectories, so we have logs, we have the access to a code base, and it runs every hour or uh uh every few 15 minutes, and it try to understand the problem. s139 00:09:23.440 --> 00:09:33.440 It deep dives in the in the logs, uh understand does the user user end up stuck, and uh send the PR, or sometimes send the Slack alert. s140 00:09:33.440 --> 00:09:47.960 And this is this is pretty uh important because sometimes the problem is uh, uh so critical so we need to fix them as soon as possible and yeah, it works pretty well. s141 00:09:47.960 --> 00:09:50.520 So, this is the one example how the PR can look like. s142 00:09:50.520 --> 00:10:02.400 So, you have a nice description, a short explanation what is happening, you have some metadata, you have a nice diagrams, mermaid or ASCII tables, maybe you will have some HTML s143 00:10:02.400 --> 00:10:12.040 artifacts that helps you to give you a glance of the problem and quickly understand what is this PR for. s144 00:10:12.040 --> 00:10:15.760 So, this is the example how a static analysis can look like. s145 00:10:15.760 --> 00:10:21.160 It helps you to quickly give you a feedback, detect if there are some critical problems. s146 00:10:21.160 --> 00:10:31.760 Sometimes it it is just like heads-up or you know, there are some warnings, there are some problems so you need to check them when you have some time. s147 00:10:31.760 --> 00:10:38.600 And as I mentioned the review agent is uh, pretty critical because it tried to check the problem in different angle. s148 00:10:38.600 --> 00:10:45.800 It always tried to criticize the problem, score the problem and as soon as it's ready, it send the PR to the human. s149 00:10:45.800 --> 00:10:53.720 And sometimes people talk about that if we need to have a human in the loop because you know, still human in the bottleneck in this case. s150 00:10:53.720 --> 00:11:01.480 In our case the the the the PR agent and the review agent send 10 times more PR than the three of us every day. s151 00:11:01.480 --> 00:11:09.080 So, we need to have some clean system how we're going to handle this to not be a bottleneck in this system. s152 00:11:09.080 --> 00:11:20.560 But from my experience as I mentioned this PR and the review agent helps me to have a nice description, some artifacts that helps me to quickly understand the problem s153 00:11:20.560 --> 00:11:25.600 and maybe I spend a few minutes to at least understand what is happening. s154 00:11:25.600 --> 00:11:29.839 So, I think at this time it's okay but maybe we'll remove it in the future. s155 00:11:29.839 --> 00:11:36.480 And yeah, people are talking about that so you need to you need you don't need to be a person like in this problem. s156 00:11:36.480 --> 00:11:40.840 Some of them prefer to remove the humans in this loop. s157 00:11:40.840 --> 00:11:44.720 But you know, the trend is that you need to close the loop first. s158 00:11:44.720 --> 00:11:52.600 So, let's make the problem when you are the bottleneck and then you can remove yourself pretty easily, I think. s159 00:11:52.600 --> 00:11:56.480 So, this is our some examples how the review agent feedback can look like. s160 00:11:56.480 --> 00:12:05.800 As you know, it requests some changes, you have some summary, you have some diagrams, explanation, what are them some risks, some edge cases. s161 00:12:05.800 --> 00:12:16.400 And sometimes after a few iterations of the code in the loop where the key where the key is ready, so we don't need you don't need to have any more changes. s162 00:12:16.400 --> 00:12:22.440 So, after that the human will jump into the loop. s163 00:12:22.440 --> 00:12:28.320 For the previous agent as I mentioned, that is especially good when you want to fix some local problems. s164 00:12:28.320 --> 00:12:39.400 For our case, it runs every hour with the chance on the one hour window of the logs and it doesn't have any high-level understanding of the problem. s165 00:12:39.400 --> 00:12:51.320 So, we found found out that we need to have another system that helps us to to get to get like more than high-level explanation, some kind of a health of the system. s166 00:12:51.320 --> 00:12:55.800 And you know, visibility is the easiest piece. s167 00:12:55.800 --> 00:13:02.880 Before the agent, it was impossible, so you cannot deep dive or summarize some of the top conversations. s168 00:13:02.880 --> 00:13:12.600 But right now, we can have a system that helps you to to score every conversations, understand what's happening and give you a high-level summary of the system. s169 00:13:12.600 --> 00:13:19.440 So, this is the session analyzer and the main goal is to just give me the score of the health of the system, right? s170 00:13:19.440 --> 00:13:31.640 So, we try to check every conversation, runs a lot of state agents, it spend a lot of tokens, but yeah, it helps us to detect some patterns connecting the dots s171 00:13:31.640 --> 00:13:36.160 and try to understand some high-level patterns. s172 00:13:36.160 --> 00:13:49.960 And yeah, it also detects some like logical problems, what is the cause, how many tool calls did you use, how many sub-agents, how many summary happened or whatever. s173 00:13:49.960 --> 00:13:55.200 But also it gives some AI insights, some entities, so you have an understanding of the problem. s174 00:13:55.200 --> 00:14:00.320 So right, as I mentioned, one of the main problem is that you lose the control of the user launch the production. s175 00:14:00.320 --> 00:14:10.760 So you need to have the system that helps you to control, at least to have some understanding what is happening, how it looks like, is healthy or what or not. s176 00:14:10.760 --> 00:14:16.600 So Actually, let me let me show you. s177 00:14:16.600 --> 00:14:19.400 One of the example how how it looks like. s178 00:14:19.400 --> 00:14:33.280 So actually we built it ourselves, so there are a lot of other companies and tools that provide the same kind of system, but I prefer to build it myself because I know what I'm interested in for, what I'm looking for. s179 00:14:33.280 --> 00:14:41.800 So as you can see, we have the health of the system, we have number of sessions that were analyzed, the cost, we have the average score, some success rate. s180 00:14:41.800 --> 00:14:50.680 We have some trends, we have the AI insights, and this is the most important part when the AI when the agent try to connect the dots, find the patterns. s181 00:14:50.680 --> 00:15:02.280 If there are some critical ones, you have a description for each of them, what is this, why it matters, what are the root cause, how many sessions affected, and some recommendation fix. s182 00:15:02.280 --> 00:15:08.880 So right now this is just a anonymized data, but it comes from a from a real conversations. s183 00:15:08.880 --> 00:15:24.800 So we have a score distribution, each company number of sessions or cost we have, the sentiment analysis, some entities, we have analytics about tool calls, all the success rate, how many rejected and why. s184 00:15:24.800 --> 00:15:28.640 And you have a detailed analysis of the each session. s185 00:15:28.640 --> 00:15:31.840 So, it ranks each one of them, scores. s186 00:15:31.840 --> 00:15:42.080 It is how many times it was running, how many messages, how many tool calls, how many summary, and you have a detailed explanation for each of them, what was the problem. s187 00:15:42.080 --> 00:15:51.920 It is something that you need attention, and believe it helps a lot, and it helps you to check and watch across hundreds of uh conversations. s188 00:15:51.920 --> 00:16:01.200 So, the main goal of this dashboard and the system is not to fix some uh um specific problems or bug. s189 00:16:01.200 --> 00:16:07.040 This is more than a high-level that helps you to watch across hundreds or thousands of real conversations. s190 00:16:07.040 --> 00:16:12.800 So, you can run it once in a week, two times in a week, or something like that. s191 00:16:12.800 --> 00:16:13.160 All right. s192 00:16:13.160 --> 00:16:22.640 So, the next one is that mm So, the problem of the previous uh agent that I mentioned is like the first one helps you to detect the specific problem. s193 00:16:22.640 --> 00:16:34.240 The second one is to give you a high-level understanding, but both of them works from a angle of the logs or a uh code or a session itself, but you need to have s194 00:16:34.240 --> 00:16:36.960 some kind of user perspective, right? s195 00:16:36.960 --> 00:16:49.240 So, that's why we have some computer use agent that helps you to uh to open the browser, uh to log in, and try to simulate the customer itself, because sometimes you have some problems in the UI. s196 00:16:49.240 --> 00:16:54.960 You need to um check if everything looks good. s197 00:16:54.960 --> 00:17:01.240 Uh and uh yeah, sometimes code and logs cannot help you uh in this way. s198 00:17:01.240 --> 00:17:03.640 So, this is uh example how it looks like. s199 00:17:03.640 --> 00:17:06.520 So, you used Codex uh sometimes. s200 00:17:06.520 --> 00:17:10.480 So, it used um uh the browser itself. s201 00:17:10.480 --> 00:17:14.199 Uh Actually, this is pretty slow. s202 00:17:14.199 --> 00:17:28.880 Uh and we try to build a specific skill that know our uh website, our DOM, how it looks like, and that is much faster, and it can open the website, login, open the sessions, send the messages, check what is happening. s203 00:17:28.880 --> 00:17:34.120 Uh, and also check, how it looks like, what what are the some artifacts. s204 00:17:34.120 --> 00:17:36.520 And, yeah, it was pretty good. s205 00:17:36.520 --> 00:17:43.320 But, yeah, you you you need to know that it it it will spend a lot of a lot of tokens. s206 00:17:43.480 --> 00:17:52.880 And also, for all of those problems, you need to remember that, you need to give access to all the kind of tools that is needed. s207 00:17:52.880 --> 00:17:59.240 So, you need to have a lot of trajectories and metrics, database, UI, so as the humans need, right? s208 00:17:59.240 --> 00:18:03.880 So, you need to give all context, all possibilities to understand what is happening. s209 00:18:03.880 --> 00:18:16.320 For example, for the computer use agent, when detect some problems, it should be able to analyze the trajectories, check the database, to understand what happened. s210 00:18:16.320 --> 00:18:30.080 And, you know, that's why I'm calling it a meta harness, the whole system, when everything is connected, and the PR or the answer from the agents are depend on the depend on the real problem, s211 00:18:30.080 --> 00:18:33.360 and they're not casing what is happening. s212 00:18:33.360 --> 00:18:50.640 So, the most is not the model alone, and you need to build the agent, or a system, or a harness around it, which which watches itself, understand, improve, and help you at least to at least to monitor what is happening, s213 00:18:50.640 --> 00:19:00.560 but also in the ideal case, to close the loop, send automatic PR, notifications, whatever, and help you to speed up the process. s214 00:19:00.560 --> 00:19:03.440 So, shipping is the easiest part today. s215 00:19:03.440 --> 00:19:12.400 If you want to if you want to build a production agent, you need to close the loop first, because somehow people are not talking about how important it is s216 00:19:12.400 --> 00:19:24.720 what happens after you launch, so everyone can have the same model, everyone can have the same agent or harness, but you need to have some internal system that helps you in this process, s217 00:19:24.720 --> 00:19:31.520 detect the problems, and give you sense what is happening, and helps to make the product better. s218 00:19:31.520 --> 00:19:33.760 Thank you.