The Missing Layer After Launch - Raphael Kalandadze, Wandero AI

AI Engineer · 19 min · 218 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00All right.
00:01So, you built an agent, you launched it, everything works pretty well in a demo, everyone is happy.
00:06But now let me ask you a few simple questions.
00:09So, how do you know if it's actually working out there?
00:12How do you watch across hundreds or thousands of real conversations every day?
00:16How do you feel or understand the health of the system?
00:20How do you make it better?
00:21How do you find the holes that you don't know are there yet?
00:24And that's the thing, right?
00:25So, most of the talks about the agents end up the moment when you ship.
00:29So, apparently it did work, the end.
00:31But I think the shipping is the moment when the real work begins.
00:35And somehow only a few people are talking about that.
00:38And I'm calling it the missing layer.
00:40So, let's dive into it.
00:42So, that's the world that we're living.
00:44So, you can create the whole product, you can create a whole startup in a couple of days, in a couple of weeks.
00:49You can write 100,000 of lines of code.
00:52You can spend a lot of tokens.
00:54And to be honest, it is the easiest part today with the help of the latest models.
00:58But I think the shipping is the moment when the real work begins.
01:02Because you need to close the loop as soon as possible.
01:05So, after you launch, you need to have some control and understanding of the system.
01:09And uh from my experience, the loop is uh at least as important as the product itself, sometimes even more.
01:16Because the tight feedback is the one that helps you to make product better every single day.
01:22And that's the missing layer, and that's what the rest of this talk is all about.
01:26So, what happens after you launch?
01:28And this is not something surprising.
01:30We had the same questions in the classical old software.
01:33You need to monitor what is happening, you need to understand how it behaves, you need to have some logs to detect the problems and fix them.
01:41Uh and for agentic systems, each one of those are even harder, and sometimes uh they turn into something genuinely new.
01:50So, let's talk about why this is hard and why this is hard now.
01:54So, the agent is not a normal software, right?
01:56You You don't have a few features and several buttons.
01:59You don't have a predefined flow that you can test before you go to the to the live.
02:05Uh and the coverage is endless.
02:07So, think about like cloud code or codex, they can do a giant range of stuff whatever user needs and most of the agents do the same, right?
02:16So, you give the instructions and they can handle it.
02:18And you cannot write all the conversations in advance.
02:23So, this leads to a detailed part of the problem.
02:26The part that keeps me up all night.
02:28Which is you lose the feel for your own system.
02:31So, after you build the product, you need to have some kind of understanding, does it get better or worse?
02:38So, we need to monitor and understand what is happening.
02:41And the problem is that the normal the normal safety nets don't save you here.
02:46Uh and believe me, we tried a few stuff.
02:49We we we build the unit test.
02:52We had some regex, some rule-based checks.
02:55We even created some scripts to simulate the customer conversation.
02:59Uh and yeah, it helps in some ways.
03:03but at the end of the day, it is like only one slice of the whole problem.
03:08Because customers always do something different.
03:12Uh you cannot write it all down.
03:14They are too many and they are all different.
03:18And Harrison for it in the same way.
03:20So, you don't know what your agent will do until it is in the production.
03:25So, why this is a new problem?
03:27So, as you know, LLMs are not deterministic.
03:31The same input can have a different path.
03:34Even slight modification of the input can cause a different trajectory and the coverage is endless.
03:39You cannot list it all down.
03:40You cannot pretest until you go in the production.
03:44The second and the scariest one is the failure height itself.
03:48So, let's think about what happens when the agent is running for a long time.
03:52It struggled in the middle of a task.
03:54Uh it has some problems but it was lucky.
03:57Uh it was recovered.
03:59It finds some workarounds.
04:01Uh try some other tool calls or whatever.
04:04And you don't get any red alerts, any problems on a dashboard.
04:08Everything looks fine.
04:09But you know, this is a early warning for you.
04:13This is a problem that uh that is hidden uh and that lives in your code base.
04:18And uh you need to fix it as soon as possible because if you're talking about uh reliable agents, uh it shouldn't be depend on the luck, right?
04:27And uh also as an Anthropic mentioned in the blog post, sometimes the agent love to make marketing features as complete uh without checking they actually worked.
04:38The next one is uh the tool calling, right?
04:40So, the tool surface is a huge.
04:43So, long-running task needs a hundred of tools and thousands of tools.
04:47Um sometimes it have a uh several summarization in the middle.
04:51It use some sub agent, writes a lot of code.
04:53It use a terminal for sure.
04:55It calls some other company services, third-party libraries.
04:59And uh they they work pretty different way uh all the time.
05:04So, actually you don't know what you're looking for.
05:07So, sometimes finish doesn't mean it is helpful for the user.
05:11Maybe there was any there wasn't any problem.
05:13So, agent finished, everything looks good.
05:16The answer was successful.
05:18But what happens like for example for our use case, sometimes uh user asked to build the itinerary.
05:24Agent run the flow.
05:25It built the trip.
05:27But it took a different service.
05:29It made a lot of mistakes in the calculating the price.
05:32So, user is not happy.
05:33So, technically it's successful but still failing the task.
05:37And as I mentioned, the unit test don't save you here.
05:41And production is the place when you learn what you need to uh what you need to test on the first place.
05:46All right.
05:47so what happens after you launch the product, right?
05:49So you need to monitor and you need to understand and improve the system.
05:54Uh and what's the source of the truth, right?
05:57This is a log.
05:57Everyone has the logs.
05:58Everyone loves the logs.
06:00You have structured information.
06:02And uh you know, those machines are the best to understand and explore the logs much better than any human, much faster, taking rise and scripts to filter the giant of walls.
06:14Um And this is the most obvious way that you can hand it to the agent.
06:20Uh but as soon as you start working on that, maybe you build an agent or skill or whatever, you you quickly understand this is not as easy as you imagine
06:29because it needs a lot of reasoning.
06:31You need to understand the problem to uh differentiate if it's a real bug or a noise, to deep dive into the fresh trajectory, understand if it's a a symptom or a root cause.
06:43So actually, you'll find out that the operate operating agent itself is an agent problem.
06:50So this is a look like end-to-end.
06:52So we have a traces, you have trajectories, you give the agent to access the code base.
06:57It diagnose the problem, understand what is happening, and send the PR.
07:01And then you can have a skill or sub-agent or or whatever that controls the PR uh because you know, most of the agents are pretty eager to send the PR.
07:11They love to fix the problems.
07:12We prefer to have a separate agent which has a like fresh context.
07:17It try to check um the PR on different angle, on different view, run the um the focus test.
07:27And uh it is not biased of the problem itself.
07:30It tries to criticize call the PR.
07:32And uh sometimes it uh requests the changes.
07:36Maybe it close the PR directly and it helps us to to filter those problems.
07:42Uh and uh then you have the human in the loop, sometimes you don't.
07:47Uh and we can talk about this later.
07:49But, you know, this is the most obvious thing that you can hand it to the agent, which is which seems like a pretty obvious for most of the people.
07:57And uh a lot of teams use the same practice, but I think people don't appreciate how important it is, and you you need to spend some time on that.
08:07You need to calibrate.
08:08You need to make it reliable to trust the loop.
08:12And this is the the fastest loop that we ever had.
08:15So, you know, after you build this simple uh simple system, you uh already have the feeling and understanding how behaves, what is happening.
08:26You detect the problem, some local fixes as soon as possible, and you can have the PR in PR ready in half an hour, and you can uh easily understand what is happening.
08:37So, let me walk you through uh how I'm handling this.
08:40Everything is not optimal, but I think this this will help you to to get some point.
08:46So, actually we have a two main flow.
08:48The first one is the the fastest loop that helps you to to detect and uh fix the problems as soon as possible.
08:56And the another one is like more on a zoom out that that helps you to have a on a high-level understanding, and it helps you to have hand on a pulse.
09:07So, the first one is a log monitoring agent, the one that I already mentioned.
09:10So, you have trajectories, so we have logs, we have the access to a code base, and it runs every hour or uh uh every few 15 minutes, and it try to understand the problem.
09:23It deep dives in the in the logs, uh understand does the user user end up stuck, and uh send the PR, or sometimes send the Slack alert.
09:33And this is this is pretty uh important because sometimes the problem is uh, uh so critical so we need to fix them as soon as possible and yeah, it works pretty well.
09:47So, this is the one example how the PR can look like.
09:50So, you have a nice description, a short explanation what is happening, you have some metadata, you have a nice diagrams, mermaid or ASCII tables, maybe you will have some HTML
10:02artifacts that helps you to give you a glance of the problem and quickly understand what is this PR for.
10:12So, this is the example how a static analysis can look like.
10:15It helps you to quickly give you a feedback, detect if there are some critical problems.
10:21Sometimes it it is just like heads-up or you know, there are some warnings, there are some problems so you need to check them when you have some time.
10:31And as I mentioned the review agent is uh, pretty critical because it tried to check the problem in different angle.
10:38It always tried to criticize the problem, score the problem and as soon as it's ready, it send the PR to the human.
10:45And sometimes people talk about that if we need to have a human in the loop because you know, still human in the bottleneck in this case.
10:53In our case the the the the PR agent and the review agent send 10 times more PR than the three of us every day.
11:01So, we need to have some clean system how we're going to handle this to not be a bottleneck in this system.
11:09But from my experience as I mentioned this PR and the review agent helps me to have a nice description, some artifacts that helps me to quickly understand the problem
11:20and maybe I spend a few minutes to at least understand what is happening.
11:25So, I think at this time it's okay but maybe we'll remove it in the future.
11:29And yeah, people are talking about that so you need to you need you don't need to be a person like in this problem.
11:36Some of them prefer to remove the humans in this loop.
11:40But you know, the trend is that you need to close the loop first.
11:44So, let's make the problem when you are the bottleneck and then you can remove yourself pretty easily, I think.
11:52So, this is our some examples how the review agent feedback can look like.
11:56As you know, it requests some changes, you have some summary, you have some diagrams, explanation, what are them some risks, some edge cases.
12:05And sometimes after a few iterations of the code in the loop where the key where the key is ready, so we don't need you don't need to have any more changes.
12:16So, after that the human will jump into the loop.
12:22For the previous agent as I mentioned, that is especially good when you want to fix some local problems.
12:28For our case, it runs every hour with the chance on the one hour window of the logs and it doesn't have any high-level understanding of the problem.
12:39So, we found found out that we need to have another system that helps us to to get to get like more than high-level explanation, some kind of a health of the system.
12:51And you know, visibility is the easiest piece.
12:55Before the agent, it was impossible, so you cannot deep dive or summarize some of the top conversations.
13:02But right now, we can have a system that helps you to to score every conversations, understand what's happening and give you a high-level summary of the system.
13:12So, this is the session analyzer and the main goal is to just give me the score of the health of the system, right?
13:19So, we try to check every conversation, runs a lot of state agents, it spend a lot of tokens, but yeah, it helps us to detect some patterns connecting the dots
13:31and try to understand some high-level patterns.
13:36And yeah, it also detects some like logical problems, what is the cause, how many tool calls did you use, how many sub-agents, how many summary happened or whatever.
13:49But also it gives some AI insights, some entities, so you have an understanding of the problem.
13:55So right, as I mentioned, one of the main problem is that you lose the control of the user launch the production.
14:00So you need to have the system that helps you to control, at least to have some understanding what is happening, how it looks like, is healthy or what or not.
14:10So Actually, let me let me show you.
14:16One of the example how how it looks like.
14:19So actually we built it ourselves, so there are a lot of other companies and tools that provide the same kind of system, but I prefer to build it myself because I know what I'm interested in for, what I'm looking for.
14:33So as you can see, we have the health of the system, we have number of sessions that were analyzed, the cost, we have the average score, some success rate.
14:41We have some trends, we have the AI insights, and this is the most important part when the AI when the agent try to connect the dots, find the patterns.
14:50If there are some critical ones, you have a description for each of them, what is this, why it matters, what are the root cause, how many sessions affected, and some recommendation fix.
15:02So right now this is just a anonymized data, but it comes from a from a real conversations.
15:08So we have a score distribution, each company number of sessions or cost we have, the sentiment analysis, some entities, we have analytics about tool calls, all the success rate, how many rejected and why.
15:24And you have a detailed analysis of the each session.
15:28So, it ranks each one of them, scores.
15:31It is how many times it was running, how many messages, how many tool calls, how many summary, and you have a detailed explanation for each of them, what was the problem.
15:42It is something that you need attention, and believe it helps a lot, and it helps you to check and watch across hundreds of uh conversations.
15:51So, the main goal of this dashboard and the system is not to fix some uh um specific problems or bug.
16:01This is more than a high-level that helps you to watch across hundreds or thousands of real conversations.
16:07So, you can run it once in a week, two times in a week, or something like that.
16:12All right.
16:13So, the next one is that mm So, the problem of the previous uh agent that I mentioned is like the first one helps you to detect the specific problem.
16:22The second one is to give you a high-level understanding, but both of them works from a angle of the logs or a uh code or a session itself, but you need to have
16:34some kind of user perspective, right?
16:36So, that's why we have some computer use agent that helps you to uh to open the browser, uh to log in, and try to simulate the customer itself, because sometimes you have some problems in the UI.
16:49You need to um check if everything looks good.
16:54Uh and uh yeah, sometimes code and logs cannot help you uh in this way.
17:01So, this is uh example how it looks like.
17:03So, you used Codex uh sometimes.
17:06So, it used um uh the browser itself.
17:10Uh Actually, this is pretty slow.
17:14Uh and we try to build a specific skill that know our uh website, our DOM, how it looks like, and that is much faster, and it can open the website, login, open the sessions, send the messages, check what is happening.
17:28Uh, and also check, how it looks like, what what are the some artifacts.
17:34And, yeah, it was pretty good.
17:36But, yeah, you you you need to know that it it it will spend a lot of a lot of tokens.
17:43And also, for all of those problems, you need to remember that, you need to give access to all the kind of tools that is needed.
17:52So, you need to have a lot of trajectories and metrics, database, UI, so as the humans need, right?
17:59So, you need to give all context, all possibilities to understand what is happening.
18:03For example, for the computer use agent, when detect some problems, it should be able to analyze the trajectories, check the database, to understand what happened.
18:16And, you know, that's why I'm calling it a meta harness, the whole system, when everything is connected, and the PR or the answer from the agents are depend on the depend on the real problem,
18:30and they're not casing what is happening.
18:33So, the most is not the model alone, and you need to build the agent, or a system, or a harness around it, which which watches itself, understand, improve, and help you at least to at least to monitor what is happening,
18:50but also in the ideal case, to close the loop, send automatic PR, notifications, whatever, and help you to speed up the process.
19:00So, shipping is the easiest part today.
19:03If you want to if you want to build a production agent, you need to close the loop first, because somehow people are not talking about how important it is
19:12what happens after you launch, so everyone can have the same model, everyone can have the same agent or harness, but you need to have some internal system that helps you in this process,
19:24detect the problems, and give you sense what is happening, and helps to make the product better.
19:31Thank you.