WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=BOCjybvJYM4

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/BOCjybvJYM4.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:13.840 --> 00:00:15.040
Hello everyone.

s2
00:00:15.040 --> 00:00:20.480
Welcome to the workshop on continuously improving agents with Langfuse.

s3
00:00:20.480 --> 00:00:23.760
Um you can already go to lenuse.com/workshop.

s4
00:00:23.760 --> 00:00:33.600
There are steps there that we will go over today and that way you can also later at home when you want to revisit it go over it yourself and there are also some extra steps that we won't cover today

s5
00:00:33.600 --> 00:00:47.600
that you can find then as well we'll show this QR code later in the presentation uh again so continuously improving agents what does it mean we will be talking a lot about

s6
00:00:47.600 --> 00:01:05.280
um different processes that teams use to then uh to improve their agents or to detect changes and improve their agents in different ways and you can change these together into a continuous loop and as we know in San Francisco we really like loops

s7
00:01:05.280 --> 00:01:21.280
and so this process and this set of these set of processes together you can it's also often called AI engineering so what is AI engineering and why do we need it and how is it different from traditional engineering

s8
00:01:21.280 --> 00:01:26.320
the main difference is that AI agents are not deter deterministic.

s9
00:01:26.320 --> 00:01:37.040
So your system your status of your system can look healthy while the responses or the actions your agent is taking are wrong.

s10
00:01:37.040 --> 00:01:38.799
Sometimes it's actually wrong.

s11
00:01:38.799 --> 00:01:42.960
Sometimes it's also not qualitative enough.

s12
00:01:44.720 --> 00:01:48.479
And so um this is what AI engineering solves.

s13
00:01:48.479 --> 00:02:04.799
For the people who are not familiar, um, Langfuse offers a platform to actually execute on these processes and, uh, we are open source, model agnostic, framework agnostic, so you can actually use us with whatever setup you already have.

s14
00:02:04.880 --> 00:02:17.040
There are some large companies that already use day-to-day, but we also have a large community of smaller individual users who really help us because they're working on many different setups and agent projects.

s15
00:02:17.040 --> 00:02:26.080
And this is definitely one of the um ways we keep up to date with where this very fastm moving field is going.

s16
00:02:26.080 --> 00:02:34.879
By the way, uh this workshop is given by Annabelle and me today and Mark, our founder, is also somewhere in the room and there he is.

s17
00:02:34.879 --> 00:02:41.840
And we will also be walking around to help you um during the workshop if you need anything.

s18
00:02:42.080 --> 00:02:52.720
So AI engineering um we like to show or visualize the set of processes that that people do in this graph.

s19
00:02:52.720 --> 00:02:55.440
Uh this is what we call the AI engineering loop.

s20
00:02:55.440 --> 00:02:59.599
The exact setup depends a bit on the setup also of companies.

s21
00:02:59.599 --> 00:03:05.360
Um it depends on different needs but high level you can categorize them in these five steps.

s22
00:03:05.360 --> 00:03:10.400
And the main thing here to um remember is we have two big parts.

s23
00:03:10.400 --> 00:03:26.879
We have an online part tracing and monitoring and an offline part building data sets experimenting and evaluating not in production and we are today mainly going to focus on the online part here tracing and monitoring.

s24
00:03:26.879 --> 00:03:37.760
So tracing you instrument your application you deploy your app to production and you will see traces coming in of what your agent is doing how it's interacting with users.

s25
00:03:37.760 --> 00:03:49.760
Based on that you will be monitoring these traces and monitoring is on the one hand automatic with evaluators and with dashboards but also the human feedback part cannot be underestimated

s26
00:03:49.760 --> 00:03:50.400
here.

s27
00:03:50.400 --> 00:04:01.120
You will also be collecting implicit or explicit explicit user feedback and also manually be going through some samples of these traces.

s28
00:04:01.120 --> 00:04:06.799
This together I don't know if you can see oh the yellow um box is actually quite clear.

s29
00:04:06.799 --> 00:04:07.360
Great.

s30
00:04:07.360 --> 00:04:12.319
Um, this together the online part of the AI engineering loop is already a lot.

s31
00:04:12.319 --> 00:04:22.000
A lot of teams that are not in high risk industries can actually just start with these two and based on what they find during monitoring of their traces can already

s32
00:04:22.000 --> 00:04:28.160
um make simple fixes or improvements to their agents, deploy these to production and see their traces coming in.

s33
00:04:28.160 --> 00:04:31.520
This is a great first step and it will already get you very far.

s34
00:04:31.520 --> 00:04:36.080
Um, and it it's a great way to iterate on your agents fast.

s35
00:04:36.080 --> 00:04:47.440
Once your product matures a bit, your setup matures or you are in a high-risk environment where any mistake has bad consequences, um it makes sense to also work on the offline part.

s36
00:04:47.440 --> 00:04:49.040
So that's really for structured testing.

s37
00:04:49.040 --> 00:04:51.680
So your data sets are your structured test cases.

s38
00:04:51.680 --> 00:04:56.080
These will be heavily inspired by the way by what you see during monitoring.

s39
00:04:56.080 --> 00:04:59.600
And when you want to make a change, you run an experiment on your data sets.

s40
00:04:59.600 --> 00:05:02.400
You evaluate the results of the experiments.

s41
00:05:02.400 --> 00:05:06.720
And only when the results are better, you didn't introduce any regressions.

s42
00:05:06.720 --> 00:05:09.199
That's when you deploy to production.

s43
00:05:09.199 --> 00:05:15.759
Before we go into the workshop, we very quickly want to focus on the two parts in the online loop.

s44
00:05:15.759 --> 00:05:19.680
Um, so that's tracing and monitoring because this is really important.

s45
00:05:19.680 --> 00:05:22.000
Everything starts with your trace.

s46
00:05:22.000 --> 00:05:28.160
So because if your trace is bad, you will not be catching everything during the monitoring step.

s47
00:05:28.160 --> 00:05:33.280
If you don't catch everything during the monitoring steps, your data sets will be not as good as they could be.

s48
00:05:33.280 --> 00:05:41.759
You will not make the best informed decisions on what to evaluate, etc. So, it's really important to take enough time to make sure your traces are set up well.

s49
00:05:41.759 --> 00:05:44.080
So, what's how do you build a trace?

s50
00:05:44.080 --> 00:05:54.639
A trace exists um consists of observations and observations are different steps an agent is taking while creating an answer for the user.

s51
00:05:54.639 --> 00:05:59.360
These consist of LLM calls or these can also be tool calls or other things.

s52
00:05:59.360 --> 00:06:01.600
These can also be nested.

s53
00:06:01.600 --> 00:06:06.720
Um each observation also has a lot of data attached and this depends on the kind of observation.

s54
00:06:06.720 --> 00:06:19.199
For an LM call, this is usually the duration, token cost, um token usage model that was used, a prompt version that was used, everything you can think of that will actually help you assess the quality

s55
00:06:19.199 --> 00:06:21.280
um of your LM call.

s56
00:06:21.280 --> 00:06:24.240
And for tool calls it can be completely different.

s57
00:06:24.240 --> 00:06:32.240
For example, if you do a web search, one data point can be the query that was used to do this web search.

s58
00:06:32.240 --> 00:06:35.919
So this is really important so that you can then do monitoring.

s59
00:06:35.919 --> 00:06:40.880
Well, monitoring there are a couple of different functions of monitoring.

s60
00:06:40.880 --> 00:06:46.000
One is you want to have aggregate metrics to then see overall contingency over time.

s61
00:06:46.000 --> 00:06:48.160
How is the health of your system?

s62
00:06:48.160 --> 00:06:52.639
both things like the cost of your system for example but also the quality.

s63
00:06:52.639 --> 00:07:01.360
So if you set up evaluators uh to measure the quality of your responses uh you want to make sure this is not going down over time.

s64
00:07:01.360 --> 00:07:05.280
A second very important one is signal detection via user feedback.

s65
00:07:05.280 --> 00:07:17.199
You will have as your application scales a lot of traces and will be impossible to see um or go through everything and user feedback is a great way to surface the ones that actually are important to look at.

s66
00:07:17.199 --> 00:07:18.960
Third one is feedback and evals.

s67
00:07:18.960 --> 00:07:29.280
So you will set up automatic evaluators and uh also you will do some human annotation to make sure um you catch new things.

s68
00:07:29.280 --> 00:07:30.479
One important thing here.

s69
00:07:30.479 --> 00:07:33.680
So you have the automatic evaluators.

s70
00:07:33.680 --> 00:07:36.240
These on the one hand are very scalable.

s71
00:07:36.240 --> 00:07:38.720
You can keep on doing them on all your traces.

s72
00:07:38.720 --> 00:07:46.720
On the other hand, you will you're less likely to catch very new or nuanced cases that you didn't think about on by yourself.

s73
00:07:46.720 --> 00:07:57.039
But user feedback and human annotation might scale a bit less well, but these are where you will detect really things you haven't thought about yourself.

s74
00:07:57.039 --> 00:08:05.199
So this is what we'll focus on today and my colleague Annabelle will go through a practical example and Mark and I will be walking around.

s75
00:08:05.199 --> 00:08:07.120
If you need help, just raise your hands.

s76
00:08:07.120 --> 00:08:16.879
So today's focus as L already mentioned is all about how to effectively working with a trace of your application also how to decide what you actually want to monitor.

s77
00:08:16.879 --> 00:08:31.039
So what is interesting for you in your application and not like generally and then also um how you can leverage coding agents to uh 10x yourself a little bit but also maybe where you should uh be in the loop and not let the coding agent do everything on its own.

s78
00:08:31.039 --> 00:08:40.959
So uh let's dive into it and um to do so uh we first want to meet our sample application for today and we brought for you specs the dad IT support agent.

s79
00:08:40.959 --> 00:08:54.320
So this uh mini repo came out of a a mini project um I did uh for my dad because my dad always asks me questions on WhatsApp uh about how how to set up his phone and also how to set up all kinds of things with his printer or his computer.

s80
00:08:54.320 --> 00:08:59.839
Um so I built this little agent that is replying to him um in the production version via WhatsApp.

s81
00:08:59.839 --> 00:09:02.160
In this case, we now have a a nice little UI.

s82
00:09:02.160 --> 00:09:11.760
So, how you can imagine working it is we have a user, my dad or whatever dad and um the dad is asking specs a question and specs has a few tools available.

s83
00:09:11.760 --> 00:09:13.680
So, we can get support context.

s84
00:09:13.680 --> 00:09:19.680
So, some um information about what kind of phone is dad using and what kind of operating system is it on?

s85
00:09:19.680 --> 00:09:21.279
What kind of apps does he have?

s86
00:09:21.279 --> 00:09:23.440
And we also have a search help library.

s87
00:09:23.440 --> 00:09:34.959
saw some frequently asked questions where there's some context on how to go about solving certain issues and um yeah it can use these tools and then we also use the uh use openai the API endpoint

s88
00:09:34.959 --> 00:09:39.360
uh to generate replies and get a reply back to our uh dad in this case.

s89
00:09:39.360 --> 00:09:50.640
So this application we're going to first um look at traces together but then also think about what is actually interesting that we want to know about uh to make sure dad gets the support he actually wants to have.

s90
00:09:50.640 --> 00:09:53.360
So um yeah, you can scan the QR code here.

s91
00:09:53.360 --> 00:09:58.880
Um you will find the workshop repository and um I'll get uh the setup together with you.

s92
00:09:58.880 --> 00:10:01.279
So no worries and we're going to go through step by step.

s93
00:10:01.279 --> 00:10:07.120
There's also documentation in case you get lost, you could also reference there and if you want to explore more afterwards.

s94
00:10:07.120 --> 00:10:11.600
Uh there's also be going to be some more um modules available.

s95
00:10:11.600 --> 00:10:19.519
So let's go to lenfuse.comworkshop and we can see that um yeah this is a a markdown page here.

s96
00:10:19.519 --> 00:10:31.519
Uh but if you click here on the lenfuse/langfuse workshop, we actually get to git repository and right here on um yeah, you can clone uh the repo into your um IDE.

s97
00:10:31.519 --> 00:10:32.880
I already did that.

s98
00:10:32.880 --> 00:10:40.800
Um so it's prepared and we can see here that the first thing um we need to get set up is of course some uh end variables.

s99
00:10:40.800 --> 00:10:47.839
And for this um let's create a or together on languis.com.

s100
00:10:49.200 --> 00:10:50.800
We can go to launch app.

s101
00:10:50.800 --> 00:10:52.399
I already have an organization.

s102
00:10:52.399 --> 00:10:58.560
Um but you can go to uh launch app or also if you're on the page here and then uh select a region.

s103
00:10:58.560 --> 00:11:00.399
It doesn't matter which one you're taking.

s104
00:11:00.399 --> 00:11:05.680
And then um you can create an account and as soon as you're in there uh what you will see is this page.

s105
00:11:05.680 --> 00:11:16.079
So there will be a project autocreated for you and you will land um on this page and there you can get your um API keys by uh clicking on create new API keys.

s106
00:11:16.079 --> 00:11:22.160
Um, so maybe I'll give you just a second uh to get to our website and get this um get this set up.

s107
00:11:22.160 --> 00:11:26.240
And this will be the the very first thing we have to do together.

s108
00:11:30.079 --> 00:11:40.320
And I'm going to show also here the steps that you will also find on our um workshop website.

s109
00:11:43.600 --> 00:11:47.519
So quickly signing up, creating a new account.

s110
00:11:47.519 --> 00:11:52.959
And for everyone who now did that already, um we brought uh for today an open AI key.

s111
00:11:52.959 --> 00:11:54.240
So you can scan it.

s112
00:11:54.240 --> 00:11:56.160
We're going to rotate it afterwards, of course.

s113
00:11:56.160 --> 00:12:06.640
Um but for today, um this is your API key to use and you can add this into your N uh file and then we'll uh get started.

s114
00:12:11.680 --> 00:12:11.920
Yeah.

s115
00:12:11.920 --> 00:12:15.920
So just copying the NF sample and then getting there.

s116
00:12:28.160 --> 00:12:29.040
Okay.

s117
00:12:29.040 --> 00:12:34.639
So when we have the uh repo cloned, the first thing we have to do is um we have to quickly install the dependencies.

s118
00:12:34.639 --> 00:12:45.200
So we're running here a TypeScript application that is using the OpenAI um agent SDK and uh yeah like this we can quickly install all the dependencies um it is available

s119
00:12:45.200 --> 00:12:56.959
and then we can also start it by running npm rundev and I'll pull this out here.

s120
00:12:56.959 --> 00:12:59.360
It's opening here directly.

s121
00:12:59.360 --> 00:13:10.959
I'm also going to open it here and we can see our um yeah our sample application and by the way if I'm a little too fast like here are the steps uh in the setup page on lenfus.com/workshop

s122
00:13:10.959 --> 00:13:18.320
um how you can also get there um step by step by adjusting the um end file and then running uh the application.

s123
00:13:18.320 --> 00:13:19.920
So let's see what we have here.

s124
00:13:19.920 --> 00:13:29.519
we have our um dad support agent and it's already greeting us uh with certain questions if I have maybe questions and also we have some um sample questions that dad might ask

s125
00:13:29.519 --> 00:13:37.360
um so we can say here how do I turn on my Bluetooth on my iPhone and then specs is taking a second and using in the background the tools

s126
00:13:37.360 --> 00:13:50.639
um that we're like needing um to get there and then we replies to our dad in this case uh now with a step-by-step in instructions on how to um yeah how to turn on Bluetooth on the device

s127
00:13:50.639 --> 00:13:53.200
and um what happened now in the background.

s128
00:13:53.200 --> 00:13:57.120
That is now uh the interesting part um that we want to look into.

s129
00:13:57.120 --> 00:14:06.560
And if we now click back into tracing and this is now also what should happen for you if you run the application and ask a single question then uh we see the first trace popping up.

s130
00:14:06.560 --> 00:14:14.560
So every row here is an observation and we can also see that they have little different icons and these different icons mean that they are different things.

s131
00:14:14.560 --> 00:14:26.959
So if we click into one of them, we can see that the whole thing is rooted in our um our agent observation and then below we have a few generations but also tool calls where certain context is pulled.

s132
00:14:26.959 --> 00:14:34.399
Um so this is now already a fully instrumented application and there's also a step-by-step um guide on on how to get this actually set up.

s133
00:14:34.399 --> 00:14:38.000
But we also have a lengthy skill that you can use to automatically get it set up.

s134
00:14:38.000 --> 00:14:44.639
And for today, we actually want to focus on working with those traces instead of step by step uh setting them up.

s135
00:14:44.639 --> 00:14:46.880
So, um yeah, here we have our trace.

s136
00:14:46.880 --> 00:14:59.120
And if we now uh get back to our um lenfuse.com/workshop page, we can go into the monitoring workshop uh module, which is the part that we actually want to check out together today as lot already mentioned.

s137
00:14:59.120 --> 00:15:04.560
So, um in our context of our dad support app, what does monitoring actually mean?

s138
00:15:04.560 --> 00:15:10.399
So whenever dad is interacting with the application, we of course don't see what he does uh because we don't sit next to him.

s139
00:15:10.399 --> 00:15:11.920
That's the whole point about the agent.

s140
00:15:11.920 --> 00:15:20.079
H but we actually want to track what's going on and we don't only want to track um what kind of things the agent is doing but we actually want to make sense of it.

s141
00:15:20.079 --> 00:15:24.880
So we thought about what are actually things that can go wrong in a support application.

s142
00:15:24.880 --> 00:15:29.199
And the first one is maybe the dad is disagreeing with the application.

s143
00:15:29.199 --> 00:15:33.519
So for example um if I'm asking a question, how do I set up Bluetooth?

s144
00:15:33.519 --> 00:15:36.959
Um, then I'm wondering, okay, um, how does it actually work?

s145
00:15:36.959 --> 00:15:37.680
Where's the button?

s146
00:15:37.680 --> 00:15:46.000
And dad is, um, asking this, and we might want to catch this because there was something maybe not clear enough, not detailed enough, um, for the question to actually be properly,

s147
00:15:46.000 --> 00:15:47.279
um, done.

s148
00:15:47.279 --> 00:15:50.720
So, um, let's get this set up together.

s149
00:15:50.720 --> 00:15:58.560
We're h starting with the um, yeah, with copying this git command here, get checkout checkpoint for monitoring.

s150
00:15:58.560 --> 00:16:02.959
And then we can get back into our um, IDE.

s151
00:16:03.440 --> 00:16:05.040
and run this command.

s152
00:16:05.040 --> 00:16:10.560
And we can see now that we swapping or switching to the uh monitoring uh step of our workshop.

s153
00:16:10.560 --> 00:16:12.399
So it's a checkpoint.

s154
00:16:12.399 --> 00:16:18.959
And now we want to wire up those um those evaluators in our application.

s155
00:16:18.959 --> 00:16:26.959
So um if we go down a little bit, we can see that the first thing um that we have to do together is configuring the Lenfuse evaluator model.

s156
00:16:26.959 --> 00:16:31.120
So I'm going to jump into our uh application here.

s157
00:16:31.120 --> 00:16:36.800
I'm going to close the tracing part and I'm going to hop into our evaluator tab.

s158
00:16:36.800 --> 00:16:41.279
So here I can see I can get started with my evaluators and there's two different possibilities.

s159
00:16:41.279 --> 00:16:57.199
I can either set up codebased evaluators for more deterministic checks about my application and I can also get set up with some LLM as a Dutch evaluators um which are like more for the implicit things that you might want to get out of the conversation history

s160
00:16:57.199 --> 00:16:59.199
uh and actually want to find out there.

s161
00:16:59.199 --> 00:17:08.480
And we can also see that we have some uh templates here and that our uh user disagreement one um is actually one template that we are uh providing here.

s162
00:17:08.480 --> 00:17:14.559
So if we click on here we can see okay we need to get up uh set up with the um LLM connection.

s163
00:17:14.559 --> 00:17:21.600
So I'm going to jump over here and very quickly stop sharing because shouldn't see the API key.

s164
00:17:24.880 --> 00:17:27.280
laptop died.

s165
00:17:28.960 --> 00:17:44.880
So this is in the end um a possibility to use a certain model to evaluate um what's going on and to give one second.

s166
00:17:50.080 --> 00:17:51.200
Okay.

s167
00:17:51.200 --> 00:17:51.440
Yeah.

s168
00:17:51.440 --> 00:18:00.160
So what I now did is I um set up a new LLM connection and in this case I'm using again the OpenAI API key.

s169
00:18:01.760 --> 00:18:03.600
Go back.

s170
00:18:03.600 --> 00:18:04.080
Yeah.

s171
00:18:04.080 --> 00:18:08.559
Um so I connected the OpenAI API key now that we um set up together.

s172
00:18:08.559 --> 00:18:17.919
And um in instead you can also connect all kinds of other uh models of course and you can also um use your own proxy in case you have a have a gateway configured yourself.

s173
00:18:17.919 --> 00:18:21.039
And now we're going to um jump back into our evaluator page.

s174
00:18:21.039 --> 00:18:22.880
We're going to click on create evaluator.

s175
00:18:22.880 --> 00:18:28.960
And now select our template again and select the model.

s176
00:18:28.960 --> 00:18:36.160
And what we can see here now is that the first thing we can do together is we can filter for the actual um observations that we want to target.

s177
00:18:36.160 --> 00:18:45.039
And in this case uh we want to go um not for the generation um but for the agent that is the root um of the whole uh trace that we're having here.

s178
00:18:45.039 --> 00:18:47.600
And then you can also see that there are some other options.

s179
00:18:47.600 --> 00:18:50.000
So you can also target tool calls or generations.

s180
00:18:50.000 --> 00:18:53.679
But in our specific case, we want to look at the um input only.

s181
00:18:53.679 --> 00:19:00.080
So we want to see uh the the message of the user um being looked at.

s182
00:19:00.080 --> 00:19:16.320
So looking at our instructions here again, um we have a certain way to target those different different parts um of the observation.

s183
00:19:16.320 --> 00:19:25.520
So we're taking the agent, we have our evaluator prompt here, and down here we can see that um via um curled brackets, we're actually pulling in um context here.

s184
00:19:25.520 --> 00:19:35.520
And to target the part of the trace that is actually relevant for us, uh we want to set up this JSON path here and make sure that we're targeting the part um that is that is correct in this case.

s185
00:19:35.520 --> 00:19:40.000
So for the conversation history, we actually want to target the whole message array.

s186
00:19:40.000 --> 00:19:45.840
So, we're going to copy this here and we're packing it in here.

s187
00:19:45.840 --> 00:19:49.679
And what we can see down here now is that it's mapping in the message history of your trace.

s188
00:19:49.679 --> 00:19:56.799
So, it's pulling together what was the interaction, what did that ask, what the tools um do, and then also what kind of reply was generated.

s189
00:19:56.799 --> 00:20:08.960
And as a second part of um detecting if a user disagrees um we of course um need to also target the last message that is coming from the user and like this

s190
00:20:08.960 --> 00:20:22.880
also get information here and we can see that down here now we're mapping how do I turn on Bluetooth on my phone into this um evaluator and if we now

s191
00:20:22.880 --> 00:20:26.160
execute this we have our first live running evaluator.

s192
00:20:26.160 --> 00:20:33.360
So you can imagine if now a trace is coming in where um dad is saying okay how do I turn on Bluetooth and then he asks as a follow-up

s193
00:20:33.360 --> 00:20:43.600
oh but this button is actually not there I don't know where to find it and um it's not true what you're saying then we would catch this because this is actually an interesting event a signal for us to understand

s194
00:20:43.600 --> 00:20:53.440
okay where are things going wrong and probably like in the first 500 questions there it's not like all 500 fail but maybe there's five or six or seven different ones

s195
00:20:53.440 --> 00:20:58.240
where we're not clear enough and that might need some more detail to actually go deeper into it.

s196
00:20:58.240 --> 00:21:01.840
So user disagreement is our uh first example.

s197
00:21:01.840 --> 00:21:05.200
Let's look at what also might be interesting.

s198
00:21:05.200 --> 00:21:11.679
And in our case, we now have one um where we're catching with a codebased evaluator if maybe there's some all caps used.

s199
00:21:11.679 --> 00:21:20.000
So if I think about how I'm interacting with cloud or codeex sometimes if I'm getting frustrated, I might just use all caps to get my point across.

s200
00:21:20.000 --> 00:21:22.720
And I think many of you have also done this in chat applications.

s201
00:21:22.720 --> 00:21:28.480
And we see especially in chat applications that this is actually a good way of catching some frustration from the user.

s202
00:21:28.480 --> 00:21:30.480
And um in this case it's all caps.

s203
00:21:30.480 --> 00:21:38.080
But as an alternative this could also be maybe cuss words or um yeah certain terms that you should maybe not usually use in a chat.

s204
00:21:38.080 --> 00:21:42.080
Um but when it's an agent you might tend to rather um go for it.

s205
00:21:42.080 --> 00:21:49.840
And uh for this we now want to set up a codebased evaluator so we can actually make sure uh we get that uh when he's frustrated.

s206
00:21:49.840 --> 00:21:59.039
So um as we can see here we now have a little um Python snippet here and if we go down a little bit um we can see uh that here the actual evaluator

s207
00:21:59.039 --> 00:22:04.559
function is defined and um it's targeting again a part of the JSON path.

s208
00:22:04.559 --> 00:22:14.480
So it's like extracting uh the certain user message and then it's checking if there's all caps in there and in case it's doing all caps then we return a um a boolean score true.

s209
00:22:14.480 --> 00:22:19.120
So let's copy this here so we can get this wired up.

s210
00:22:19.120 --> 00:22:24.159
And I'm only copying the bottom part because the upper part is actually static.

s211
00:22:24.159 --> 00:22:29.919
And I'm setting up an additional evaluator in this case now a codebased one.

s212
00:22:29.919 --> 00:22:32.240
And um we also support TypeScript code.

s213
00:22:32.240 --> 00:22:35.840
Uh so in case you're rather operating on TypeScript, this is possible.

s214
00:22:35.840 --> 00:22:39.280
But we now want to go for um Python code.

s215
00:22:39.280 --> 00:22:44.159
And um we call it all caps.

s216
00:22:44.480 --> 00:22:46.480
frustrated.

s217
00:22:46.799 --> 00:22:56.240
And since we're going for a boolean score, let's also add a little is um so we always know if it's true, then it's um he is actually frustrated uh with all caps.

s218
00:22:56.240 --> 00:23:13.440
And then I'm going to also down here replace this part um with our of course copying it again.

s219
00:23:22.720 --> 00:23:28.240
And of course down here you can um write all kinds of custom code um that you want to go for.

s220
00:23:28.240 --> 00:23:36.000
So whatever you want to evaluate if it's certain words you're looking for or uh for example also if you have some brand name that always should be written the right way.

s221
00:23:36.000 --> 00:23:38.880
So for example for length use also sometimes the f is written big.

s222
00:23:38.880 --> 00:23:40.080
We would usually write it small.

s223
00:23:40.080 --> 00:23:44.720
So you could also catch these kind of things to see where maybe things are going wrong.

s224
00:23:44.720 --> 00:23:46.559
And yeah let's get this set up.

s225
00:23:46.559 --> 00:23:50.320
So we have our second evaluator created.

s226
00:23:50.320 --> 00:23:55.039
But then of course we also need to think about what kind of observations do we want to target.

s227
00:23:55.039 --> 00:23:56.960
So let's look again here.

s228
00:23:56.960 --> 00:24:00.159
Um we say okay of course we want to run it on live observations.

s229
00:24:00.159 --> 00:24:02.400
So we want to know what's going on in production.

s230
00:24:02.400 --> 00:24:08.960
And we're again targeting the um the observation type agent.

s231
00:24:09.679 --> 00:24:16.720
And here I can select this going for the agent one again.

s232
00:24:16.720 --> 00:24:21.600
And here on the right side, we can also s that see that we can do a small little test run on it.

s233
00:24:21.600 --> 00:24:29.039
And here we see that in this case it's um returning a false because we're here only asking the questions about the turning on and off Bluetooth.

s234
00:24:29.039 --> 00:24:33.600
And as soon as we're happy with it, uh we can then uh click on execute.

s235
00:24:33.600 --> 00:24:42.880
Uh just as a short note here like right now we're running on observations but this of course is also possible to run on all kinds of experiments that you might might want to do in an offline

s236
00:24:42.880 --> 00:24:46.559
improvement process instead of running it on live production data.

s237
00:24:46.559 --> 00:24:52.320
So yeah right now we're happy with uh just running it on live observations and let's also click on execute here.

s238
00:24:52.320 --> 00:24:52.880
Nice.

s239
00:24:52.880 --> 00:24:55.279
Uh so let's check how this actually turned out.

s240
00:24:55.279 --> 00:24:58.159
Uh because we of course want to know if it also works.

s241
00:24:58.159 --> 00:25:04.400
And for this I am jumping back into our IT support agent.

s242
00:25:04.400 --> 00:25:10.000
Let's kick it off first with a how do I turn on Bluetooth again.

s243
00:25:10.880 --> 00:25:16.880
We're taking a second and then we're getting our replies.

s244
00:25:16.960 --> 00:25:17.919
Zooming in.

s245
00:25:17.919 --> 00:25:19.760
Yeah, I can't try.

s246
00:25:19.760 --> 00:25:21.039
It's also small for me.

s247
00:25:21.039 --> 00:25:26.640
I was earlier like I should have brought more glasses and yeah, now we can see it.

s248
00:25:26.640 --> 00:25:27.279
Perfect.

s249
00:25:27.279 --> 00:25:35.600
Um, so let's get frustrated this one first.

s250
00:25:35.600 --> 00:25:40.880
And of course, our agent is somehow dealing with it because we have some specifications in the system prompt.

s251
00:25:40.880 --> 00:25:44.880
And I might also um need of course the reply first.

s252
00:25:44.880 --> 00:25:46.000
So let's see.

s253
00:25:46.000 --> 00:25:51.520
We get some more details and then I can maybe ask.

s254
00:25:52.480 --> 00:26:15.440
No, there is no such Bluetooth sign just to disagree a little bit with the agent and he says that I'm right.

s255
00:26:15.440 --> 00:26:20.640
So let's see how this turns out in our traces.

s256
00:26:20.640 --> 00:26:28.480
Okay, we see we have a few more um now here coming in and let's check maybe on this one.

s257
00:26:28.480 --> 00:26:31.679
Okay, we also have some evaluators coming in already.

s258
00:26:31.679 --> 00:26:35.760
Let's dive in a bit deeper into what's actually going on here.

s259
00:26:35.760 --> 00:26:41.360
So uh we can see that this is now not the Yeah, we have our user message.

s260
00:26:41.360 --> 00:26:42.480
How do I turn on Bluetooth?

s261
00:26:42.480 --> 00:26:48.799
we get a reply and then we can see that this is the output we're getting back.

s262
00:26:48.799 --> 00:26:49.760
Okay, dad, no worries.

s263
00:26:49.760 --> 00:26:51.919
Let's try the other um way first.

s264
00:26:51.919 --> 00:26:56.880
So here we had some um disagreement going on and up here we can see actually this little score.

s265
00:26:56.880 --> 00:27:02.080
So on the top left is probably still small zooming in.

s266
00:27:02.080 --> 00:27:09.360
Yeah, we can see now that our root observation like the overall agent is actually tagged with a user disagreement um true.

s267
00:27:09.360 --> 00:27:19.360
Uh because here the the user actually said okay it's actually um in this case also all caps um is actually not true and it didn't work out.

s268
00:27:19.360 --> 00:27:23.200
And uh we can see this here now and have like this also signals.

s269
00:27:23.200 --> 00:27:38.240
So we could in theory now filter for the observations that are having a um maybe user disagreement true or also um all caps frustration uh true and like this find the ones that are actually interesting for us to work with.

s270
00:27:39.360 --> 00:27:45.760
Okay, so these are the ones where we get something out of the more or less um production behavior um of the people.

s271
00:27:45.760 --> 00:27:57.120
There's also one additional thing that is often interesting to monitor in a production application mainly being um if someone is trying to do something that is actually not supposed to be done um with the respective uh application.

s272
00:27:57.120 --> 00:27:59.200
So we call this out of scope request.

s273
00:27:59.200 --> 00:28:06.399
So in our assistant prompt um we are specifying here that the agent is supposed to help with all kinds of phone IT support questions.

s274
00:28:06.399 --> 00:28:09.440
Uh but of course dad also has different kinds of devices.

s275
00:28:09.440 --> 00:28:14.559
So maybe a printer also computer that he wants to um get uh get questions answered with.

s276
00:28:14.559 --> 00:28:18.960
But right now we're scoped um to this um specific scope.

s277
00:28:18.960 --> 00:28:28.799
So for this we can also um set up these kinds of evaluators and we are calling them in this case now out of scope request.

s278
00:28:28.799 --> 00:28:30.799
And here we have a different uh dynamic.

s279
00:28:30.799 --> 00:28:34.240
We're not looking at only the last user message or the message history.

s280
00:28:34.240 --> 00:28:36.960
Um but here we're actually also looking at the system prompt.

s281
00:28:36.960 --> 00:28:42.880
And because we are defining all of this via this JSON path, we have the possibility to target this also specifically.

s282
00:28:42.880 --> 00:28:48.320
So, um, one final time, let's get this evaluator, uh, set up.

s283
00:28:48.720 --> 00:28:53.039
And for this, we're jumping into our, um, evaluators page.

s284
00:28:53.039 --> 00:28:54.640
We can click on evaluators.

s285
00:28:54.640 --> 00:29:01.919
And once again, um, we can select our out of scope request template.

s286
00:29:02.799 --> 00:29:09.279
We now want to target um, generations because we also want to reference the system prompt.

s287
00:29:11.200 --> 00:29:13.919
And let's now map our variables again.

s288
00:29:13.919 --> 00:29:22.240
So we are taking the system prompt which in this case is the very first message that is sent in this interaction.

s289
00:29:29.200 --> 00:29:36.799
And then we're also taking our last user message that is coming up.

s290
00:29:39.360 --> 00:29:48.240
And in this prompt now we define okay we also wanna we want to always compare like what the system is actually supposed to do and what the agent is compared to do uh supposed to do

s291
00:29:48.240 --> 00:29:51.840
um to what the user is actually trying to do.

s292
00:29:51.840 --> 00:30:12.320
So this is now wired in not mapping the variables right now and let's execute this one as well.

s293
00:30:12.320 --> 00:30:12.640
Okay.

s294
00:30:12.640 --> 00:30:21.600
So whenever now dev would try to do something with the application that he's actually not supposed to and then we can know about this and then we can based on this information take an informed decision.

s295
00:30:21.600 --> 00:30:30.640
Do we either want to fix this so or expand the scope of our application or do we maybe have an issue in communicating to our users what they're actually supposed to do with our application

s296
00:30:30.640 --> 00:30:40.640
and in this case of course it's only the IT support but maybe in bigger scope application there's sometimes things people are trying to do and really misunderstand how the system is supposed to use

s297
00:30:40.640 --> 00:30:49.840
supposed to be used and like this we can uh yeah catch these kind of things um that are going on and go into more um detail here.

s298
00:30:49.840 --> 00:30:50.559
Perfect.

s299
00:30:50.559 --> 00:30:55.520
So this is how we can set up evaluators and monitors um for our application.

s300
00:30:55.520 --> 00:31:03.760
And I think what you can uh clearly see here is that this is not about um copying just what's out there and what can use.

s301
00:31:03.760 --> 00:31:13.279
And of course like many of these things you could now reuse for chat application but it's really about thinking what is what are the typical fail modes we will um endure in our kind of application

s302
00:31:13.279 --> 00:31:19.120
and not about like trying to copy paste u what's out there but really looking into what's going on.

s303
00:31:19.120 --> 00:31:30.480
So whenever we would want to work uh work with this now we could for example now go for um use like using these um scores now to also filter for certain

s304
00:31:30.480 --> 00:31:36.799
um for certain traces and results.

s305
00:31:36.880 --> 00:31:44.559
Here we have two and like this get like an insight on how often these things are actually going wrong.

s306
00:31:44.799 --> 00:31:45.519
Perfect.

s307
00:31:45.519 --> 00:31:56.159
So um like this we can wire up monitors and see u what's going on and um let's now seat some production production data in to see what is actually

s308
00:31:56.159 --> 00:32:00.320
um happening and how this performs in our app.

s309
00:32:00.320 --> 00:32:16.000
And for this uh we go down to the end of our um mon monitoring module and we have a script here to seat some um production information and like this have a bit more data to actually look at.

s310
00:32:16.000 --> 00:32:24.240
So I'm jumping back into my um repository and I'm copying our command.

s311
00:32:24.240 --> 00:32:29.039
And what you can imagine is happening in the background is we have just a set of traces as a data set.

s312
00:32:29.039 --> 00:32:41.519
And this information um also includes some scores and we are seeding this um into our um into our application into our language instance um directly.

s313
00:32:41.600 --> 00:32:46.960
So, let's see if it already is seated.

s314
00:32:57.360 --> 00:33:02.240
Taking out our score filter again.

s315
00:33:02.799 --> 00:33:06.559
And we can see now, yes, there's new things coming in.

s316
00:33:06.559 --> 00:33:13.279
We actually now have a lot of um a lot of data, multiple pages of traces.

s317
00:33:13.279 --> 00:33:23.120
And now the question is of course how can we work with this and how we can we find out um what is actually going wrong or also what is going um right in our application.

s318
00:33:23.120 --> 00:33:33.600
And um all of this data is usually then used to inform what you actually want to make sure that it's also working in an um offline uh improvement process.

s319
00:33:33.600 --> 00:33:43.919
So these production traces and especially the ones that you find that are interesting to look at h can then be used to build up a data set that you can use um in your offline improvement process

s320
00:33:43.919 --> 00:33:46.240
uh to to make a better application.

s321
00:33:46.240 --> 00:33:51.440
Um but today uh we want to focus on the uh monitoring part and the tracing part.

s322
00:33:51.440 --> 00:33:58.640
So let's see how we can maybe use our coding agent to find the larger scale issues that are in this application.

s323
00:33:58.640 --> 00:34:02.720
And for this um let's install the lenfuse skill together.

s324
00:34:02.720 --> 00:34:23.679
Uh so we can um use this and for this we have a little command up here in the setup step and I I usually go actually with just um telling my coding agent uh to install it for me but you can equally also just um install it separately.

s325
00:34:23.679 --> 00:34:52.960
So we have a CLI and we also have our skill and this can be installed then in my case now with claude trusting my application and what is now going on is that it's pulling our CLI and our CLI is exposing all APIs that we have so you can always access your traces your scores your data sets

s326
00:34:52.960 --> 00:35:03.280
experience ment runs but also start them through your um yeah command line directly or also with your coding agent together and like this work more effectively um with lenfuse

s327
00:35:03.280 --> 00:35:15.440
and this is actually also when you are trying to instrument your application in the first place and try to get your first traces and this is something you can uh use to automatically add tracing to your application and get your first insights

s328
00:35:15.440 --> 00:35:27.839
so here I'm now um installing language skill and uh just to see how it's done then running I'm going uh over to codex I already installed it in the background and what I now want to find out what is actually going wrong in my traces.

s329
00:35:27.839 --> 00:35:31.839
So what are the kind of information um steps that are going wrong?

s330
00:35:31.839 --> 00:35:34.400
What kind of um scores do we actually have in there?

s331
00:35:34.400 --> 00:35:58.400
And I'm asking my coding agent here now please use the length fuse skill to find issues in our traces and naturally cudex will take now some time uh but just to understand what's going on um cudex is now accessing

s332
00:35:58.400 --> 00:36:05.760
um the live traces that we have in our application so it's looking for what kind of questions are asked what kind of scores do we find in there?

s333
00:36:05.760 --> 00:36:14.800
Um what is maybe a lot of all caps frustration um going on there also where did the user disagree but also beyond what we officially set up.

s334
00:36:14.800 --> 00:36:24.880
So it's also checking on is there maybe issues with our latency is there issues with our tool calls um anything that has been going wrong in terms of what we didn't want to

s335
00:36:24.880 --> 00:36:40.000
um so to find where it's silently f failing in production h even though we don't want it um wanted to do this of course and since this is now going to um just run a little bit I think this is a good moment to also um wait for codeex and just take a few questions

s336
00:36:40.000 --> 00:36:50.960
um in case you have any okay let's see if the um if we're there yeah uh we actually got report and I could save this now as a markdown but I'm just going to um very briefly

s337
00:36:50.960 --> 00:37:00.960
show you the one that I prepared and I'm swearing I'm not p hacking and we can also look at the proper one but just wanted to show you how this looks like.

s338
00:37:00.960 --> 00:37:07.599
So what our coding agent now did is ran it ran over the um production traces that we we seated in there.

s339
00:37:07.599 --> 00:37:15.359
It was like around 120 traces or it was 70 traces and it actually looked at all the different dimensions where things could be going wrong.

s340
00:37:15.359 --> 00:37:24.320
And we get a quite um detailed report now um about like how many traces were there, how many observations did we have also how many of those were tool calls

s341
00:37:24.320 --> 00:37:34.960
and what we have in there and then it also generated us some findings and the first finding here is actually that um we have a silent retrieval quality failure and search help library.

s342
00:37:34.960 --> 00:37:54.160
So this is actually something we didn't evaluate for um but uh we also didn't realize before that our um lookup of information uh tool uh where we're looking like what kind of um question could what kind of answer could be the right answer that is actually um silently failing and not actually

s343
00:37:54.160 --> 00:37:55.520
uh doing what it's supposed to do.

s344
00:37:55.520 --> 00:37:57.440
So that's the first one we see.

s345
00:37:57.440 --> 00:38:02.000
Um but we also see that um the retrieval misses are increasing latency and cost.

s346
00:38:02.000 --> 00:38:15.920
So we have some issue in like tool retry calls um where we are detecting some kind of um problems and we can also see that there's um an evaluated coverage gap that uh is across like recent traffic

s347
00:38:15.920 --> 00:38:30.320
and um we we actually don't uh evaluate everything but we only get um signal from a subset of the traces um that are going on there and just briefly verifying um if Codex now also came up with the

s348
00:38:30.320 --> 00:38:30.800
same.

s349
00:38:30.800 --> 00:38:42.320
Okay, it also detected that the prompt version is um is off and then it also says about the latency text for the retries and we have the monitoring cover split.

s350
00:38:42.320 --> 00:38:42.560
Yeah.

s351
00:38:42.560 --> 00:38:51.760
And like this we can find um those issues in our application but we can then also make sure that we find a way to um to fix them and uh like this

s352
00:38:51.760 --> 00:39:00.400
um yeah improve the the application over time and make sure um that we yeah that we fix the issues that are actually coming up.

s353
00:39:00.400 --> 00:39:05.599
Okay, so we get the proposal here to save this.

s354
00:39:05.599 --> 00:39:10.960
Um, but actually let's see if we can get also a fix in here.

s355
00:39:10.960 --> 00:39:12.880
So what is the most interesting part?

s356
00:39:12.880 --> 00:39:21.680
Um, I actually want to get rid of the search help library one because if it's not really helping us to use those tools, then we can also um remove this.

s357
00:39:21.680 --> 00:39:37.280
Um, and for this we can also use our skill our search help library tool.

s358
00:39:37.280 --> 00:39:47.359
And because we now have the repository context, it actually can also help us um fix um issues in the code and not only um in our Langu setup.

s359
00:39:47.359 --> 00:39:56.320
And yeah, now it's getting um it's getting back to work and um fixing this uh for us in our application.

s360
00:39:56.320 --> 00:40:02.079
But let's now take a moment to zoom out and understand what actually needs to happen um with this.

s361
00:40:02.079 --> 00:40:07.520
And for this um we're coming back into our um slides here again.

s362
00:40:07.520 --> 00:40:13.280
Uh so what we now focused on today is actually the part of catching signals from our production traces.

s363
00:40:13.280 --> 00:40:19.359
So we initially set up our tracing and made sure that we have a step-by-step view into what is um actually going on.

s364
00:40:19.359 --> 00:40:20.880
What kind of tools is our agent using?

s365
00:40:20.880 --> 00:40:24.800
What's the inputs and outputs and h get a signal from this.

s366
00:40:24.800 --> 00:40:36.079
But what does need to happen with those kind of signals to make a continuously improving agent over time and uh for this like on the one hand we need to maintain our evaluation criteria.

s367
00:40:36.079 --> 00:40:43.839
So um we had this question for example on how can we make sure that it's like not overindexing on something um or like not really telling us anything.

s368
00:40:43.839 --> 00:40:51.920
And um yeah from this signal we need to make sure that over time we actually have a good um setup of our evaluators and get the information we need to get.

s369
00:40:51.920 --> 00:40:55.359
Um but most importantly we also need to maintain our data set.

s370
00:40:55.359 --> 00:41:01.359
So when you initially build an application um you might think you know what kind of things that people are trying to do.

s371
00:41:01.359 --> 00:41:08.240
Um but over time you will realize that there's a lot of edge cases that you did not account for and not cover um beforehand.

s372
00:41:08.240 --> 00:41:22.079
And that's why you need um to over time uh also maintain your data set um create good production examples um add them to your offline events and then make sure that you also can iterate and whenever you make a change also ship

s373
00:41:22.079 --> 00:41:31.920
um with confidence so you can make it part of your CI/CD and uh make sure that is um yeah actually not regressing whenever you introduce a change.

s374
00:41:33.200 --> 00:41:41.440
So um we today only covered the part of the AI engineering loop uh where we are looking into the online evaluations.

s375
00:41:41.440 --> 00:41:44.079
Uh but actually of course um there's more to it.

s376
00:41:44.079 --> 00:41:50.319
So it's all about uh making sure you know what's going on, you catch the interesting parts.

s377
00:41:50.319 --> 00:42:02.880
um if it's a user disagreement or all caps or if it's cussing um or also some kind of quality evaluations that you can do and out of this um you're building and especially also maintaining data sets over time.

s378
00:42:02.880 --> 00:42:04.400
So all of this is not a one-off.

s379
00:42:04.400 --> 00:42:13.760
You actually need to make sure that you are like putting in the reps and close the loop continuously to to also have then a data set that represents what your user users are doing.

s380
00:42:13.760 --> 00:42:19.440
And on those data sets you then want to um run experiments and evaluate those experiments.

s381
00:42:19.440 --> 00:42:31.920
Um so you always have a good uh indication of how good the quality of your of your app is and um whenever you deploy then also know um that that you can ship with confidence.

s382
00:42:31.920 --> 00:42:35.760
So this is the core part of our workshop.

s383
00:42:35.760 --> 00:42:37.920
I just wanted to briefly mention some things.

s384
00:42:37.920 --> 00:42:40.079
So today we only looked into that part.

s385
00:42:40.079 --> 00:42:48.720
Uh but on our website you can see in this length workshop that this is actually a eightstep um uh workshop where you are guided through every single module.

s386
00:42:48.720 --> 00:42:54.560
So, we can only touch upon this one and this one today, but there's also prompt management to Lenfuse.

s387
00:42:54.560 --> 00:42:56.720
You can set up those data sets.

s388
00:42:56.720 --> 00:43:04.560
And um this is a step-by-step walkthrough where you're guided through um uh all of these steps and like this uh can get it set up properly.

s389
00:43:04.560 --> 00:43:13.280
And um we of course also recommend to to uh to use the the skill um to get the setup together with your coding agent.

s390
00:43:13.280 --> 00:43:19.200
And if we uh were you and we would have to start tomorrow um this is exactly what we would uh be doing.

s391
00:43:19.200 --> 00:43:27.839
So we would take our repository um we would um use the length view skill to get up set up with tracing to really know what's happening in your application

s392
00:43:27.839 --> 00:43:29.359
um via this command.

s393
00:43:29.359 --> 00:43:45.359
We then would review traces manually and really understand uh what people are doing and then um yeah finally get into the motion of building out our traces and set um email setup for our production traces to then get also into the motion of um optimizing

s394
00:43:45.359 --> 00:43:47.280
offline.

s395
00:43:47.280 --> 00:43:49.119
All right, thank you so much everyone.

s396
00:43:49.119 --> 00:44:04.079
Um I'm taking a few more questions but thank you so much for participating and Mark is around lot is around I am around and we're super happy um to discuss anything with you and we also have a expo booth so please come around the next few days and uh chat with us

s397
00:44:20.640 --> 00:44:22.640
[music]