WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=eAXxdtNlK04 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/eAXxdtNlK04.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.560 --> 00:00:02.320 Hi everyone, my name is Annabelle. s2 00:00:02.320 --> 00:00:08.800 I'm a growth engineer at Lenfuse and we're the largest open source observability and evaluation platform for your AI system. s3 00:00:08.800 --> 00:00:19.680 And today I'm going to share about how you should stop burning your tokens and why you should start building in domain expertise early in into your loop design, but also into your overall um application design s4 00:00:19.680 --> 00:00:24.320 to make sure you're continuously improving and updating your application. s5 00:00:24.720 --> 00:00:29.439 It is June 2026 and the whole internet is about uh loops right now. s6 00:00:29.439 --> 00:00:32.480 So we have Boris Journey saying he doesn't write any prompts anymore. s7 00:00:32.480 --> 00:00:37.680 He has loops and Peter Steinberger being like you should be designing loops and not prompt your agents. s8 00:00:37.680 --> 00:00:43.120 And I mean the whole Kapathy auto research and auto improvement topic blew up earlier this year already. s9 00:00:43.120 --> 00:00:46.399 But all of them are coming a little bit from this de developer perspective. s10 00:00:46.399 --> 00:00:49.440 And um coding is one for for a good reason. s11 00:00:49.440 --> 00:00:59.760 One of the cases where this whole um automatically reaching a goal uh worked quite well because they've always had a at least one um target function that wasite quite clear. s12 00:00:59.760 --> 00:01:02.879 And this was uh does the code compile or not. s13 00:01:02.879 --> 00:01:08.960 And of course just because code compiles doesn't mean it's great and doesn't mean um that all the features are exactly how you wanted them. s14 00:01:08.960 --> 00:01:11.760 But you at least know that you shipped something that worked. s15 00:01:11.760 --> 00:01:15.760 And of course this was then expanded over time to to expand this target function. s16 00:01:15.760 --> 00:01:30.479 But essentially it's a does it work or does it not in all other fields and especially if you're building AI applications for some kind of domains like medicinal compliance or healthcare or all kinds of chatbots s17 00:01:30.479 --> 00:01:38.560 these target functions are uh not nearly as clear and a target that you give an agent is actually also always incomplete. s18 00:01:38.560 --> 00:01:49.920 So um you might initially think that you're trying to head down here uh but actually your optimal destination is up there and to get there and to understand this it takes quite some time and to figure this out s19 00:01:49.920 --> 00:02:00.640 and it's yeah it's just inherently a difficult problem especially if you are in a field where a clear yes or no like does the code compile um doesn't really uh cut it. s20 00:02:00.640 --> 00:02:11.280 So it is very difficult but at the same time we at length you see that the teams who are investing heavily here in the middle so uh making sure they capture what they actually want to work so the target function s21 00:02:11.280 --> 00:02:27.520 and build this out and make sure they have good evaluators that are evaluating this are the ones who manage to continuously upgrade and improve their application over time and also ship with confidence because if you know um it's working as you intended to h then you can also sleep um well at night in case you push a s22 00:02:27.520 --> 00:02:28.720 code change. s23 00:02:28.720 --> 00:02:36.959 Okay, so now we know it's super important and you should be doing it, but at the same kind it's time it's almost impossible to actually h get it right. s24 00:02:36.959 --> 00:02:45.840 Uh so we were setting out and wondering what's the clearest cut target function we can find um for an agent to use and also if you run an auto optimization s25 00:02:45.840 --> 00:02:57.360 on it h what can we learn about the role of target functions from from running it on it and the clearest cut target function we could find was a single label classification task that has has a very clearcut s26 00:02:57.360 --> 00:03:01.120 yes or no. So for example, let's say we have an item to categorize. s27 00:03:01.120 --> 00:03:02.000 There's a true label. s28 00:03:02.000 --> 00:03:03.599 For example, oh, it's an order. s29 00:03:03.599 --> 00:03:06.800 And then we have a set of available labels, order, complaint, inquiry. s30 00:03:06.800 --> 00:03:09.519 And our classifier then assigns um one label. s31 00:03:09.519 --> 00:03:13.840 And you can very clearly say is the true label equal to the predicted label. s32 00:03:13.840 --> 00:03:17.360 And if yes, um then this one is right and the next one might be wrong. s33 00:03:17.360 --> 00:03:23.360 And overall you can calculate an accuracy value and get a very clear signal from here. s34 00:03:23.360 --> 00:03:28.560 So we put this target function together with an agent and also with an optimizer. s35 00:03:28.560 --> 00:03:32.720 And how this minimal loop looked like I'm going to share with you now. s36 00:03:32.720 --> 00:03:35.920 So here's our minimal self-optimization loop. s37 00:03:35.920 --> 00:03:39.040 Up there in the first two rows we can see our target function. s38 00:03:39.040 --> 00:03:42.879 In our case we opted for classified um archive papers. s39 00:03:42.879 --> 00:03:55.120 So it's a set of papers that based on their primary um based on their title and abstract h got a primary label from the author um to categorize it for the other people in research. s40 00:03:55.120 --> 00:04:05.280 And um yeah, we're going to uh we have the the ground truth here and we have it in 200 um items in a fit data set, 100 in a validate data set, and 300 in a test data set. s41 00:04:05.280 --> 00:04:07.760 Um just to also make sure we're not overfitting. s42 00:04:07.760 --> 00:04:12.080 Then we have our agent, which is more or less just a a simple prompt. s43 00:04:12.080 --> 00:04:22.400 um uh based on GPD5 for nano because we wanted to see how a very small and cheap model performs on the auto improvement because the good ones actually got really really good but also very expensive s44 00:04:22.400 --> 00:04:30.400 um over time and we have this flash list of labels and then we have our optimization process that runs through uh cloud code leveraging cloud us 4.8 eight. s45 00:04:30.400 --> 00:04:40.479 So one of the frontier models and it proposes the prompt updates and has this context reference the GPT54 prompting guide as well as a task MD that is describing the loop. s46 00:04:40.479 --> 00:04:42.560 Looking a little bit closer on how this looks like. s47 00:04:42.560 --> 00:04:48.160 Here we can see our target function in this case especially our fit data set uh within length view. s48 00:04:48.160 --> 00:04:53.600 So we see an input column where the title and the abstracts are inside as well as an expected output column. s49 00:04:53.600 --> 00:05:05.360 So what kind of uh label should be applied and um on the those three different data sets the the idea is on the fit data set you run it you look for the errors and especially error clusters s50 00:05:05.360 --> 00:05:20.000 and um opus should then formulate hypothesis for prompt updates in the validation set we then check if those prompt updates actually also generalize to unseen data and then finally when we're like reaching a plateau or our stopping criteria s51 00:05:20.000 --> 00:05:28.240 um then we're running it on a test set to see how well we actually um generalize for untouched data that was not part of the training process. s52 00:05:28.240 --> 00:05:34.240 Our base prompt is a very flat list of labels and just a simple task classify this paper with a label. s53 00:05:34.240 --> 00:05:39.840 Um of course if uh if we would write the prompt we would probably add some pros how to think about it and all of this. s54 00:05:39.840 --> 00:05:48.880 But we just wanted to see what happens if we use the very base version um of this prompt and how the the system is dealing with it. s55 00:05:48.880 --> 00:05:55.840 And our leap loop is a step-by-step instructed uh loop um done through a task markdown file. s56 00:05:55.840 --> 00:06:02.400 Uh so the overall idea of the loop is first you run the base prompt on the fit and in the validate set to get like a baseline accuracy. s57 00:06:02.400 --> 00:06:07.039 You score it per item and overall then on the fit data set you do this error analysis. s58 00:06:07.039 --> 00:06:10.240 So you really look into what kind of categories frequently are done right? s59 00:06:10.240 --> 00:06:19.280 what kind of um item pairs are maybe often confused and also what's then the uh underlying abstract and title um belonging to it. s60 00:06:19.280 --> 00:06:21.199 So we can maybe find patterns there. s61 00:06:21.199 --> 00:06:34.800 Then the uh then float um proposes an update uh for the biggest error category and publishes a new prompt change and then reruns and only um accepts if this uh also improves on the validation set. s62 00:06:34.800 --> 00:06:36.160 And we have two stopping criteria. s63 00:06:36.160 --> 00:06:40.720 Either it's 15 runs uh completed or we have 92% accuracy reached. s64 00:06:40.720 --> 00:06:48.400 And as soon as this is happening or um it stops for some other reason, then we perform a final uh run on a test data set to see it. s65 00:06:48.400 --> 00:06:51.600 So yeah, here on the right you can see all of this is described in a markdown. s66 00:06:51.600 --> 00:06:55.840 We point at the data sets and we not even give it the test data set link. s67 00:06:55.840 --> 00:07:00.560 Um so it will not look into it before we wanted to. s68 00:07:00.560 --> 00:07:01.680 So what happened? s69 00:07:01.680 --> 00:07:12.880 Um yeah, I mean overall we can see here that there's a a tendency to increase uh but also we can see that um yeah we reached a 15% uptake uh from our baseline which is actually quite solid. s70 00:07:12.880 --> 00:07:25.759 Um the first run ended at uh 68% accuracy and on our third fourth iteration um we went all the way to 83% which then also afterwards um plateaued a bit. s71 00:07:25.759 --> 00:07:32.960 Overall it kept this level around 80% and we wondering okay actually we wanted the perfect clearcut target function. s72 00:07:32.960 --> 00:07:41.919 Uh but then looking deeper into the data and how those labels are chosen we realize okay there's um some creative freedom for the authors to choose what kind of label they want. s73 00:07:41.919 --> 00:07:52.319 Uh so even though description might make sense uh they might have chosen in that moment that they actually want to go for a different label and that's then also of course not recorded in here. s74 00:07:52.319 --> 00:08:00.879 And we also saw this uh improvement then generalized to the um test data set on yeah to 80.2%. s75 00:08:00.879 --> 00:08:14.080 Um so the generalization was also on the 300 item unseen data um quite good and uh in a way that you could say okay this actually worked and um the most interesting thing uh we thought is that the first iteration s76 00:08:14.080 --> 00:08:17.759 immediately gained 10% and then it was only a little bit movement. s77 00:08:17.759 --> 00:08:29.759 So uh it somehow got a lot of information from this very first run already and made a made a big uptick uh that we can can see here um up until the yeah very close to the final result also s78 00:08:29.759 --> 00:08:33.440 and we um wanted to look closer into what it actually did here. s79 00:08:33.440 --> 00:08:38.000 So here we can see on the left side the um flat label list prompt. s80 00:08:38.000 --> 00:08:45.360 So here on the left again a screenshot from our platform where we managed the prompts and on the right side we can see what happened. s81 00:08:45.360 --> 00:08:50.080 So the very first thing it um added is a general classification approach. s82 00:08:50.080 --> 00:08:56.880 So how should the model think about um the behavior of uh choosing one of these labels? s83 00:08:56.880 --> 00:09:00.720 So when what's what's the main goal of doing so? s84 00:09:00.720 --> 00:09:07.920 Then there's some information on how to decide between two very similar classes and then also some um related confused patterns. s85 00:09:07.920 --> 00:09:21.279 So it's uh basically telling it um when to choose which and under which conditions and how to go about it and to rather go for the more specific label than for the more broad label because there's some implicit testing in the label structure as well. s86 00:09:21.279 --> 00:09:30.240 And then it also added some um examples for the the item pairs that it um yeah frequently missed or frequently mclassified. s87 00:09:30.240 --> 00:09:32.000 So this is what uh we can see here. s88 00:09:32.000 --> 00:09:42.320 Uh so overall the approach was um improvement loop added rules and examples and I would have probably I was surprised I would have probably um spontaneously added descriptions to the labels. s89 00:09:42.320 --> 00:09:49.279 Uh but the prompting guide or the model together with prompting guide um decided that actually the right approach is here and I mean it worked. s90 00:09:49.279 --> 00:09:53.040 So uh I'm not going to going to interfere here. s91 00:09:53.040 --> 00:10:01.440 Um yeah what we wanted to dive deeper a bit more then is what actually happened in this one 10% uptick uh that we have. s92 00:10:01.440 --> 00:10:05.200 So let me remove myself a little bit because we need the bottom right corner. s93 00:10:05.200 --> 00:10:13.519 So this is the um the reasoning step that clude oppus 4.8 took uh in this one jump from 68% to 78%. s94 00:10:13.519 --> 00:10:18.640 And what we can see here is that it has a very clear okay we have 64 errors. s95 00:10:18.640 --> 00:10:20.480 These are the dominating patterns. s96 00:10:20.480 --> 00:10:24.480 These are the number one answer sync and this is the biggest confusion. s97 00:10:24.480 --> 00:10:30.560 So also what are the two labels that are most frequently confused and where there's like um issues coming up. s98 00:10:30.560 --> 00:10:36.720 And then it's forming a hypothesis and saying okay this loop will extract ex exactly address this pattern. s99 00:10:36.720 --> 00:10:46.959 So it got a very clearcut um clearly quantifiable and reliable failure mode because they were just wrong on a yes no right wrong basis. s100 00:10:46.959 --> 00:10:49.519 Um they also had it had a lot of data to look at. s101 00:10:49.519 --> 00:11:02.959 we had a 200 item um fit data set which probably in most real cases um yeah it takes some time to get there and to also like make sure that it was only 10 labels so each of them got covered like s102 00:11:02.959 --> 00:11:18.959 um enough times that that it can get a signal from it um yeah and like this it could form a datab hypothesis and ended up uh returning a 10% gain on the first uh run and then I mean it was a bit of movement and we also found a better version then uh but the biggest jump was s103 00:11:18.959 --> 00:11:26.800 uh just the very first one from a very clear-cut signal, we could have probably stopped there and already have very good baseline. s104 00:11:26.959 --> 00:11:38.240 So um given this right wrong high signal feedback works really really well on this classification test but also knowing that this is not um the case for all others like there's barely any deterministic s105 00:11:38.240 --> 00:11:47.519 yes no target functions in most cases you will run it this time and next time you get a different um next time you run the same evaluator you get a different s106 00:11:47.519 --> 00:11:59.040 answer from the same kind of evaluation you ran um just because yeah LLM as a judge is also the nondeterministic we were wondering how can we translate this right wrong high signal feedback into other applications. s107 00:11:59.040 --> 00:12:10.320 So how can we try to shape this in a way it also works for your um vertical domain expert AI agent or for your uh automation of requirements extraction in s108 00:12:10.320 --> 00:12:14.263 medicine device um compliance or whatever your use case might be. s109 00:12:14.263 --> 00:12:14.720 [snorts] s110 00:12:14.720 --> 00:12:27.519 And um for this um we overall see the approach working that um while the market for for a good reason likes those um evaluators like correctness or helpfulness or hallucination s111 00:12:27.519 --> 00:12:32.240 which like early in the days um were were a good um thing to go for. s112 00:12:32.240 --> 00:12:46.079 Um this is actually if you're trying to um autoimprove against this a rather low signal especially also because it's put on a often on a scale out of like between zero and one and 10 one and five and s113 00:12:46.079 --> 00:12:49.519 for this to really work properly you need to define each of those numbers. s114 00:12:49.519 --> 00:12:52.320 What does it mean in which context would you need which number? s115 00:12:52.320 --> 00:12:54.399 Uh which kind of criteria need to be met. s116 00:12:54.399 --> 00:12:56.399 Uh but this is most of the time also not done. s117 00:12:56.399 --> 00:13:08.399 So it's just choosing a number between zero and one and um depending on the uh context it might just uh totally change perspective and is therefore rather low signal and also probably inconsistent s118 00:13:08.399 --> 00:13:09.839 um across runs. s119 00:13:09.839 --> 00:13:18.240 What we see uh working instead is uh really looking into um what are the quality criteria that you want to work uh for your application. s120 00:13:18.240 --> 00:13:20.160 So what does good mean? s121 00:13:20.160 --> 00:13:38.480 Uh so for example instead of correctness the answer is based on the knowledge base yes no. So for example, if there's a snippet about internal information in the answer and you can check is it actually also in the ret retrieved context we did in the previous step or also if you're frequently struggling with cor using your correct s122 00:13:38.480 --> 00:13:43.040 brand voice you could look into okay um is it this time correct? s123 00:13:43.040 --> 00:13:45.680 Did we make sure that our name was written correctly? s124 00:13:45.680 --> 00:13:52.160 I've also seen companies that have English names that are checking for did we not accidentally translate our name to Spanish? s125 00:13:52.160 --> 00:13:53.680 things like this. s126 00:13:53.680 --> 00:14:01.920 And also um uh one thing that works well because it's also this categorization yes no is maybe some known failure modes. s127 00:14:01.920 --> 00:14:08.720 So which out of these five types of failures uh happened here and um can you categorize it? s128 00:14:08.720 --> 00:14:10.399 And these are only a few examples. s129 00:14:10.399 --> 00:14:23.040 Um we usually see that these are created by looking into the data, understanding what good means, what are things that are important to you and especially to your domain experts and creating these high signal um feedback loops. s130 00:14:23.040 --> 00:14:31.199 Um because if you can't do code compiles, you need to wrap your head a little bit differently around um what is good and what is not. s131 00:14:31.199 --> 00:14:33.279 So um this is the one side. s132 00:14:33.279 --> 00:14:35.839 The other part is of course um the volume that you need. s133 00:14:35.839 --> 00:14:39.600 We had a like luckily we could have used thousands of examples. s134 00:14:39.600 --> 00:14:41.199 there's so many papers out there. s135 00:14:41.199 --> 00:14:54.160 We opted for like 200 train 100 validate um 300 test um just because we wanted to see that it properly generalizes um but we made the experience that this already got gave back a lot of um high signal feedback s136 00:14:54.160 --> 00:15:01.680 um so depending on the complexity of your application and the steps in there of course it might differ um but having the volume there and having the high signal feedback s137 00:15:01.680 --> 00:15:03.760 um oops are the foundation. s138 00:15:03.760 --> 00:15:05.440 So how do you get there? s139 00:15:05.440 --> 00:15:14.000 So the most important thing we see is working with your experts and this is like repeated advice and everyone's saying it and everyone's saying talk to your customers. s140 00:15:14.000 --> 00:15:21.519 Um but here it's something you can actually start to encode the domain expertise into very concrete examples. s141 00:15:21.519 --> 00:15:24.560 So use them to create examples what should come out of it. s142 00:15:24.560 --> 00:15:31.199 Look at those samples run sample runs together with them and also ask okay but why is it like this here and here um the other way? s143 00:15:31.199 --> 00:15:43.360 Uh because like this you can often also get information out of them that they implicitly think is clear and also um really understand um how the decisions are taken there and like this identify failure modes of your application s144 00:15:43.360 --> 00:15:50.800 and also define what good means because you probably need both and then use them to find these high signal evaluators. s145 00:15:50.800 --> 00:15:59.839 Then um as soon as you have this baseline and um you might hit production um review this data and don't review it only with your coding agents but review it as a human. s146 00:15:59.839 --> 00:16:08.480 look through it, understand like what is the the scope initially and um uh what is then being done with this application. s147 00:16:08.480 --> 00:16:09.680 This can shift over time. s148 00:16:09.680 --> 00:16:11.040 The failure modes can change. s149 00:16:11.040 --> 00:16:26.000 Um people might try to do other things and it's potential for feature um expansion and to really understand how the system works and fails in production um is necessary to to cover the typical failure modes and catch them uh before your customers do. s150 00:16:26.000 --> 00:16:31.519 And then um finally um think about the whole thing as a system that generalizes. s151 00:16:31.519 --> 00:16:38.320 So you're trying to build up examples that are representative of what you might want to hit uh in production at some point. s152 00:16:38.320 --> 00:16:44.160 And um for this to properly generalize, you need to uh bake in mechanisms for validation. s153 00:16:44.160 --> 00:16:49.360 Uh so like in traditional machine learning context, um the validation approach is actually quite typical. s154 00:16:49.360 --> 00:16:55.040 So people um all the time make sure that they're actually validating if uh things don't overfitit. s155 00:16:55.040 --> 00:17:08.559 And um this combined with like a real mechanism combined with like the instructions in your loop and giving the system an escape hatch instead of having it work for hours and hours uh hitting a wall and uh burning tokens. s156 00:17:08.559 --> 00:17:23.919 Um yeah are very important divas um to actually uh make sure that you're not um just burning tokens but actually looping towards um a system where a human and your agent together collaborate on an improving system. s157 00:17:23.919 --> 00:17:24.319 Perfect. s158 00:17:24.319 --> 00:17:26.559 Thank you so much for uh listening to me. s159 00:17:26.559 --> 00:17:28.799 Um very excited that you took the time. s160 00:17:28.799 --> 00:17:33.520 And if you want to know more about Lenfuse and also how we think about these topics, you can find more on length.com. s161 00:17:33.520 --> 00:17:38.160 And I'm super excited to meet some of you at some point in person.