WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=4loPnxvWWhg NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/4loPnxvWWhg.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:14.000 --> 00:00:14.559 All right. s2 00:00:14.559 --> 00:00:16.240 Hello everybody. s3 00:00:16.240 --> 00:00:17.279 Thank you for coming. s4 00:00:17.279 --> 00:00:18.720 I'm Dan Bujorn. s5 00:00:18.720 --> 00:00:21.840 I'm a senior data scientist at Lease End. s6 00:00:21.840 --> 00:00:22.720 Lease End. s7 00:00:22.720 --> 00:00:32.880 We connect people who are coming to the end of their auto lease with financing uh options so that they can buy out their lease and keep their car. s8 00:00:32.880 --> 00:00:47.120 Now, as part of this uh we uh built a an LLM based application in late 2024 to help our customers connect with uh with our sales team. s9 00:00:47.120 --> 00:00:51.520 This application allowed them to send messages through text. s10 00:00:51.520 --> 00:00:58.960 uh they could ask questions about the sales process, they could schedule calls, uh they could get reminders, all of this stuff. s11 00:00:58.960 --> 00:01:12.400 Our first solution used a workflowbased approach uh built on top of uh a rag system where we searched a vector database of messages that we had already seen and classified s12 00:01:12.400 --> 00:01:15.200 with the customer's intent. s13 00:01:15.200 --> 00:01:26.240 So for example, uh a message saying call me tomorrow would be classified as the customer wants to uh wants to talk later. s14 00:01:26.240 --> 00:01:33.040 A message saying I've got time now would be classified as uh the customer wants to talk right now. s15 00:01:33.040 --> 00:01:36.960 U this has worked but not super amazing. s16 00:01:36.960 --> 00:01:48.720 there's a lot of nuance in in messages and and conversation and this rags approach just couldn't quite pick up on that nuance and so we started to look for s17 00:01:48.720 --> 00:02:01.600 uh for new options to improve this and naturally being a data scientist my first thought was hey let's start fine-tuning this seemed like a fun thing to do and I was sure that this was the right call s18 00:02:01.600 --> 00:02:14.800 um there's a few reasons for that first of all uh we needed better accuracy Our entire system uh was built upon us getting the user's intent correct. s19 00:02:14.800 --> 00:02:16.239 Did they want to talk now? s20 00:02:16.239 --> 00:02:17.599 Do they want to schedule a call? s21 00:02:17.599 --> 00:02:19.440 Do they want to opt out? s22 00:02:19.440 --> 00:02:21.360 All of this hinged on that decision. s23 00:02:21.360 --> 00:02:25.680 And so we needed to make sure that we got that first and foremost. s24 00:02:25.680 --> 00:02:30.000 Next, uh we could use smaller models with fine-tuning. s25 00:02:30.000 --> 00:02:34.080 And so this would lower the cost and also lower latency. s26 00:02:34.080 --> 00:02:46.319 So this was really important for us because we were uh responding to thousands of messages a day in real time and so it it uh would help us scale a lot. s27 00:02:46.319 --> 00:02:58.720 Then next like I said we were classifying the intent of the user and so this was a very narrow structured task that we were trying to do and so it lent itself very nicely to supervised fine-tuning. s28 00:02:58.720 --> 00:03:07.599 uh we would bucket uh that conversation in one of six different categories and the model would learn the differences between those. s29 00:03:07.599 --> 00:03:10.720 Uh so seemed like a great option there. s30 00:03:10.720 --> 00:03:20.080 Lastly, uh I believe that this would help us have a little bit more control over our destiny with the the model providers. s31 00:03:20.080 --> 00:03:31.360 The idea was that we had the data and all we would need to do is pass that into a new model, go through the fine-tuning process and we could get s32 00:03:31.360 --> 00:03:35.280 similar results no matter uh what we decided to use. s33 00:03:35.280 --> 00:03:38.159 So we could be model agnostic. s34 00:03:38.159 --> 00:03:41.200 So this was the approach that we took. s35 00:03:41.200 --> 00:03:51.120 Um, and I built a pipeline to collect examples, run LLM as judge, uh, classifications to label our data. s36 00:03:51.120 --> 00:03:58.480 I'd manually review that, create holdout sets, go through the fine-tuning process, check my metrics. s37 00:03:58.480 --> 00:04:00.799 This was a data scientist's dream. s38 00:04:00.799 --> 00:04:03.519 And, uh, the numbers sure helped. s39 00:04:03.519 --> 00:04:11.040 Within a year, this application had helped us bring in $12 million of revenue at a 50x ROI. s40 00:04:11.040 --> 00:04:20.400 Um, it was pretty awesome, but uh, the whole time it was quietly accumulating debt underneath that we didn't see. s41 00:04:20.400 --> 00:04:27.520 So, I want to show a couple examples of how this application could get things wrong. s42 00:04:27.520 --> 00:04:44.160 Uh first of all uh the confused confirmer is a situation where um when customers set up an appointment with a sales rep, we send them a confirmation message to let them know that it's been scheduled and give them the details of that. s43 00:04:44.160 --> 00:04:48.400 Uh so a conversation may look like this. s44 00:04:48.400 --> 00:04:56.800 We reach out and say, "Hi Tracy, just confirming your lease and call with your advisor is set for Thursday at 2 p.m. We'll call you then." s45 00:04:56.800 --> 00:05:01.040 Tracy then sends us a message back saying, "Sounds good." s46 00:05:01.040 --> 00:05:04.320 And then our LLM responds with, "Great. s47 00:05:04.320 --> 00:05:06.080 I'm calling you right now. s48 00:05:06.080 --> 00:05:07.919 Uh, it's not what we want. s49 00:05:07.919 --> 00:05:12.080 We just confirmed a an appointment for a following day." s50 00:05:12.080 --> 00:05:14.240 And then all of a sudden, we start calling them. s51 00:05:14.240 --> 00:05:19.680 This led to frustrated customers and some missed opportunities. s52 00:05:19.680 --> 00:05:24.800 The next one, um, I've come to lovingly call the overeager puppy. s53 00:05:24.800 --> 00:05:29.199 Um the the conversation looks like this. s54 00:05:29.199 --> 00:05:36.240 So first, hi James, this is Alex with Lease End reaching out about your upcoming lease maturity. s55 00:05:36.240 --> 00:05:38.720 James then says, hi, good morning. s56 00:05:38.720 --> 00:05:41.600 And good morning. s57 00:05:41.600 --> 00:05:43.199 I'm giving you a call. s58 00:05:43.199 --> 00:05:53.120 Um just like a puppy that gets so excited that somebody's giving it attention, our model decided to to give a call right there. s59 00:05:53.120 --> 00:05:56.320 Um, obviously this is not what James wanted. s60 00:05:56.320 --> 00:05:58.560 This actually did happen in production. s61 00:05:58.560 --> 00:06:01.520 Um, very embarrassing there. s62 00:06:01.520 --> 00:06:06.639 Uh, but this is a these are a couple examples of of where it went wrong. s63 00:06:06.639 --> 00:06:09.759 And and don't get me wrong, the the app did well. s64 00:06:09.759 --> 00:06:17.440 The revenue numbers show that that it was working, but it could also mess up pretty spectacularly. s65 00:06:17.440 --> 00:06:24.880 Um the big issue wasn't how to fix it, but how to make it the fix manageable. s66 00:06:24.880 --> 00:06:28.560 The the fine-tuning process was pretty complex. s67 00:06:28.560 --> 00:06:33.840 Uh first we needed to gather examples of the problems that we started to see. s68 00:06:33.840 --> 00:06:43.440 Um then we needed to ask ourselves uh do we have enough examples for uh to go through fine-tuning? s69 00:06:43.440 --> 00:06:46.639 If not, we synthesized those examples. s70 00:06:46.639 --> 00:06:51.759 uh we passed it through an LLM, it created some some possible examples there. s71 00:06:51.759 --> 00:06:59.360 We'd have to validate those, which was a very manual process uh because we wanted to make sure it had the best training data possible. s72 00:06:59.360 --> 00:07:09.599 And then once we had enough, uh we labeled those with the the categorization bins and we validated validated those through a manual review. s73 00:07:09.599 --> 00:07:14.160 Surprisingly, the fine-tuning process was the shortest part of all of this. s74 00:07:14.160 --> 00:07:23.520 uh normally took about an hour depending on the size of the data that we had but uh we never got it on the first iteration. s75 00:07:23.520 --> 00:07:36.960 Uh normally what happened was we would uh we would fine-tune and we'd evaluate this and uh we fix the problem that we were just trying to solve but then we caused regressions and other things. s76 00:07:36.960 --> 00:07:47.199 And so this turned into kind of a whack-a-mole process where we would solve something new, but then other old issues kept popping up that we had to to whack down. s77 00:07:47.199 --> 00:07:56.479 Um, this whole process took about a week to gather the data, label everything, go through the fine-tuning process, and iterate and then deploy. s78 00:07:56.479 --> 00:07:58.319 So it was costly. s79 00:07:58.319 --> 00:08:04.639 Um, therefore, we needed to triage all of these issues that we ran into. s80 00:08:04.639 --> 00:08:09.520 We asked ourselves three questions before we did any any retraining. s81 00:08:09.520 --> 00:08:12.080 How frequent is the issue? s82 00:08:12.080 --> 00:08:14.960 Is it something that customers are seeing every day? s83 00:08:14.960 --> 00:08:16.560 Is it one off? s84 00:08:16.560 --> 00:08:23.599 Um, one big exception to this was if it was hurting the customer experience too much. s85 00:08:23.599 --> 00:08:36.479 So for example of this would be uh somebody repeatedly stating what their uh their preference for a call time is and then the uh the model ignoring that. s86 00:08:36.479 --> 00:08:40.560 Another one would be a customer scheduling a call. s87 00:08:40.560 --> 00:08:51.200 We tell them that we've scheduled it for them but we don't return the payload in in the proper way and so the the call never gets scheduled and so we don't follow up with them. s88 00:08:51.200 --> 00:08:53.920 So these kinds of things needed to be fixed right away. s89 00:08:53.920 --> 00:09:02.320 But before we did that, uh we asked the last question, is there any thing that we can do in order to prevent a retrain? s90 00:09:02.320 --> 00:09:11.040 Can we have some kind of a band-aid fix to get out there so we don't have to go through a whole week-long process uh for one or two issues? s91 00:09:11.040 --> 00:09:19.120 And so we we ranked our own bugs uh based on how much customer pain we could tolerate at the moment. s92 00:09:19.120 --> 00:09:23.519 Um, so not a great situation to be in with a production system. s93 00:09:23.519 --> 00:09:27.920 This led to what I've come to call the calcification tax. s94 00:09:27.920 --> 00:09:32.880 Uh, the more we used the model, the more rigid everything became. s95 00:09:32.880 --> 00:09:35.279 This manifested in a couple different ways. s96 00:09:35.279 --> 00:09:38.480 First, we were locked into our model. s97 00:09:38.480 --> 00:09:45.680 You remember when I said that uh fine-tuning would give us uh more freedom in what model we did? s98 00:09:45.680 --> 00:09:47.519 That was not the case. s99 00:09:47.519 --> 00:09:58.560 Um within providers there's nuance between one model version to another and so that changes the the training data that you need to provide it. s100 00:09:58.560 --> 00:10:06.080 Um across model provisors uh it's extremely different the structure of the data you need to pass to it be different. s101 00:10:06.080 --> 00:10:13.200 The amount of the training data to get good results the way to interact with the training interface. s102 00:10:13.200 --> 00:10:18.959 All of this caused a lot of complexity and so it was just too costly for us to switch. s103 00:10:18.959 --> 00:10:30.959 And so um to we kept it the same model for consistency because we already had a lot to do with uh with each retraining process and we couldn't afford s104 00:10:30.959 --> 00:10:34.480 to uh upgrade the model. s105 00:10:34.560 --> 00:10:40.800 So uh the other way that this locked in was architecture. s106 00:10:40.800 --> 00:10:57.279 We built this app in uh late 2024 when workflows were kind of the um gold standard if you wanted good u production results and uh the AI world moves very fast s107 00:10:57.279 --> 00:11:08.160 and we couldn't adapt to that because we were so locked into this just trying to keep it running and we couldn't take advantage of the new architectures um and s108 00:11:08.160 --> 00:11:15.200 and improve performance that Okay, so earlier this year I had an aha moment. s109 00:11:15.200 --> 00:11:27.440 Um, we started using cloud code for our coding tasks and I noticed that we never needed to change the model depending on what task we're using. s110 00:11:27.440 --> 00:11:33.279 Um, we just changed the skill, the resources that we passed it, the context. s111 00:11:33.279 --> 00:11:37.920 Um, you drop in the better better context, you get better results. s112 00:11:37.920 --> 00:11:43.040 And I thought, why can't we do this with our messaging app? s113 00:11:43.040 --> 00:11:49.040 Um, this was obviously difficult for me to admit because I was the champion for fine-tuning. s114 00:11:49.040 --> 00:11:55.839 And, uh, luckily we were able to piggyback on a project that was already happening. s115 00:11:55.839 --> 00:12:07.279 Um, and so we migrated our workflow approach to a series of skills, tools, and resources that the skills could could load into or load up. s116 00:12:07.279 --> 00:12:09.360 um and get that context. s117 00:12:09.360 --> 00:12:18.800 And so we pushed this as one of our first production tests of our our new agentic framework that that was being built already. s118 00:12:18.800 --> 00:12:24.320 Now uh I want to compare the process before and after our rebuild. s119 00:12:24.320 --> 00:12:35.440 Uh before we already went through the kind of the training cycle, but there was this triage cycle beforehand where we needed to make sure that we had reached a critical mass s120 00:12:35.440 --> 00:12:43.839 of problems before we would even attempt to uh fine-tune again to improve everything. s121 00:12:43.839 --> 00:12:48.480 Like I said, this took about a week, so it was a long process, costly. s122 00:12:48.480 --> 00:13:00.079 Uh after the rebuild um it was a simple process of you find a problem you adjust the simp system prompt or the skill that was affected. s123 00:13:00.079 --> 00:13:06.639 We validated performance on a curated set that we have been collecting over the time that this was in production. s124 00:13:06.639 --> 00:13:14.639 We iterate a few times and then we deploy that simply by uploading MD files to an S3 bucket. s125 00:13:14.639 --> 00:13:22.800 um this whole process from discovering a problem to deploying the fix we reduced down to less than an hour. s126 00:13:22.800 --> 00:13:34.399 So it extremely improved all of this and we could be far more reactive give our our customers way better performance or better experience there. s127 00:13:34.480 --> 00:13:38.399 Now I'll be honest it did cost us a little bit more per message. s128 00:13:38.399 --> 00:13:40.399 we were using better models. s129 00:13:40.399 --> 00:13:48.800 Um, so the API costs were a little higher, but accuracy went way up. s130 00:13:48.800 --> 00:13:53.440 I said before that accuracy was the key to to getting all of this right. s131 00:13:53.440 --> 00:13:55.120 Um, and we did that. s132 00:13:55.120 --> 00:14:00.320 Accuracy uh was far better with this than it ever was with fine-tuning. s133 00:14:00.320 --> 00:14:09.120 Um, next, like I said, we reduced our uh our fixed process from days down to minutes. s134 00:14:09.120 --> 00:14:17.760 Next, we were able to unfreeze our model and finally get that freedom from a vendor that we never had with fine-tuning. s135 00:14:17.760 --> 00:14:22.079 Um, our agentic framework was built model agnostic. s136 00:14:22.079 --> 00:14:27.760 So, we could use open AI, we can use uh anthropic, we can use any other model that we want. s137 00:14:27.760 --> 00:14:32.079 The important part is the context that we're providing to that model. s138 00:14:32.079 --> 00:14:45.600 And then lastly, while it cost us a little more per message, the total cost went down because we were spending far less time trying to keep it up and running and s139 00:14:45.600 --> 00:14:49.680 fine-tuning to uh to keep it working properly. s140 00:14:49.680 --> 00:14:56.959 So before you fine-tune, I'd ask you, can you cross your reason off of this list? s141 00:14:56.959 --> 00:15:00.079 So, I thought we would get better accuracy. s142 00:15:00.079 --> 00:15:04.880 Uh, the rebuild beat the fine-tuned model. s143 00:15:04.880 --> 00:15:07.920 I thought we would get lower cost of the volume we were doing. s144 00:15:07.920 --> 00:15:10.959 Um, I was looking at the wrong costs. s145 00:15:10.959 --> 00:15:16.720 We we paid more per message, but the total cost ended up going down with our rebuild, lower latency. s146 00:15:16.720 --> 00:15:26.160 We we did see marginal gains on these smaller models, but they were so small that in practice it really didn't make any difference. s147 00:15:26.160 --> 00:15:30.320 And then maybe you've got a narrow or structured task. s148 00:15:30.320 --> 00:15:34.880 Our textbook case still became tech debt. s149 00:15:35.279 --> 00:15:37.279 And lastly, vendor control. s150 00:15:37.279 --> 00:15:40.720 It's not as simple as just plugging the data in. s151 00:15:40.720 --> 00:15:49.759 The other two situations where you you might have privacy and data control or you need some offline offline solution. s152 00:15:49.759 --> 00:15:56.240 I would say this these are the situations where a fine-tuned model may be useful. s153 00:15:56.240 --> 00:15:57.839 But you need to be cautious. s154 00:15:57.839 --> 00:16:00.000 There are other solutions out there. s155 00:16:00.000 --> 00:16:07.519 Um but um you need to make sure that it's not uh not causing issues in the long run. s156 00:16:07.519 --> 00:16:18.639 So finally fine-tune only when you literally cannot call a frontier model and even then your decision still has to beat the the tax. s157 00:16:18.639 --> 00:16:20.800 Thank you.