Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards — Dan Bjornn, Lease End

AI Engineer · 16 min · 157 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:14All right.
00:14Hello everybody.
00:16Thank you for coming.
00:17I'm Dan Bujorn.
00:18I'm a senior data scientist at Lease End.
00:21Lease End.
00:22We connect people who are coming to the end of their auto lease with financing uh options so that they can buy out their lease and keep their car.
00:32Now, as part of this uh we uh built a an LLM based application in late 2024 to help our customers connect with uh with our sales team.
00:47This application allowed them to send messages through text.
00:51uh they could ask questions about the sales process, they could schedule calls, uh they could get reminders, all of this stuff.
00:58Our first solution used a workflowbased approach uh built on top of uh a rag system where we searched a vector database of messages that we had already seen and classified
01:12with the customer's intent.
01:15So for example, uh a message saying call me tomorrow would be classified as the customer wants to uh wants to talk later.
01:26A message saying I've got time now would be classified as uh the customer wants to talk right now.
01:33U this has worked but not super amazing.
01:36there's a lot of nuance in in messages and and conversation and this rags approach just couldn't quite pick up on that nuance and so we started to look for
01:48uh for new options to improve this and naturally being a data scientist my first thought was hey let's start fine-tuning this seemed like a fun thing to do and I was sure that this was the right call
02:01um there's a few reasons for that first of all uh we needed better accuracy Our entire system uh was built upon us getting the user's intent correct.
02:14Did they want to talk now?
02:16Do they want to schedule a call?
02:17Do they want to opt out?
02:19All of this hinged on that decision.
02:21And so we needed to make sure that we got that first and foremost.
02:25Next, uh we could use smaller models with fine-tuning.
02:30And so this would lower the cost and also lower latency.
02:34So this was really important for us because we were uh responding to thousands of messages a day in real time and so it it uh would help us scale a lot.
02:46Then next like I said we were classifying the intent of the user and so this was a very narrow structured task that we were trying to do and so it lent itself very nicely to supervised fine-tuning.
02:58uh we would bucket uh that conversation in one of six different categories and the model would learn the differences between those.
03:07Uh so seemed like a great option there.
03:10Lastly, uh I believe that this would help us have a little bit more control over our destiny with the the model providers.
03:20The idea was that we had the data and all we would need to do is pass that into a new model, go through the fine-tuning process and we could get
03:31similar results no matter uh what we decided to use.
03:35So we could be model agnostic.
03:38So this was the approach that we took.
03:41Um, and I built a pipeline to collect examples, run LLM as judge, uh, classifications to label our data.
03:51I'd manually review that, create holdout sets, go through the fine-tuning process, check my metrics.
03:58This was a data scientist's dream.
04:00And, uh, the numbers sure helped.
04:03Within a year, this application had helped us bring in $12 million of revenue at a 50x ROI.
04:11Um, it was pretty awesome, but uh, the whole time it was quietly accumulating debt underneath that we didn't see.
04:20So, I want to show a couple examples of how this application could get things wrong.
04:27Uh first of all uh the confused confirmer is a situation where um when customers set up an appointment with a sales rep, we send them a confirmation message to let them know that it's been scheduled and give them the details of that.
04:44Uh so a conversation may look like this.
04:48We reach out and say, "Hi Tracy, just confirming your lease and call with your advisor is set for Thursday at 2 p.m. We'll call you then."
04:56Tracy then sends us a message back saying, "Sounds good."
05:01And then our LLM responds with, "Great.
05:04I'm calling you right now.
05:06Uh, it's not what we want.
05:07We just confirmed a an appointment for a following day."
05:12And then all of a sudden, we start calling them.
05:14This led to frustrated customers and some missed opportunities.
05:19The next one, um, I've come to lovingly call the overeager puppy.
05:24Um the the conversation looks like this.
05:29So first, hi James, this is Alex with Lease End reaching out about your upcoming lease maturity.
05:36James then says, hi, good morning.
05:38And good morning.
05:41I'm giving you a call.
05:43Um just like a puppy that gets so excited that somebody's giving it attention, our model decided to to give a call right there.
05:53Um, obviously this is not what James wanted.
05:56This actually did happen in production.
05:58Um, very embarrassing there.
06:01Uh, but this is a these are a couple examples of of where it went wrong.
06:06And and don't get me wrong, the the app did well.
06:09The revenue numbers show that that it was working, but it could also mess up pretty spectacularly.
06:17Um the big issue wasn't how to fix it, but how to make it the fix manageable.
06:24The the fine-tuning process was pretty complex.
06:28Uh first we needed to gather examples of the problems that we started to see.
06:33Um then we needed to ask ourselves uh do we have enough examples for uh to go through fine-tuning?
06:43If not, we synthesized those examples.
06:46uh we passed it through an LLM, it created some some possible examples there.
06:51We'd have to validate those, which was a very manual process uh because we wanted to make sure it had the best training data possible.
06:59And then once we had enough, uh we labeled those with the the categorization bins and we validated validated those through a manual review.
07:09Surprisingly, the fine-tuning process was the shortest part of all of this.
07:14uh normally took about an hour depending on the size of the data that we had but uh we never got it on the first iteration.
07:23Uh normally what happened was we would uh we would fine-tune and we'd evaluate this and uh we fix the problem that we were just trying to solve but then we caused regressions and other things.
07:36And so this turned into kind of a whack-a-mole process where we would solve something new, but then other old issues kept popping up that we had to to whack down.
07:47Um, this whole process took about a week to gather the data, label everything, go through the fine-tuning process, and iterate and then deploy.
07:56So it was costly.
07:58Um, therefore, we needed to triage all of these issues that we ran into.
08:04We asked ourselves three questions before we did any any retraining.
08:09How frequent is the issue?
08:12Is it something that customers are seeing every day?
08:14Is it one off?
08:16Um, one big exception to this was if it was hurting the customer experience too much.
08:23So for example of this would be uh somebody repeatedly stating what their uh their preference for a call time is and then the uh the model ignoring that.
08:36Another one would be a customer scheduling a call.
08:40We tell them that we've scheduled it for them but we don't return the payload in in the proper way and so the the call never gets scheduled and so we don't follow up with them.
08:51So these kinds of things needed to be fixed right away.
08:53But before we did that, uh we asked the last question, is there any thing that we can do in order to prevent a retrain?
09:02Can we have some kind of a band-aid fix to get out there so we don't have to go through a whole week-long process uh for one or two issues?
09:11And so we we ranked our own bugs uh based on how much customer pain we could tolerate at the moment.
09:19Um, so not a great situation to be in with a production system.
09:23This led to what I've come to call the calcification tax.
09:27Uh, the more we used the model, the more rigid everything became.
09:32This manifested in a couple different ways.
09:35First, we were locked into our model.
09:38You remember when I said that uh fine-tuning would give us uh more freedom in what model we did?
09:45That was not the case.
09:47Um within providers there's nuance between one model version to another and so that changes the the training data that you need to provide it.
09:58Um across model provisors uh it's extremely different the structure of the data you need to pass to it be different.
10:06The amount of the training data to get good results the way to interact with the training interface.
10:13All of this caused a lot of complexity and so it was just too costly for us to switch.
10:18And so um to we kept it the same model for consistency because we already had a lot to do with uh with each retraining process and we couldn't afford
10:30to uh upgrade the model.
10:34So uh the other way that this locked in was architecture.
10:40We built this app in uh late 2024 when workflows were kind of the um gold standard if you wanted good u production results and uh the AI world moves very fast
10:57and we couldn't adapt to that because we were so locked into this just trying to keep it running and we couldn't take advantage of the new architectures um and
11:08and improve performance that Okay, so earlier this year I had an aha moment.
11:15Um, we started using cloud code for our coding tasks and I noticed that we never needed to change the model depending on what task we're using.
11:27Um, we just changed the skill, the resources that we passed it, the context.
11:33Um, you drop in the better better context, you get better results.
11:37And I thought, why can't we do this with our messaging app?
11:43Um, this was obviously difficult for me to admit because I was the champion for fine-tuning.
11:49And, uh, luckily we were able to piggyback on a project that was already happening.
11:55Um, and so we migrated our workflow approach to a series of skills, tools, and resources that the skills could could load into or load up.
12:07um and get that context.
12:09And so we pushed this as one of our first production tests of our our new agentic framework that that was being built already.
12:18Now uh I want to compare the process before and after our rebuild.
12:24Uh before we already went through the kind of the training cycle, but there was this triage cycle beforehand where we needed to make sure that we had reached a critical mass
12:35of problems before we would even attempt to uh fine-tune again to improve everything.
12:43Like I said, this took about a week, so it was a long process, costly.
12:48Uh after the rebuild um it was a simple process of you find a problem you adjust the simp system prompt or the skill that was affected.
13:00We validated performance on a curated set that we have been collecting over the time that this was in production.
13:06We iterate a few times and then we deploy that simply by uploading MD files to an S3 bucket.
13:14um this whole process from discovering a problem to deploying the fix we reduced down to less than an hour.
13:22So it extremely improved all of this and we could be far more reactive give our our customers way better performance or better experience there.
13:34Now I'll be honest it did cost us a little bit more per message.
13:38we were using better models.
13:40Um, so the API costs were a little higher, but accuracy went way up.
13:48I said before that accuracy was the key to to getting all of this right.
13:53Um, and we did that.
13:55Accuracy uh was far better with this than it ever was with fine-tuning.
14:00Um, next, like I said, we reduced our uh our fixed process from days down to minutes.
14:09Next, we were able to unfreeze our model and finally get that freedom from a vendor that we never had with fine-tuning.
14:17Um, our agentic framework was built model agnostic.
14:22So, we could use open AI, we can use uh anthropic, we can use any other model that we want.
14:27The important part is the context that we're providing to that model.
14:32And then lastly, while it cost us a little more per message, the total cost went down because we were spending far less time trying to keep it up and running and
14:45fine-tuning to uh to keep it working properly.
14:49So before you fine-tune, I'd ask you, can you cross your reason off of this list?
14:56So, I thought we would get better accuracy.
15:00Uh, the rebuild beat the fine-tuned model.
15:04I thought we would get lower cost of the volume we were doing.
15:07Um, I was looking at the wrong costs.
15:10We we paid more per message, but the total cost ended up going down with our rebuild, lower latency.
15:16We we did see marginal gains on these smaller models, but they were so small that in practice it really didn't make any difference.
15:26And then maybe you've got a narrow or structured task.
15:30Our textbook case still became tech debt.
15:35And lastly, vendor control.
15:37It's not as simple as just plugging the data in.
15:40The other two situations where you you might have privacy and data control or you need some offline offline solution.
15:49I would say this these are the situations where a fine-tuned model may be useful.
15:56But you need to be cautious.
15:57There are other solutions out there.
16:00Um but um you need to make sure that it's not uh not causing issues in the long run.
16:07So finally fine-tune only when you literally cannot call a frontier model and even then your decision still has to beat the the tax.
16:18Thank you.