Stop Fine-Tuning to Fix Retrieval Problems — Anant Srivastava

AI Engineer · 20 min · 223 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13How do you decide what to put where?
00:18For most enterprise AI systems, the interesting engineing problem is not the model itself.
00:26The model consumes tokens and produces tokens.
00:31Your knowledge lives in files, databases, and APIs.
00:36The key architecture decision is how that knowledge reaches inference um from the prompt or the context through the retrieval or memory layer or by model adaptations such as fine-tuning.
00:53Today I'm going to argue that this is a decision that most enterprise teams make by accident.
01:04And the path to rec to to making it on purpose is to recognize that this is not a ladder that you climb.
01:13These are three different tools for three different jobs.
01:20So how is this an accident?
01:22Well, there are two mechanisms.
01:24One is the obvious one.
01:25The other one not so much.
01:28So the obvious one is that teams escalate.
01:33You don't get the right answer from your model.
01:36The easy thing is go put something in the prompt like go fix the prompt.
01:41Still wrong?
01:42Maybe there is something with the memory system or the retrieval system.
01:46Let's take a look there.
01:49Even then, if you're wrong, hey, you know what?
01:51Let's fine-tune the model.
01:53Doesn't happen very often, but yes, sometimes teams reach out to fine-tune the model.
02:00There is a layer underneath it.
02:02The teams that are not escalating are still making these decisions every day.
02:09Every edit that you make to the prompt decides what behavior lives in the prompt.
02:17Every document that you index decides what knowledge is held in the memory.
02:23And every training sample that you bake in will every every training sample that you identify will bake something into the weights.
02:36Um all these are architecture decisions.
02:40None like that.
02:42Okay, let me make it a little concrete for you all.
02:45Um, so and I've seen this play out actually.
02:49Uh, an internal support assistant just released.
02:55In a few weeks, the product manager makes some changes to the prompt to fix the tone of the support assistant.
03:03After a few weeks, the support team adds refund policy documents to your retrieval on memory systems.
03:10I'll talk about retrieval and memory.
03:11I'm teaching it.
03:12I'm I'm treating it all as memory, but I'll come to that.
03:16Um, and then since your model is not your your agent is not giving you the right answers, an engineer shuffs the product catalog into the prompt.
03:29And after a few weeks, the machine learning team decides to train the model or fine-tune the model on support tickets from 6 months ago from up to 6 months ago.
03:43All decisions look fine look okay except one by the way.
03:48But even I have made such decisions in my in my journey.
03:51I have made such decisions of doing such things without really giving it a lot of thought.
03:56Um, so when the new product product catalog has launched, looks easy, right?
04:05Oh, the engineer has shoved the product catalog into the prompt.
04:08Let's put the new product catalog there.
04:10And you would think that everything is working fine, but your model or your agent is still giving you product names that don't even exist.
04:21How did that happen?
04:23Well, the product catalog leaked into the weights when you fine-tuned it on your support tickets.
04:31And that is the accident.
04:33Your model, your your architecture was accumulated.
04:39It was not designed.
04:41That is the accident.
04:43It is what 6 months of normal product work led to.
04:49Nobody owned it.
04:50Meaning everybody owned a piece of it.
04:52Okay.
04:55Um, nobody asked the diagnostic question.
05:00Nobody asked where this should where this thing should live.
05:04That diagnostic those diagnostic questions are the lens we are going to talk about and the rest of the talk is is about the lens.
05:13Okay.
05:15So the first part is prompt.
05:18What is the job of the prompt?
05:19The job of the prompt is behavior, is tone, is the persona of the agent.
05:27Things that are small, stable, and editable.
05:31The wrong job for the prompt is to store facts.
05:36You don't put product catalog into the prompt because you are unnecessarily providing context to your model and paying the cost for it.
05:44And when you have too many tokens, you run into lost in the middle problem.
05:49The diagnostic for prompt is is the small knowledge small stable and about how to behave rather than what to know.
06:01A concrete example for a good prompt use case is a customer support agent for a product SAS.
06:12Right now this agent is has a professional tone.
06:19It is it offers human escalation after three failures and it provides you concrete next steps.
06:30None of this changes per query per user.
06:34This is this this exact thing belongs to the prompt.
06:39And one other thing is prompt is something I'm not only talking about system prompt.
06:43I'm talking about authored instructions for your for your model.
06:48So they could also come from MD files.
06:50They could also come from other things, right?
06:52But this is what I mean by prompt here.
06:58The second path is memory.
07:01The job of memory is to ground your model or your agent in in knowledge that is current, large and citable.
07:13Current meaning it changes faster than you can fine-tune.
07:17It changes every now and then.
07:21Large is because it cannot be held anywhere.
07:23It cannot be held in your prompt.
07:25and citable because production AI systems need to be able to point to the source of the knowledge.
07:31This is the this is what belongs to memory.
07:36Now when I use the term memory people think of agent memory knowledge that is acquired about people about people who about a person who is interacting with the agent right
07:47what I'm also referring to here is external memory which is which you retrieve using rag and all that like the refund policies right it's not being managed by your agent it is managed by an external application
08:01that is included here so I'm talking both about the agent memory and the memory Mor of the enterprise.
08:07Okay.
08:08The wrong job of for memory is shoving behavior in or thinking about how to reason with the memory.
08:20Now, of course, you can have a graphical kind of rag which will help the agent do reasoning, but that's it.
08:27That's where you stop.
08:30For reasoning, you got to have the right model.
08:33Because if your model cannot reason over two or three or five documents, it won't be able to reason over 50 that you provide.
08:39The more context doesn't help if it's not able to reason.
08:42So the diagnostic or the question you need to ask is is the knowledge too large to fit changing faster than I can retrain.
08:51And another important aspect is whether it has some access control whether it is scope per user.
08:59You have got to have that in memory.
09:01If there is some kind of access control that this user cannot see the other person's information or the data or whatever, you've got to have that knowledge in memory because then you can control who sees what.
09:18A good example for storing something in external memory or memory is a code assistant, right?
09:25Which has access to your all your repos as an organization.
09:33So uh would you fine-tune your model to learn the code?
09:39No. It's it's a simple one, right?
09:41It changes very often.
09:42You wouldn't do that.
09:43You'd finetune your model to learn reflexes, not facts typically.
09:47And would you stuff it in the prompt?
09:49No, you don't stuff such things into the prompt.
09:51You may stuff or you may add certain things that are important in the context window, right?
09:57That's something different.
09:58But you wouldn't stuff the entire code base into the prompt.
10:01Okay?
10:01Or or even a part of it if it has no use there.
10:06Uh now the other important thing is um when you build a rack system right like for for for your code for your code you need to make sure that you are chunking properly you are using code aware chunking
10:21with a perhaps uh and then the other important thing is use filtering.
10:27You should denormalize.
10:28You should perhaps denormalize things such as who is the what is the repo this code is for.
10:34You get a code chunk.
10:35You have to create metadata about which repo it belongs to.
10:39What is the user what are the users who are allowed to commit and all that because that is what is going to help you get the exact data that you need
10:48versus creating a rag mush where you are returning a lot of chunks back from the database without and and then confusing the model.
10:59So again the diagnostic is something that's too large to fit changing the faster than you can retrain the model and it's scoped at a certain level.
11:08It could be it could be a user, it could be a repo or whatever.
11:11All that knowledge belongs to memory.
11:14It could be external memory which is your like your applications database or it could be your agent memory as well.
11:24Now let's come to the interest more interesting part which is weights.
11:30So what belongs to the weight?
11:36Something that stopped changing.
11:38That is what belongs to the weight.
11:39And the only axis that drives this slide is rate of change.
11:47You can lock in what has stopped changing and perhaps retrain.
11:51And I'm not saying just lock in what stopped changing and just go retrain your model.
11:55No, that's not what you do all the time.
11:56You would you have to have a very strong reason to retrain your model.
12:01So what you do is you lock in what's changing and then go from there.
12:04But if you get it wrong, then you could freeze a boundary that's still moving.
12:09Okay?
12:11And it's not as subtle as like, hey, can I re would I re Nobody's going to retrain their model on like or fine-tune, I meant not retrain, sorry, fine-tune their model on pricing list.
12:21Nobody does that.
12:22It is more subtle than that, right?
12:24And a current version of that is um there is an let's say there is an internal document assistant and the support team or the or your team starts decides to fine-tune
12:40the model because it's not giving the right answers.
12:43So to teach the domain, it says, you know what, let's fine-tune this model on on our documentation.
12:50Let's fine-tune the model on or or maybe not all your all their documentation but runbooks and procedures.
12:56And what happens with that?
12:57There is an issue.
12:58If you do that when a your model will be giving you information that's old.
13:08This was actually a retrieval problem.
13:09If the model was not giving the right answer, what the model needed was the right chunks, the right pieces of documentation.
13:17So basically instead of fixing the retrieval problem, the team actually went ahead and fine-tuned the model.
13:24That's not what you should do.
13:26Runbooks, procedures, these are facts.
13:28These belong to either your me either your agent memory not there but of course your external memory.
13:34That is where these things should reside.
13:38Not if the if the model is not
13:40good afternoon.
13:42Hello.
13:42Oh sorry that was I was like who's that?
13:46So if your if your model is not giving the right answers, fix the retrieval problem.
13:50Don't just go ahead and fine-tune your model.
13:52Okay.
13:53Uh and if you flip it, same axis.
13:57If you reverse it, u a good use case where I could fine-tune is let's say it's a hard ambiguous problem such as content moderation or claims processing.
14:10Right?
14:11When you begin, you can use a frontier model that makes recommendations and humans correct it although inconsistently.
14:20Over a period of time, patterns will emerge.
14:23Patterns will emerge and you will have sort of a center which is sort of fixed and stable.
14:30Okay?
14:31And then you will have the contested edge.
14:35the the rates of human overrides will flatline over after a certain point and you'll you will most likely emerge in a pattern where you have a contested edge where you still need humans but a center which is pretty stable.
14:50Okay, now that center is something you can fine-tune your model on.
14:55Okay, but when you do that you have to monitor for drift.
14:59Is there a drift in my center?
15:00And if that happens then you're going to run into problem.
15:04And um actually a stable version of that is I don't know have you heard of medical claims coding?
15:10I think we must have like ICD 10 codes doctor's notes are mapped to like codes for different purposes like insurance purposes and all that.
15:19So Mount Si Mount Sinai and IMO health actually
15:26there are 70,000 of such codes by the way.
15:29So would you fine-tune your model to learn those codes?
15:33They don't change so often.
15:35You won't.
15:36That's fact.
15:37That's knowledge.
15:37That's you should not be looking at fine-tuning your model.
15:41One of the obvious reasons is is that they change sometimes.
15:45And the other is you will need a large test training data to even fine-tune your model on it.
15:51Okay.
15:51What you fine-tune our model is on the reflex whether whether it is understanding the format of the input and whether it is able to and when it you have to make selection across the formats like which one is the right one that's where you fine-tune your model to do that.
16:11Okay.
16:15So the diagnostic for fine-tuning is has it has the information stopped changing have the humans converged and more important you should either h have one of these two reasons
16:29is the is are you fine-tuning because your model is not capable or are you fine-tuning to save cost and a lot of times it's not capability frontier models are quite capable it usually comes down to cost
16:43where you can have a small if you have nice pattern in your work, you can actually perhaps use a small model, fine-tune it and run the volume cheap, right?
16:56So, if you don't remember anything from the talk, this is a simple one, right?
17:04This table helps me decide.
17:06You can b you can bounce your current AI system against this and and decide this is the lens basically that helps you decide what lives where.
17:16Of course there can be exceptions.
17:18I'm not saying this is always the rule.
17:20There can be exceptions to this but prompt is more about behavior.
17:24Memory is about what to know and facts and weights are about how to reason.
17:29Okay.
17:32And this is the architecture.
17:37Your prompt or your context window, your memory and your finetuning needs to circulate.
17:44You need to build a right harness around it so that these things can circulate.
17:50Your prompt or your context window has information.
17:54It generates signals.
17:55Some of them becomes durable memory.
17:57Okay.
17:59Now when you are restarting a new session things get pulled out of from the memory into the prompt or context window and that's when you're going from the memory to the prompt.
18:11The more so this is lot of teams have built this architecture but what people are looking to are focusing on building is the right side the memory to fine-tuning flow.
18:20Okay.
18:21Now what happens is when you build an agentic system over a period of time you will have retrieval patterns.
18:28You will have formats that you think the model should reflexively know.
18:34So those things move from your memory to fine-tuning once you have fine-tuned.
18:43So that's memory to weights.
18:46Once you have fine-tuned what is worth retrieving changes if you have changed if you have fine- tuned your model on the formats of like notes and the ICD codes
18:57you don't have to retrieve certain examples from your memory or external memory each time and provide it to the model because now the model knows the format reflexively you don't have to provide examples to it that hey this is the format or that is the format so that's weights to memory
19:14and so basically This becomes a loop over a period of time and the agent gets better at its job by the virtue of doing its job.
19:25Okay, that's the architecture.
19:31So what I would like to conclude with is the model is is the easy part.
19:38What you build around the model, the harness around the model that helps you store the right information at the right place and circulate among them is the key architecture that you got to build.
19:51Thank you.
19:51Thanks for coming up.