WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=qflLT3SoVbw NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/qflLT3SoVbw.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:13.120 --> 00:00:16.480 How do you decide what to put where? s3 00:00:18.720 --> 00:00:26.480 For most enterprise AI systems, the interesting engineing problem is not the model itself. s4 00:00:26.480 --> 00:00:31.199 The model consumes tokens and produces tokens. s5 00:00:31.199 --> 00:00:36.239 Your knowledge lives in files, databases, and APIs. s6 00:00:36.399 --> 00:00:53.760 The key architecture decision is how that knowledge reaches inference um from the prompt or the context through the retrieval or memory layer or by model adaptations such as fine-tuning. s7 00:00:53.760 --> 00:01:04.000 Today I'm going to argue that this is a decision that most enterprise teams make by accident. s8 00:01:04.000 --> 00:01:13.200 And the path to rec to to making it on purpose is to recognize that this is not a ladder that you climb. s9 00:01:13.200 --> 00:01:17.360 These are three different tools for three different jobs. s10 00:01:20.159 --> 00:01:22.159 So how is this an accident? s11 00:01:22.159 --> 00:01:24.320 Well, there are two mechanisms. s12 00:01:24.320 --> 00:01:25.600 One is the obvious one. s13 00:01:25.600 --> 00:01:28.320 The other one not so much. s14 00:01:28.320 --> 00:01:32.960 So the obvious one is that teams escalate. s15 00:01:33.200 --> 00:01:36.320 You don't get the right answer from your model. s16 00:01:36.320 --> 00:01:41.040 The easy thing is go put something in the prompt like go fix the prompt. s17 00:01:41.040 --> 00:01:42.320 Still wrong? s18 00:01:42.320 --> 00:01:46.640 Maybe there is something with the memory system or the retrieval system. s19 00:01:46.640 --> 00:01:49.119 Let's take a look there. s20 00:01:49.119 --> 00:01:51.920 Even then, if you're wrong, hey, you know what? s21 00:01:51.920 --> 00:01:53.920 Let's fine-tune the model. s22 00:01:53.920 --> 00:02:00.640 Doesn't happen very often, but yes, sometimes teams reach out to fine-tune the model. s23 00:02:00.960 --> 00:02:02.799 There is a layer underneath it. s24 00:02:02.799 --> 00:02:09.679 The teams that are not escalating are still making these decisions every day. s25 00:02:09.679 --> 00:02:17.920 Every edit that you make to the prompt decides what behavior lives in the prompt. s26 00:02:17.920 --> 00:02:23.760 Every document that you index decides what knowledge is held in the memory. s27 00:02:23.760 --> 00:02:36.400 And every training sample that you bake in will every every training sample that you identify will bake something into the weights. s28 00:02:36.400 --> 00:02:40.800 Um all these are architecture decisions. s29 00:02:40.800 --> 00:02:42.080 None like that. s30 00:02:42.080 --> 00:02:45.360 Okay, let me make it a little concrete for you all. s31 00:02:45.360 --> 00:02:49.440 Um, so and I've seen this play out actually. s32 00:02:49.440 --> 00:02:55.519 Uh, an internal support assistant just released. s33 00:02:55.519 --> 00:03:03.280 In a few weeks, the product manager makes some changes to the prompt to fix the tone of the support assistant. s34 00:03:03.760 --> 00:03:10.159 After a few weeks, the support team adds refund policy documents to your retrieval on memory systems. s35 00:03:10.159 --> 00:03:11.599 I'll talk about retrieval and memory. s36 00:03:11.599 --> 00:03:12.560 I'm teaching it. s37 00:03:12.560 --> 00:03:16.239 I'm I'm treating it all as memory, but I'll come to that. s38 00:03:16.239 --> 00:03:29.680 Um, and then since your model is not your your agent is not giving you the right answers, an engineer shuffs the product catalog into the prompt. s39 00:03:29.840 --> 00:03:42.640 And after a few weeks, the machine learning team decides to train the model or fine-tune the model on support tickets from 6 months ago from up to 6 months ago. s40 00:03:43.680 --> 00:03:48.000 All decisions look fine look okay except one by the way. s41 00:03:48.000 --> 00:03:51.200 But even I have made such decisions in my in my journey. s42 00:03:51.200 --> 00:03:56.400 I have made such decisions of doing such things without really giving it a lot of thought. s43 00:03:56.400 --> 00:04:05.280 Um, so when the new product product catalog has launched, looks easy, right? s44 00:04:05.280 --> 00:04:08.239 Oh, the engineer has shoved the product catalog into the prompt. s45 00:04:08.239 --> 00:04:10.959 Let's put the new product catalog there. s46 00:04:10.959 --> 00:04:21.919 And you would think that everything is working fine, but your model or your agent is still giving you product names that don't even exist. s47 00:04:21.919 --> 00:04:23.840 How did that happen? s48 00:04:23.840 --> 00:04:31.120 Well, the product catalog leaked into the weights when you fine-tuned it on your support tickets. s49 00:04:31.120 --> 00:04:33.520 And that is the accident. s50 00:04:33.520 --> 00:04:39.440 Your model, your your architecture was accumulated. s51 00:04:39.440 --> 00:04:41.120 It was not designed. s52 00:04:41.120 --> 00:04:43.680 That is the accident. s53 00:04:43.680 --> 00:04:49.520 It is what 6 months of normal product work led to. s54 00:04:49.520 --> 00:04:50.720 Nobody owned it. s55 00:04:50.720 --> 00:04:52.960 Meaning everybody owned a piece of it. s56 00:04:52.960 --> 00:04:54.960 Okay. s57 00:04:55.199 --> 00:05:00.320 Um, nobody asked the diagnostic question. s58 00:05:00.320 --> 00:05:04.720 Nobody asked where this should where this thing should live. s59 00:05:04.720 --> 00:05:13.280 That diagnostic those diagnostic questions are the lens we are going to talk about and the rest of the talk is is about the lens. s60 00:05:13.280 --> 00:05:15.280 Okay. s61 00:05:15.840 --> 00:05:18.080 So the first part is prompt. s62 00:05:18.080 --> 00:05:19.520 What is the job of the prompt? s63 00:05:19.520 --> 00:05:26.960 The job of the prompt is behavior, is tone, is the persona of the agent. s64 00:05:27.120 --> 00:05:31.120 Things that are small, stable, and editable. s65 00:05:31.840 --> 00:05:36.160 The wrong job for the prompt is to store facts. s66 00:05:36.160 --> 00:05:44.560 You don't put product catalog into the prompt because you are unnecessarily providing context to your model and paying the cost for it. s67 00:05:44.560 --> 00:05:49.680 And when you have too many tokens, you run into lost in the middle problem. s68 00:05:49.840 --> 00:05:59.600 The diagnostic for prompt is is the small knowledge small stable and about how to behave rather than what to know. s69 00:06:01.840 --> 00:06:12.080 A concrete example for a good prompt use case is a customer support agent for a product SAS. s70 00:06:12.080 --> 00:06:19.520 Right now this agent is has a professional tone. s71 00:06:19.520 --> 00:06:30.000 It is it offers human escalation after three failures and it provides you concrete next steps. s72 00:06:30.000 --> 00:06:34.639 None of this changes per query per user. s73 00:06:34.880 --> 00:06:39.039 This is this this exact thing belongs to the prompt. s74 00:06:39.039 --> 00:06:43.440 And one other thing is prompt is something I'm not only talking about system prompt. s75 00:06:43.440 --> 00:06:48.400 I'm talking about authored instructions for your for your model. s76 00:06:48.400 --> 00:06:50.319 So they could also come from MD files. s77 00:06:50.319 --> 00:06:52.319 They could also come from other things, right? s78 00:06:52.319 --> 00:06:56.400 But this is what I mean by prompt here. s79 00:06:58.000 --> 00:07:01.599 The second path is memory. s80 00:07:01.680 --> 00:07:13.680 The job of memory is to ground your model or your agent in in knowledge that is current, large and citable. s81 00:07:13.680 --> 00:07:17.360 Current meaning it changes faster than you can fine-tune. s82 00:07:17.360 --> 00:07:21.120 It changes every now and then. s83 00:07:21.120 --> 00:07:23.199 Large is because it cannot be held anywhere. s84 00:07:23.199 --> 00:07:25.120 It cannot be held in your prompt. s85 00:07:25.120 --> 00:07:31.599 and citable because production AI systems need to be able to point to the source of the knowledge. s86 00:07:31.599 --> 00:07:36.479 This is the this is what belongs to memory. s87 00:07:36.479 --> 00:07:47.120 Now when I use the term memory people think of agent memory knowledge that is acquired about people about people who about a person who is interacting with the agent right s88 00:07:47.120 --> 00:08:01.280 what I'm also referring to here is external memory which is which you retrieve using rag and all that like the refund policies right it's not being managed by your agent it is managed by an external application s89 00:08:01.280 --> 00:08:07.280 that is included here so I'm talking both about the agent memory and the memory Mor of the enterprise. s90 00:08:07.280 --> 00:08:08.879 Okay. s91 00:08:08.879 --> 00:08:20.800 The wrong job of for memory is shoving behavior in or thinking about how to reason with the memory. s92 00:08:20.800 --> 00:08:27.759 Now, of course, you can have a graphical kind of rag which will help the agent do reasoning, but that's it. s93 00:08:27.759 --> 00:08:30.080 That's where you stop. s94 00:08:30.080 --> 00:08:33.279 For reasoning, you got to have the right model. s95 00:08:33.279 --> 00:08:39.120 Because if your model cannot reason over two or three or five documents, it won't be able to reason over 50 that you provide. s96 00:08:39.120 --> 00:08:42.880 The more context doesn't help if it's not able to reason. s97 00:08:42.880 --> 00:08:51.120 So the diagnostic or the question you need to ask is is the knowledge too large to fit changing faster than I can retrain. s98 00:08:51.120 --> 00:08:59.680 And another important aspect is whether it has some access control whether it is scope per user. s99 00:08:59.680 --> 00:09:01.519 You have got to have that in memory. s100 00:09:01.519 --> 00:09:15.040 If there is some kind of access control that this user cannot see the other person's information or the data or whatever, you've got to have that knowledge in memory because then you can control who sees what. s101 00:09:18.080 --> 00:09:25.440 A good example for storing something in external memory or memory is a code assistant, right? s102 00:09:25.440 --> 00:09:30.640 Which has access to your all your repos as an organization. s103 00:09:33.120 --> 00:09:39.519 So uh would you fine-tune your model to learn the code? s104 00:09:39.519 --> 00:09:41.200 No. It's it's a simple one, right? s105 00:09:41.200 --> 00:09:42.160 It changes very often. s106 00:09:42.160 --> 00:09:43.279 You wouldn't do that. s107 00:09:43.279 --> 00:09:47.519 You'd finetune your model to learn reflexes, not facts typically. s108 00:09:47.519 --> 00:09:49.200 And would you stuff it in the prompt? s109 00:09:49.200 --> 00:09:51.200 No, you don't stuff such things into the prompt. s110 00:09:51.200 --> 00:09:57.440 You may stuff or you may add certain things that are important in the context window, right? s111 00:09:57.440 --> 00:09:58.720 That's something different. s112 00:09:58.720 --> 00:10:01.680 But you wouldn't stuff the entire code base into the prompt. s113 00:10:01.680 --> 00:10:01.920 Okay? s114 00:10:01.920 --> 00:10:06.320 Or or even a part of it if it has no use there. s115 00:10:06.320 --> 00:10:21.279 Uh now the other important thing is um when you build a rack system right like for for for your code for your code you need to make sure that you are chunking properly you are using code aware chunking s116 00:10:21.279 --> 00:10:27.200 with a perhaps uh and then the other important thing is use filtering. s117 00:10:27.200 --> 00:10:28.240 You should denormalize. s118 00:10:28.240 --> 00:10:34.000 You should perhaps denormalize things such as who is the what is the repo this code is for. s119 00:10:34.000 --> 00:10:35.360 You get a code chunk. s120 00:10:35.360 --> 00:10:39.120 You have to create metadata about which repo it belongs to. s121 00:10:39.120 --> 00:10:48.640 What is the user what are the users who are allowed to commit and all that because that is what is going to help you get the exact data that you need s122 00:10:48.640 --> 00:10:58.079 versus creating a rag mush where you are returning a lot of chunks back from the database without and and then confusing the model. s123 00:10:59.040 --> 00:11:08.000 So again the diagnostic is something that's too large to fit changing the faster than you can retrain the model and it's scoped at a certain level. s124 00:11:08.000 --> 00:11:11.440 It could be it could be a user, it could be a repo or whatever. s125 00:11:11.440 --> 00:11:14.079 All that knowledge belongs to memory. s126 00:11:14.079 --> 00:11:22.079 It could be external memory which is your like your applications database or it could be your agent memory as well. s127 00:11:24.720 --> 00:11:29.360 Now let's come to the interest more interesting part which is weights. s128 00:11:30.800 --> 00:11:35.120 So what belongs to the weight? s129 00:11:36.800 --> 00:11:38.480 Something that stopped changing. s130 00:11:38.480 --> 00:11:39.920 That is what belongs to the weight. s131 00:11:39.920 --> 00:11:45.680 And the only axis that drives this slide is rate of change. s132 00:11:47.040 --> 00:11:51.920 You can lock in what has stopped changing and perhaps retrain. s133 00:11:51.920 --> 00:11:55.120 And I'm not saying just lock in what stopped changing and just go retrain your model. s134 00:11:55.120 --> 00:11:56.800 No, that's not what you do all the time. s135 00:11:56.800 --> 00:12:01.680 You would you have to have a very strong reason to retrain your model. s136 00:12:01.680 --> 00:12:04.959 So what you do is you lock in what's changing and then go from there. s137 00:12:04.959 --> 00:12:09.600 But if you get it wrong, then you could freeze a boundary that's still moving. s138 00:12:09.600 --> 00:12:11.120 Okay? s139 00:12:11.120 --> 00:12:21.360 And it's not as subtle as like, hey, can I re would I re Nobody's going to retrain their model on like or fine-tune, I meant not retrain, sorry, fine-tune their model on pricing list. s140 00:12:21.360 --> 00:12:22.240 Nobody does that. s141 00:12:22.240 --> 00:12:24.800 It is more subtle than that, right? s142 00:12:24.800 --> 00:12:40.399 And a current version of that is um there is an let's say there is an internal document assistant and the support team or the or your team starts decides to fine-tune s143 00:12:40.399 --> 00:12:43.360 the model because it's not giving the right answers. s144 00:12:43.360 --> 00:12:50.000 So to teach the domain, it says, you know what, let's fine-tune this model on on our documentation. s145 00:12:50.000 --> 00:12:56.160 Let's fine-tune the model on or or maybe not all your all their documentation but runbooks and procedures. s146 00:12:56.160 --> 00:12:57.440 And what happens with that? s147 00:12:57.440 --> 00:12:58.800 There is an issue. s148 00:12:58.800 --> 00:13:08.000 If you do that when a your model will be giving you information that's old. s149 00:13:08.000 --> 00:13:09.920 This was actually a retrieval problem. s150 00:13:09.920 --> 00:13:17.360 If the model was not giving the right answer, what the model needed was the right chunks, the right pieces of documentation. s151 00:13:17.360 --> 00:13:24.959 So basically instead of fixing the retrieval problem, the team actually went ahead and fine-tuned the model. s152 00:13:24.959 --> 00:13:26.639 That's not what you should do. s153 00:13:26.639 --> 00:13:28.800 Runbooks, procedures, these are facts. s154 00:13:28.800 --> 00:13:34.959 These belong to either your me either your agent memory not there but of course your external memory. s155 00:13:34.959 --> 00:13:38.240 That is where these things should reside. s156 00:13:38.560 --> 00:13:40.959 Not if the if the model is not s157 00:13:40.959 --> 00:13:42.240 good afternoon. s158 00:13:42.240 --> 00:13:42.639 Hello. s159 00:13:42.639 --> 00:13:46.000 Oh sorry that was I was like who's that? s160 00:13:46.000 --> 00:13:50.320 So if your if your model is not giving the right answers, fix the retrieval problem. s161 00:13:50.320 --> 00:13:52.079 Don't just go ahead and fine-tune your model. s162 00:13:52.079 --> 00:13:53.120 Okay. s163 00:13:53.120 --> 00:13:57.279 Uh and if you flip it, same axis. s164 00:13:57.279 --> 00:14:10.560 If you reverse it, u a good use case where I could fine-tune is let's say it's a hard ambiguous problem such as content moderation or claims processing. s165 00:14:10.560 --> 00:14:11.279 Right? s166 00:14:11.279 --> 00:14:19.920 When you begin, you can use a frontier model that makes recommendations and humans correct it although inconsistently. s167 00:14:20.320 --> 00:14:23.680 Over a period of time, patterns will emerge. s168 00:14:23.680 --> 00:14:30.720 Patterns will emerge and you will have sort of a center which is sort of fixed and stable. s169 00:14:30.720 --> 00:14:31.519 Okay? s170 00:14:31.519 --> 00:14:34.880 And then you will have the contested edge. s171 00:14:35.839 --> 00:14:50.079 the the rates of human overrides will flatline over after a certain point and you'll you will most likely emerge in a pattern where you have a contested edge where you still need humans but a center which is pretty stable. s172 00:14:50.079 --> 00:14:55.199 Okay, now that center is something you can fine-tune your model on. s173 00:14:55.199 --> 00:14:59.120 Okay, but when you do that you have to monitor for drift. s174 00:14:59.120 --> 00:15:00.880 Is there a drift in my center? s175 00:15:00.880 --> 00:15:04.639 And if that happens then you're going to run into problem. s176 00:15:04.639 --> 00:15:10.160 And um actually a stable version of that is I don't know have you heard of medical claims coding? s177 00:15:10.160 --> 00:15:19.440 I think we must have like ICD 10 codes doctor's notes are mapped to like codes for different purposes like insurance purposes and all that. s178 00:15:19.440 --> 00:15:26.240 So Mount Si Mount Sinai and IMO health actually s179 00:15:26.560 --> 00:15:29.600 there are 70,000 of such codes by the way. s180 00:15:29.600 --> 00:15:33.519 So would you fine-tune your model to learn those codes? s181 00:15:33.519 --> 00:15:35.040 They don't change so often. s182 00:15:35.040 --> 00:15:36.399 You won't. s183 00:15:36.399 --> 00:15:37.120 That's fact. s184 00:15:37.120 --> 00:15:37.920 That's knowledge. s185 00:15:37.920 --> 00:15:41.199 That's you should not be looking at fine-tuning your model. s186 00:15:41.199 --> 00:15:45.279 One of the obvious reasons is is that they change sometimes. s187 00:15:45.279 --> 00:15:51.199 And the other is you will need a large test training data to even fine-tune your model on it. s188 00:15:51.199 --> 00:15:51.680 Okay. s189 00:15:51.680 --> 00:16:11.279 What you fine-tune our model is on the reflex whether whether it is understanding the format of the input and whether it is able to and when it you have to make selection across the formats like which one is the right one that's where you fine-tune your model to do that. s190 00:16:11.279 --> 00:16:13.279 Okay. s191 00:16:15.440 --> 00:16:29.120 So the diagnostic for fine-tuning is has it has the information stopped changing have the humans converged and more important you should either h have one of these two reasons s192 00:16:29.120 --> 00:16:43.040 is the is are you fine-tuning because your model is not capable or are you fine-tuning to save cost and a lot of times it's not capability frontier models are quite capable it usually comes down to cost s193 00:16:43.040 --> 00:16:55.920 where you can have a small if you have nice pattern in your work, you can actually perhaps use a small model, fine-tune it and run the volume cheap, right? s194 00:16:56.639 --> 00:17:04.559 So, if you don't remember anything from the talk, this is a simple one, right? s195 00:17:04.559 --> 00:17:06.559 This table helps me decide. s196 00:17:06.559 --> 00:17:16.880 You can b you can bounce your current AI system against this and and decide this is the lens basically that helps you decide what lives where. s197 00:17:16.880 --> 00:17:18.559 Of course there can be exceptions. s198 00:17:18.559 --> 00:17:20.640 I'm not saying this is always the rule. s199 00:17:20.640 --> 00:17:24.400 There can be exceptions to this but prompt is more about behavior. s200 00:17:24.400 --> 00:17:29.120 Memory is about what to know and facts and weights are about how to reason. s201 00:17:29.120 --> 00:17:31.120 Okay. s202 00:17:32.799 --> 00:17:36.320 And this is the architecture. s203 00:17:37.280 --> 00:17:44.080 Your prompt or your context window, your memory and your finetuning needs to circulate. s204 00:17:44.080 --> 00:17:50.080 You need to build a right harness around it so that these things can circulate. s205 00:17:50.880 --> 00:17:54.080 Your prompt or your context window has information. s206 00:17:54.080 --> 00:17:55.440 It generates signals. s207 00:17:55.440 --> 00:17:57.840 Some of them becomes durable memory. s208 00:17:57.840 --> 00:17:59.679 Okay. s209 00:17:59.679 --> 00:18:11.520 Now when you are restarting a new session things get pulled out of from the memory into the prompt or context window and that's when you're going from the memory to the prompt. s210 00:18:11.520 --> 00:18:20.720 The more so this is lot of teams have built this architecture but what people are looking to are focusing on building is the right side the memory to fine-tuning flow. s211 00:18:20.720 --> 00:18:21.360 Okay. s212 00:18:21.360 --> 00:18:28.400 Now what happens is when you build an agentic system over a period of time you will have retrieval patterns. s213 00:18:28.400 --> 00:18:34.640 You will have formats that you think the model should reflexively know. s214 00:18:34.640 --> 00:18:43.200 So those things move from your memory to fine-tuning once you have fine-tuned. s215 00:18:43.200 --> 00:18:46.000 So that's memory to weights. s216 00:18:46.000 --> 00:18:57.440 Once you have fine-tuned what is worth retrieving changes if you have changed if you have fine- tuned your model on the formats of like notes and the ICD codes s217 00:18:57.440 --> 00:19:14.000 you don't have to retrieve certain examples from your memory or external memory each time and provide it to the model because now the model knows the format reflexively you don't have to provide examples to it that hey this is the format or that is the format so that's weights to memory s218 00:19:14.000 --> 00:19:25.280 and so basically This becomes a loop over a period of time and the agent gets better at its job by the virtue of doing its job. s219 00:19:25.280 --> 00:19:31.120 Okay, that's the architecture. s220 00:19:31.120 --> 00:19:37.840 So what I would like to conclude with is the model is is the easy part. s221 00:19:38.720 --> 00:19:51.039 What you build around the model, the harness around the model that helps you store the right information at the right place and circulate among them is the key architecture that you got to build. s222 00:19:51.039 --> 00:19:51.919 Thank you. s223 00:19:51.919 --> 00:19:54.559 Thanks for coming up.