WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=T0HhO4YtTfE NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/T0HhO4YtTfE.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:02.080 --> 00:00:02.960 Hi everyone. s2 00:00:02.960 --> 00:00:08.400 I'm Apurva and I'm a data scientist turned developer advocate currently at MongoDB. s3 00:00:08.400 --> 00:00:21.680 I spent the first of my career building machine learning applications for various cybersecurity use cases and I now use that applied machine learning knowledge to help AI builders successfully build AI applications s4 00:00:21.680 --> 00:00:25.160 with MongoDB and Voyage AI. s5 00:00:25.160 --> 00:00:31.520 In this talk you'll learn how to think through building AI systems end-to-end from idea to production. s6 00:00:31.520 --> 00:00:41.240 We'll take a real-world use case and walk through all the steps of designing it and hopefully in the end you'll walk away with a repeatable framework that you can apply to s7 00:00:41.240 --> 00:00:44.320 any AI system you design. s8 00:00:46.400 --> 00:00:52.480 Now, you might be thinking in the age of AI do we even need to think about what we build? s9 00:00:52.480 --> 00:00:54.720 Just wipe code it and ship it, right? s10 00:00:54.720 --> 00:00:57.280 But that's actually where I want to start. s11 00:00:57.280 --> 00:01:00.400 Now, here's the problem with that. s12 00:01:01.080 --> 00:01:09.600 Wipe coding works great when you're building for fun, the stakes are low, and you can easily eyeball whether the output of what you're building is right. s13 00:01:09.600 --> 00:01:19.080 But the moment you're building something real, something other people depend on, something with real consequences, just ship it is actually kind of dangerous. s14 00:01:19.080 --> 00:01:25.680 Uh and it's not just me saying it, folks from Anthropic and OpenAI who are so bullish on AI coding are saying the same thing. s15 00:01:25.680 --> 00:01:30.320 These are actually quotes from their talks over the past few months. s16 00:01:30.320 --> 00:01:32.120 Specs are the new code. s17 00:01:32.120 --> 00:01:42.280 The art is in defining the product requirements, the system design, and evaluate criteria so you can be confident that your AI coding buddies are building the right thing. s18 00:01:42.280 --> 00:01:44.360 And the rest of the talk is just about that. s19 00:01:44.360 --> 00:01:47.240 How do we do this well? s20 00:01:49.520 --> 00:01:52.440 It's useful to think about it as a framework. s21 00:01:52.440 --> 00:01:53.280 Four phases. s22 00:01:53.280 --> 00:01:55.160 You start with product requirements. s23 00:01:55.160 --> 00:01:59.960 What are you actually building, for whom, and what are the constraints? s24 00:01:59.960 --> 00:02:07.479 Then system design, the data, the architecture, the patterns that actually help you meet these requirements. s25 00:02:07.479 --> 00:02:09.840 Then comes evaluation and monitoring. s26 00:02:09.840 --> 00:02:15.920 How do you know what's being built actually works before and also after you ship? s27 00:02:15.920 --> 00:02:29.160 And finally, how do you optimize for cost, not just for accuracy, but also for cost, latency, and reliability before you ship and or as you find gaps in production. s28 00:02:30.360 --> 00:02:42.959 Instead of talking abstractly through a framework, I thought why not apply it to a real-world use case so you can see how exactly each decision flows from one stage of the framework to the next. s29 00:02:42.959 --> 00:02:48.440 So let's take the example of a health insurance claims review system. s30 00:02:49.600 --> 00:03:06.320 So adjudicating health insurance claims has historically been an extremely manual process where human medical reviewers have to cross-reference clinical guidelines, insurance coverage policies, patient history, and stuff to decide whether or not s31 00:03:06.320 --> 00:03:12.760 a medical treatment, procedure, or medication is covered by a patient's health insurance policy. s32 00:03:12.760 --> 00:03:21.040 Now, if you live somewhere that has free health care, even there there's things that are covered and things that are not by your health care system. s33 00:03:21.040 --> 00:03:26.680 So you're still subject to some version of this process. s34 00:03:27.320 --> 00:03:37.560 Now, our goal with building this system is to see if we can improve or simplify the experience for medical reviewers with the help of AI. s35 00:03:38.239 --> 00:03:45.280 So let's see what the product requirements stage of the framework for this application might look like. s36 00:03:45.720 --> 00:03:49.320 So first thing to do is quantify the business problem. s37 00:03:49.320 --> 00:04:02.160 The business problem should focus on something specific, uh clearly state who the users of the application are, uh the current state of things, and quantify the users' pain point. s38 00:04:02.160 --> 00:04:08.880 It should not prescribe what the system is going to be, whether it's going to be an agent, a multi-agent system, something else. s39 00:04:08.880 --> 00:04:11.560 Uh that will come later. s40 00:04:14.000 --> 00:04:18.600 So, here's what the business problem for our claims review system could look like. s41 00:04:18.600 --> 00:04:31.720 Medical reviewers at MDB Health spend an average of 2 days processing claim review requests, which is four times the industry standard for non-urgent cases, and 12 times the industry standard for urgent ones. s42 00:04:31.720 --> 00:04:38.080 Delays at this scale postpone patient care, uh particularly for time-sensitive treatment. s43 00:04:38.080 --> 00:04:41.520 So, as you can see, this business problem is user-specific. s44 00:04:41.520 --> 00:04:44.560 It says this is meant for medical reviewers. s45 00:04:44.560 --> 00:04:54.160 Uh it states the current state, which is like this manual process that leads to them spending 2 days processing uh claim reviews. s46 00:04:54.160 --> 00:04:57.840 It's measurable because it states some baselines. s47 00:04:57.840 --> 00:04:59.200 Uh it's solution-agnostic. s48 00:04:59.200 --> 00:05:08.280 It doesn't really tell you uh what the system should look like, uh and it's very focused on a specific problem. s49 00:05:10.919 --> 00:05:14.600 Next, you want to gather any business constraints for the application. s50 00:05:14.600 --> 00:05:28.280 So, any regulatory uh compliance requirements, any constraints on data leaving your organization, procurement constraints, meaning are certain vendors uh not approved for use within the organization. s51 00:05:28.280 --> 00:05:33.000 Uh all of these would be good to know before you even start designing the system. s52 00:05:33.000 --> 00:05:36.560 So, for our application, say these are some of the business constraints. s53 00:05:36.560 --> 00:05:40.480 Patient data has to stay within the approved cloud environment. s54 00:05:40.480 --> 00:05:44.760 Um only models available on the approved cloud can be used. s55 00:05:44.760 --> 00:05:49.760 Um there's some use cases where human review is required. s56 00:05:49.760 --> 00:05:54.560 So, any complex use cases require a senior physician's review. s57 00:05:54.560 --> 00:05:56.800 Um what else? s58 00:05:56.800 --> 00:06:00.840 Any denial decisions must be uh reviewed by a human reviewer. s59 00:06:00.840 --> 00:06:04.160 So, there's cases which you can't fully automate with AI. s60 00:06:04.160 --> 00:06:06.960 So, yeah, all of these are useful design inputs. s61 00:06:06.960 --> 00:06:10.800 So, you want to have these sooner rather than later. s62 00:06:14.520 --> 00:06:18.000 Also good to know of any performance requirements. s63 00:06:18.000 --> 00:06:22.440 Uh do we need sub-millisecond responses or do we have some leeway there? s64 00:06:22.440 --> 00:06:26.040 Uh how much can we spend per month on LLM inference? s65 00:06:26.040 --> 00:06:29.840 Any uptime SLAs to be aware of? s66 00:06:31.080 --> 00:06:37.640 Next, if you're building an AI product, identifying the role of AI uh in your product is helpful. s67 00:06:37.640 --> 00:06:41.120 And I like to think about the role of AI along three dimensions. s68 00:06:41.120 --> 00:06:44.840 First, is AI critical to your product or complementary? s69 00:06:44.840 --> 00:06:54.280 In this case, complementary since um claims are already happening without claim reviews already happening without AI, but they are just slower. s70 00:06:54.720 --> 00:06:57.800 Next, will this be a reactive or proactive system? s71 00:06:57.800 --> 00:07:03.800 Meaning, is it triggered by a user or an event or does it proactively do things? s72 00:07:03.800 --> 00:07:12.200 In this case, it's going to be a reactive system since uh it's triggered by a claims application actually being submitted. s73 00:07:12.200 --> 00:07:17.240 And finally, what's the level of autonomy uh for AI in the application? s74 00:07:17.240 --> 00:07:21.640 So, in our case, the business constraints prescribe human review in certain cases. s75 00:07:21.640 --> 00:07:26.520 So, our system can be semi-autonomous at max. s76 00:07:28.040 --> 00:07:39.320 And finally, as part of defining what you're solving for, you also want to come up with one to two uh success metrics aligned with the business problem so you can capture what success looks like. s77 00:07:39.320 --> 00:07:46.440 A good success metric is specific, measurable, achievable, relevant, and time-bound. s78 00:07:46.440 --> 00:07:50.360 And here's an example of a success metric for our claims review application. s79 00:07:50.360 --> 00:07:59.720 So, it says it says reduce the average processing time for urgent claim review requests from 2 days to 1 hour within 90 days of launch. s80 00:07:59.720 --> 00:08:02.360 So, it's super specific. s81 00:08:02.360 --> 00:08:09.200 It's measurable because it's saying uh the processing time needs to be 1 hour. s82 00:08:09.200 --> 00:08:20.800 Uh it seems achievable because um there'll be AI in the loop, so hopefully any information gathering uh getting an initial recommended recommendation will be faster. s83 00:08:20.800 --> 00:08:32.320 Uh it's very relevant and rooted in the business problem from before, and it's time-bound because it gives you uh a window for when we need to start seeing success. s84 00:08:35.440 --> 00:08:39.039 Okay, so at this point we figured out uh the what. s85 00:08:39.039 --> 00:08:41.680 Let's look at the how. s86 00:08:41.680 --> 00:08:45.320 The first thing you want to think about here is your data strategy. s87 00:08:45.320 --> 00:08:48.680 What does uh what data does your application need? s88 00:08:48.680 --> 00:08:50.640 Do you even have access to it? s89 00:08:50.640 --> 00:08:52.400 Where does it reside? s90 00:08:52.400 --> 00:08:58.080 Uh is it ready for consumption in its raw form, and so on? s91 00:08:58.960 --> 00:09:07.280 So, for our claims review application, these are the data sources that we might need, where they reside, and what their raw access formats might look like. s92 00:09:07.280 --> 00:09:14.640 So, we need uh clinical guidelines probably, uh which describe what care is appropriate for a given condition. s93 00:09:14.640 --> 00:09:22.680 Uh MDB Health's coverage policies that state what MDB Health will and will not pay for. s94 00:09:22.720 --> 00:09:32.720 And finally, the patient's claims history to know any uh past conditions, uh procedures that were performed, when they were performed, and so on. s95 00:09:33.320 --> 00:09:45.840 Let's assume that the clinical guidelines and coverage policies are stored as PDFs in something like Confluence and claims are stored in MongoDB as and when they are processed. s96 00:09:46.680 --> 00:09:49.920 The next thing to think about is data update frequency. s97 00:09:49.920 --> 00:09:52.600 How often do the data sources update? s98 00:09:52.600 --> 00:10:02.760 Now, if you're doing any sort of data processing on these raw data sources and you typically are, these pipelines are going to have to run at the right cadence to keep up. s99 00:10:02.760 --> 00:10:10.880 Otherwise, our system is going to work off of stale information and that's not good, especially in such a high-stakes use case. s100 00:10:11.640 --> 00:10:15.200 So, let's work this out for our claims review system. s101 00:10:15.200 --> 00:10:19.440 Clinical guide These probably update annually. s102 00:10:19.440 --> 00:10:28.160 Uh internal coverage policies may be quarterly and the patient's claim history needs to be updated whenever a new claim is processed. s103 00:10:28.160 --> 00:10:39.600 And in our success metrics, if you recall, uh we've said we want to aim for a 1-hour time frame for adjudicating urgent claims, so these will probably get updated hourly. s104 00:10:42.400 --> 00:10:47.880 Next, can the data be consumed directly by our application or does it need any processing? s105 00:10:47.880 --> 00:11:04.120 For clinical guidelines, which are typically long documents, you probably want to chunk them, embed the chunks, and also extract metadata such as procedure names, publication dates to make it suitable for retrieval using something like vector search or hybrid search. s106 00:11:04.120 --> 00:11:19.120 Same goes for coverage policies, but for patient claims history, which is already well-formatted and stored in MongoDB, we might just need to remove personally identifiable information before passing it to an LLM. s107 00:11:20.680 --> 00:11:26.280 Next, we want to think about what retrieval techniques would be best suited for each of the data sources. s108 00:11:26.280 --> 00:11:32.600 And you kind of already start thinking about this in the data processing step, but let's quickly formalize this. s109 00:11:32.600 --> 00:11:45.720 Now, clinical guidelines and coverage policies are well suited for vector search, but they can contain medical terminologies, such as diagnosis codes, procedure codes, that might not be picked up by vector search. s110 00:11:45.720 --> 00:11:51.720 So, um but these will be picked up by metadata filters or something like keyword search. s111 00:11:51.720 --> 00:12:00.400 So, for these, you'd either want to use vector search with metadata pre-filtering or hybrid search. s112 00:12:01.280 --> 00:12:11.160 For getting the patient's claims history, you can do an exact match on the patient name or ID, whatever the patient identifier is in the system. s113 00:12:12.720 --> 00:12:16.520 Next, you want to think about what the system architecture looks like. s114 00:12:16.520 --> 00:12:20.120 Now, it can be tempting to jump straight to building an agent. s115 00:12:20.120 --> 00:12:31.360 There's so much hype around them, or let a coding agent decide what the system architecture should look like, but you risk ending up with an over-engineered system by doing this. s116 00:12:31.360 --> 00:12:42.000 So, what you instead want to do is start with the simplest design, evaluate it, find gaps during the evaluation, and iterate from there. s117 00:12:42.839 --> 00:12:52.720 To decide what the system should look like, let's first map out how an insurance claim would flow through the system, because this will help us inform the architecture. s118 00:12:52.720 --> 00:12:59.600 So, first, the claim request and any clinical notes from the physician are received by our system. s119 00:12:59.600 --> 00:13:12.120 Then the system needs to retrieve any clinical guidelines and coverage policies that are relevant to the current claim, and also retrieve the patient's claims history. s120 00:13:12.400 --> 00:13:25.280 Then all of this information, along with the claim itself, and any system prompts on how the LLM should generate go about generating the recommendations, all of these are passed to the LLM, s121 00:13:25.280 --> 00:13:34.680 Um, and it uses those to generate uh, a recommendation of whether the claim is claim should be approved or rejected. s122 00:13:34.880 --> 00:13:45.160 Now, if it's a complex case, our uh, system or the LLM at this point should uh, send its initial recommendation um, to a senior physician. s123 00:13:45.160 --> 00:13:54.080 And if the LLM's recommendation is to deny the claim, then it should also be escalated to a human medical reviewer. s124 00:13:54.800 --> 00:14:06.160 And finally, the final decision should be logged to MongoDB, which is where we are storing our claim documents uh, along with a reasoning for why the claim was approved or rejected. s125 00:14:10.000 --> 00:14:19.040 Now, let's talk about some of the most common AI design patterns that we see uh, in AI systems today and see which of those might apply to our system. s126 00:14:19.040 --> 00:14:23.840 So, the first one is rag, uh, short for retrieval augmented generation. s127 00:14:23.840 --> 00:14:26.120 You all probably know it at this point. s128 00:14:26.120 --> 00:14:33.440 Um, basically, in these systems, you simply augment and LLM's pre-trained knowledge with external knowledge sources. s129 00:14:33.440 --> 00:14:37.400 So, information from external knowledge sources. s130 00:14:38.000 --> 00:14:51.880 Then, there's AI agents where you give uh, a single LLM or multiple LLMs in the case of multi-agent systems full autonomy to complete tasks end-to-end with the help of a set of tools. s131 00:14:52.760 --> 00:15:00.680 And then, there's agentic systems, which cover uh, a broad spectrum of semi-autonomous AI systems. s132 00:15:01.560 --> 00:15:13.440 A common architectural pattern for uh, these types of systems are control flows, where LLMs perform certain tasks within the workflow, but the sequence of steps in the workflow itself is predetermined s133 00:15:13.440 --> 00:15:17.160 by design or by code. s134 00:15:18.120 --> 00:15:30.920 Then, there's LLM as a router, where LLMs have limited autonomy in that their job is simply to categorize incoming requests and route them to different downstream workflows. s135 00:15:32.000 --> 00:15:42.160 Then you have human in the loop where an LLM or agent can take certain actions, but it needs human intervention somewhere in the process. s136 00:15:42.720 --> 00:15:44.560 And finally, there's fine-tuning. s137 00:15:44.560 --> 00:16:01.400 So, if you're observing that the LLM's failures are behavioral rather than data or orchestration issues, or if you need superior performance on a domain-specific task, then fine-tuning is a good technique to explore. s138 00:16:02.240 --> 00:16:12.760 I don't have the time to go too deep into these here, but if you'd like to learn more about these design patterns, I'm going to leave this resource here for you to check out later. s139 00:16:18.200 --> 00:16:21.000 So, we've thought through what the workflow of this system should look like. s140 00:16:21.000 --> 00:16:25.040 We've spoken about a few different AI design patterns. s141 00:16:25.040 --> 00:16:32.880 Now, let's apply those and decide what design patterns um we'd need to implement for our system. s142 00:16:32.880 --> 00:16:42.200 So, since we need to retrieve clinical guidelines, coverage policies, and stuff to inform the decision, this is This definitely looks like a rag component. s143 00:16:42.200 --> 00:16:44.680 So, we need some form of rag. s144 00:16:44.680 --> 00:16:56.200 The workflow is pre-structured in that the LLM first provides its recommendation on whether or not a claim should be approved or rejected, and in very specific and well-defined cases, s145 00:16:56.200 --> 00:16:58.640 it needs to escalate claims to human reviewers. s146 00:16:58.640 --> 00:17:02.480 So, this sounds like a controlled workflow. s147 00:17:02.480 --> 00:17:11.839 You could also argue that this looks a little bit like LLM as a router, especially if you use an additional LLM call to decide what a complex case looks like, s148 00:17:11.839 --> 00:17:13.600 unless this is already well-defined. s149 00:17:13.600 --> 00:17:20.560 So, yeah, this could be an LLM as a router, but uh for now let's just go with control flow. s150 00:17:21.839 --> 00:17:24.760 And finally, there's also an element of human in the loop, right? s151 00:17:24.760 --> 00:17:29.400 Since human reviewers are very much a part of the workflow. s152 00:17:30.920 --> 00:17:39.720 And finally, in the design phase, you also want to think about the end user experience and how you'll collect feedback from your users to improve the system. s153 00:17:39.720 --> 00:17:48.040 Here are some questions to ask yourself in this stage to help think through uh what the UX and feedback mechanism should look like. s154 00:17:48.040 --> 00:17:50.920 So, first, what does the system take as input? s155 00:17:50.920 --> 00:17:56.200 In our case, it's going to be a claim request form of some kind. s156 00:17:56.280 --> 00:17:58.720 Then, what does the system produce as output? s157 00:17:58.720 --> 00:18:09.320 So, in our case, it's going to be uh an approval or denial verdict along with an explanation for why uh a claim was approved or rejected. s158 00:18:09.480 --> 00:18:10.640 Where does the system live? s159 00:18:10.640 --> 00:18:12.400 Does it have a standalone app? s160 00:18:12.400 --> 00:18:15.240 Is it embedded in an existing website? s161 00:18:15.240 --> 00:18:17.080 Is it a Slack bot? s162 00:18:17.080 --> 00:18:18.440 Uh and so on. s163 00:18:18.440 --> 00:18:27.240 Uh so, for our application, it's very likely going to be embedded within MDB Health's main website, let's say. s164 00:18:27.240 --> 00:18:29.360 Uh what triggers the system? s165 00:18:29.360 --> 00:18:33.240 So, in our case, very clearly, a claim submission. s166 00:18:33.360 --> 00:18:34.640 What's the human's role? s167 00:18:34.640 --> 00:18:41.800 So, in our case, they are going to review the system's assessment and make the final decision in some cases. s168 00:18:41.960 --> 00:18:44.440 Um how does the system explain itself? s169 00:18:44.440 --> 00:18:56.920 In our case, um it explains itself by providing citations, so quoting uh the clinical guidelines and coverage policies that uh informed the system's assessment. s170 00:18:57.480 --> 00:19:00.320 And finally, how does it give feedback? s171 00:19:00.320 --> 00:19:05.160 So, remember, for our system, um the consumers are medical reviewers. s172 00:19:05.160 --> 00:19:18.320 Um so, they can provide feedback by overriding the verdict of the LLM and also log a reason for why Uh they can also we can maybe also have them flag any irrelevant citations. s173 00:19:18.320 --> 00:19:33.560 So, if their LM is hallucinating um clinical guidelines and policies or yeah, quoting the wrong ones for a particular claim, then we can have them flag those as well. s174 00:19:37.040 --> 00:19:37.520 All right. s175 00:19:37.520 --> 00:19:47.000 So, at this point we've decided what the architecture of our system should look like, but we also need to decide what our stack should look like. s176 00:19:47.000 --> 00:20:00.080 Once again, it might be tempting to have a coding agent recommend what tools to use, but they might not necessarily apply to your system or you might not be able to use them based on the constraints that we identified during s177 00:20:00.080 --> 00:20:01.440 the product requirement stage. s178 00:20:01.440 --> 00:20:10.680 So, we had constraints around models need to be supported by our cloud provider and things like that. s179 00:20:12.280 --> 00:20:22.640 I'm not going to get into this too much since our constraints are well, hypothetical, but I just want to mention that this is the next step and the different tooling decisions to be made s180 00:20:22.640 --> 00:20:25.480 such as what model to use. s181 00:20:25.480 --> 00:20:33.080 There's a bunch of data processing tools out there that have some nice out of the box data processing capabilities. s182 00:20:33.600 --> 00:20:43.680 If retrieval is part of your system, then and you're using things like vector search, then you might need to decide what embedding models to use, what vector database to use. s183 00:20:43.680 --> 00:20:47.920 Um then there's orchestration frameworks and so on. s184 00:20:47.920 --> 00:20:52.760 So, I'm just going to leave this here for reference later on. s185 00:20:56.360 --> 00:20:59.400 Next, let's talk about evaluation and monitoring. s186 00:20:59.400 --> 00:21:03.680 How do you know that what you've built works before and also after you ship? s187 00:21:03.680 --> 00:21:10.760 So, evaluation is before, monitoring is after you ship, and you need both of them. s188 00:21:12.160 --> 00:21:19.880 But before we get into evaluation metrics, let's talk about guardrails because these weren't really really a thing pre-LLMs. s189 00:21:20.000 --> 00:21:33.680 The reason they've become a topic of conversation is because unlike traditional software, LLM-based systems are going to are probabilistic systems and can produce outputs that are unexpected, incorrect, or even harmful. s190 00:21:33.680 --> 00:21:40.040 And guardrails are an attempt to mitigate these risks and ensure that your system behaves within acceptable boundaries. s191 00:21:40.040 --> 00:21:43.960 And what those boundaries are are something you need to define. s192 00:21:43.960 --> 00:21:50.640 So for inputs to your system, the goal is usually to detect invalid, irrelevant, or harmful inputs. s193 00:21:50.640 --> 00:22:00.920 Uh so for our claims review system, an input like write me a poem is irrelevant and our system should reject this claim. s194 00:22:00.920 --> 00:22:08.600 Similarly, for outputs, our goal is to detect invalid, incorrect, or hallucinated, uh or harmful outputs. s195 00:22:08.600 --> 00:22:20.520 So for our claims application, we'll consider uh a response to be invalid if the system doesn't provide citations for uh what informed an approval or denial. s196 00:22:20.520 --> 00:22:23.520 Now guardrail compliance is not the only thing you want to measure. s197 00:22:23.520 --> 00:22:34.120 You want to measure the quality of the responses, uh create and measure domain-specific accuracy metrics, and also general system health. s198 00:22:34.200 --> 00:22:40.040 So for input guardrail compliance, uh for our system, we can calculate a claim rejection rate. s199 00:22:40.040 --> 00:22:47.080 How often is our uh system rejecting claims because they don't obey uh our input guardrails. s200 00:22:47.080 --> 00:22:56.320 Now if it's rejecting too many times, then that's a call for investigation, but if you didn't measure this in the first place, then you won't have anything to investigate, so s201 00:22:56.320 --> 00:22:57.280 uh there we go. s202 00:22:57.280 --> 00:22:58.680 That's why you need metrics. s203 00:22:58.680 --> 00:23:07.160 For output guardrail compliance, we can have something like a missing citation rate, how many times is the LLM producing responses without citations. s204 00:23:07.160 --> 00:23:20.480 For response quality, we can measure faithfulness, which is whether or not the claims approval or rejection is actually rooted in the retrieved uh information like clinical guidelines and policies. s205 00:23:24.920 --> 00:23:29.320 Then you want to define at least one domain or application-specific metric. s206 00:23:29.320 --> 00:23:36.440 So, in our case, something like claim processing time might be a good one because that's really uh our North Star. s207 00:23:36.840 --> 00:23:40.960 And system-level metrics are usually uh around overall system health. s208 00:23:40.960 --> 00:23:49.720 So, uh what's your system's average token cost, token usage, uh average number of turns in a conversation. s209 00:23:49.720 --> 00:23:58.240 Uh so, here just to not have another latency metric, uh I put down something like cost for recommendation. s210 00:24:00.680 --> 00:24:08.960 So, once you evaluated your system and ship your product in front of users, then comes monitoring the system for regressions in real time. s211 00:24:08.960 --> 00:24:23.480 Now, in production, you still want to track the metrics that you used for uh offline evaluation, so the metrics that we talked about, but you might also be able to track additional metrics that can act as uh implicit indicators of your product's s212 00:24:23.480 --> 00:24:31.760 success or like how uh uh how happy users are with your product and overall system health. s213 00:24:31.760 --> 00:24:37.880 For example, for the claims review app, you can track how often a human reviewer overrides the AI verdict. s214 00:24:37.880 --> 00:24:42.440 Uh now, you want this rate to be low uh because that means the system is not doing its job. s215 00:24:42.440 --> 00:24:50.640 So, if you ever start seeing is seeing it increasing about above a certain threshold, then you need to go and investigate. s216 00:24:50.640 --> 00:24:56.680 Or you can also track something like how long does it take for a human to review the AI recommendation. s217 00:24:56.680 --> 00:25:03.400 If it takes too long, which means um this could mean that the responses are very verbose, confusing. s218 00:25:03.400 --> 00:25:10.640 So, yeah, these would be uh some good indicators to track uh once you ship your product to production. s219 00:25:12.520 --> 00:25:19.480 Now, once you have a working prototype that you've evaluated, accuracy looks good, it's time to move to production, right? s220 00:25:19.480 --> 00:25:24.520 Well, what about cost, latency, and reliability? s221 00:25:24.520 --> 00:25:30.120 Uh these constraints become absolutely non-negotiable as you're moving your product to production. s222 00:25:30.120 --> 00:25:39.960 So, you might need to do a few more iterations, evaluations, and testing uh between accuracy looks good and moving to production. s223 00:25:40.520 --> 00:25:44.280 So, let's talk about some optimization techniques for each of these. s224 00:25:44.280 --> 00:25:46.720 So, optimizing for accuracy. s225 00:25:46.720 --> 00:25:55.000 In LLM-based applications, it really comes down to optimizing the information that uh ends up in the LLM's context window. s226 00:25:55.000 --> 00:26:03.200 And here are some techniques for you to refer to later, but let's quickly talk about which of these uh might apply to our claims review application. s227 00:26:03.200 --> 00:26:05.640 So, prompt engineering, of course. s228 00:26:05.640 --> 00:26:10.680 Uh reranking is another good one because uh retrieval is a big component of it. s229 00:26:10.680 --> 00:26:22.560 So, uh a reranker will make sure that the information being retrieved um is surfacing information in the right order of relevance, which is important when working with LLMs. s230 00:26:22.560 --> 00:26:33.360 Uh we are also persisting patient history uh to MongoDB, so our application already has some form of memory, but you could also think about other types of information that might be s231 00:26:33.360 --> 00:26:36.680 uh useful to persist across sessions. s232 00:26:36.680 --> 00:26:41.040 Here are some techniques for cost and latency optimization. s233 00:26:41.440 --> 00:26:49.120 For our use case, semantic caching could be useful to expedite decisions based on uh similar claims from the past. s234 00:26:49.120 --> 00:26:54.600 Uh batch processing, perhaps, to process claims in batches instead of one at a time. s235 00:26:54.600 --> 00:26:58.080 And finally, optimizing for reliability. s236 00:26:58.320 --> 00:27:11.200 Most of these would be useful for our use case since they're mainly solving for API failures, but I want to specifically call out structured outputs, which would be a good one to make sure that our system is always producing s237 00:27:11.200 --> 00:27:16.960 a structured output which contains not just the decision, but also the citations. s238 00:27:17.400 --> 00:27:23.520 This brings me to the end of my talk, but I wanted to leave you with a few takeaways. s239 00:27:24.640 --> 00:27:29.640 Think deeply about your product's requirements before having AI generate any code. s240 00:27:29.640 --> 00:27:35.120 The product spec is the hard part now, it's not the code anymore. s241 00:27:40.280 --> 00:27:47.960 Your latency budget, your cost ceiling, any regulatory requirements, all of these shape every architectural decision downstream. s242 00:27:47.960 --> 00:27:55.920 So, good to identify business and performance constraints before starting to design your application. s243 00:27:57.880 --> 00:28:00.640 Design the simplest system that meets your needs. s244 00:28:00.640 --> 00:28:03.360 Evaluate it and then iterate from there. s245 00:28:03.360 --> 00:28:14.080 The most common mistake I see is over-engineering the solution before knowing what's actually failing or not even evaluating what's actually failing. s246 00:28:14.320 --> 00:28:16.840 And that brings me to evaluation. s247 00:28:16.840 --> 00:28:19.400 Build evaluation in from the start. s248 00:28:19.400 --> 00:28:23.200 You can't improve what you can't measure. s249 00:28:23.320 --> 00:28:38.440 Link to our GenAI cookbook that has a ton of examples of different retrieval technique, agentic design patterns, and some of the evaluation and optimization techniques that I was talking about. s250 00:28:40.640 --> 00:28:44.160 And yeah, thank you so much for taking the time to listen to my talk. s251 00:28:44.160 --> 00:28:46.920 I'll see you at the conference. s252 00:28:47.720 --> 00:28:49.720 conference.