AI System Design: From Idea to Production - Apoorva Joshi, MongoDB

AI Engineer · 28 min · 252 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:02Hi everyone.
00:02I'm Apurva and I'm a data scientist turned developer advocate currently at MongoDB.
00:08I spent the first of my career building machine learning applications for various cybersecurity use cases and I now use that applied machine learning knowledge to help AI builders successfully build AI applications
00:21with MongoDB and Voyage AI.
00:25In this talk you'll learn how to think through building AI systems end-to-end from idea to production.
00:31We'll take a real-world use case and walk through all the steps of designing it and hopefully in the end you'll walk away with a repeatable framework that you can apply to
00:41any AI system you design.
00:46Now, you might be thinking in the age of AI do we even need to think about what we build?
00:52Just wipe code it and ship it, right?
00:54But that's actually where I want to start.
00:57Now, here's the problem with that.
01:01Wipe coding works great when you're building for fun, the stakes are low, and you can easily eyeball whether the output of what you're building is right.
01:09But the moment you're building something real, something other people depend on, something with real consequences, just ship it is actually kind of dangerous.
01:19Uh and it's not just me saying it, folks from Anthropic and OpenAI who are so bullish on AI coding are saying the same thing.
01:25These are actually quotes from their talks over the past few months.
01:30Specs are the new code.
01:32The art is in defining the product requirements, the system design, and evaluate criteria so you can be confident that your AI coding buddies are building the right thing.
01:42And the rest of the talk is just about that.
01:44How do we do this well?
01:49It's useful to think about it as a framework.
01:52Four phases.
01:53You start with product requirements.
01:55What are you actually building, for whom, and what are the constraints?
01:59Then system design, the data, the architecture, the patterns that actually help you meet these requirements.
02:07Then comes evaluation and monitoring.
02:09How do you know what's being built actually works before and also after you ship?
02:15And finally, how do you optimize for cost, not just for accuracy, but also for cost, latency, and reliability before you ship and or as you find gaps in production.
02:30Instead of talking abstractly through a framework, I thought why not apply it to a real-world use case so you can see how exactly each decision flows from one stage of the framework to the next.
02:42So let's take the example of a health insurance claims review system.
02:49So adjudicating health insurance claims has historically been an extremely manual process where human medical reviewers have to cross-reference clinical guidelines, insurance coverage policies, patient history, and stuff to decide whether or not
03:06a medical treatment, procedure, or medication is covered by a patient's health insurance policy.
03:12Now, if you live somewhere that has free health care, even there there's things that are covered and things that are not by your health care system.
03:21So you're still subject to some version of this process.
03:27Now, our goal with building this system is to see if we can improve or simplify the experience for medical reviewers with the help of AI.
03:38So let's see what the product requirements stage of the framework for this application might look like.
03:45So first thing to do is quantify the business problem.
03:49The business problem should focus on something specific, uh clearly state who the users of the application are, uh the current state of things, and quantify the users' pain point.
04:02It should not prescribe what the system is going to be, whether it's going to be an agent, a multi-agent system, something else.
04:08Uh that will come later.
04:14So, here's what the business problem for our claims review system could look like.
04:18Medical reviewers at MDB Health spend an average of 2 days processing claim review requests, which is four times the industry standard for non-urgent cases, and 12 times the industry standard for urgent ones.
04:31Delays at this scale postpone patient care, uh particularly for time-sensitive treatment.
04:38So, as you can see, this business problem is user-specific.
04:41It says this is meant for medical reviewers.
04:44Uh it states the current state, which is like this manual process that leads to them spending 2 days processing uh claim reviews.
04:54It's measurable because it states some baselines.
04:57Uh it's solution-agnostic.
04:59It doesn't really tell you uh what the system should look like, uh and it's very focused on a specific problem.
05:10Next, you want to gather any business constraints for the application.
05:14So, any regulatory uh compliance requirements, any constraints on data leaving your organization, procurement constraints, meaning are certain vendors uh not approved for use within the organization.
05:28Uh all of these would be good to know before you even start designing the system.
05:33So, for our application, say these are some of the business constraints.
05:36Patient data has to stay within the approved cloud environment.
05:40Um only models available on the approved cloud can be used.
05:44Um there's some use cases where human review is required.
05:49So, any complex use cases require a senior physician's review.
05:54Um what else?
05:56Any denial decisions must be uh reviewed by a human reviewer.
06:00So, there's cases which you can't fully automate with AI.
06:04So, yeah, all of these are useful design inputs.
06:06So, you want to have these sooner rather than later.
06:14Also good to know of any performance requirements.
06:18Uh do we need sub-millisecond responses or do we have some leeway there?
06:22Uh how much can we spend per month on LLM inference?
06:26Any uptime SLAs to be aware of?
06:31Next, if you're building an AI product, identifying the role of AI uh in your product is helpful.
06:37And I like to think about the role of AI along three dimensions.
06:41First, is AI critical to your product or complementary?
06:44In this case, complementary since um claims are already happening without claim reviews already happening without AI, but they are just slower.
06:54Next, will this be a reactive or proactive system?
06:57Meaning, is it triggered by a user or an event or does it proactively do things?
07:03In this case, it's going to be a reactive system since uh it's triggered by a claims application actually being submitted.
07:12And finally, what's the level of autonomy uh for AI in the application?
07:17So, in our case, the business constraints prescribe human review in certain cases.
07:21So, our system can be semi-autonomous at max.
07:28And finally, as part of defining what you're solving for, you also want to come up with one to two uh success metrics aligned with the business problem so you can capture what success looks like.
07:39A good success metric is specific, measurable, achievable, relevant, and time-bound.
07:46And here's an example of a success metric for our claims review application.
07:50So, it says it says reduce the average processing time for urgent claim review requests from 2 days to 1 hour within 90 days of launch.
07:59So, it's super specific.
08:02It's measurable because it's saying uh the processing time needs to be 1 hour.
08:09Uh it seems achievable because um there'll be AI in the loop, so hopefully any information gathering uh getting an initial recommended recommendation will be faster.
08:20Uh it's very relevant and rooted in the business problem from before, and it's time-bound because it gives you uh a window for when we need to start seeing success.
08:35Okay, so at this point we figured out uh the what.
08:39Let's look at the how.
08:41The first thing you want to think about here is your data strategy.
08:45What does uh what data does your application need?
08:48Do you even have access to it?
08:50Where does it reside?
08:52Uh is it ready for consumption in its raw form, and so on?
08:58So, for our claims review application, these are the data sources that we might need, where they reside, and what their raw access formats might look like.
09:07So, we need uh clinical guidelines probably, uh which describe what care is appropriate for a given condition.
09:14Uh MDB Health's coverage policies that state what MDB Health will and will not pay for.
09:22And finally, the patient's claims history to know any uh past conditions, uh procedures that were performed, when they were performed, and so on.
09:33Let's assume that the clinical guidelines and coverage policies are stored as PDFs in something like Confluence and claims are stored in MongoDB as and when they are processed.
09:46The next thing to think about is data update frequency.
09:49How often do the data sources update?
09:52Now, if you're doing any sort of data processing on these raw data sources and you typically are, these pipelines are going to have to run at the right cadence to keep up.
10:02Otherwise, our system is going to work off of stale information and that's not good, especially in such a high-stakes use case.
10:11So, let's work this out for our claims review system.
10:15Clinical guide These probably update annually.
10:19Uh internal coverage policies may be quarterly and the patient's claim history needs to be updated whenever a new claim is processed.
10:28And in our success metrics, if you recall, uh we've said we want to aim for a 1-hour time frame for adjudicating urgent claims, so these will probably get updated hourly.
10:42Next, can the data be consumed directly by our application or does it need any processing?
10:47For clinical guidelines, which are typically long documents, you probably want to chunk them, embed the chunks, and also extract metadata such as procedure names, publication dates to make it suitable for retrieval using something like vector search or hybrid search.
11:04Same goes for coverage policies, but for patient claims history, which is already well-formatted and stored in MongoDB, we might just need to remove personally identifiable information before passing it to an LLM.
11:20Next, we want to think about what retrieval techniques would be best suited for each of the data sources.
11:26And you kind of already start thinking about this in the data processing step, but let's quickly formalize this.
11:32Now, clinical guidelines and coverage policies are well suited for vector search, but they can contain medical terminologies, such as diagnosis codes, procedure codes, that might not be picked up by vector search.
11:45So, um but these will be picked up by metadata filters or something like keyword search.
11:51So, for these, you'd either want to use vector search with metadata pre-filtering or hybrid search.
12:01For getting the patient's claims history, you can do an exact match on the patient name or ID, whatever the patient identifier is in the system.
12:12Next, you want to think about what the system architecture looks like.
12:16Now, it can be tempting to jump straight to building an agent.
12:20There's so much hype around them, or let a coding agent decide what the system architecture should look like, but you risk ending up with an over-engineered system by doing this.
12:31So, what you instead want to do is start with the simplest design, evaluate it, find gaps during the evaluation, and iterate from there.
12:42To decide what the system should look like, let's first map out how an insurance claim would flow through the system, because this will help us inform the architecture.
12:52So, first, the claim request and any clinical notes from the physician are received by our system.
12:59Then the system needs to retrieve any clinical guidelines and coverage policies that are relevant to the current claim, and also retrieve the patient's claims history.
13:12Then all of this information, along with the claim itself, and any system prompts on how the LLM should generate go about generating the recommendations, all of these are passed to the LLM,
13:25Um, and it uses those to generate uh, a recommendation of whether the claim is claim should be approved or rejected.
13:34Now, if it's a complex case, our uh, system or the LLM at this point should uh, send its initial recommendation um, to a senior physician.
13:45And if the LLM's recommendation is to deny the claim, then it should also be escalated to a human medical reviewer.
13:54And finally, the final decision should be logged to MongoDB, which is where we are storing our claim documents uh, along with a reasoning for why the claim was approved or rejected.
14:10Now, let's talk about some of the most common AI design patterns that we see uh, in AI systems today and see which of those might apply to our system.
14:19So, the first one is rag, uh, short for retrieval augmented generation.
14:23You all probably know it at this point.
14:26Um, basically, in these systems, you simply augment and LLM's pre-trained knowledge with external knowledge sources.
14:33So, information from external knowledge sources.
14:38Then, there's AI agents where you give uh, a single LLM or multiple LLMs in the case of multi-agent systems full autonomy to complete tasks end-to-end with the help of a set of tools.
14:52And then, there's agentic systems, which cover uh, a broad spectrum of semi-autonomous AI systems.
15:01A common architectural pattern for uh, these types of systems are control flows, where LLMs perform certain tasks within the workflow, but the sequence of steps in the workflow itself is predetermined
15:13by design or by code.
15:18Then, there's LLM as a router, where LLMs have limited autonomy in that their job is simply to categorize incoming requests and route them to different downstream workflows.
15:32Then you have human in the loop where an LLM or agent can take certain actions, but it needs human intervention somewhere in the process.
15:42And finally, there's fine-tuning.
15:44So, if you're observing that the LLM's failures are behavioral rather than data or orchestration issues, or if you need superior performance on a domain-specific task, then fine-tuning is a good technique to explore.
16:02I don't have the time to go too deep into these here, but if you'd like to learn more about these design patterns, I'm going to leave this resource here for you to check out later.
16:18So, we've thought through what the workflow of this system should look like.
16:21We've spoken about a few different AI design patterns.
16:25Now, let's apply those and decide what design patterns um we'd need to implement for our system.
16:32So, since we need to retrieve clinical guidelines, coverage policies, and stuff to inform the decision, this is This definitely looks like a rag component.
16:42So, we need some form of rag.
16:44The workflow is pre-structured in that the LLM first provides its recommendation on whether or not a claim should be approved or rejected, and in very specific and well-defined cases,
16:56it needs to escalate claims to human reviewers.
16:58So, this sounds like a controlled workflow.
17:02You could also argue that this looks a little bit like LLM as a router, especially if you use an additional LLM call to decide what a complex case looks like,
17:11unless this is already well-defined.
17:13So, yeah, this could be an LLM as a router, but uh for now let's just go with control flow.
17:21And finally, there's also an element of human in the loop, right?
17:24Since human reviewers are very much a part of the workflow.
17:30And finally, in the design phase, you also want to think about the end user experience and how you'll collect feedback from your users to improve the system.
17:39Here are some questions to ask yourself in this stage to help think through uh what the UX and feedback mechanism should look like.
17:48So, first, what does the system take as input?
17:50In our case, it's going to be a claim request form of some kind.
17:56Then, what does the system produce as output?
17:58So, in our case, it's going to be uh an approval or denial verdict along with an explanation for why uh a claim was approved or rejected.
18:09Where does the system live?
18:10Does it have a standalone app?
18:12Is it embedded in an existing website?
18:15Is it a Slack bot?
18:17Uh and so on.
18:18Uh so, for our application, it's very likely going to be embedded within MDB Health's main website, let's say.
18:27Uh what triggers the system?
18:29So, in our case, very clearly, a claim submission.
18:33What's the human's role?
18:34So, in our case, they are going to review the system's assessment and make the final decision in some cases.
18:41Um how does the system explain itself?
18:44In our case, um it explains itself by providing citations, so quoting uh the clinical guidelines and coverage policies that uh informed the system's assessment.
18:57And finally, how does it give feedback?
19:00So, remember, for our system, um the consumers are medical reviewers.
19:05Um so, they can provide feedback by overriding the verdict of the LLM and also log a reason for why Uh they can also we can maybe also have them flag any irrelevant citations.
19:18So, if their LM is hallucinating um clinical guidelines and policies or yeah, quoting the wrong ones for a particular claim, then we can have them flag those as well.
19:37All right.
19:37So, at this point we've decided what the architecture of our system should look like, but we also need to decide what our stack should look like.
19:47Once again, it might be tempting to have a coding agent recommend what tools to use, but they might not necessarily apply to your system or you might not be able to use them based on the constraints that we identified during
20:00the product requirement stage.
20:01So, we had constraints around models need to be supported by our cloud provider and things like that.
20:12I'm not going to get into this too much since our constraints are well, hypothetical, but I just want to mention that this is the next step and the different tooling decisions to be made
20:22such as what model to use.
20:25There's a bunch of data processing tools out there that have some nice out of the box data processing capabilities.
20:33If retrieval is part of your system, then and you're using things like vector search, then you might need to decide what embedding models to use, what vector database to use.
20:43Um then there's orchestration frameworks and so on.
20:47So, I'm just going to leave this here for reference later on.
20:56Next, let's talk about evaluation and monitoring.
20:59How do you know that what you've built works before and also after you ship?
21:03So, evaluation is before, monitoring is after you ship, and you need both of them.
21:12But before we get into evaluation metrics, let's talk about guardrails because these weren't really really a thing pre-LLMs.
21:20The reason they've become a topic of conversation is because unlike traditional software, LLM-based systems are going to are probabilistic systems and can produce outputs that are unexpected, incorrect, or even harmful.
21:33And guardrails are an attempt to mitigate these risks and ensure that your system behaves within acceptable boundaries.
21:40And what those boundaries are are something you need to define.
21:43So for inputs to your system, the goal is usually to detect invalid, irrelevant, or harmful inputs.
21:50Uh so for our claims review system, an input like write me a poem is irrelevant and our system should reject this claim.
22:00Similarly, for outputs, our goal is to detect invalid, incorrect, or hallucinated, uh or harmful outputs.
22:08So for our claims application, we'll consider uh a response to be invalid if the system doesn't provide citations for uh what informed an approval or denial.
22:20Now guardrail compliance is not the only thing you want to measure.
22:23You want to measure the quality of the responses, uh create and measure domain-specific accuracy metrics, and also general system health.
22:34So for input guardrail compliance, uh for our system, we can calculate a claim rejection rate.
22:40How often is our uh system rejecting claims because they don't obey uh our input guardrails.
22:47Now if it's rejecting too many times, then that's a call for investigation, but if you didn't measure this in the first place, then you won't have anything to investigate, so
22:56uh there we go.
22:57That's why you need metrics.
22:58For output guardrail compliance, we can have something like a missing citation rate, how many times is the LLM producing responses without citations.
23:07For response quality, we can measure faithfulness, which is whether or not the claims approval or rejection is actually rooted in the retrieved uh information like clinical guidelines and policies.
23:24Then you want to define at least one domain or application-specific metric.
23:29So, in our case, something like claim processing time might be a good one because that's really uh our North Star.
23:36And system-level metrics are usually uh around overall system health.
23:40So, uh what's your system's average token cost, token usage, uh average number of turns in a conversation.
23:49Uh so, here just to not have another latency metric, uh I put down something like cost for recommendation.
24:00So, once you evaluated your system and ship your product in front of users, then comes monitoring the system for regressions in real time.
24:08Now, in production, you still want to track the metrics that you used for uh offline evaluation, so the metrics that we talked about, but you might also be able to track additional metrics that can act as uh implicit indicators of your product's
24:23success or like how uh uh how happy users are with your product and overall system health.
24:31For example, for the claims review app, you can track how often a human reviewer overrides the AI verdict.
24:37Uh now, you want this rate to be low uh because that means the system is not doing its job.
24:42So, if you ever start seeing is seeing it increasing about above a certain threshold, then you need to go and investigate.
24:50Or you can also track something like how long does it take for a human to review the AI recommendation.
24:56If it takes too long, which means um this could mean that the responses are very verbose, confusing.
25:03So, yeah, these would be uh some good indicators to track uh once you ship your product to production.
25:12Now, once you have a working prototype that you've evaluated, accuracy looks good, it's time to move to production, right?
25:19Well, what about cost, latency, and reliability?
25:24Uh these constraints become absolutely non-negotiable as you're moving your product to production.
25:30So, you might need to do a few more iterations, evaluations, and testing uh between accuracy looks good and moving to production.
25:40So, let's talk about some optimization techniques for each of these.
25:44So, optimizing for accuracy.
25:46In LLM-based applications, it really comes down to optimizing the information that uh ends up in the LLM's context window.
25:55And here are some techniques for you to refer to later, but let's quickly talk about which of these uh might apply to our claims review application.
26:03So, prompt engineering, of course.
26:05Uh reranking is another good one because uh retrieval is a big component of it.
26:10So, uh a reranker will make sure that the information being retrieved um is surfacing information in the right order of relevance, which is important when working with LLMs.
26:22Uh we are also persisting patient history uh to MongoDB, so our application already has some form of memory, but you could also think about other types of information that might be
26:33uh useful to persist across sessions.
26:36Here are some techniques for cost and latency optimization.
26:41For our use case, semantic caching could be useful to expedite decisions based on uh similar claims from the past.
26:49Uh batch processing, perhaps, to process claims in batches instead of one at a time.
26:54And finally, optimizing for reliability.
26:58Most of these would be useful for our use case since they're mainly solving for API failures, but I want to specifically call out structured outputs, which would be a good one to make sure that our system is always producing
27:11a structured output which contains not just the decision, but also the citations.
27:17This brings me to the end of my talk, but I wanted to leave you with a few takeaways.
27:24Think deeply about your product's requirements before having AI generate any code.
27:29The product spec is the hard part now, it's not the code anymore.
27:40Your latency budget, your cost ceiling, any regulatory requirements, all of these shape every architectural decision downstream.
27:47So, good to identify business and performance constraints before starting to design your application.
27:57Design the simplest system that meets your needs.
28:00Evaluate it and then iterate from there.
28:03The most common mistake I see is over-engineering the solution before knowing what's actually failing or not even evaluating what's actually failing.
28:14And that brings me to evaluation.
28:16Build evaluation in from the start.
28:19You can't improve what you can't measure.
28:23Link to our GenAI cookbook that has a ton of examples of different retrieval technique, agentic design patterns, and some of the evaluation and optimization techniques that I was talking about.
28:40And yeah, thank you so much for taking the time to listen to my talk.
28:44I'll see you at the conference.
28:47conference.