200 Million Patient Interactions Later — Vivek Muppalla, Hippocratic AI

AI Engineer · 20 min · 251 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hope everyone's having a great conference.
00:14Super excited to get started.
00:17Uh how many of you all just quick show of hands have ever gotten a proactive healthcare call from your provider?
00:23Yeah, looks like no one.
00:25Me neither.
00:26Uh I'm uh Vivek of a few hands over there.
00:28I'm Vivek.
00:29I run uh engineering at Hippocratic.
00:32Uh we've built a product that calls patients and can have clinical conversations.
00:36Uh and we're over 200 million conversations in at this point.
00:41Uh here's the reality of like healthcare uh across all of human history, the entire system has been built on scarcity, right?
00:48Not enough clinicians, not enough time, not enough money.
00:51And hence the word triage.
00:53We're always trying to fig- figure out who amongst us is the sickest and deserves to get care and attention.
01:00As an engineer, I like to think about the math.
01:03And for most of human history, the math has just never been in our favor.
01:08Until now.
01:09We're finally at the point where we've seen enough of technology progress.
01:14And we also have AI systems that can do clinically safe conversations.
01:18And the cost for these conversations continues to drop.
01:22So the math now flips.
01:23We can stop rationing, and you don't have to have calls just for the sickest 5%, but you can call everyone.
01:30So that's the mission.
01:31We're trying to build clinically safe abundance for all.
01:35And it's not about replacing clinicians.
01:37In fact, what we're trying to do is help clinicians reach everyone that they couldn't do before because there was simply not enough of them to do this work.
01:46So how are we doing this?
01:49Um our product and our company is predicated on three things.
01:52First, do no harm.
01:55Next, patient first, and access for all.
01:59Um and here's a cool thing.
02:00Uh this is plastered across our walls in our company, uh but it's not just a poster.
02:04Uh this is an oath every employee in the company takes.
02:07I did too on on my first day.
02:09Uh and we built our product uh based on this oath.
02:13Um and this is working at scale.
02:17Uh at this point, we've had 200 million clinical interactions.
02:21We've had zero significant safety incidents.
02:23We've deployed in over 60 plus health systems and have an 8.5 on 10 patient satisfaction rating.
02:30Now, uh I could put an architecture diagram and explain how all of this works.
02:35And don't worry, we'll do that.
02:36But before we get there, uh I want to show you the product in action.
02:40So, we'll have one of our agents interact with the patient so you get a feel for what we're up to.
02:48I'm your AI care manager.
02:50I'm calling to check in on how you're doing with your health monitoring devices.
02:54[music]
02:54Do you have time to chat now?
02:57I I need to know if I'm talking to a human being or a some kind of computer.
03:03You sound like either one.
03:05I understand your concern, miss.
03:08I'm an AI, which stands for artificial intelligence.
03:11Okay.
03:12How are you doing today?
03:14[laughter]
03:16I was tired earlier, but I'm somewhat revived.
03:21I'm incredibly amused.
03:23Delivering at this level of care required building clinical capabilities you won't find
03:26[music]
03:27anywhere else.
03:28On the safety side, accurate drug name recognition,
03:31Could you tell me the name of the medication you're currently on?
03:34Prazosin.
03:36And that's one of them, but I don't remember the name.
03:38Could you help me out by spelling the name of this medication?
03:40[music]
03:41P r o z i n, I n, I guess.
03:44It seems like you might be talking about prazosin.
03:47Does that sound
03:48Yeah.
03:49Vital sign clarification,
03:51Let me say it again it was 121 over 45 91.
03:58Just to confirm, your blood pressure is 121 over 45 and your heart rate is 91, right?
04:04[music]
04:05Right.
04:06Med stoppage detection.
04:08What about the nifedipine tablet?
04:10Were you able to stop taking that one as instructed?
04:13Well, I took it this morning.
04:15Well, no, I took it last night because I didn't have the other prescription.
04:20And I don't think that would keep my blood pressure down.
04:23I understand your concern about your blood pressure and it's great that you're thinking ahead.
04:28I would recommend that you do talk to your primary care doctor about the nifedipine and get their guidance on what to do.
04:34And real-time nurse escalation when clinically necessary.
04:38How are you feeling?
04:39Okay, just a little tired and weak.
04:43It's quite common to feel this way after being in the hospital.
04:46Could you tell me more about it?
04:48My legs feel weak when I walk.
04:50Got it.
04:51Along with the weakness in your legs, have you noticed any chest discomfort or difficulty breathing?
04:57I just was this was having shortness of breath for a while.
05:00I'm so sorry to hear that you're experiencing shortness of breath.
05:04Given your symptoms, it would be best to have a nurse speak with you immediately.
05:10Yeah, so that's our sneak peek into our product and so we'll get into why this is hard and why a generic stack doesn't work for us.
05:18Okay, so if I were to strip out all of the complexity around the audio and speaking back, it boils down to two things, intelligence and latency, right?
05:27On the top left, you see some of our ever-evolving smart intelligent models.
05:33These models are fantastic at what they do.
05:35They're clinically safe.
05:36But unfortunately, many of these models take tens of seconds to respond, sometimes over a minute.
05:42And that's completely useless when we're trying to have a two-way conversation on a telephone.
05:48On the bottom right, we have models that are fantastic.
05:52They're really fast, but then they're just not at the same level of clinical accuracy for us to have a safe conversation.
05:59When we started building out this product, where we wanted to be was on the top right corner.
06:03And in order to accomplish this, what we had to do was build a vertically integrated stack ground up bit by bit, optimizing every part of the stock stack.
06:13When we started out building our own models, we also had several seconds of like latency.
06:18But hundreds of optimizations later, where we landed at was a product that is insanely fast, but also doesn't lose its intelligence.
06:28And we continue to benchmark all of this consistently.
06:31All of these results are on our website and more.
06:34Um and there's three typical ways in which folks build uh voice systems.
06:39One, you can use an ensemble of models, or you use cascaded models, or you use speech-to-speech real-time models.
06:46Again, all of these models and architectures are great at different things.
06:51As an example, let me take two specific benchmarks among so many hundreds that are critical for our workflows.
06:58Uh lab results check and IVR navigation.
07:01Now, these are things that you don't typically hear about or see in the most common benchmarks.
07:06And most of the generic models out there don't perform well at those.
07:10And we needed our models and our product to be over 99% accuracy on those specific benchmarks amongst other things.
07:17Uh and that's one of the key reasons why we continued down investing our own stack.
07:25And latency is key to this entire system.
07:28So, every time we work on an optimization, we buy back some latency.
07:32And we just don't bank that latency.
07:34We use that extra gap now to pack more intelligence into the overall system, such that we can have a more reliable conversation with the patient.
07:44Um and then we do that, we go back, work on more optimization, and that flywheel compounds.
07:49So, what seems like a tug-of-war between latency and intelligence for us is a compounding flywheel.
07:55So, what does this entire machine look like?
07:58Uh this is Polaris.
07:59This is our constellation architecture.
08:01Left to right, you have the system that hears.
08:04The middle is the brain that reasons.
08:07The right is the system that talks back.
08:10And all of this round trip needs to be really fast.
08:13So, on the left-hand side, what you're seeing are collection of models uh that are used for speech detection.
08:19So, everything from bilingual switching to background noise detection um to contextual understanding of the conversation.
08:27The brain isn't a singular model.
08:29We in fact run 31 models at any given point of time for every conversation.
08:35So, we have one central model that's hard like have a handling the conversation, where we have other 30 specialist models, everything from labs to medications to scheduling uh that are feeding input into this model.
08:48And then finally, on the output side, we have a custom personality, voice, HD quality, and clinical documentation engine that make sure the patient gets back all of the right information,
09:01and all of the same information flows back into the health system.
09:06The reason we have the system is because we see a singular model being as like one point of failure, and that's just unacceptable for a patient conversation.
09:15Uh this architecture gives us the redundancy and safety that we otherwise couldn't get to.
09:19And the patient never can tell the different.
09:21It's pretty seamless from their perspective.
09:24Let me double click into some of these systems for you.
09:27Uh first, we'll talk about our uh ASR system.
09:31So, the real world, as all of us know, is fairly loud and noisy, uh but most of the audio benchmarks get recorded in a quiet room.
09:39And something we learned the hard way was most of what looks like model reasoning failures end up actually being model mishearing things, right?
09:49So, the Spanish C or yes gets transcribed as the alphabet C, or you could have an Arabic drug name that gets over 30% in inaccuracy in terms of like word error rate.
10:00So, we had to build a system to to combat all of these challenges.
10:05A typical speech to text system just takes in the audio and then outputs some text.
10:09So, ours is a decoder-only audio LLM system.
10:14So, we take in the audio, but in addition to that, we're giving it two additional pieces of data.
10:19One is the context around the conversation thus far, and second, it's the domain knowledge around that entire conversation.
10:26And that is sort of like the trick that makes this work for us.
10:30So, under the hood, what's happening?
10:32So, we have First, we have a encoder, which is a year.
10:35So, we took an open-source Whisper V3 large turbo model and fine-tuned that on millions of clinical conversations to get to the accuracy we need.
10:44Next, we have a conformer projector.
10:47What this does is takes the audio and compacts it and projects it into tokens that the language model would understand.
10:54But, the key over here is it also maintains all the prosody.
10:58So, all the pauses and the stresses are maintained.
11:01So, the model hears not just the what, but also the how.
11:06Next, we're passing in the context around, let's say, the medications this patient's taking in, or the task at hand.
11:13Are we trying to fill in a form?
11:15What kind of like details that need to go in there and the policy?
11:19And how this helps is now when a patient mentions a medication name, we aren't guessing from like an infinite list of medications, but we have the chance to optimize around a finite list, and that helps in getting the word error rate down.
11:34Next, we also have contextual biasing as a part of like the training.
11:38So, during the training process, uh we have millions of these synthetic patients with their addresses, phone numbers, uh and other details uh that are fed into the training process.
11:50So, as a concrete example, let's say someone mentions their address on a conversation as 1,100 Geary Boulevard.
11:56Now, that specific utterance is ripe for phonetic garbage for most ASR systems, but given we're feeding that as additional context to the audio LLM, we have a high degree of accuracy and are able to get to the right transcription,
12:11which then feeds into the brain accurately.
12:14Further, what we see is for most clinical conversations, many of the patient responses are mono mono words, right?
12:22Uh and those often get transcribed incorrectly.
12:24A now becomes a no, or a five becomes a fine.
12:28And in a patient conversation, that's catastrophic.
12:31So, what we do is a secondary round of scoring, specifically when we see a patient has said only a singular word, and again, we use the context of the entire conversation to do this.
12:41So, all of these like help us bring down the medical word accuracy rate by over uh 50% um from what we see as standard off-the-shelf models.
12:51In addition to that, uh the specific system we've built allows us to uh check in the latency, and at a P99, it's three times faster than every other system out there.
13:03Next, let's look into the constellation, uh the brain.
13:07So, as I said, we have 31 models live running in uh at any given point of time.
13:12So, how do we do this without falling apart on latency?
13:16Uh the answer is a bit counterintuitive.
13:17We actually run every single one of these models in parallel.
13:20Uh but the trick over here is uh every single model does a really quick check to see if they even need to say something on this conversation, right?
13:29So, every specialist first decides, "Hey, do I need to speak?"
13:33If not, it's a short circuit, and that's what helps us keep us in the budget.
13:38So, that's the synchronous part.
13:39In addition to that, we also have asynchronous run models running in the background that are doing verification.
13:46And let's take two specific examples to walk through how we use these models.
13:50Again, going back to our medication example, for example, if the patient talks about ibuprofen or is talking about another medication, that's when the med engine specialist will kick in, and it will say, "Oh, I have something to say over here."
14:04Main model will pass in that context to the main model.
14:06The main model with a lung along with everything else that already knows around the goal and the conversation will take this input from the specialist and steer the conversation as appropriate.
14:16Next, like many other agentic systems, we heavily use like tool calls, and I think pretty much every one of us has experienced tool calls have innumerable failure modes.
14:27So, we have these like verifiers in the background that are running to make sure all the parameters into the tool calls and the responses and the tool calls are accurate.
14:37And for use cases like scheduling, this is what's gotten us to get to a 3949 accuracy.
14:42And finally, for certain use cases, we're also running these verifiers offline where possible.
14:48So, again, going back to scheduling, sometimes we have the luxury of time to go back and check to make sure if there was an inaccurate appointment, we can course correct that on the back end or call a patient back and apologize.
15:01All of these changes would be meaningless if we didn't optimize our inference stack, and we have a fantastic research and engineering team that lives and breathes on this problem.
15:12And for inference itself, quality is our constraint, and speed is the work.
15:16So, we can never compromise on the quality of the output.
15:20Every speed optimization has to be lossless.
15:24We've worked on like many many of these like optimizations.
15:26Let me talk you through three specific ones that we found to be the most meaningful.
15:31One is four-bit quantization.
15:33So basically what we've done is we've shrunk down the math for model computation to be from 16-bit to four-bit, again lossless, and this has like really helped us with overall latency.
15:44The next is speculative decoding.
15:46So we have a smaller model that's generating the tokens ahead of time, and then the main model just checks this in one go, and this also has greatly improved latency for us.
15:56Finally, a KV cache compression system.
15:59So as these conversations get like really long in length, we've figured out a way to keep a large chunk of these conversations warm on cache, giving us an over 96%
16:09hit rate, and then also helps us with the prefill portion, which goes 18 times faster.
16:14So this style of across-the-board board optimizations is what's getting us ahead during the product development phase.
16:24Now, evals are obviously key as we've seen across many of these talks.
16:29So some quick math, right?
16:31For example, let's take the scheduling use case again.
16:34With over 10,000 calls a day, a one 1% failure rate like sounds all right.
16:39Most agentic systems would claim that's great for them.
16:45For us, even the 99% is pretty bad because 1% error means 100 people a day are going to get the wrong appointment type, and it's physically bad because they'll show up
16:56on the wrong date, wrong time, but worse off, they could actually be missing a critical appointment, and that's unacceptable from our standpoint.
17:03So it's not just an annoyance, right?
17:05Um And the challenge over here is in order to catch this like statistically, the math like is like incredibly challenging, right?
17:13You need about 450 tests to be 99% sure that you can catch this like 1% error rate Um and 1900 tests to be able to see that you've caught it like 10 times.
17:24So, you can't purely rely on synthetic data from our experience to be able to get to the scale of accuracy.
17:31So, we use a combination of human beings and synthetic data.
17:35So, we have over 7,000 trained clinicians on our platform who are continuously helping us like evaluate our platform.
17:41They've done over 700,000 close to 800,000 clinical conversations to help us figure out the accuracy of our system and we do this on a continuous basis.
17:53Finally, coming to safety.
17:56We've shipped five versions of our product thus far and each one's gotten better compared to the previous one.
18:03So, we don't just grade the output of our models on the correctness or validity of it.
18:09So, we use the same scale that as they do for human beings.
18:13So, they're graded across correctness, no harm, minor harm, severe harm, and death.
18:18And across five generations where we're at with our Polaris system is a 99.89% accuracy with respect to like no harm.
18:29And humans on the same rubric are at about 81%.
18:32It's not because like we're terrible, but AI systems don't get tired and unfortunately we do.
18:38And we also don't have the luxury of having 30 plus supervisors helping us at any given point of time.
18:45And finally, safe is not enough, right?
18:48A key part of the product is the empathy.
18:51For human beings to open up to AI systems, we needed to make sure the voice and how we actually lead these conversations are empathetic.
19:00And as with most other things during our product journey, we found that there weren't actually accurate or good benchmarks to measure the empathy of these systems.
19:10So, we actually built one which is called heart.
19:12The paper for this is public.
19:14Encourage you guys to like read it.
19:16And we we had to make sure the end system was not just safe, but also empathetic so that people would actually prefer it and use it.
19:27And finally, all of this like comes down to the people.
19:30We wouldn't be here if we didn't invest in people from an AI native perspective.
19:36Everything in our company across the last couple of years has been transformed our hiring processes to how we use like tokens.
19:43We still have unlimited tokens for everyone at our company.
19:46We also have two great programs for anyone who comes into Hippocratic.
19:50For early career engineers, we have an agent deployment residency program where they're trained to write some of the best like health care safest agents.
19:59And then for experienced software engineers, we have an AI residency program where we help you start like training models and build the next great system along with us.
20:08So yeah, we're told you got to pick two of these options around quality, speed, and safety.
20:14We didn't and we decided to go with all of them and we're hoping the rest of you should come and build with us.
20:20Thank you.