WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=AN65uc645mE NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/AN65uc645mE.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.920 --> 00:00:14.880 Hope everyone's having a great conference. s3 00:00:14.880 --> 00:00:17.000 Super excited to get started. s4 00:00:17.000 --> 00:00:23.960 Uh how many of you all just quick show of hands have ever gotten a proactive healthcare call from your provider? s5 00:00:23.960 --> 00:00:25.040 Yeah, looks like no one. s6 00:00:25.040 --> 00:00:26.200 Me neither. s7 00:00:26.200 --> 00:00:28.720 Uh I'm uh Vivek of a few hands over there. s8 00:00:28.720 --> 00:00:29.400 I'm Vivek. s9 00:00:29.400 --> 00:00:32.040 I run uh engineering at Hippocratic. s10 00:00:32.040 --> 00:00:36.960 Uh we've built a product that calls patients and can have clinical conversations. s11 00:00:36.960 --> 00:00:41.000 Uh and we're over 200 million conversations in at this point. s12 00:00:41.000 --> 00:00:48.240 Uh here's the reality of like healthcare uh across all of human history, the entire system has been built on scarcity, right? s13 00:00:48.240 --> 00:00:51.840 Not enough clinicians, not enough time, not enough money. s14 00:00:51.840 --> 00:00:53.560 And hence the word triage. s15 00:00:53.560 --> 00:01:00.880 We're always trying to fig- figure out who amongst us is the sickest and deserves to get care and attention. s16 00:01:00.880 --> 00:01:03.440 As an engineer, I like to think about the math. s17 00:01:03.440 --> 00:01:08.560 And for most of human history, the math has just never been in our favor. s18 00:01:08.560 --> 00:01:09.760 Until now. s19 00:01:09.760 --> 00:01:14.240 We're finally at the point where we've seen enough of technology progress. s20 00:01:14.240 --> 00:01:18.800 And we also have AI systems that can do clinically safe conversations. s21 00:01:18.800 --> 00:01:22.160 And the cost for these conversations continues to drop. s22 00:01:22.160 --> 00:01:23.840 So the math now flips. s23 00:01:23.840 --> 00:01:30.600 We can stop rationing, and you don't have to have calls just for the sickest 5%, but you can call everyone. s24 00:01:30.600 --> 00:01:31.960 So that's the mission. s25 00:01:31.960 --> 00:01:35.160 We're trying to build clinically safe abundance for all. s26 00:01:35.160 --> 00:01:37.880 And it's not about replacing clinicians. s27 00:01:37.880 --> 00:01:46.560 In fact, what we're trying to do is help clinicians reach everyone that they couldn't do before because there was simply not enough of them to do this work. s28 00:01:46.560 --> 00:01:49.200 So how are we doing this? s29 00:01:49.200 --> 00:01:52.960 Um our product and our company is predicated on three things. s30 00:01:52.960 --> 00:01:55.120 First, do no harm. s31 00:01:55.120 --> 00:01:59.400 Next, patient first, and access for all. s32 00:01:59.400 --> 00:02:00.800 Um and here's a cool thing. s33 00:02:00.800 --> 00:02:04.960 Uh this is plastered across our walls in our company, uh but it's not just a poster. s34 00:02:04.960 --> 00:02:07.560 Uh this is an oath every employee in the company takes. s35 00:02:07.560 --> 00:02:09.800 I did too on on my first day. s36 00:02:09.800 --> 00:02:13.920 Uh and we built our product uh based on this oath. s37 00:02:13.920 --> 00:02:17.000 Um and this is working at scale. s38 00:02:17.000 --> 00:02:21.160 Uh at this point, we've had 200 million clinical interactions. s39 00:02:21.160 --> 00:02:23.920 We've had zero significant safety incidents. s40 00:02:23.920 --> 00:02:30.920 We've deployed in over 60 plus health systems and have an 8.5 on 10 patient satisfaction rating. s41 00:02:30.920 --> 00:02:35.280 Now, uh I could put an architecture diagram and explain how all of this works. s42 00:02:35.280 --> 00:02:36.959 And don't worry, we'll do that. s43 00:02:36.959 --> 00:02:40.560 But before we get there, uh I want to show you the product in action. s44 00:02:40.560 --> 00:02:47.360 So, we'll have one of our agents interact with the patient so you get a feel for what we're up to. s45 00:02:48.920 --> 00:02:50.640 I'm your AI care manager. s46 00:02:50.640 --> 00:02:54.231 I'm calling to check in on how you're doing with your health monitoring devices. s47 00:02:54.231 --> 00:02:54.600 [music] s48 00:02:54.600 --> 00:02:57.000 Do you have time to chat now? s49 00:02:57.000 --> 00:03:03.440 I I need to know if I'm talking to a human being or a some kind of computer. s50 00:03:03.440 --> 00:03:05.959 You sound like either one. s51 00:03:05.959 --> 00:03:08.400 I understand your concern, miss. s52 00:03:08.400 --> 00:03:11.360 I'm an AI, which stands for artificial intelligence. s53 00:03:11.360 --> 00:03:12.480 Okay. s54 00:03:12.480 --> 00:03:14.537 How are you doing today? s55 00:03:14.537 --> 00:03:16.519 [laughter] s56 00:03:16.519 --> 00:03:21.320 I was tired earlier, but I'm somewhat revived. s57 00:03:21.320 --> 00:03:23.000 I'm incredibly amused. s58 00:03:23.000 --> 00:03:26.982 Delivering at this level of care required building clinical capabilities you won't find s59 00:03:26.982 --> 00:03:27.280 [music] s60 00:03:27.280 --> 00:03:28.280 anywhere else. s61 00:03:28.280 --> 00:03:31.440 On the safety side, accurate drug name recognition, s62 00:03:31.440 --> 00:03:34.600 Could you tell me the name of the medication you're currently on? s63 00:03:34.600 --> 00:03:36.239 Prazosin. s64 00:03:36.239 --> 00:03:38.440 And that's one of them, but I don't remember the name. s65 00:03:38.440 --> 00:03:40.737 Could you help me out by spelling the name of this medication? s66 00:03:40.737 --> 00:03:41.760 [music] s67 00:03:41.760 --> 00:03:44.880 P r o z i n, I n, I guess. s68 00:03:44.880 --> 00:03:47.760 It seems like you might be talking about prazosin. s69 00:03:47.760 --> 00:03:48.680 Does that sound s70 00:03:48.680 --> 00:03:49.239 Yeah. s71 00:03:49.239 --> 00:03:51.000 Vital sign clarification, s72 00:03:51.000 --> 00:03:58.840 Let me say it again it was 121 over 45 91. s73 00:03:58.840 --> 00:04:04.317 Just to confirm, your blood pressure is 121 over 45 and your heart rate is 91, right? s74 00:04:04.317 --> 00:04:05.040 [music] s75 00:04:05.040 --> 00:04:06.200 Right. s76 00:04:06.200 --> 00:04:08.280 Med stoppage detection. s77 00:04:08.280 --> 00:04:10.120 What about the nifedipine tablet? s78 00:04:10.120 --> 00:04:13.560 Were you able to stop taking that one as instructed? s79 00:04:13.560 --> 00:04:15.480 Well, I took it this morning. s80 00:04:15.480 --> 00:04:20.079 Well, no, I took it last night because I didn't have the other prescription. s81 00:04:20.079 --> 00:04:23.640 And I don't think that would keep my blood pressure down. s82 00:04:23.640 --> 00:04:28.240 I understand your concern about your blood pressure and it's great that you're thinking ahead. s83 00:04:28.240 --> 00:04:34.800 I would recommend that you do talk to your primary care doctor about the nifedipine and get their guidance on what to do. s84 00:04:34.800 --> 00:04:38.520 And real-time nurse escalation when clinically necessary. s85 00:04:38.520 --> 00:04:39.960 How are you feeling? s86 00:04:39.960 --> 00:04:43.000 Okay, just a little tired and weak. s87 00:04:43.000 --> 00:04:46.480 It's quite common to feel this way after being in the hospital. s88 00:04:46.480 --> 00:04:48.320 Could you tell me more about it? s89 00:04:48.320 --> 00:04:50.800 My legs feel weak when I walk. s90 00:04:50.800 --> 00:04:51.400 Got it. s91 00:04:51.400 --> 00:04:57.120 Along with the weakness in your legs, have you noticed any chest discomfort or difficulty breathing? s92 00:04:57.120 --> 00:05:00.840 I just was this was having shortness of breath for a while. s93 00:05:00.840 --> 00:05:04.520 I'm so sorry to hear that you're experiencing shortness of breath. s94 00:05:04.520 --> 00:05:09.600 Given your symptoms, it would be best to have a nurse speak with you immediately. s95 00:05:10.560 --> 00:05:18.040 Yeah, so that's our sneak peek into our product and so we'll get into why this is hard and why a generic stack doesn't work for us. s96 00:05:18.040 --> 00:05:27.640 Okay, so if I were to strip out all of the complexity around the audio and speaking back, it boils down to two things, intelligence and latency, right? s97 00:05:27.640 --> 00:05:33.000 On the top left, you see some of our ever-evolving smart intelligent models. s98 00:05:33.000 --> 00:05:35.400 These models are fantastic at what they do. s99 00:05:35.400 --> 00:05:36.840 They're clinically safe. s100 00:05:36.840 --> 00:05:42.760 But unfortunately, many of these models take tens of seconds to respond, sometimes over a minute. s101 00:05:42.760 --> 00:05:48.200 And that's completely useless when we're trying to have a two-way conversation on a telephone. s102 00:05:48.200 --> 00:05:52.160 On the bottom right, we have models that are fantastic. s103 00:05:52.160 --> 00:05:59.320 They're really fast, but then they're just not at the same level of clinical accuracy for us to have a safe conversation. s104 00:05:59.320 --> 00:06:03.680 When we started building out this product, where we wanted to be was on the top right corner. s105 00:06:03.680 --> 00:06:13.680 And in order to accomplish this, what we had to do was build a vertically integrated stack ground up bit by bit, optimizing every part of the stock stack. s106 00:06:13.680 --> 00:06:18.680 When we started out building our own models, we also had several seconds of like latency. s107 00:06:18.680 --> 00:06:28.480 But hundreds of optimizations later, where we landed at was a product that is insanely fast, but also doesn't lose its intelligence. s108 00:06:28.800 --> 00:06:31.640 And we continue to benchmark all of this consistently. s109 00:06:31.640 --> 00:06:34.880 All of these results are on our website and more. s110 00:06:34.880 --> 00:06:39.680 Um and there's three typical ways in which folks build uh voice systems. s111 00:06:39.680 --> 00:06:46.880 One, you can use an ensemble of models, or you use cascaded models, or you use speech-to-speech real-time models. s112 00:06:46.880 --> 00:06:51.840 Again, all of these models and architectures are great at different things. s113 00:06:51.840 --> 00:06:58.240 As an example, let me take two specific benchmarks among so many hundreds that are critical for our workflows. s114 00:06:58.240 --> 00:07:01.280 Uh lab results check and IVR navigation. s115 00:07:01.280 --> 00:07:06.560 Now, these are things that you don't typically hear about or see in the most common benchmarks. s116 00:07:06.560 --> 00:07:10.520 And most of the generic models out there don't perform well at those. s117 00:07:10.520 --> 00:07:17.960 And we needed our models and our product to be over 99% accuracy on those specific benchmarks amongst other things. s118 00:07:17.960 --> 00:07:24.240 Uh and that's one of the key reasons why we continued down investing our own stack. s119 00:07:25.600 --> 00:07:28.840 And latency is key to this entire system. s120 00:07:28.840 --> 00:07:32.880 So, every time we work on an optimization, we buy back some latency. s121 00:07:32.880 --> 00:07:34.760 And we just don't bank that latency. s122 00:07:34.760 --> 00:07:44.280 We use that extra gap now to pack more intelligence into the overall system, such that we can have a more reliable conversation with the patient. s123 00:07:44.280 --> 00:07:49.080 Um and then we do that, we go back, work on more optimization, and that flywheel compounds. s124 00:07:49.080 --> 00:07:55.520 So, what seems like a tug-of-war between latency and intelligence for us is a compounding flywheel. s125 00:07:55.520 --> 00:07:58.120 So, what does this entire machine look like? s126 00:07:58.120 --> 00:07:59.240 Uh this is Polaris. s127 00:07:59.240 --> 00:08:01.800 This is our constellation architecture. s128 00:08:01.800 --> 00:08:04.920 Left to right, you have the system that hears. s129 00:08:04.920 --> 00:08:07.400 The middle is the brain that reasons. s130 00:08:07.400 --> 00:08:10.040 The right is the system that talks back. s131 00:08:10.040 --> 00:08:13.480 And all of this round trip needs to be really fast. s132 00:08:13.480 --> 00:08:19.400 So, on the left-hand side, what you're seeing are collection of models uh that are used for speech detection. s133 00:08:19.400 --> 00:08:27.520 So, everything from bilingual switching to background noise detection um to contextual understanding of the conversation. s134 00:08:27.520 --> 00:08:29.880 The brain isn't a singular model. s135 00:08:29.880 --> 00:08:35.080 We in fact run 31 models at any given point of time for every conversation. s136 00:08:35.080 --> 00:08:48.680 So, we have one central model that's hard like have a handling the conversation, where we have other 30 specialist models, everything from labs to medications to scheduling uh that are feeding input into this model. s137 00:08:48.680 --> 00:09:01.240 And then finally, on the output side, we have a custom personality, voice, HD quality, and clinical documentation engine that make sure the patient gets back all of the right information, s138 00:09:01.240 --> 00:09:06.000 and all of the same information flows back into the health system. s139 00:09:06.640 --> 00:09:15.280 The reason we have the system is because we see a singular model being as like one point of failure, and that's just unacceptable for a patient conversation. s140 00:09:15.280 --> 00:09:19.560 Uh this architecture gives us the redundancy and safety that we otherwise couldn't get to. s141 00:09:19.560 --> 00:09:21.760 And the patient never can tell the different. s142 00:09:21.760 --> 00:09:24.160 It's pretty seamless from their perspective. s143 00:09:24.160 --> 00:09:27.480 Let me double click into some of these systems for you. s144 00:09:27.480 --> 00:09:31.520 Uh first, we'll talk about our uh ASR system. s145 00:09:31.520 --> 00:09:39.360 So, the real world, as all of us know, is fairly loud and noisy, uh but most of the audio benchmarks get recorded in a quiet room. s146 00:09:39.360 --> 00:09:49.000 And something we learned the hard way was most of what looks like model reasoning failures end up actually being model mishearing things, right? s147 00:09:49.000 --> 00:10:00.560 So, the Spanish C or yes gets transcribed as the alphabet C, or you could have an Arabic drug name that gets over 30% in inaccuracy in terms of like word error rate. s148 00:10:00.560 --> 00:10:05.000 So, we had to build a system to to combat all of these challenges. s149 00:10:05.000 --> 00:10:09.840 A typical speech to text system just takes in the audio and then outputs some text. s150 00:10:09.840 --> 00:10:14.280 So, ours is a decoder-only audio LLM system. s151 00:10:14.280 --> 00:10:19.160 So, we take in the audio, but in addition to that, we're giving it two additional pieces of data. s152 00:10:19.160 --> 00:10:26.600 One is the context around the conversation thus far, and second, it's the domain knowledge around that entire conversation. s153 00:10:26.600 --> 00:10:30.040 And that is sort of like the trick that makes this work for us. s154 00:10:30.040 --> 00:10:32.120 So, under the hood, what's happening? s155 00:10:32.120 --> 00:10:35.840 So, we have First, we have a encoder, which is a year. s156 00:10:35.840 --> 00:10:44.320 So, we took an open-source Whisper V3 large turbo model and fine-tuned that on millions of clinical conversations to get to the accuracy we need. s157 00:10:44.320 --> 00:10:47.480 Next, we have a conformer projector. s158 00:10:47.480 --> 00:10:54.400 What this does is takes the audio and compacts it and projects it into tokens that the language model would understand. s159 00:10:54.400 --> 00:10:58.120 But, the key over here is it also maintains all the prosody. s160 00:10:58.120 --> 00:11:01.360 So, all the pauses and the stresses are maintained. s161 00:11:01.360 --> 00:11:06.240 So, the model hears not just the what, but also the how. s162 00:11:06.240 --> 00:11:13.920 Next, we're passing in the context around, let's say, the medications this patient's taking in, or the task at hand. s163 00:11:13.920 --> 00:11:15.920 Are we trying to fill in a form? s164 00:11:15.920 --> 00:11:19.960 What kind of like details that need to go in there and the policy? s165 00:11:19.960 --> 00:11:34.480 And how this helps is now when a patient mentions a medication name, we aren't guessing from like an infinite list of medications, but we have the chance to optimize around a finite list, and that helps in getting the word error rate down. s166 00:11:34.480 --> 00:11:38.960 Next, we also have contextual biasing as a part of like the training. s167 00:11:38.960 --> 00:11:50.360 So, during the training process, uh we have millions of these synthetic patients with their addresses, phone numbers, uh and other details uh that are fed into the training process. s168 00:11:50.360 --> 00:11:56.800 So, as a concrete example, let's say someone mentions their address on a conversation as 1,100 Geary Boulevard. s169 00:11:56.800 --> 00:12:11.320 Now, that specific utterance is ripe for phonetic garbage for most ASR systems, but given we're feeding that as additional context to the audio LLM, we have a high degree of accuracy and are able to get to the right transcription, s170 00:12:11.320 --> 00:12:14.160 which then feeds into the brain accurately. s171 00:12:14.160 --> 00:12:22.120 Further, what we see is for most clinical conversations, many of the patient responses are mono mono words, right? s172 00:12:22.120 --> 00:12:24.480 Uh and those often get transcribed incorrectly. s173 00:12:24.480 --> 00:12:28.040 A now becomes a no, or a five becomes a fine. s174 00:12:28.040 --> 00:12:31.280 And in a patient conversation, that's catastrophic. s175 00:12:31.280 --> 00:12:41.960 So, what we do is a secondary round of scoring, specifically when we see a patient has said only a singular word, and again, we use the context of the entire conversation to do this. s176 00:12:41.960 --> 00:12:51.640 So, all of these like help us bring down the medical word accuracy rate by over uh 50% um from what we see as standard off-the-shelf models. s177 00:12:51.640 --> 00:13:03.640 In addition to that, uh the specific system we've built allows us to uh check in the latency, and at a P99, it's three times faster than every other system out there. s178 00:13:03.880 --> 00:13:07.200 Next, let's look into the constellation, uh the brain. s179 00:13:07.200 --> 00:13:12.600 So, as I said, we have 31 models live running in uh at any given point of time. s180 00:13:12.600 --> 00:13:16.040 So, how do we do this without falling apart on latency? s181 00:13:16.040 --> 00:13:17.640 Uh the answer is a bit counterintuitive. s182 00:13:17.640 --> 00:13:20.800 We actually run every single one of these models in parallel. s183 00:13:20.800 --> 00:13:29.880 Uh but the trick over here is uh every single model does a really quick check to see if they even need to say something on this conversation, right? s184 00:13:29.880 --> 00:13:33.400 So, every specialist first decides, "Hey, do I need to speak?" s185 00:13:33.400 --> 00:13:38.160 If not, it's a short circuit, and that's what helps us keep us in the budget. s186 00:13:38.160 --> 00:13:39.960 So, that's the synchronous part. s187 00:13:39.960 --> 00:13:46.360 In addition to that, we also have asynchronous run models running in the background that are doing verification. s188 00:13:46.360 --> 00:13:50.880 And let's take two specific examples to walk through how we use these models. s189 00:13:50.880 --> 00:14:04.000 Again, going back to our medication example, for example, if the patient talks about ibuprofen or is talking about another medication, that's when the med engine specialist will kick in, and it will say, "Oh, I have something to say over here." s190 00:14:04.000 --> 00:14:06.880 Main model will pass in that context to the main model. s191 00:14:06.880 --> 00:14:16.520 The main model with a lung along with everything else that already knows around the goal and the conversation will take this input from the specialist and steer the conversation as appropriate. s192 00:14:16.520 --> 00:14:27.240 Next, like many other agentic systems, we heavily use like tool calls, and I think pretty much every one of us has experienced tool calls have innumerable failure modes. s193 00:14:27.240 --> 00:14:37.040 So, we have these like verifiers in the background that are running to make sure all the parameters into the tool calls and the responses and the tool calls are accurate. s194 00:14:37.040 --> 00:14:42.800 And for use cases like scheduling, this is what's gotten us to get to a 3949 accuracy. s195 00:14:42.800 --> 00:14:48.240 And finally, for certain use cases, we're also running these verifiers offline where possible. s196 00:14:48.240 --> 00:15:00.640 So, again, going back to scheduling, sometimes we have the luxury of time to go back and check to make sure if there was an inaccurate appointment, we can course correct that on the back end or call a patient back and apologize. s197 00:15:01.720 --> 00:15:12.960 All of these changes would be meaningless if we didn't optimize our inference stack, and we have a fantastic research and engineering team that lives and breathes on this problem. s198 00:15:12.960 --> 00:15:16.920 And for inference itself, quality is our constraint, and speed is the work. s199 00:15:16.920 --> 00:15:20.280 So, we can never compromise on the quality of the output. s200 00:15:20.280 --> 00:15:24.040 Every speed optimization has to be lossless. s201 00:15:24.040 --> 00:15:26.840 We've worked on like many many of these like optimizations. s202 00:15:26.840 --> 00:15:31.160 Let me talk you through three specific ones that we found to be the most meaningful. s203 00:15:31.160 --> 00:15:33.200 One is four-bit quantization. s204 00:15:33.200 --> 00:15:44.120 So basically what we've done is we've shrunk down the math for model computation to be from 16-bit to four-bit, again lossless, and this has like really helped us with overall latency. s205 00:15:44.120 --> 00:15:46.040 The next is speculative decoding. s206 00:15:46.040 --> 00:15:56.800 So we have a smaller model that's generating the tokens ahead of time, and then the main model just checks this in one go, and this also has greatly improved latency for us. s207 00:15:56.800 --> 00:15:59.040 Finally, a KV cache compression system. s208 00:15:59.040 --> 00:16:09.600 So as these conversations get like really long in length, we've figured out a way to keep a large chunk of these conversations warm on cache, giving us an over 96% s209 00:16:09.600 --> 00:16:14.840 hit rate, and then also helps us with the prefill portion, which goes 18 times faster. s210 00:16:14.840 --> 00:16:24.000 So this style of across-the-board board optimizations is what's getting us ahead during the product development phase. s211 00:16:24.000 --> 00:16:29.520 Now, evals are obviously key as we've seen across many of these talks. s212 00:16:29.520 --> 00:16:31.360 So some quick math, right? s213 00:16:31.360 --> 00:16:34.200 For example, let's take the scheduling use case again. s214 00:16:34.200 --> 00:16:39.720 With over 10,000 calls a day, a one 1% failure rate like sounds all right. s215 00:16:39.720 --> 00:16:45.000 Most agentic systems would claim that's great for them. s216 00:16:45.000 --> 00:16:56.160 For us, even the 99% is pretty bad because 1% error means 100 people a day are going to get the wrong appointment type, and it's physically bad because they'll show up s217 00:16:56.160 --> 00:17:03.480 on the wrong date, wrong time, but worse off, they could actually be missing a critical appointment, and that's unacceptable from our standpoint. s218 00:17:03.480 --> 00:17:05.439 So it's not just an annoyance, right? s219 00:17:05.439 --> 00:17:13.880 Um And the challenge over here is in order to catch this like statistically, the math like is like incredibly challenging, right? s220 00:17:13.880 --> 00:17:24.079 You need about 450 tests to be 99% sure that you can catch this like 1% error rate Um and 1900 tests to be able to see that you've caught it like 10 times. s221 00:17:24.079 --> 00:17:31.440 So, you can't purely rely on synthetic data from our experience to be able to get to the scale of accuracy. s222 00:17:31.440 --> 00:17:35.320 So, we use a combination of human beings and synthetic data. s223 00:17:35.320 --> 00:17:41.320 So, we have over 7,000 trained clinicians on our platform who are continuously helping us like evaluate our platform. s224 00:17:41.320 --> 00:17:53.160 They've done over 700,000 close to 800,000 clinical conversations to help us figure out the accuracy of our system and we do this on a continuous basis. s225 00:17:53.640 --> 00:17:56.920 Finally, coming to safety. s226 00:17:56.920 --> 00:18:03.960 We've shipped five versions of our product thus far and each one's gotten better compared to the previous one. s227 00:18:03.960 --> 00:18:09.000 So, we don't just grade the output of our models on the correctness or validity of it. s228 00:18:09.000 --> 00:18:13.120 So, we use the same scale that as they do for human beings. s229 00:18:13.120 --> 00:18:18.440 So, they're graded across correctness, no harm, minor harm, severe harm, and death. s230 00:18:18.440 --> 00:18:29.000 And across five generations where we're at with our Polaris system is a 99.89% accuracy with respect to like no harm. s231 00:18:29.000 --> 00:18:32.760 And humans on the same rubric are at about 81%. s232 00:18:32.760 --> 00:18:38.880 It's not because like we're terrible, but AI systems don't get tired and unfortunately we do. s233 00:18:38.880 --> 00:18:45.760 And we also don't have the luxury of having 30 plus supervisors helping us at any given point of time. s234 00:18:45.840 --> 00:18:48.960 And finally, safe is not enough, right? s235 00:18:48.960 --> 00:18:51.400 A key part of the product is the empathy. s236 00:18:51.400 --> 00:19:00.600 For human beings to open up to AI systems, we needed to make sure the voice and how we actually lead these conversations are empathetic. s237 00:19:00.600 --> 00:19:10.560 And as with most other things during our product journey, we found that there weren't actually accurate or good benchmarks to measure the empathy of these systems. s238 00:19:10.560 --> 00:19:12.880 So, we actually built one which is called heart. s239 00:19:12.880 --> 00:19:14.760 The paper for this is public. s240 00:19:14.760 --> 00:19:16.720 Encourage you guys to like read it. s241 00:19:16.720 --> 00:19:27.160 And we we had to make sure the end system was not just safe, but also empathetic so that people would actually prefer it and use it. s242 00:19:27.160 --> 00:19:30.800 And finally, all of this like comes down to the people. s243 00:19:30.800 --> 00:19:36.280 We wouldn't be here if we didn't invest in people from an AI native perspective. s244 00:19:36.280 --> 00:19:43.360 Everything in our company across the last couple of years has been transformed our hiring processes to how we use like tokens. s245 00:19:43.360 --> 00:19:46.040 We still have unlimited tokens for everyone at our company. s246 00:19:46.040 --> 00:19:50.880 We also have two great programs for anyone who comes into Hippocratic. s247 00:19:50.880 --> 00:19:59.480 For early career engineers, we have an agent deployment residency program where they're trained to write some of the best like health care safest agents. s248 00:19:59.480 --> 00:20:08.360 And then for experienced software engineers, we have an AI residency program where we help you start like training models and build the next great system along with us. s249 00:20:08.360 --> 00:20:14.360 So yeah, we're told you got to pick two of these options around quality, speed, and safety. s250 00:20:14.360 --> 00:20:20.400 We didn't and we decided to go with all of them and we're hoping the rest of you should come and build with us. s251 00:20:20.400 --> 00:20:22.640 Thank you.