WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=fnLBmfsI_Fg NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/fnLBmfsI_Fg.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.000 --> 00:00:06.960 Hi, I'm Ornella, and that's Joel, and we built Ace, a live AI voice tutor. s2 00:00:06.960 --> 00:00:14.040 It run on a small model on purpose, and I want to tell you more why that's not a compromise. s3 00:00:14.280 --> 00:00:16.720 Quick gut check. s4 00:00:19.280 --> 00:00:25.520 That silence on a voice call, that the difference between a tutor and a broken up. s5 00:00:25.520 --> 00:00:31.400 When a voice agent pause for even a second, your brain says it's dead. s6 00:00:31.400 --> 00:00:38.800 So, when the answer fit a leader of every instant stay, which for the smallest, biggest model. s7 00:00:38.800 --> 00:00:43.280 In voice, that instant is actually a backward. s8 00:00:43.960 --> 00:00:48.920 Because our budget was never IQ, it's millisecond. s9 00:00:49.280 --> 00:00:54.560 The AI model need to start talking in about 950 milliseconds. s10 00:00:54.560 --> 00:01:03.760 A frontier model that think for a full second has already lost the room, no matter how good the answer is. s11 00:01:04.760 --> 00:01:10.320 So, we made the model small and took the hardest part jobs away from it. s12 00:01:10.320 --> 00:01:14.000 It doesn't decide when happen what's happen in the lesson. s13 00:01:14.000 --> 00:01:17.200 It had It doesn't track what the student knows. s14 00:01:17.200 --> 00:01:19.520 It doesn't explain what's next. s15 00:01:19.520 --> 00:01:26.200 We have a system in place to do that, and they hand the model a summary every turn. s16 00:01:26.200 --> 00:01:30.640 What's left for the model is one thing it's really good at, talking. s17 00:01:30.640 --> 00:01:32.240 And that's the theory. s18 00:01:32.240 --> 00:01:35.760 Joel, go on and show them what it actually feel like. s19 00:01:35.760 --> 00:01:40.760 Yeah, if maybe I can add some color to what Ornella was mentioning. s20 00:01:40.760 --> 00:01:49.320 So, if you think about the models of today, especially the frontier model, let's take Claude 4.7, which is uh from Anthropic. s21 00:01:49.320 --> 00:01:51.480 The model is really good at reasoning. s22 00:01:51.480 --> 00:02:00.440 You can give it a problem, in this case a lesson, and it can reason through it, it can reason through what the student asking and it can come up with the answer. s23 00:02:00.440 --> 00:02:04.000 But that is actually precisely the problem. s24 00:02:04.000 --> 00:02:12.520 Because the reasoning can take couple of seconds and those seconds are really valuable when you are building voice applications. s25 00:02:12.520 --> 00:02:25.480 So what we are doing is saying, "Hey, let's extract all of the thinking away from the model so that the model focuses on only what matters, which is speaking in our case." s26 00:02:25.480 --> 00:02:29.320 So all of the thinking is extracted into a state machine. s27 00:02:29.320 --> 00:02:34.480 So for Ace, we have thought about all the scenarios that are needed for a lesson. s28 00:02:34.480 --> 00:02:45.440 We have built a state machine that is able to coordinate each step to the next and we also added intelligent layer on top to derive some of the mastery s29 00:02:45.440 --> 00:02:49.120 that a student might need for the lesson to be complete. s30 00:02:49.120 --> 00:03:01.519 So everything when it comes to what happens next, when it comes to what needs to be displayed, when it comes to how to actually answer a question, it's all done outside of the model. s31 00:03:01.519 --> 00:03:07.080 And we simply feed that output to the model to speak out. s32 00:03:07.080 --> 00:03:12.400 And so let's go ahead and look at an example and see how that works in real time. s33 00:03:12.400 --> 00:03:17.200 So the first video here is without the implementation we've done. s34 00:03:17.200 --> 00:03:28.040 So it's a simple Opus 4.7. We ask a very simple question and as you can see the model is thinking, it's reasoning and it takes couple of seconds to return the answer back to the user. s35 00:03:28.040 --> 00:03:36.440 Hey, what In this video, we've added all everything we just talked about on Haiku 4.5, which is a much smaller model. s36 00:03:36.440 --> 00:03:41.959 Same question, but now you see that the answer comes in about 900 milliseconds. s37 00:03:41.959 --> 00:03:45.920 And so that's the beauty of building around the model. s38 00:03:45.920 --> 00:04:00.080 So by removing all of the thinking, all of the logic, all of the reasoning from the model and actually putting it within the code, we actually saves a lot of time and allows us to use smaller models which are cost effective s39 00:04:00.080 --> 00:04:04.520 and actually better at real-time voice applications. s40 00:04:04.520 --> 00:04:15.480 And as you can see, this feels almost instant and again that's because all of the smart parts have already happened prior to the model actually speaking. s41 00:04:15.480 --> 00:04:20.400 But I have to be honest because this isn't necessarily free. s42 00:04:20.400 --> 00:04:21.920 It has a cost, right? s43 00:04:21.920 --> 00:04:35.040 A small model like the Haiku 4.5, if it doesn't have any scaffolding, tend to drift on long structure and really needs strict rules in order to be able to stay organized. s44 00:04:35.040 --> 00:04:38.000 So the scaffolding piece is the price. s45 00:04:38.000 --> 00:04:41.360 But the good thing is you pay it once and in code, right? s46 00:04:41.360 --> 00:04:43.919 Not on every single turn. s47 00:04:43.919 --> 00:04:45.680 So here's a rule. s48 00:04:45.680 --> 00:04:54.640 Pick the fastest model that your latency budget allows and then spend the rest of your time actually building the scaffolding. s49 00:04:54.640 --> 00:04:54.753 So s50 00:04:54.753 --> 00:04:54.919 [snorts] s51 00:04:54.919 --> 00:04:56.040 in our case, right? s52 00:04:56.040 --> 00:05:04.640 Maybe you build a state machine, you build a reasoning process, you think about scenarios, what happens if this happens, how should your model handle it? s53 00:05:04.640 --> 00:05:16.080 Everything that comes with the logic, everything that comes with the harnessing, you do that outside of the model and then allowing the model to focus on that one thing that is really good at. s54 00:05:16.080 --> 00:05:28.360 And so that's true for voice applications like A is, that's true for real-time applications where latency is of priority and that's really true for anything that is high volume, s55 00:05:28.360 --> 00:05:28.680 right? s56 00:05:28.680 --> 00:05:33.840 In those cases, the model is the smallest part of the system. s57 00:05:33.840 --> 00:05:41.360 So this is Joel and Ornella and we are building A is again and if you have any questions, let us know. s58 00:05:41.360 --> 00:05:42.640 Thank you. s59 00:05:42.640 --> 00:05:44.880 Thank you.