WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=m24UKZomm7k NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/m24UKZomm7k.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.000 --> 00:00:01.400 Hi, I'm Onela. s2 00:00:01.400 --> 00:00:02.520 That's Joel. s3 00:00:02.520 --> 00:00:09.840 And we built Ace, a live AI voice tutor that runs a full lesson start to finish reliably. s4 00:00:09.840 --> 00:00:13.600 The trick is LLM is not in charge. s5 00:00:13.600 --> 00:00:17.720 If you have shipped a multi-step agent, you know this moment. s6 00:00:17.720 --> 00:00:23.920 It's near the demo, then a real user gets in and halfway through the agent decide it's done. s7 00:00:23.920 --> 00:00:26.640 Or skip a step, or even loops. s8 00:00:26.640 --> 00:00:29.960 The demo never show you that. s9 00:00:30.000 --> 00:00:35.640 And the first fix everyone reaches for is prompt this harder, add more holes. s10 00:00:35.640 --> 00:00:38.920 But reliability was never a prompting problem. s11 00:00:38.920 --> 00:00:40.840 It's a control problem. s12 00:00:40.840 --> 00:00:42.400 Think of it this way. s13 00:00:42.400 --> 00:00:46.160 The model is the talent, and the harness is the director. s14 00:00:46.160 --> 00:00:53.320 The model is brilliant at delivering a line, but it's really terrible at remembering if it's on step three of six. s15 00:00:53.320 --> 00:00:56.400 So we stop asking it to. s16 00:00:56.400 --> 00:01:04.600 A lesson is a small state machine with intro, teach, check, grade, advance, and wrap. s17 00:01:04.600 --> 00:01:07.800 Each step sends the model a neural contract. s18 00:01:07.800 --> 00:01:10.520 Do this one thing, return it. s19 00:01:10.520 --> 00:01:16.440 The harness validates what's comes back, advance the state, and decide what's next. s20 00:01:16.440 --> 00:01:18.920 The model never decide where we are. s21 00:01:18.920 --> 00:01:20.280 That's the design. s22 00:01:20.280 --> 00:01:24.360 Joel is going to show you the harness thing. s23 00:01:24.360 --> 00:01:25.040 Yeah. s24 00:01:25.040 --> 00:01:40.280 So, when we think about the frontier models of today, let's take for example um Opus 4.7 Cloud from Anthropic, you see that oftentimes people leverage the model for essentially everything. s25 00:01:40.280 --> 00:01:43.960 For the thinking, for the processing, right? s26 00:01:43.960 --> 00:01:46.120 And for everything in between. s27 00:01:46.120 --> 00:01:57.240 While that can be good, it's not always effective in situations like ours, where we are building a live AI tutor that is speaking back and forth with students. s28 00:01:57.240 --> 00:01:57.680 Right? s29 00:01:57.680 --> 00:02:07.720 For something like this, we had a need to actually build something that is reliable, something that is cost-effective, and something that is fast. s30 00:02:07.720 --> 00:02:08.000 Right? s31 00:02:08.000 --> 00:02:13.640 So, this is where the idea of leveraging the concept of harness engineering has come in. s32 00:02:13.640 --> 00:02:27.160 Where instead of having a model that is really intelligent, sort of go through everything for us, we will build all of these steps that are needed and provide only the input required s33 00:02:27.160 --> 00:02:30.280 for the model to execute a specific scenario. s34 00:02:30.280 --> 00:02:34.840 So, when we were building ACE, we actually thought very deeply about state machines. s35 00:02:34.840 --> 00:02:35.120 Right? s36 00:02:35.120 --> 00:02:37.120 What is the step right now? s37 00:02:37.120 --> 00:02:41.000 And what is the possible steps that could come after? s38 00:02:41.000 --> 00:02:51.560 And within each of the steps, what are concrete things that we can provide to the model so that it is confined to that specific action, it is confined to that specific s39 00:02:51.560 --> 00:02:56.440 step at that particular moment, and only execute what needs to be done. s40 00:02:56.440 --> 00:03:07.800 So, by doing this, instead of having a very heavy model like a 4.7, we were actually able to rely on something like a Haiku 4.5, which is a much smaller model, s41 00:03:07.800 --> 00:03:19.440 doesn't have as much reasoning capabilities, but because of the harnessing around it, it's still able to perform at the level in which we expect, saving money, saving time, and saving latency. s42 00:03:19.440 --> 00:03:26.239 So, let's go ahead and play this recording, which will show us logs um about a particular lesson. s43 00:03:26.239 --> 00:03:34.187 So, as you can see in this video, especially on the right side, we see logs on all of the different harnessing that are happening. s44 00:03:34.187 --> 00:03:35.600 [clears throat] Right? s45 00:03:35.600 --> 00:03:45.360 So, for example, we see that there's harnessing for a section, which provides input to the model about exactly what to speak about, what to do. s46 00:03:45.360 --> 00:03:48.480 We have harnessing about drawing on the whiteboard. s47 00:03:48.480 --> 00:03:51.920 We have harnessing that deals with clearing the queue. s48 00:03:51.920 --> 00:03:57.360 We have steps to how to end the lesson and everything in between, right? s49 00:03:57.360 --> 00:04:06.480 So, everything that would allow us to actually build the lesson in a way that is reliable even if there's a new scenario that comes in. s50 00:04:06.480 --> 00:04:09.280 We try to incorporate that in our state machine. s51 00:04:09.280 --> 00:04:12.480 We try to incorporate that with within the lesson, right? s52 00:04:12.480 --> 00:04:20.600 So, again, the model all it worries about is given an input, it knows which action to take and it provides the output of that action. s53 00:04:20.600 --> 00:04:21.160 Right? s54 00:04:21.160 --> 00:04:25.480 And so, the model never really um has to think. s55 00:04:25.480 --> 00:04:29.080 It proposes, but ultimately it is the harness that decides. s56 00:04:29.080 --> 00:04:33.320 And so, for his A specifically, there are three things that we wanted to think about. s57 00:04:33.320 --> 00:04:35.640 Like, when is the lesson done is one, right? s58 00:04:35.640 --> 00:04:37.280 Did the student actually get it right? s59 00:04:37.280 --> 00:04:40.080 Like, did they actually learn in the way they were supposed to? s60 00:04:40.080 --> 00:04:41.760 And what comes next, right? s61 00:04:41.760 --> 00:04:51.360 And so, everything that comes within those three categories, all of the different questions, all of the different actions that the models needs to take, we have engineered that outside s62 00:04:51.360 --> 00:04:52.560 of the model, right? s63 00:04:52.560 --> 00:04:57.080 So, again, it's an input, the model receives it and gives us an output, right? s64 00:04:57.080 --> 00:05:02.680 And so, this is very, very important and we have found this to be very remarkable. s65 00:05:02.680 --> 00:05:02.880 Right? s66 00:05:02.880 --> 00:05:06.960 So, again, the the this is applicable to really everything, right? s67 00:05:06.960 --> 00:05:11.680 It's applicable to something like A that is a voice model. s68 00:05:11.680 --> 00:05:15.360 It's applicable to coding agents, to Ops Run Books. s69 00:05:15.360 --> 00:05:17.440 Um it's applicable to onboarding flows, right? s70 00:05:17.440 --> 00:05:19.640 The same rule applies, right? s71 00:05:19.640 --> 00:05:25.919 We want to find a way to not let the model think, but building abstractions around it. s72 00:05:25.919 --> 00:05:34.960 So, a good way to to remember on whether you should use this abstraction is essentially to think about the reliability of your agent, right? s73 00:05:34.960 --> 00:05:40.120 If it's somewhat of a coin flip, then you want to take the control flow out of the model. s74 00:05:40.120 --> 00:05:53.440 You want the model to not make as many decisions as it should and instead build those decisions around the model and simply feed an easy input so that the model can easily produce an output, s75 00:05:53.440 --> 00:05:53.760 right? s76 00:05:53.760 --> 00:05:56.480 So, don't let the model talk, right? s77 00:05:56.480 --> 00:05:59.640 Or actually let it talk, but don't let it drive. s78 00:05:59.640 --> 00:06:01.480 So, we're Joel and Ornella. s79 00:06:01.480 --> 00:06:05.200 This is Ace, and if you have any questions, please let us know. s80 00:06:05.200 --> 00:06:06.360 Thank you. s81 00:06:06.360 --> 00:06:08.600 Thank you.