Your Voice Agent Doesn't Need a Frontier Model - Joel Allou & Ornella Bahidika, Microsoft

AI Engineer · 5 min · 59 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:00Hi, I'm Ornella, and that's Joel, and we built Ace, a live AI voice tutor.
  2. 00:06It run on a small model on purpose, and I want to tell you more why that's not a compromise.
  3. 00:14Quick gut check.
  4. 00:19That silence on a voice call, that the difference between a tutor and a broken up.
  5. 00:25When a voice agent pause for even a second, your brain says it's dead.
  6. 00:31So, when the answer fit a leader of every instant stay, which for the smallest, biggest model.
  7. 00:38In voice, that instant is actually a backward.
  8. 00:43Because our budget was never IQ, it's millisecond.
  9. 00:49The AI model need to start talking in about 950 milliseconds.
  10. 00:54A frontier model that think for a full second has already lost the room, no matter how good the answer is.
  11. 01:04So, we made the model small and took the hardest part jobs away from it.
  12. 01:10It doesn't decide when happen what's happen in the lesson.
  13. 01:14It had It doesn't track what the student knows.
  14. 01:17It doesn't explain what's next.
  15. 01:19We have a system in place to do that, and they hand the model a summary every turn.
  16. 01:26What's left for the model is one thing it's really good at, talking.
  17. 01:30And that's the theory.
  18. 01:32Joel, go on and show them what it actually feel like.
  19. 01:35Yeah, if maybe I can add some color to what Ornella was mentioning.
  20. 01:40So, if you think about the models of today, especially the frontier model, let's take Claude 4.7, which is uh from Anthropic.
  21. 01:49The model is really good at reasoning.
  22. 01:51You can give it a problem, in this case a lesson, and it can reason through it, it can reason through what the student asking and it can come up with the answer.
  23. 02:00But that is actually precisely the problem.
  24. 02:04Because the reasoning can take couple of seconds and those seconds are really valuable when you are building voice applications.
  25. 02:12So what we are doing is saying, "Hey, let's extract all of the thinking away from the model so that the model focuses on only what matters, which is speaking in our case."
  26. 02:25So all of the thinking is extracted into a state machine.
  27. 02:29So for Ace, we have thought about all the scenarios that are needed for a lesson.
  28. 02:34We have built a state machine that is able to coordinate each step to the next and we also added intelligent layer on top to derive some of the mastery
  29. 02:45that a student might need for the lesson to be complete.
  30. 02:49So everything when it comes to what happens next, when it comes to what needs to be displayed, when it comes to how to actually answer a question, it's all done outside of the model.
  31. 03:01And we simply feed that output to the model to speak out.
  32. 03:07And so let's go ahead and look at an example and see how that works in real time.
  33. 03:12So the first video here is without the implementation we've done.
  34. 03:17So it's a simple Opus 4.7. We ask a very simple question and as you can see the model is thinking, it's reasoning and it takes couple of seconds to return the answer back to the user.
  35. 03:28Hey, what In this video, we've added all everything we just talked about on Haiku 4.5, which is a much smaller model.
  36. 03:36Same question, but now you see that the answer comes in about 900 milliseconds.
  37. 03:41And so that's the beauty of building around the model.
  38. 03:45So by removing all of the thinking, all of the logic, all of the reasoning from the model and actually putting it within the code, we actually saves a lot of time and allows us to use smaller models which are cost effective
  39. 04:00and actually better at real-time voice applications.
  40. 04:04And as you can see, this feels almost instant and again that's because all of the smart parts have already happened prior to the model actually speaking.
  41. 04:15But I have to be honest because this isn't necessarily free.
  42. 04:20It has a cost, right?
  43. 04:21A small model like the Haiku 4.5, if it doesn't have any scaffolding, tend to drift on long structure and really needs strict rules in order to be able to stay organized.
  44. 04:35So the scaffolding piece is the price.
  45. 04:38But the good thing is you pay it once and in code, right?
  46. 04:41Not on every single turn.
  47. 04:43So here's a rule.
  48. 04:45Pick the fastest model that your latency budget allows and then spend the rest of your time actually building the scaffolding.
  49. 04:54So
  50. 04:54[snorts]
  51. 04:54in our case, right?
  52. 04:56Maybe you build a state machine, you build a reasoning process, you think about scenarios, what happens if this happens, how should your model handle it?
  53. 05:04Everything that comes with the logic, everything that comes with the harnessing, you do that outside of the model and then allowing the model to focus on that one thing that is really good at.
  54. 05:16And so that's true for voice applications like A is, that's true for real-time applications where latency is of priority and that's really true for anything that is high volume,
  55. 05:28right?
  56. 05:28In those cases, the model is the smallest part of the system.
  57. 05:33So this is Joel and Ornella and we are building A is again and if you have any questions, let us know.
  58. 05:41Thank you.
  59. 05:42Thank you.