{
  "video": {
    "id": "fnLBmfsI_Fg",
    "title": "Your Voice Agent Doesn't Need a Frontier Model - Joel Allou & Ornella Bahidika, Microsoft",
    "duration": 345,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "fnLBmfsI_Fg",
    "title": "Your Voice Agent Doesn't Need a Frontier Model - Joel Allou & Ornella Bahidika, Microsoft",
    "one_liner": "Microsoft's Joel Allou and Ornella Bahidika built Ace, a live AI voice tutor, by stripping all reasoning out of the model into a state machine so a small model (Haiku 4.5) can start speaking in ~900ms instead of the couple of seconds a frontier model spends thinking.",
    "summary": "Ace is a live AI voice tutor that runs on a small model on purpose, because in voice the constraint is latency, not intelligence — the model has to start talking in about 950 milliseconds or the pause reads as a dead call. The team removed everything hard from the model: it doesn't decide what happens next in the lesson, doesn't track what the student knows, and doesn't explain what comes next; a state machine plus an 'intelligent layer' that derives student mastery does all of that and hands the model a summary each turn to speak out. They show two videos: plain Opus 4.7 visibly reasoning for seconds, versus the same question on Haiku 4.5 with the scaffolding answering in about 900ms. The honest cost is that a small model without scaffolding drifts on long structure and needs strict rules — but you pay that price once, in code, not on every turn.",
    "key_points": [
      "The latency budget for a voice agent is ~950ms to start talking; when an agent pauses even a second, 'your brain says it's dead'.",
      "Frontier models like Claude 4.7 are good at reasoning through a lesson and a student's question — and that reasoning, taking a couple of seconds, is precisely the problem in voice.",
      "All the thinking is extracted out of the model into a state machine: for Ace they enumerated every scenario a lesson needs and built a machine that coordinates each step to the next.",
      "On top of the state machine sits an 'intelligent layer' that derives the mastery a student needs for a lesson to be complete.",
      "What happens next, what gets displayed, and how to answer a question are all decided outside the model; the model is simply fed that output to speak.",
      "Live comparison: plain Opus 4.7 takes a couple of seconds of visible reasoning on a simple question; the same question with the scaffolding on Haiku 4.5 answers in about 900ms.",
      "The admitted cost: a small model like Haiku 4.5 with no scaffolding drifts on long structure and needs strict rules to stay organized — but the scaffolding is paid once, in code, not per turn.",
      "The smaller model is also cheaper, so the pattern generalizes to real-time and high-volume applications, not just voice."
    ],
    "takeaways": [
      "Choose the model by your latency budget first: 'pick the fastest model that your latency budget allows', then spend the remaining effort on scaffolding.",
      "Move orchestration, state tracking and reasoning into deterministic code — a state machine over enumerated scenarios — and leave the model only the job of speaking.",
      "Hand the model a freshly computed summary every turn rather than expecting it to carry lesson state or decide what comes next.",
      "Budget engineering time for strict rules and scaffolding when dropping to a small model, since it will otherwise drift on long structured interactions.",
      "Apply the same 'build around the model' pattern to any real-time or high-volume system, treating the model as the smallest part of the system."
    ],
    "topics": [
      "voice-agents",
      "latency",
      "small-models",
      "state-machines",
      "agent-architecture",
      "edtech",
      "cost-optimization"
    ],
    "tools": [
      "Ace",
      "Claude Opus 4.7 (stated as 'Claude 4.7' / 'Opus 4.7')",
      "Claude Haiku 4.5",
      "Anthropic",
      "Microsoft"
    ],
    "quotes": [
      {
        "text": "Because our budget was never IQ, it's millisecond.",
        "at": "00:43",
        "url": "https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=43s"
      },
      {
        "text": "A frontier model that think for a full second has already lost the room, no matter how good the answer is.",
        "at": "00:54",
        "url": "https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=54s"
      },
      {
        "text": "What's left for the model is one thing it's really good at, talking.",
        "at": "01:26",
        "url": "https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=86s"
      },
      {
        "text": "Pick the fastest model that your latency budget allows and then spend the rest of your time actually building the scaffolding.",
        "at": "04:45",
        "url": "https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=285s"
      }
    ],
    "words": 1032
  },
  "summary_url": "/#fnLBmfsI_Fg",
  "transcript": {
    "html": "/transcripts/fnLBmfsI_Fg.html",
    "txt": "/transcripts/fnLBmfsI_Fg.txt",
    "vtt": "/transcripts/fnLBmfsI_Fg.vtt"
  }
}