{
  "video": {
    "id": "m24UKZomm7k",
    "title": "Don't Let the LLM Drive - Ornella Bahidika & Joel Allou, Microsoft",
    "duration": 368,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "m24UKZomm7k",
    "title": "Don't Let the LLM Drive - Ornella Bahidika & Joel Allou, Microsoft",
    "one_liner": "Ornella Bahidika and Joel Allou built Ace, a live AI voice tutor, by making the lesson a state machine the harness owns — the model only executes one confined step at a time, which let them swap a heavy Opus 4.7-class model for Haiku 4.5 without losing reliability.",
    "summary": "The talk argues that multi-step agents fail in production not because the prompt is weak but because the model is holding the control flow: it decides it's done, skips a step, or loops. Their fix is \"harness engineering\" — a lesson is a small state machine (intro, teach, check, grade, advance, wrap), each step hands the model a narrow contract to do one thing and return it, and the harness validates the output, advances the state and picks what's next. They show logs from a real Ace lesson with separate harnessing for the section content, whiteboard drawing, clearing the queue and ending the lesson. Because the decisions live outside the model, they run on Haiku 4.5 instead of a frontier model and still hit their bar on reliability, cost and latency.",
    "key_points": [
      "The failure mode everyone recognises: it works near the demo, then a real user gets in and halfway through the agent decides it's done, skips a step, or loops — \"the demo never show you that.\"",
      "The first fix people reach for — prompt it harder, add more rules — is the wrong axis: \"reliability was never a prompting problem. It's a control problem.\"",
      "Framing: the model is the talent, the harness is the director. The model is brilliant at delivering a line but terrible at remembering it's on step three of six, so they stop asking it to.",
      "A lesson is modelled as a small state machine with the states intro, teach, check, grade, advance and wrap; each step sends the model a narrow contract (\"do this one thing, return it\"), the harness validates what comes back, advances the state and decides what's next. The model never decides where we are.",
      "Because the control flow was pulled out, they dropped from a heavy frontier model (they name Anthropic's Opus 4.7) to Haiku 4.5 — much smaller, less reasoning capability — and it still performs at the expected level, saving money, time and latency.",
      "Demo: a recorded lesson with logs on the right showing distinct harnessing for what to speak about in a section, drawing on the whiteboard, clearing the queue, and how to end the lesson.",
      "Three questions they deliberately engineered outside the model: when is the lesson done, did the student actually get it, and what comes next.",
      "The pattern generalises beyond voice — they cite coding agents, ops run books and onboarding flows.",
      "Decision rule: if your agent's reliability is \"somewhat of a coin flip,\" take the control flow out of the model."
    ],
    "takeaways": [
      "When a multi-step agent is unreliable, stop escalating the prompt — move the step tracking and next-step decision into an explicit state machine you own.",
      "Give the model one narrow contract per step (specific input in, specific action out) and have the harness validate the output before advancing, rather than letting the model self-report progress.",
      "Enumerate the control questions your agent keeps getting wrong — is it done, did it succeed, what's next — and engineer each one outside the model.",
      "Once the harness carries the control flow, re-test on a much smaller model: they went from an Opus-class model to Haiku 4.5 and kept the quality bar while cutting cost and latency.",
      "Use coin-flip reliability as the trigger: if the agent succeeds about half the time, that's the signal to remove decisions from the model instead of adding rules to it."
    ],
    "topics": [
      "agents",
      "harness-engineering",
      "state-machines",
      "reliability",
      "voice-agents",
      "small-models",
      "latency",
      "cost-optimization"
    ],
    "tools": [
      "Ace",
      "Microsoft",
      "Anthropic",
      "Claude Opus 4.7",
      "Haiku 4.5"
    ],
    "quotes": [
      {
        "text": "But reliability was never a prompting problem. It's a control problem.",
        "at": "00:35",
        "url": "https://www.youtube.com/watch?v=m24UKZomm7k&t=35s"
      },
      {
        "text": "The model is the talent, and the harness is the director. The model is brilliant at delivering a line, but it's really terrible at remembering if it's on step three of six. So we stop asking it to.",
        "at": "00:41",
        "url": "https://www.youtube.com/watch?v=m24UKZomm7k&t=41s"
      },
      {
        "text": "The model never really has to think. It proposes, but ultimately it is the harness that decides.",
        "at": "04:23",
        "url": "https://www.youtube.com/watch?v=m24UKZomm7k&t=263s"
      },
      {
        "text": "So, don't let the model talk, right? Or actually let it talk, but don't let it drive.",
        "at": "05:53",
        "url": "https://www.youtube.com/watch?v=m24UKZomm7k&t=353s"
      }
    ],
    "words": 1141
  },
  "summary_url": "/#m24UKZomm7k",
  "transcript": {
    "html": "/transcripts/m24UKZomm7k.html",
    "txt": "/transcripts/m24UKZomm7k.txt",
    "vtt": "/transcripts/m24UKZomm7k.vtt"
  }
}