Don't Let the LLM Drive - Ornella Bahidika & Joel Allou, Microsoft

AI Engineer · 6 min · 81 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:00Hi, I'm Onela.
  2. 00:01That's Joel.
  3. 00:02And we built Ace, a live AI voice tutor that runs a full lesson start to finish reliably.
  4. 00:09The trick is LLM is not in charge.
  5. 00:13If you have shipped a multi-step agent, you know this moment.
  6. 00:17It's near the demo, then a real user gets in and halfway through the agent decide it's done.
  7. 00:23Or skip a step, or even loops.
  8. 00:26The demo never show you that.
  9. 00:30And the first fix everyone reaches for is prompt this harder, add more holes.
  10. 00:35But reliability was never a prompting problem.
  11. 00:38It's a control problem.
  12. 00:40Think of it this way.
  13. 00:42The model is the talent, and the harness is the director.
  14. 00:46The model is brilliant at delivering a line, but it's really terrible at remembering if it's on step three of six.
  15. 00:53So we stop asking it to.
  16. 00:56A lesson is a small state machine with intro, teach, check, grade, advance, and wrap.
  17. 01:04Each step sends the model a neural contract.
  18. 01:07Do this one thing, return it.
  19. 01:10The harness validates what's comes back, advance the state, and decide what's next.
  20. 01:16The model never decide where we are.
  21. 01:18That's the design.
  22. 01:20Joel is going to show you the harness thing.
  23. 01:24Yeah.
  24. 01:25So, when we think about the frontier models of today, let's take for example um Opus 4.7 Cloud from Anthropic, you see that oftentimes people leverage the model for essentially everything.
  25. 01:40For the thinking, for the processing, right?
  26. 01:43And for everything in between.
  27. 01:46While that can be good, it's not always effective in situations like ours, where we are building a live AI tutor that is speaking back and forth with students.
  28. 01:57Right?
  29. 01:57For something like this, we had a need to actually build something that is reliable, something that is cost-effective, and something that is fast.
  30. 02:07Right?
  31. 02:08So, this is where the idea of leveraging the concept of harness engineering has come in.
  32. 02:13Where instead of having a model that is really intelligent, sort of go through everything for us, we will build all of these steps that are needed and provide only the input required
  33. 02:27for the model to execute a specific scenario.
  34. 02:30So, when we were building ACE, we actually thought very deeply about state machines.
  35. 02:34Right?
  36. 02:35What is the step right now?
  37. 02:37And what is the possible steps that could come after?
  38. 02:41And within each of the steps, what are concrete things that we can provide to the model so that it is confined to that specific action, it is confined to that specific
  39. 02:51step at that particular moment, and only execute what needs to be done.
  40. 02:56So, by doing this, instead of having a very heavy model like a 4.7, we were actually able to rely on something like a Haiku 4.5, which is a much smaller model,
  41. 03:07doesn't have as much reasoning capabilities, but because of the harnessing around it, it's still able to perform at the level in which we expect, saving money, saving time, and saving latency.
  42. 03:19So, let's go ahead and play this recording, which will show us logs um about a particular lesson.
  43. 03:26So, as you can see in this video, especially on the right side, we see logs on all of the different harnessing that are happening.
  44. 03:34[clears throat] Right?
  45. 03:35So, for example, we see that there's harnessing for a section, which provides input to the model about exactly what to speak about, what to do.
  46. 03:45We have harnessing about drawing on the whiteboard.
  47. 03:48We have harnessing that deals with clearing the queue.
  48. 03:51We have steps to how to end the lesson and everything in between, right?
  49. 03:57So, everything that would allow us to actually build the lesson in a way that is reliable even if there's a new scenario that comes in.
  50. 04:06We try to incorporate that in our state machine.
  51. 04:09We try to incorporate that with within the lesson, right?
  52. 04:12So, again, the model all it worries about is given an input, it knows which action to take and it provides the output of that action.
  53. 04:20Right?
  54. 04:21And so, the model never really um has to think.
  55. 04:25It proposes, but ultimately it is the harness that decides.
  56. 04:29And so, for his A specifically, there are three things that we wanted to think about.
  57. 04:33Like, when is the lesson done is one, right?
  58. 04:35Did the student actually get it right?
  59. 04:37Like, did they actually learn in the way they were supposed to?
  60. 04:40And what comes next, right?
  61. 04:41And so, everything that comes within those three categories, all of the different questions, all of the different actions that the models needs to take, we have engineered that outside
  62. 04:51of the model, right?
  63. 04:52So, again, it's an input, the model receives it and gives us an output, right?
  64. 04:57And so, this is very, very important and we have found this to be very remarkable.
  65. 05:02Right?
  66. 05:02So, again, the the this is applicable to really everything, right?
  67. 05:06It's applicable to something like A that is a voice model.
  68. 05:11It's applicable to coding agents, to Ops Run Books.
  69. 05:15Um it's applicable to onboarding flows, right?
  70. 05:17The same rule applies, right?
  71. 05:19We want to find a way to not let the model think, but building abstractions around it.
  72. 05:25So, a good way to to remember on whether you should use this abstraction is essentially to think about the reliability of your agent, right?
  73. 05:34If it's somewhat of a coin flip, then you want to take the control flow out of the model.
  74. 05:40You want the model to not make as many decisions as it should and instead build those decisions around the model and simply feed an easy input so that the model can easily produce an output,
  75. 05:53right?
  76. 05:53So, don't let the model talk, right?
  77. 05:56Or actually let it talk, but don't let it drive.
  78. 05:59So, we're Joel and Ornella.
  79. 06:01This is Ace, and if you have any questions, please let us know.
  80. 06:05Thank you.
  81. 06:06Thank you.