{
  "video": {
    "id": "OmyqelTwpOo",
    "title": "How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit",
    "duration": 1255,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "OmyqelTwpOo",
    "title": "How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit",
    "one_liner": "A voice agent isn't just a chatbot with a speaker attached. To pick a stack, measure time-to-first-audio and time-to-first-sentence, run slow tools asynchronously so the conversation keeps going, and benchmark the whole conversation on your own simulated callers instead of picking models from leaderboards.",
    "summary": "Jesse Hall of LiveKit argues that the goal for real-time voice agents is to be human-like, not just fast. Latency is a fixed budget: about 1.5 seconds between the caller's last syllable and the agent's first audible response still holds a call, and 500-600 ms starts to feel human. He shows how async tool calls keep a conversation flowing through interruptions, using a hotel-booking demo, and how an audio-aware turn detector cuts down on cutting users off. Leaderboards score individual components on clean audio, but you ship a conversation. So you should build your own benchmark by running simulated callers against a complete agent and swapping models in and out.",
    "key_points": [
      "A traditional voice agent is a cascaded pipeline of speech-to-text, an LLM and text-to-speech. Speech-to-speech real-time models can replace that pipeline, but they are 'not quite there for every use case yet'.",
      "Leaderboards score components on clean audio, but errors compound down the pipeline. If the caller says 'the 3rd' and speech-to-text hears 'the 30th', the LLM plans for the 30th. Real callers are on speakerphone in traffic with kids in the back seat.",
      "TTS time to first byte is misleading. One dashboard showed 210 ms, yet LiveKit tested popular providers that send up to 3/4 of a second of silence after the first byte. Measure time to first audio instead.",
      "LLM time to first token is also misleading, because TTS needs a whole sentence before it can start speaking. Measure time to first sentence: the LLM that produces tokens fastest is not always the one that produces the first sentence fastest.",
      "You can't hide slow backend tool calls (lookups, payment status, disputes). Instead, do what a human would: say 'give me a second' or keep asking questions while the tool runs. A one-line async/await change to a tool call 'hands the mic back' to the agent. The hotel-booking demo kept going through an interruption, saving each detail to a real backend without waiting for confirmation.",
      "LiveKit's turn detector listens to the audio (rhythm, pitch, intonation) as well as the transcript. It cut users off 4.5% of the time, against 9.9% for the leading alternative, in LiveKit's own use cases. Example: '500... and $55.23', where a transcript-only detector would stop at 500 after the pause. The evals are open source.",
      "The method: put simulated callers against a complete working agent. The callers pursue goals, change their mind, interrupt, mumble, and have background noise added. Text simulations test business logic; audio simulations test speech recognition and turn-taking. Score the final outcome (did the correct reservation or authorized amount hit the database?) and the steps taken (10 steps instead of 2, hallucinated function calls). Then swap models while giving every model byte-identical input.",
      "On the hotel benchmark, the top LLM was Gemma 4 31B on LiveKit Inference. The same model on another provider was slower, because LiveKit optimizes its inference serving for voice workloads."
    ],
    "takeaways": [
      "Set a latency budget for what the caller actually hears (about 1.5 s still works, 500-600 ms feels human). Measure time to first audio and time to first sentence, not time to first byte or first token.",
      "Make slow tool calls async so the agent keeps listening and talking while the backend works. Correct course if a tool errors, and hand off to a human as a last resort.",
      "For financial flows, confirm critical dates and amounts and verify identity before any account is accessed.",
      "Build a personalized benchmark: run the complete conversation through simulated callers, text and audio, against your agent. Test final state, privacy safeguards, unnecessary tool calls and turn-taking, swapping models over identical inputs.",
      "Turn every failure you find into a regression scenario. Start with one workflow you care about, listen to the conversations, and check what happened on the backend."
    ],
    "topics": [
      "voice agents",
      "latency",
      "turn detection",
      "interruptions",
      "evals",
      "simulation",
      "tool calling",
      "financial services"
    ],
    "tools": [
      "LiveKit",
      "LiveKit Inference",
      "OpenAI",
      "ChatGPT advanced voice mode",
      "Gemma 4 31B"
    ],
    "quotes": [
      {
        "text": "So a byte does not equal audio.",
        "at": "07:17",
        "url": "https://www.youtube.com/watch?v=OmyqelTwpOo&t=437s"
      },
      {
        "text": "leaderboards, they score these models, these components, but we're shipping a conversation.",
        "at": "05:19",
        "url": "https://www.youtube.com/watch?v=OmyqelTwpOo&t=319s"
      },
      {
        "text": "So, there are as many benchmarks now as there are agents.",
        "at": "17:50",
        "url": "https://www.youtube.com/watch?v=OmyqelTwpOo&t=1070s"
      },
      {
        "text": "a voice agent isn't just a chatbot with a speaker attached.",
        "at": "19:15",
        "url": "https://www.youtube.com/watch?v=OmyqelTwpOo&t=1155s"
      }
    ],
    "words": 4134
  },
  "summary_url": "/#OmyqelTwpOo",
  "transcript": {
    "html": "/transcripts/OmyqelTwpOo.html",
    "txt": "/transcripts/OmyqelTwpOo.txt",
    "vtt": "/transcripts/OmyqelTwpOo.vtt"
  }
}