{
  "video": {
    "id": "j4AmIK6OvHc",
    "title": "Latency Is a Budget. Humanlike Is the Goal. — Jesse Hall, LiveKit",
    "duration": 1021,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "j4AmIK6OvHc",
    "title": "Latency Is a Budget. Humanlike Is the Goal. — Jesse Hall, LiveKit",
    "one_liner": "For voice agents, latency is a fixed budget (about 1.5s end to end at most, around 600ms to feel human), not the goal. Spend it on the most humanlike stack you can fit, and choose that stack by testing the whole conversation, not by reading component leaderboards.",
    "summary": "Jesse Hall argues that the usual fix for voice agents, which is more speed, is the wrong answer. The goal is humanlike behaviour: listening while talking, telling a finished thought from a pause, and rolling with changes of mind. Speed only sets the budget. He describes the 'leaderboard trap', where models are scored as components on clean audio but shipped as a cascade. He also shows how measured latency (time to first byte or token) differs from perceived latency (time to first audio or first complete sentence). He demos async tool calls in a hotel-booking agent, presents LiveKit's audio-based turn detector, and introduces an open-source end-to-end benchmark you can run against your own agent.",
    "key_points": [
      "Voice pipelines chain three model slots: STT → LLM → TTS. Errors flow downhill. If STT hears 'Tuesday the 30th' instead of 'Tuesday the 3rd', the LLM confidently books the wrong day. Real-time speech-to-speech models exist but aren't right for every use case.",
      "Leaderboard trap: scores come from clean audio in a quiet room, but real callers are on speakerphone, in traffic, with kids in the back seat. 'What is the best STT/LLM/TTS?' is a different question from 'which stack should I ship?'",
      "TTS time to first byte is misleading. LiveKit measured popular providers sending hundreds of milliseconds of silence at the start of their audio, and up to 3/4 of a second in the worst case. Time to first audio is what the caller hears.",
      "LLM time to first token is a text metric. TTS needs a complete first sentence, and the fastest token streamer isn't necessarily the first model to finish a sentence.",
      "Tool calls can take 2, 10 or 40+ seconds, and no model choice can hide that. Async tools fix it: a normal tool with one extra 'first update' line hands the mic back to the agent while the tool keeps working, so 'the conversation and the tool stop sharing a thread.' The Live Oak Hotel demo handled an interruption and a mid-booking date change.",
      "LiveKit's turn detector works on the audio itself (rhythm, pitch, intonation). At the same latency budget, the best alternative cuts users off 9.9% of the time and LiveKit's detector 4.5%. The eval harness is open source.",
      "The latency window: end-of-turn detection fires from audio before the transcript arrives, the LLM runs on a provisional transcript (saving about half a second when the final words match), and TTS starts on the first complete sentence. Up to 1.5s end to end holds the conversation but is 'right on the edge'; around 600ms 'starts to feel really human'.",
      "The benchmark runs hundreds of simulated callers (who interrupt, mumble and change their mind) against a complete open-source hotel agent with a real backend. It scores human-equivalent outcomes: correct reservation in the DB, cut-offs, correct names and dates, invented tool calls, leaked personal data. It also penalises unnecessary questions and extra steps, because completion alone is gameable. Stacks are swapped with byte-identical inputs. A tuned Gemma 4 31B on LiveKit Inference passes the voice evals at 88% and starts speaking in about 380ms, over 2x faster than the next model; it was due to launch the next day."
    ],
    "takeaways": [
      "Set your latency budget first (≤1.5s end to end, aiming for ~600ms), then spend whatever is left on model quality, not on speed that doesn't matter.",
      "Measure perceived latency: time to first audio and time to first complete sentence, not TTFB or TTFT on a vendor dashboard.",
      "Make any tool call that takes over a second asynchronous, so the agent can buy time ('give me a sec') and keep listening.",
      "Test the whole conversation, not the components: run simulated noisy, interrupting, mind-changing callers against your own agent and score outcomes as you would for a human. Then keep testing in production.",
      "Use audio-based end-of-turn detection and start the LLM on provisional transcripts to save time inside the window."
    ],
    "topics": [
      "voice agents",
      "latency",
      "evals",
      "benchmarks",
      "turn detection",
      "tool calling",
      "real-time ai"
    ],
    "tools": [
      "LiveKit",
      "LiveKit Inference",
      "ChatGPT Advanced Voice Mode",
      "OpenAI",
      "Gemma 4 31B"
    ],
    "quotes": [
      {
        "text": "Leaderboards score components, but you are shipping a cascade, a pipeline.",
        "at": "04:07",
        "url": "https://www.youtube.com/watch?v=j4AmIK6OvHc&t=247s"
      },
      {
        "text": "The dashboard says it's really fast, but your ears say it's slow.",
        "at": "06:18",
        "url": "https://www.youtube.com/watch?v=j4AmIK6OvHc&t=378s"
      },
      {
        "text": "The conversation and the tool stop sharing a thread.",
        "at": "08:12",
        "url": "https://www.youtube.com/watch?v=j4AmIK6OvHc&t=492s"
      },
      {
        "text": "A voice agent isn't a chatbot with a speaker stacked onto it.",
        "at": "15:33",
        "url": "https://www.youtube.com/watch?v=j4AmIK6OvHc&t=933s"
      }
    ],
    "words": 3007
  },
  "summary_url": "/#j4AmIK6OvHc",
  "transcript": {
    "html": "/transcripts/j4AmIK6OvHc.html",
    "txt": "/transcripts/j4AmIK6OvHc.txt",
    "vtt": "/transcripts/j4AmIK6OvHc.vtt"
  }
}