{
  "video": {
    "id": "vblnYHzBgS4",
    "title": "5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo",
    "duration": 1606,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "vblnYHzBgS4",
    "title": "5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo",
    "one_liner": "A Plivo founder's field report from over a billion monthly voice calls: the five things that break when a voice agent leaves the demo — latency, brittle transcription, data collection, TTS normalization and turn-taking — plus the specific model sizes, thresholds and layers they use to fix each.",
    "summary": "Venky B, founder/CEO of Plivo, argues that stitching STT → LLM → TTS with turn detection on LiveKit or Pipecat gets you a demo, not a production agent, and walks through five failure modes his team sees in customer deployments. The core engineering claim is that voice agents must balance cost, intelligence and latency, and that because thinking has to be off, the last year of LLM reasoning gains simply don't apply — so self-hosted open-source models (Qwen 3.5, Gemma 4) under 300ms beat both frontier APIs and expensive dedicated Groq/Cerebras capacity. He then argues that transcription, data collection and TTS should each be wrapped in your own normalization layer rather than trusted to the vendor engine, and that data collection in particular should be modelled like Pydantic/Zod form fields with field-level unit tests instead of prompt engineering.",
    "key_points": [
      "Latency: most teams target the advertised sub-550ms time-to-first-audio, but actually land at 750ms–1.2s; past 1.2s users start hanging up.",
      "The irony of voice agents: the talking LLM almost always has thinking turned off, so a year of reasoning/RL advances doesn't apply — only better instruction following and tool calling carry over.",
      "Frontier models (OpenAI, Claude, Gemini) run ~450–500ms p50 TTFT but spike to 1.2–1.3s at P90/P95; Groq and Cerebras need dedicated capacity that is very expensive and booked out 12 months in advance, which is a big bet that your model still exists in a year.",
      "Plivo's answer is self-hosted open source: Qwen 3.5 and Gemma 4 consistently under 300ms. For multilingual, Gemma 4 is 2.5–3x better than Qwen 3.5 on token fertility (tokens needed per word), so time-to-words is much faster. A 3–4B mixture-of-experts gets you 90% of the way out of the box but is hard to fine-tune without breaking; if you need fine-tuning for a domain like healthcare, start at 8B–12B. Some teams run a mixture of models: a small ~3B conversational model plus a larger model for tool calling.",
      "Assume transcription is brittle: state-of-the-art engines hit 4–6% WER on known eval sets but double digits on real noisy calls with accents and domain vocabulary. Predictable break points are proper nouns, jargon, phone numbers with dropped digits, wrong substitutions, long addresses and code-switched languages (English written in Devanagari, Hindi written in Roman), which corrupt the LLM and then the TTS downstream.",
      "Transcription fixes: dynamic keyword boosting tied to call state rather than a static list for the whole call (a context stuffed with keywords starts hallucinating); post-process transcripts with an LLM, which has the domain context the ASR lacks — it knows the 'E' in a phone number is a 3; normalize transliteration with an LLM or an open-source neural transliteration engine so the LLM always gets clean text regardless of which engine you use.",
      "Data collection is where 50–60% of agents fail; treating it as a voice UX/data-model problem — Python dataclasses, Pydantic, Zod, form fields — took accuracy from 30% to 95%. Decide the shape before you ask: a phone-number-typed field knows the digit count, allowed values and can validate or smart-guess-and-confirm; hard names need letter-by-letter spelling and confirmation rules; relative values like 'next week Wednesday 8' become tractable as a datetime field resolved against the current date via tool calling.",
      "Evals should treat each field as a unit test rather than running hundreds of end-to-end agent tests to discover one broken field; this got them to 95–97% accuracy with no fine-tuning.",
      "Put a normalization layer between the LLM and the TTS instead of feeding LLM output straight in: strip emojis and markdown (LiveKit/Pipecat do this behind flags), set custom pronunciation dictionaries for brands and acronyms, drop to 0.7–0.8x speed when enunciating an email, phone number or name, and normalize emails, currency and dates yourself so you can swap TTS vendors or survive one going down.",
      "Barge-in and back-channeling do not require a speech-to-speech model — Plivo does all of it in a speech pipeline."
    ],
    "takeaways": [
      "Stop defaulting to frontier LLM APIs for the conversational turn; benchmark a self-hosted Qwen 3.5 or Gemma 4 at 3–4B MoE against a sub-300ms budget, and pick Gemma 4 if you're multilingual. Only move to 8B–12B if you actually need to fine-tune for a domain.",
      "Build your own layers on both sides of the LLM: a transcript-cleaning layer (dynamic per-state keyword boosting + LLM post-processing + transliteration normalization) and a pre-TTS normalization layer (emoji/markdown stripping, pronunciation dictionaries, speed control, email/currency/date expansion) so you are not locked to any one STT or TTS vendor.",
      "Model every piece of data you collect as a typed, constrained field with validation and a confirmation strategy — like Pydantic or a UI form — instead of hoping a longer prompt makes the LLM extract it from a transcript.",
      "Move your evals down to the field level and run them as unit tests, rather than debugging via end-to-end conversation tests.",
      "Use a pronunciation smoke test on your own hard cases — the speaker checks whether an agent can say 'Balasubramanian' and 'Plivo' before he takes it seriously."
    ],
    "topics": [
      "voice-agents",
      "latency",
      "speech-to-text",
      "tts",
      "open-source-llms",
      "evals",
      "structured-data-collection",
      "production-ai"
    ],
    "tools": [
      "Plivo",
      "LiveKit",
      "Pipecat",
      "Qwen 3.5",
      "Gemma 4",
      "Groq",
      "Cerebras",
      "OpenAI",
      "Claude",
      "Gemini",
      "Pydantic",
      "Python dataclasses",
      "Zod",
      "TypeScript",
      "SIP trunking"
    ],
    "quotes": [
      {
        "text": "So all the advancements we've had in the LLM layer in the last one year like none of that even apply here now.",
        "at": "07:20",
        "url": "https://www.youtube.com/watch?v=vblnYHzBgS4&t=440s"
      },
      {
        "text": "assume your transcriptions are going to be brittle",
        "at": "13:39",
        "url": "https://www.youtube.com/watch?v=vblnYHzBgS4&t=819s"
      },
      {
        "text": "This is where I think 50 to 60% of AI agents mess up pretty badly.",
        "at": "18:04",
        "url": "https://www.youtube.com/watch?v=vblnYHzBgS4&t=1084s"
      },
      {
        "text": "so my first test is if it cannot pronounce my last name or my company's name it's already dropping the ball",
        "at": "24:43",
        "url": "https://www.youtube.com/watch?v=vblnYHzBgS4&t=1483s"
      }
    ],
    "words": 5493
  },
  "summary_url": "/#vblnYHzBgS4",
  "transcript": {
    "html": "/transcripts/vblnYHzBgS4.html",
    "txt": "/transcripts/vblnYHzBgS4.txt",
    "vtt": "/transcripts/vblnYHzBgS4.vtt"
  }
}