{
  "video": {
    "id": "42Bz0TUfeOQ",
    "title": "The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents, Arize",
    "duration": 1080,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "42Bz0TUfeOQ",
    "title": "The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents, Arize",
    "one_liner": "A voice-agent transcript can look like the agent corrected itself while the real call had 2.4 seconds of dead air, the agent talking over the caller and a refund issued for order 40 instead of 14. To catch failures like that, keep the audio, the transcript, the trace and audio-based evals together in one session view.",
    "summary": "The speaker is a product manager at Arize. He argues that voice agents are growing fast but are among the hardest systems to debug, because 'the log lies': latency, turn-taking, interruptions, tone and transcription drift don't show up in text logs. His main example is an AI-generated refund call. He shows how a session-level trace with inline audio playback, timing metrics and the agent's tool calls exposes failures the transcript hides. He also argues for evals that run on the audio itself rather than LLM-as-a-judge on transcripts, and ends with an observe → evaluate → improve loop. In that loop, SRE agents propose and validate fixes as PRs for a human to approve.",
    "key_points": [
      "Refund demo: the transcript reads as 'refund for order 14' → 'started a refund for order 40' → 'No, 14', which looks like the agent corrected itself. The audio and trace show 2.4 seconds of dead air before the first response, the agent talking over the caller in turn three, a flat robotic tone, and a tool call that actually refunded order 40.",
      "A useful debugging view has three things: (1) audio, transcript and trace in one place; (2) the whole multi-turn session in a single trace view, with the session ID stitching every turn and websocket event into a timeline tagged user or assistant; (3) metrics such as time to first audio/token, audio token cost, interruption events and sentiment.",
      "Arize has mapped audio semantic conventions onto OpenInference, its open-source, OTel-compliant GenAI conventions (first released in 2023). Websocket events from providers like OpenAI Realtime and Gemini Live become one provider-agnostic, queryable schema covering the session lifecycle, audio input, transcript, conversation items, output, response lifecycle and token counts. It is 'one auto instrumenter' with 'no bespoke parsing per vendor', and you can send the data to any backend, including Grafana.",
      "Evals specific to audio matter: classifying tone and sentiment from the actual audio (positive/neutral/negative), time-to-first-audio SLAs for customer service, interruptions, inaccuracy, transcription drift and task success. These should run against the audio using audio models, not as LLM-as-a-judge over the transcript.",
      "The industry is moving from LLM-as-a-judge toward agent-as-a-judge: judges that call tools and third-party systems, for example running RBAC checks that the agent only accessed systems it was allowed to.",
      "You can create evals in Arize through 'skills' with a plain-language prompt, e.g. 'Hey Claude, create an audio eval for me for sentiment analysis using GPT audio and catch frustrated user tone on those audio spans.' The CLI then queries the spans, filters to audio, creates the eval, deploys it and runs it live against incoming traffic.",
      "Putting eval scores on the spans themselves allows complex filtering. Monitors on those evals can trigger investigation agents that look for failing tool calls, unhandled product areas, new edge cases or missing tools.",
      "Arize's newly released 'agent experiments' feature is described as 'Postman with tracing'. The pictured future: an SRE agent finds a latency cause (tool results weren't being truncated, so context got overloaded), adds truncation, stands the fix up in dev, replays the failed traces against it, and opens a PR with a latency report card for a human to approve or deny."
    ],
    "takeaways": [
      "Don't debug voice agents from transcripts alone. Instrument the audio (the speaker suggests the open-source OpenInference instrumenter, sent to Arize or any OTel backend) so you can replay calls inline next to the trace and tool calls.",
      "Track time to first audio and interruption events as first-class metrics, and alert when they break your latency SLAs.",
      "Run evals against the audio itself (tone/sentiment, interruptions, transcription drift, task success), not only LLM-as-a-judge over transcripts. Attach the scores to the spans so you can filter on them.",
      "Use a provider-agnostic schema so you can query across all your traces and swap realtime providers (OpenAI Realtime, Gemini Live) as new models come out, without writing a parser per vendor.",
      "Give voice agents the same observe → evaluate → improve rigor as coding agents, including replaying failed traces against a candidate fix to confirm it works before approving it."
    ],
    "topics": [
      "voice agents",
      "observability",
      "tracing",
      "evals",
      "latency",
      "opentelemetry",
      "agent-as-a-judge",
      "self-healing"
    ],
    "tools": [
      "Arize",
      "Arize AX",
      "OpenInference",
      "OpenTelemetry",
      "Grafana",
      "OpenAI GPT Realtime",
      "GPT audio",
      "Google Gemini Live",
      "Vercel",
      "Claude",
      "Postman",
      "White Castle",
      "Bojangles",
      "McDonald's",
      "Robinhood"
    ],
    "quotes": [
      {
        "text": "Uh, and the log lies.",
        "at": "02:01",
        "url": "https://www.youtube.com/watch?v=42Bz0TUfeOQ&t=121s"
      },
      {
        "text": "I'd be pretty pissed off if I was at McDonald's at 2:00 a.m. and I got 600 cheeseburgers.",
        "at": "02:30",
        "url": "https://www.youtube.com/watch?v=42Bz0TUfeOQ&t=150s"
      },
      {
        "text": "you're not just reading transcripts, you're listening to audio",
        "at": "08:20",
        "url": "https://www.youtube.com/watch?v=42Bz0TUfeOQ&t=500s"
      },
      {
        "text": "Agent experiments, you could think of it kind of like Postman with tracing.",
        "at": "14:35",
        "url": "https://www.youtube.com/watch?v=42Bz0TUfeOQ&t=875s"
      }
    ],
    "words": 3822
  },
  "summary_url": "/#42Bz0TUfeOQ",
  "transcript": {
    "html": "/transcripts/42Bz0TUfeOQ.html",
    "txt": "/transcripts/42Bz0TUfeOQ.txt",
    "vtt": "/transcripts/42Bz0TUfeOQ.vtt"
  }
}