{
  "video": {
    "id": "OpY6MmZFeHo",
    "title": "Voice Agents Can Just Do Things — Charlie Guo, OpenAI",
    "duration": 950,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "OpY6MmZFeHo",
    "title": "Voice Agents Can Just Do Things — Charlie Guo, OpenAI",
    "one_liner": "OpenAI DevEx's Charlie Guo argues the big misconception about voice agents is that they must talk back — the underexplored win is speech-to-action, where voice drives your existing app's API endpoints and React hooks as tools and the model responds with UI, not words.",
    "summary": "Guo lays out three remixable modes of voice interaction — speech-to-speech, speech-to-action, and event-to-speech — and argues speech-to-action is the biggest 'capability overhang' because a model can respond by changing UI state, showing a notification, or moving a ghost cursor rather than speaking. He makes the case for native audio-in/audio-out models over the chained transcribe→LLM→TTS pipeline, since transcription throws away tone, cadence, interruption cues and background noise, and because ChatGPT's first two chained voice modes had significantly higher latency than native advanced voice mode. He closes with a plug for GPT Realtime 2 (reasoning before speaking, parallel tool calls, preambles, wake words) and a reframed starting question for builders.",
    "key_points": [
      "Three emerging modes: speech-to-speech (live coaching/language practice, 'concierge experiences' i.e. customer support plus-plus, live translation), speech-to-action (model uses tools), and event-to-speech (model receives an event and talks to you). They aren't new — the Moviefone showtimes hotline was arguably speech-to-speech, in-car GPS navigation was event-to-speech — and they're meant to be remixed, not chosen between.",
      "Speech-to-action is 'the next capability overhang': form filling ('affects a significant GDP of the internet' — talk for five minutes and get 90% of a government form, then check it), creative tools where 'my taste exceeds my capability' because he can't drive Photoshop or Ableton the way he can drive Codex, and generalized computer use — 'why am I not just talking to the entire computer?'",
      "Retrofitting voice onto existing software is easier than developers think: a modern web app already exposes nouns and verbs — API endpoints and React hooks — each of which converts relatively easily into a tool the model can call, letting users drive existing software by voice (still with guardrails and safety checks).",
      "A voice model's response doesn't have to be speech: notifications, popups, state changes like a button colour or drop shadow, highlighted text, or the ghost-cursor animation in the Codex app all communicate what's happening without words.",
      "Event-to-speech through-lines: hands-free/screen-free use where attention is diverted (e.g. a recipe app while cooking) and proactive outreach — treating voice as the top rung of an escalation path: animate something, then pop something up, then speak to get the user's attention. No developer would show all their logs, and none would speak them either.",
      "Accessibility is the underlying theme: Guo knows multiple developers who lost hand mobility or finger dexterity and thought their programming careers were over; with LLMs, coding agents and voice agents they now generate orders of magnitude more code than before.",
      "Why native audio beats the chained (transcribe → LLM → text → TTS) pipeline: the realtime model family does no transcription and is trained on native audio as tokens, audio in / audio out, with the industry also letting go of the turn-based abstraction in favour of continuous streaming. Transcription loses tone, cadence, emotional impact, interruption attempts and background noise — citing the old saying that 55% of face-to-face information is body language, 38% tone of voice, and 7% the actual words. Quantitatively, ChatGPT's first two chained voice modes had significantly higher latency than native advanced voice mode.",
      "GPT Realtime 2 (released a couple of months before the talk) brings reasoning to audio so the model can 'think before it speaks' with a tunable reasoning budget, plus parallel tool calls, longer context, better domain understanding, more natural voices, more steerability, and wake words you can name (e.g. 'hey, Marin'). Because thinking and tools add latency, preambles let you prompt the model to warn the user — like a travel agent saying 'I'm going to go check flight prices, give me a couple seconds' — while it acts in the background."
    ],
    "takeaways": [
      "Don't start from 'what kind of voice agent am I building?' — start from 'what is the role of voice and audio in this interaction?', then ask what the model can perceive, what context it has, which tools are available and which it should execute safely, whether it should communicate now or wait, and whether to answer visually or by speaking.",
      "Turn the API endpoints and React hooks you already have into model tools so users can drive your existing app by voice — add guardrails and safety checks, but don't treat voice as a rewrite.",
      "Let the model reply through the UI: notifications, popups, colour/drop-shadow state changes, text highlights, a moving cursor. Reserve speech for the top of an escalation path when quieter channels fail to get attention.",
      "Prefer native audio-token models (audio in, audio out, continuous streaming) over a chained STT→LLM→TTS pipeline — you keep tone, cadence, interruptions and background noise, and you cut latency.",
      "When you give a realtime model reasoning budget or parallel tool calls, prompt it with preambles so it tells the user what it's doing instead of leaving dead air."
    ],
    "topics": [
      "voice agents",
      "realtime api",
      "tool calling",
      "multimodality",
      "latency",
      "accessibility",
      "computer use",
      "ux design"
    ],
    "tools": [
      "OpenAI",
      "GPT Realtime 2",
      "OpenAI realtime model family",
      "ChatGPT advanced voice mode",
      "Codex",
      "Codex app computer use",
      "Photoshop",
      "Ableton",
      "Spotify",
      "Google Maps",
      "CarPlay",
      "Moviefone"
    ],
    "quotes": [
      {
        "text": "speech is not the only way that a voice model has to respond.",
        "at": "01:02",
        "url": "https://www.youtube.com/watch?v=OpY6MmZFeHo&t=62s"
      },
      {
        "text": "I actually almost titled this talk uh voice is the next capability overhang because I think there is just a vast vast amount of stuff um that we could be doing in this category that we are not currently doing.",
        "at": "04:15",
        "url": "https://www.youtube.com/watch?v=OpY6MmZFeHo&t=255s"
      },
      {
        "text": "if the models are just getting good enough to do everything on a computer that a human can do, like why am I talking to an app? Why am I talking to a terminal? Why am I not just talking to the entire computer?",
        "at": "06:08",
        "url": "https://www.youtube.com/watch?v=OpY6MmZFeHo&t=368s"
      },
      {
        "text": "I do believe that AGI will be spoken, not typed.",
        "at": "15:21",
        "url": "https://www.youtube.com/watch?v=OpY6MmZFeHo&t=921s"
      }
    ],
    "words": 3671
  },
  "summary_url": "/#OpY6MmZFeHo",
  "transcript": {
    "html": "/transcripts/OpY6MmZFeHo.html",
    "txt": "/transcripts/OpY6MmZFeHo.txt",
    "vtt": "/transcripts/OpY6MmZFeHo.vtt"
  }
}