{
  "video": {
    "id": "18Um2VjMM_g",
    "title": "Speech-to-Speech Model Research at Google DeepMind — Valeria Wu Fon & Tom Ouyang, Google DeepMind",
    "duration": 1002,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "18Um2VjMM_g",
    "title": "Speech-to-Speech Model Research at Google DeepMind — Valeria Wu Fon & Tom Ouyang, Google DeepMind",
    "one_liner": "Google DeepMind's Gemini speech-to-speech team argues that native multimodal pre-training — not cascaded ASR→LLM→TTS pipelines — is what makes universal voice agents possible, demoed via 70+ language live translation in Google Meet, Search Live, the Live API and real-time avatars.",
    "summary": "Valeria Wu Fon (product lead) and Tom Ouyang (engineer) trace speech modelling from pre-2018 component stacks (feature extraction, acoustic, pronunciation and language modelling, second-pass rescoring) through 2018-era end-to-end ASR to today's natively multimodal Gemini models, where audio, video and text share one token embedding space learned in pre-training. Their argument is that capabilities which used to require bespoke engineering — streaming multilingual translation, speaker-voice preservation, noise robustness — now largely fall out of pre-training and become a prompting task. They frame the North Star as a Venn diagram of conversationality, intelligence and multimodality, note that pushing one knob (e.g. high thinking for intelligence) breaks another (latency, time to first audio), and show four demos from one promptable model: Meet live translation, Search Live shopping, an insurance roadside-assistance voice agent, and a Citi avatar pilot from Cloud Next.",
    "key_points": [
      "Pre-2018 ASR stacked separate components — feature extraction, acoustic modelling, pronunciation modelling, language modelling and second-pass rescoring; around 2018 these collapsed into end-to-end neural systems, but those still only did speech→transcription: tone, emotion, speed, word biasing and images all had to be built yourself, capping how far the system could scale.",
      "Gemini pre-training uses multimodal interleaved examples (e.g. a text prompt plus a sequence of video and audio inputs, asking the model to summarise a bedtime story and annotate timestamps), so ASR, TTS, video captioning and agentic tasks are all learned in one unified token embedding space — YouTube videos with captions are cited as effectively limitless training data.",
      "Streaming live translation quality is described as about as good as offline systems that see the full utterance from the beginning — something classically very hard with cascaded systems — while simultaneously doing multilingual switching without knowing the languages in advance, preserving the source speaker's voice, handling multiple speakers and staying noise-robust.",
      "The North Star is a three-way trade-off: conversationality (low latency, snappy, natural), intelligence (task completion, instruction following, reasoning) and multimodality (audio, video, screen sharing, PDFs in and out) — turning thinking up improves intelligence on evals but damages latency and time to first audio.",
      "The majority of Gemini users are non-English speakers, so ITN and localisation work targets far more than EN-US; the Search Live demo localises to Spain Spanish and deliberately leaves 'mid-century' in English because the term is commonly known in English.",
      "Live translation in Google Meet supports 70+ languages, starts translating right after the speaker begins rather than turn-by-turn, so users aren't afraid to interrupt.",
      "'Proactive audio' lets the model decide when not to respond to external input — it won't cut itself short because of background noise or another person talking, on the assumption conversations happen on a train or a walk, not in a sealed office.",
      "One model serves very different products: Search Live (real-time video+audio in, tool call to pull search cards), an insurance roadside-assistance agent needing alphanumeric accuracy for registration plates and postcodes, and a Citi pilot at Cloud Next with customisable real-time avatars — hyperrealistic human through cartoon — with low-latency multilingual lip-sync."
    ],
    "takeaways": [
      "Stop assembling ASR + LLM + TTS cascades for voice agents; a natively multimodal speech-to-speech model gets real-time translation, paralinguistics and multi-speaker handling from pre-training rather than from your glue code.",
      "Treat capability as a prompt, not an architecture: the same model handles streaming translation, image+audio Q&A or an embodied avatar agent depending on how you prompt it — build for switching between conversation modes rather than one pipeline per mode.",
      "Measure the trade-off explicitly — if you raise the thinking budget for intelligence, check what it does to latency and time to first audio, because evals showing better reasoning will hide a worse conversation.",
      "Design for noisy, on-the-go environments: use features like proactive audio so the model doesn't truncate itself on background speech, and test alphanumeric accuracy (postcodes, registration plates) for agent use cases.",
      "Don't scope quality work to EN-US — most users are non-English speakers, so ITN, localisation and knowing which loanwords to leave untranslated are core, not polish."
    ],
    "topics": [
      "speech-to-speech",
      "voice agents",
      "multimodal",
      "gemini",
      "translation",
      "latency",
      "asr",
      "avatars"
    ],
    "tools": [
      "Google DeepMind",
      "Gemini",
      "Gemini Live",
      "Search Live",
      "Live API",
      "Google Meet",
      "Google Cloud",
      "Cloud Next",
      "Citi",
      "YouTube"
    ],
    "quotes": [
      {
        "text": "we believe that AGI will not be typed, that it will be spoken",
        "at": "15:53",
        "url": "https://www.youtube.com/watch?v=18Um2VjMM_g&t=953s"
      },
      {
        "text": "again, it would be very hard to try to engineer this, but then with the LLM and Gemini models, this almost becomes a prompting task",
        "at": "05:21",
        "url": "https://www.youtube.com/watch?v=18Um2VjMM_g&t=321s"
      },
      {
        "text": "once you move one of the knobs, it's very easy for the other knobs to kind of like mess up",
        "at": "07:47",
        "url": "https://www.youtube.com/watch?v=18Um2VjMM_g&t=467s"
      },
      {
        "text": "Because we realized that the majority of these conversations are not happening in an office room with like no noise, you know, in a sealed environment. They're happening on the go, on the train, while you're on a walk.",
        "at": "13:51",
        "url": "https://www.youtube.com/watch?v=18Um2VjMM_g&t=831s"
      }
    ],
    "words": 3648
  },
  "summary_url": "/#18Um2VjMM_g",
  "transcript": {
    "html": "/transcripts/18Um2VjMM_g.html",
    "txt": "/transcripts/18Um2VjMM_g.txt",
    "vtt": "/transcripts/18Um2VjMM_g.vtt"
  }
}