{ "video": { "id": "MBHOH1NmDqc", "title": "Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI", "duration": 794, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "MBHOH1NmDqc", "title": "Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI", "one_liner": "EliseAI's Bo Li walks through the three tricks — a speculative streaming+batch transcriber, background agents that fake the main agent's tool calls, and a TTS prefix cache — that let a cascaded voice agent feel real-time while still running slow frontier LLMs.", "summary": "Li frames cascaded voice agents through his self-driving background: transcription is perception, the LLM is planning, TTS is controls, and each layer can be optimised for speed without giving up frontier-level intelligence. He walks a single caller utterance (\"Sure, it's Elise Trial, date of birth 2303\") through all three layers, showing a fast streaming transcriber being corrected by a slower context-aware one, eager LLM generations that are cancelled and retriggered, tool calls hoisted into background agents, and cached audio prefixes played while the real TTS engine is still generating. He closes with a recorded end-to-end call where an OBGYN receptionist agent books an ultrasound, and a pitch for EliseAI, which works on housing and healthcare.", "key_points": [ "Cascaded voice agents map onto the self-driving stack: perception (transcription, in place of camera/lidar bounding boxes), planning (the LLM), and controls (turning text into audio instead of turning a trajectory into steering).", "The \"streaming speculative transcriber\" layers a fast streaming transcriber (Flux) with a slower, more accurate batch transcriber (Scribe V2); the corrective pass is skipped when it agrees, and cancelled outright when newer audio arrives, because more context beats an older accurate result.", "Scribe V2 uses the context of the agent's question (\"can you provide your name and date of birth\") to correct the streaming transcript into a name and a date of birth; late detections that are only punctuation are ignored.", "Because frontier LLMs are slow, the goal is to cut round trips, and tool calling is the main cause of extra inferences — so background agents make the tool calls and their results are pushed into the main agent's context, making it think it already called the tool.", "Each transcript detection triggers an eager agent generation that is held back until the user is confirmed to have finished speaking; the eager tool-call-free generation gets cancelled and retriggered once the background agent resolves the name and date of birth.", "The background tool call does real work: correcting mis-transcriptions of names and doing phonetic matching.", "The TTS layer keeps a prefix cache of audio for previously generated word sequences; it waits ~3 words before hitting, so a common opener like \"you said your name is\" hits, and the name itself causes the cache miss.", "On the miss, the cached audio is played immediately while Cartesia (a TTS engine with WebSocket support) generates the full sentence with natural prosody unaware of the cache; the overlapping portion of Cartesia's audio is suppressed and only the remainder is played, leaving at most a hiccup the user won't notice.", "A recorded demo call shows the agent handling an ultrasound booking end to end: name/DOB capture, new-patient check, texting an insurance upload link, waiting on \"give me a second, let me check my calendar\", and offering a Tuesday July 7th 2pm or 3pm slot with Dr. Avery Stone." ], "takeaways": [ "Don't choose between a fast and an accurate transcriber — run both, let the streaming one drive latency, and let the slower context-aware one issue corrections that are cancelled whenever fresher audio arrives.", "Speculatively generate agent responses on every partial transcript but hold the emission until end-of-utterance, so a cancelled guess costs nothing and a correct one is already finished.", "Move tool calls off the critical path into background agents and inject the results into the main agent's context as if it had called them itself, to cut LLM round trips.", "Cache audio for common response prefixes, gate the hit on a few words, and play the cached audio while the real TTS finishes — then suppress the overlapping generated audio and splice in the remainder.", "Treat latency as a harness problem: most of the perceived speed came from orchestration tricks around the models, not from choosing faster, dumber models." ], "topics": [ "voice agents", "latency", "speech-to-text", "text-to-speech", "caching", "tool calling", "agent architecture", "healthcare" ], "tools": [ "EliseAI (transcribed as \"Xnor.ai\")", "Flux", "Scribe V2", "Cartesia", "Twitter/X (@EliseAI)" ], "quotes": [ { "text": "to me, cascaded voice agents makes a lot of sense when you view it in lens of kind of breaking it down into perception", "at": "00:50", "url": "https://www.youtube.com/watch?v=MBHOH1NmDqc&t=50s" }, { "text": "push the tools back into the context of the main agent so that it thinks it made the tool call, but um but it it it really didn't.", "at": "04:11", "url": "https://www.youtube.com/watch?v=MBHOH1NmDqc&t=251s" }, { "text": "at this point, the user hears the agent and user doesn't really know what's going on. They just looks like really fast response times to them.", "at": "08:23", "url": "https://www.youtube.com/watch?v=MBHOH1NmDqc&t=503s" }, { "text": "there's a lot of effort that can be done in the harness to really kind of get a um natural conversation, which is what we're after.", "at": "11:49", "url": "https://www.youtube.com/watch?v=MBHOH1NmDqc&t=709s" } ], "words": 2390 }, "summary_url": "/#MBHOH1NmDqc", "transcript": { "html": "/transcripts/MBHOH1NmDqc.html", "txt": "/transcripts/MBHOH1NmDqc.txt", "vtt": "/transcripts/MBHOH1NmDqc.vtt" } }