{ "video": { "id": "z1dqv74SpUs", "title": "Voice agents with Realtime Video — Sidney Primas, LemonSlice", "duration": 1596, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "z1dqv74SpUs", "title": "Voice agents with Realtime Video — Sidney Primas, LemonSlice", "one_liner": "LemonSlice's CTO explains how they turn a video world model into a real-time, interactive avatar: causal attention masking so it can only look at the past, 30 denoising steps collapsed to 1, a private fix for error accumulation over 8–16 hour continuous generations, and inference cheap enough to cost about the same as a voice model.", "summary": "Sidney Primas frames LemonSlice's goal as breaking the \"Avatar Turing test\" — an avatar indistinguishable from a human on a video call — and argues the way there is to take general video/world models and focus them on humans, so emotions, hands, object interactions, physics and micro-expressions come more or less for free rather than being bolted on. He walks through the three things that make a video model usable as a live agent: training it causally with an attention mask so it never sees future audio, distilling ~30 denoising steps down to one, and solving error accumulation so a stream can run for hours without reset. The company sells the visual layer as an API (customers bring their own LLM and voice), and the current work is an \"emotion engine\" that predicts the right action and emotion at the right moment. His long-term bet is a single end-to-end \"EQ model\" taking user video and audio in and emitting avatar video and audio, paired with a separate IQ model for reasoning and tool calls.", "key_points": [ "Launched the day before the talk with Microsoft: a real-time Teddy Roosevelt avatar in a replica Oval Office at his presidential library — full body, hands, actions, movement. President Trump was scheduled for one quick interaction and stayed 10 minutes, leaving and coming back to talk to the avatar again.", "The architectural bet is world models focused on humans rather than the approach other avatar companies use: harder to train and deploy, but full-body movement, object interactions, micro-expressions and physics (demo shows earrings swinging and water moving) emerge instead of being engineered.", "Avatars are built from a single image — photorealistic, Pixar-style or cartoon — and clothes and scene can be changed mid-call on the same inference setup.", "Audio is the lever for expressiveness: most audio encoders are trained on audiobooks, which are monotone and unemotional, so LemonSlice sources emotional audio data and builds its own audio embeddings to drive the video DiT model.", "Two changes turn a video model into a live agent: training with an attention mask so it can only attend to the past (no future audio at inference), and cutting denoising from ~30 steps to a single step.", "Error accumulation is the hard problem — each generated block feeds back its own error — and it matters because the Teddy avatar generates non-stop frame by frame for 8 hours with no reset, with a 16-hour deployment coming. They claim a new solution, different from what everyone else does, that they won't disclose.", "Cost per minute of generated video is roughly the same as a voice model, which he calls crazy given the pixel volume; that is what makes consumer and entertainment customers viable.", "The \"model harness\" (rendered as \"hardness\" in the transcript) — orchestrating separate threads, GPU and CPU work, queues, buffering and interrupts so the stream never stutters — is where he thinks much of the durable value in real-time applications sits.", "Next release, targeted in one to two months: an emotion engine that predicts the right action and emotion from the avatar's audio and the text it is about to say. The model can already produce these behaviors; the gap is making them controllable and deterministic enough to time to a conversation.", "They have not yet built their avatar Turing test; they plan to design one with real people, run it this year, expect to fail it this year, and publish it so others can take it." ], "takeaways": [ "If you are streaming a video model interactively, train it causally with an attention mask from the start — a bidirectional model that peeks at future latents cannot be run live — and budget your engineering on error accumulation, not just per-frame quality.", "Don't reuse off-the-shelf audio encoders for expressive generation; they're trained on audiobooks and will flatten your emotions. Invest in emotional audio data and your own embeddings.", "Distill the denoising schedule aggressively (30 steps → 1) as the primary real-time lever, then treat the CPU/GPU thread orchestration, queues and interrupt handling as first-class product work rather than plumbing.", "If you already run a voice agent, adding a video layer is now roughly cost-neutral — LemonSlice sells it as an API where you keep your own LLM and voice — so the visual layer is worth trying for language learning, sales calls and similar use cases.", "Plan for a split architecture: a high-EQ end-to-end audio/video model for interaction, fed by a separate high-IQ model doing the tool calling and deep thinking. Expect end-to-end EQ models on the market in two to three years." ], "topics": [ "avatars", "world models", "real-time video generation", "diffusion distillation", "voice agents", "multimodal", "inference optimization", "emotion modeling" ], "tools": [ "LemonSlice", "Microsoft", "video DiT (diffusion transformer) model", "audio encoders / audio embeddings" ], "quotes": [ { "text": "he was scheduled to be there for, you know, a quick minute, one interaction, and he actually stayed for 10 minutes.", "at": "03:45", "url": "https://www.youtube.com/watch?v=z1dqv74SpUs&t=225s" }, { "text": "most audio encoders today are trained on basically audiobooks, which is very monotone, very simple, don't have a lot of emotions.", "at": "09:37", "url": "https://www.youtube.com/watch?v=z1dqv74SpUs&t=577s" }, { "text": "the teddy avatar is generating continuously non-stop frame by frame for 8 hours straight with like no reset throughout the entire process.", "at": "12:44", "url": "https://www.youtube.com/watch?v=z1dqv74SpUs&t=764s" }, { "text": "we've been able to make the models small enough and efficient enough so that the costs are about the same as a voice model.", "at": "14:06", "url": "https://www.youtube.com/watch?v=z1dqv74SpUs&t=846s" }, { "text": "If you can describe it in words, you can generate those actions and emotions.", "at": "22:00", "url": "https://www.youtube.com/watch?v=z1dqv74SpUs&t=1320s" } ], "words": 4647 }, "summary_url": "/#z1dqv74SpUs", "transcript": { "html": "/transcripts/z1dqv74SpUs.html", "txt": "/transcripts/z1dqv74SpUs.txt", "vtt": "/transcripts/z1dqv74SpUs.vtt" } }