Voice agents with Realtime Video — Sidney Primas, LemonSlice

AI Engineer · 26 min · 274 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12My name is Sydney.
00:14I'm the CTO and founder of Lemon Slice.
00:19And Lemon Slice is on a mission to break the Avatar Turing test.
00:27What we mean by this is making an Avatar that is indistinguishable from a human on a video call.
00:39And all of this is of course making the Avatar photo-realistic.
00:44But there's actually a long tail of technical problems that we care a lot about that I'll be talking about today that we're planning to solve.
00:53And those problems are things like getting the emotions right, getting the object interactions right, getting the like micro-expressions right, and even figuring out an internal state for these Avatars
01:08so that they feel like a real human being.
01:11So what I'm going to do in this Oh, this What I'm going to do in this talk is talk about our technical bet to solving the Avatar Turing test,
01:22what we've built today, and the technical the hard technical problems we've solved today, and our bet on like how we think this problem will be solved in the long term.
01:33But before we dive into that I want to give you guys a sense for what our product is and how it works.
01:41So actually yesterday was a was a pretty big and exciting day for us.
01:45We had a big big launch yesterday.
01:48We had a partnership with Microsoft that brought an ex-US president to life, uh Roosevelt.
01:57And they built his presidential library and inside of the presidential library uh they did a replica of the Oval Office and you can walk in and you can have a full interaction
02:07uh with Teddy Roosevelt in real time.
02:10Uh so definitely not uh Turing test solved yet, but like a big milestone uh towards trying to solve solve that problem.
02:18Uh and the coolest part of all of this was uh we had a special guest.
02:24Uh but hate hate him or love him, uh he showed up.
02:28Uh so we had President Trump uh who came to the opening uh of this ceremony uh and interacted uh with our avatar.
02:36So uh we actually got some footage, real footage, uh not not fake uh AI-generated footage of Trump uh interacting.
02:44So let me let me show you guys this real quick.
02:50your greatest achievement?
02:57Right then, yes.
02:59What is that answer?
03:01The canal stands as one of my proudest battles, no question.
03:04But greatness is a strange thing.
03:06It isn't always the biggest or boldest job that matters most.
03:10Panama Canal showed what America could achieve if we held steady and acted fast when the world dragged its feet.
03:17That said, I measure my greatest work by the lives improved.
03:22Mark set aside, food and drugs made safe, the square deal given to all, not just to a few.
03:29Still, when I stood in the mud watching those steam shovels, knowing ships would pass through, changing the world's map forever, I felt I'd left a mark that would last.
03:39Perfect.
03:40Uh so you guys get a sense uh of of the experience.
03:44Um the funny thing is he was scheduled to be there for, you know, a quick minute, one interaction, and he actually stayed for 10 minutes.
03:54He left He left the room and then came back again to talk to Teddy Teddy Roosevelt even more.
04:01So, you can see there like the avatars there.
04:04It's full body avatar.
04:05It has hands.
04:06It does actions.
04:06It does movement.
04:08And all of this is obviously done in real time.
04:12I'll talk a little bit more about like the why we think this is an important problem later in the talk.
04:20Um but a big part of it is actually related to this interaction.
04:25Um humans like struggle to to to focus and pay attention.
04:30It's It's often hard for a lot of people spend a bunch of time reading.
04:35And even voice it's hard to really lock in and understand what's happening.
04:39So, we think like we're biologically wired to best like understand things when there's a visual component as well.
04:47So, our bet here is in the long term we think most interactions between AI and humans will have a visual visual layer.
04:58And we're building that visual layer.
05:02Um so, now I want to talk about our approach to building these avatars.
05:11It's a very different approach than what most other avatar companies use.
05:17Essentially what we do is we take these world models and we focus them on humans.
05:25Um and the reason we take this bet is even though it's harder to get the initial model working, it's harder to train the model, it's harder to deploy the model.
05:36Once you have a model, you get all of these nice emergent properties that we all know about where you just very easily can solve things like full body movement,
05:47like object interactions, like movements in the scene, all the way down to the micro micro expressions as well and emotions.
05:58And all of those kind of don't come for free, but come more for free than when you use the other approaches that other people use.
06:06So, let me guys let me show you guys real quick the product in action, so you guys can get a sense.
06:14Uh so Let me expand this a little bit.
06:32How is it going?
06:33I am here to listen and help with anything you need.
06:36We can chat about something in particular or play a game if you would like.
06:41So, the lipstick might be off here a little bit, uh but that's just because of the AV setup here.
06:48Uh but as you can see, um we have actions, they can wave, they really can do any actions.
06:55Uh we have movement, we have hands, um and the way the way this works is it it just can use any single image to create the avatar.
07:05Uh and so yes, you can have photorealistic, uh you also can have Pixar, you also can have cartoons.
07:09Uh it just literally takes a single image and um Show you this.
07:18Oh, no. Perfect.
07:27Uh and I'll I'll show you kind of a quick interaction here as well and then and then we can talk about how we built this.
07:33Um here I within the same video call and on the same kind of like inference setup, uh we can easily change clothes, we can change the scene she's in.
07:42And I can talk to it, too.
07:46I'm not going to do it here.
07:47be trying to say story.
07:49Would you like to tell me a story or hear one?
07:52Yeah, so you can see here, too, like that there's physics on the earrings.
07:56They move.
07:57The water moves.
07:59So, like the the video model has a good understanding of of physics and like can use that to like make the avatar feel more realistic.
08:10Perfect.
08:14Now, the way this is used, just for for kind of everybody's information, is we're mostly the API layer.
08:20So, we provide an API.
08:21People bring their own LLM.
08:23People bring their own usually like voices.
08:27And then we're the the visual layer on top of it.
08:32So, people use this for language learning.
08:36People use it for like AI sales call.
08:38Basically, anything today where you have a voice agent, you now also kind of a video agent if it's on your screen or on your computer.
08:48Great.
08:50So, now let's talk about how we built this today.
08:54This is our approach.
08:57And can't talk about all the technical details, but I'm going to try to go give you enough information to make it interesting.
09:06So, what we do is we get started with training our own video DIT model.
09:11The big thing here for us that really matters is the audio.
09:16The audio turns out to be very important for getting emotions right and the facial expressions right.
09:23People care about that a lot.
09:25So, we really focus on getting great audio data that we can train on.
09:31And then actually also getting the the encoders very right.
09:37Like most audio encoders today are trained on basically audiobooks, which is very monotone, very simple, don't have a lot of emotions.
09:47So, if you want to have a very expressive model, you can't use those audio encoders and you actually have to spend a lot of time like getting the audio embeddings right so that the video model is super expressive.
09:59So, so that's kind of like step one for us.
10:01And this is basically just a video model.
10:04You know, people can call this a world model, but but it's basically a video model that understands the physics of the world.
10:11And now the big challenge is how do you take that video model and how do you make it real time and interactive?
10:17And so, the way that works is the interactivity part is interesting.
10:22So, usually video models are bidirectional, so they can look into the past, but they actually also can look into the future and like what's about to come.
10:31And they basically generate videos all at the same time by looking at all of the the latents that that are being generated.
10:39For us, you can't do that.
10:41You only can look into the past.
10:43And so, here on this like make interactive, you can see that we basically train a model with an attention mask so that the model can only look into the past.
10:54So, when you do inference, it never can see the future because the future doesn't exist because like you haven't given it those inputs yet.
11:01So, for example, it doesn't see audio in the future, it only see what has has been said in the past.
11:06And then, once you have that, we make it real time.
11:11And there's a bunch of things that goes in here, but the biggest thing to speed up video generation is usually you spend like a bunch of steps denoising these video models.
11:25So, like let's say 30 steps.
11:26You spend 30 steps like removing the noise to generate the beautiful beautiful videos.
11:32And what we need to do is go from like 30 steps, bring it out to one step.
11:36So you basically just in a single step go from like pure noise to a pure video model to make it make it real time.
11:42So those are the the two big things that like make it interactive and real time.
11:46Now the hard part here, like what makes this difficult is this problem here on the left, which is called error accumulation, which is a problem anybody in like real-time video generation
11:58and world model is familiar with.
12:00I'll show you this video and then we'll talk about it briefly.
12:04And ophthalmology at Stanford School of Medicine.
12:08So you can see I mean that's a very bad example of error accumulation, but you can see as the video continues to generate more errors introduced.
12:18And the reason for that is because since you only can look backwards, you're looking actually backwards at at videos that you've previously generated, but each video you generate or each video block you generate has some error in it.
12:33So now you're looking in the past, you're looking at the error, you're adding more error to it, and then just the error compounds over time.
12:38And this is especially hard because I mean ideally these these video models are endless.
12:45Like the teddy avatar is generating continuously non-stop frame by frame for 8 hours straight with like no reset throughout the entire process.
12:56We have another one that's going to be generating for 16 hours straight.
12:58So it's a very hard problem to not have any error accumulation over long periods of time.
13:03And um we came up with a new way to solve this problem that is different to the best of our knowledge than what everybody else does today.
13:14And so I can't talk about that yet, but because of that we can basically generate these very long videos.
13:21I mean you guys can can try it out that essentially have no error accumulation, no noticeable error accumulation in them.
13:28Um Great.
13:31Um and so the last two hard problems uh we worked on to enable this is a hardware optimization.
13:42Um There's There's a bunch of stuff that goes into this.
13:46The big thing here is uh making it low cost enough.
13:51Uh so you actually can use this.
13:54Like usually when you generate an AI video, it's 5 seconds and you just share that.
13:59Uh we have to generate minutes and hours of AI AI video and still have it be cost-effective.
14:05Uh and so the cool cool thing here is we've been able to make the models small enough and efficient enough so that the costs are about the same as a voice model.
14:14Which is crazy to me.
14:16Cuz like think about the a voice model, how much data is streamed there compared to a video model.
14:22Like it it's much more pixel heavy in a video model than with voice.
14:27Uh and the costs costs are about similar.
14:29Um so the cool thing there is this enables us to do uh consumer use cases.
14:35And a lot of our customers are consumer companies that uh basically use the video for consumer and like entertainment applications.
14:43And then the final thing to mention here is the model hardness.
14:48Um I feel like the model hardness is something that is often overlooked but is actually super important and super hard.
14:57Uh like getting the model hardness right uh is a huge technical challenge for us.
15:01And I feel like a lot of our value actually in like productizing this is in the model hardness.
15:06And the way to think about it is you just have a bunch of separate threads.
15:11Uh all of it is managing real data streaming through our system.
15:15And you have basically a bunch of stuff you do on a GPU and a bunch of stuff you do on a CPU.
15:21And you have to orchestrate this perfectly in a way that like the video always remains real time.
15:26There is never any stutter that happens inside of the video.
15:29Um and this is especially hard when you have things like interrupts, you have queues, you're buffering data, you have to clean the queues.
15:35It's all like getting this orchestration right at production at scale has been a ton of work.
15:40And honestly, I feel like over time a lot more of the value uh of like the things we build will be in like figuring out the model hardness.
15:49I think it's especially true for like any real-time applications.
15:53Um cool.
15:55So, this is This is our technical approach of like commercializing these uh world models for avatars.
16:00Uh let me talk a little bit about like what we're actually working on actively now.
16:05Um so, the big thing we want to enable next is having more of an emotional engine uh that's driving these avatars.
16:15Uh having them be more more aware of what's actually happening inside of the conversation, and then be able to react to what's happening.
16:23So, react means uh the right the right emotions at the right time for the right duration, and same same for the actions.
16:30Um and um today when you're when you interact with these avatars, you can try.
16:36It still feels a little bit I mean, it definitely still feels um uh awkward.
16:42Uh and I think a big part of that is uh they don't emotionally react to you.
16:46They they're not listening to you.
16:48They're not like emoting in the right Um and so, this is what we're trying to solve.
16:52And so, let me show you guys some of um kind of the the new videos we're able to create uh with with basically basically better emotional control.
17:05Um so, this is a real-time video.
17:07Oh my god.
17:09Goal!
17:10Goal!
17:12We won the game.
17:13I can't believe it.
17:15We won!
17:16Oh my god.
17:18Oh my god.
17:20Oh my god.
17:21So you can see like a lot like the emotions just matter a lot.
17:24They make a huge difference in like connecting with this avatar.
17:28You feel you feel much more connected with that avatar.
17:31So that's one thing and then the neck this is the next generation model not in real time yet but there's just a lot more
17:39I think the most important thing is just to take a moment and breathe.
17:43Life gets so busy, you know.
17:45Do you think you'll stay there for a while?
17:46Yeah, I really think I need to stay right here for a long time.
17:50So there there'll be a lot more basically natural interaction.
17:56Sorry.
17:57There'll be a lot more natural interactions with themselves and then also with their environment.
18:02And the big thing here is like our model can already do this today.
18:07It has the capabilities to do this.
18:09It's just not controllable enough to like make it real time with the conversation and not deterministic enough to to make it useful with the conversation.
18:17So a big part of our work here is like making these actions actually controllable at the right right time and to do that we're building this emotion engine that is just literally predicting based on like the audio input
18:29from the avatar and the text input for what the avatar is going to say what the action they should be doing at this moment in time.
18:37So target launch for this next model is and and basically one to two months is what we're targeting.
18:49Awesome.
18:50Oh All right, great.
18:54So that's where we are today.
18:56Um I just want to talk briefly about let's see what the time is.
19:00Almost done.
19:03I want to talk briefly about where we see this heading.
19:08So I strongly believe that in the end uh there'll be a single model um that is the EQ layer for AI.
19:21Uh the way you can think about this model is the model will literally take in directly uh the user video and the user audio.
19:29So, directly the use the you know, the user talking to you like you are in a FaceTime uh and put out the avatar video and audio and do all of that in an end-to-end model
19:39a in a single model end-to-end.
19:41Um internally inside of the model it will do the audio understanding generate what it's going to say uh model its own internal state like own internal emotional state uh and then based on all of that
19:58produce an output video and audio.
20:03And all of this will be trained in one model.
20:04Like this will happen.
20:06Uh we're already seeing like early papers that are like doing proof proofs of concepts around this.
20:11What we're not saying is that this EQ model will be very intelligent.
20:17Uh it'll be very it'll have very high EQ and it'll be very good at like interacting with people uh but there'll be a separate model uh that will drive the IQ that will basically give this EQ model input
20:30to like do all the magical things that AI can do today.
20:33Like do the tool calling, do the deep thinking, do all the intelligent stuff.
20:38Uh but there will be an end-to-end layer up top.
20:40And so, that's that's our bet like we're working towards towards building that uh and um we can talk more about like some of the benefits of this approach but I feel strongly that within two or three years you'll be seeing these kinds of end-to-end
20:55EQ models coming on the market.
20:58Um that's it.
21:01Um Well, maybe take some questions if there's any questions.
21:05Um yeah, go ahead.
21:08Um so Mhm.
21:31In the future?
21:32Well, I think in the future in the long term, uh it's going to be more of an internal state uh where the model will will have an internal state in latent space
21:40that like defines what the emotional state is or the goals are of the model and they will be more implicit.
21:50So, they won't be human interpretable necessarily.
21:53Um today today basically the way we think about actions and emotions is basically words.
21:59If you can describe it in words, you can generate those actions and emotions.
22:17Exactly.
22:18Exactly and track that over time.
22:20Right, like have an internal state that tracks that over over some history of time.
22:23But yes, exactly.
22:31Yes.
22:36[laughter]
22:38Uh there's a big debate about that.
22:41Uh Twit- Twitter is debating.
22:42Gavin Newsom said uh that he definitely doesn't know uh and you know, uh he got tricked by like the ghost of of uh of Teddy Roosevelt.
23:06You're asking about what the way method is or the test is?
23:11Yeah, that's a really good point.
23:12We haven't done this yet, but we're uh in the process of figuring out our own version of the Turing test for these avatars, which will just include real people.
23:23Uh and uh we're planning to run that this year.
23:27Um and we won't pass it this year, but it'll just be cool for us to track it over time to see when we can actually solve the Turing test
23:36uh and ideally allow other people to also go through the test as well.
23:42But yeah, we plan to we've planned to create a test and then publish around around the test.
23:50Oh, yeah.
24:08Uh not for us.
24:11Like that's not what we plan to do.
24:13Um maybe people can use our technology to do that.
24:17Uh but that's not our our goal.
24:19Uh but you could imagine uh people using our technology, giving it the right context, having the right harness around it, uh and then having the nice like IQ brain around it to do exactly what you're saying.
24:33But like we we don't want to solve that problem.
24:53Yeah.
24:54Um it's still we would still like it to be cheaper.
24:58Um, the good news is that it is Okay, I'm almost done.
25:01Last question.
25:02Um, the good news is it is surprisingly like going into this we didn't know how cheap this would be.
25:07I've been very surprised at how inexpensive it is.
25:09Again, like the cost of this is at the same level as an audio model in terms of what we charge for it.
25:16Um, and we're all hoping for this costs to go down.
25:20Par- part of it is for consumer you just need very low costs.
25:24Uh, and then part of it is that as the cost goes down we can do higher resolution which will help us a lot.
25:29Um, but I guess the way we're thinking about it is there's like algorithm improvements and hardware improvements.
25:36Um, and the what has been true for some time now is things have been improving on all vectors faster than expected and we're just hoping that continues basically.
25:49I think there'll also be very cool architectural updates to to move to more of like a token approach instead of a diffusion approach that will make video like this type of video generation way cheaper.
26:00Um, that's it for me.
26:02Uh, thank you guys for showing up.
26:05Um, we if you guys are we are hiring.
26:07If you guys are interested in like solving these types of technical problems come talk to me afterwards.
26:13Uh, or if you're just interested in this at all come talk to me afterwards.
26:16Appreciate it guys.
26:32[music]