WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=65X0pQ6Lmbg NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/65X0pQ6Lmbg.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.200 --> 00:00:03.080 All right. s2 00:00:03.080 --> 00:00:04.880 I'm Alan Pike. s3 00:00:04.880 --> 00:00:14.520 And today I'm going to be sharing some of what we've learned building voice in visuals out experiences using AI. s4 00:00:14.560 --> 00:00:16.040 Here we go. s5 00:00:16.040 --> 00:00:20.720 So this is Andrej Karpathy. s6 00:00:20.720 --> 00:00:31.320 And he made an argument last month that voice is the human preferred input for AIs. s7 00:00:31.320 --> 00:00:36.560 But that we prefer visuals as the output. s8 00:00:36.760 --> 00:00:40.840 And he knows a thing or two about this stuff. s9 00:00:40.840 --> 00:00:49.920 But that's not how we have been for the most part building or using AI. s10 00:00:49.920 --> 00:00:52.120 We've been typing to it. s11 00:00:52.120 --> 00:00:55.080 It's been typing back maybe with some markdown. s12 00:00:55.080 --> 00:01:08.920 But there's some real breakthroughs over the last few months where both of this visuals in and audio or the audio in and visuals out experience is now um really feasible. s13 00:01:08.920 --> 00:01:12.320 We can create really delightful experiences with it. s14 00:01:12.320 --> 00:01:16.040 The visuals outside is pretty intuitive, right? s15 00:01:16.040 --> 00:01:21.040 Of course a third of our brain is dedicated to processing visual information. s16 00:01:21.040 --> 00:01:24.000 We love looking at things. s17 00:01:24.080 --> 00:01:30.920 And models have recently got to the point where they can generate rich HTML, tool calling. s18 00:01:30.920 --> 00:01:42.200 So we can have these experiences where there's visualizations that come back, explain things, help us understand, communicate the the responses from these models. s19 00:01:42.200 --> 00:01:53.840 They can give us interactive um controls that allows to explore and understand and modify and change and and and direct the models. s20 00:01:53.840 --> 00:02:01.160 And they can even respond with beautiful illustrations and images, right? s21 00:02:01.160 --> 00:02:15.440 And so the the ceiling on the visuals out piece has really lifted and we have a lot more capability of what we can do in terms of responding from these models. s22 00:02:15.440 --> 00:02:23.880 The more controversial half of Karpathy's argument though is this idea of voice as the preferred input. s23 00:02:23.880 --> 00:02:39.560 We've long sort of idealized and fantasized about speaking to an AI and having this real-time conversation where it understands what we need and it and it reacts appropriately in real-time. s24 00:02:39.560 --> 00:02:49.840 But the experiences that most people have had so far with voice interfaces have been more like trying to get Siri to turn the lights on and it's not working s25 00:02:49.840 --> 00:03:01.840 or like this guy that is like trying to get ChatGPT voice mode to do things, but it keeps being like awkward and confused, right? s26 00:03:01.840 --> 00:03:14.000 The models that we have so far, the experiences that people most people have seen are both slow and dumb, which is like not a great combination. s27 00:03:14.000 --> 00:03:17.920 And so a lot of people are down on voices as an input. s28 00:03:18.000 --> 00:03:24.480 But speaking is the ultimate way that as humans we communicate. s29 00:03:24.480 --> 00:03:31.160 When we're speaking, we have more words per minute than when we're typing. s30 00:03:31.160 --> 00:03:35.680 But we also convey more with each word. s31 00:03:35.680 --> 00:03:46.400 There's a huge difference in between if you say to me okay versus if you say okay. s32 00:03:46.600 --> 00:03:59.440 And that's why when we have something that is really important that we need to communicate or get through to somebody, we will jump on a call. s33 00:03:59.440 --> 00:04:02.400 And we will or we'll speak in person. s34 00:04:02.400 --> 00:04:08.560 So that we can get through that high bandwidth uh communication. s35 00:04:08.680 --> 00:04:18.840 Um and so what we've done at Forest Walk is we have built an agent that is in our calls that can help us out in real time. s36 00:04:18.840 --> 00:04:29.680 And so the other day I mentioned to my co-founder in call on a call that I'd seen a bug with the Slack integration or at least I thought I had. s37 00:04:29.680 --> 00:04:34.640 And she responded that she had seen the same bug. s38 00:04:34.640 --> 00:04:40.120 And so I just said, "Okay, well, let's file that as a Linear issue." s39 00:04:40.120 --> 00:04:45.960 And the voice agent within a second responded that it had done so. s40 00:04:45.960 --> 00:04:51.200 And that feels perfectly natural when you get it really dialed in. s41 00:04:51.200 --> 00:05:04.240 That you have are speaking either incidentally or you know, with a purpose intentionally speaking to the AI and it responds in a way that is not interruptive of you. s42 00:05:04.240 --> 00:05:05.919 It doesn't need to be voice. s43 00:05:05.919 --> 00:05:10.840 It is just taking action on your intent. s44 00:05:10.840 --> 00:05:16.200 And so this is we're going to see this experience more and more in the coming months and years. s45 00:05:16.200 --> 00:05:22.280 But there's a huge barrier, a huge challenge to making this actually work and feel good. s46 00:05:22.280 --> 00:05:28.520 And that is the tyranny of latency. s47 00:05:29.080 --> 00:05:35.880 It's really difficult to get a response through that whole chain fast enough. s48 00:05:35.880 --> 00:05:51.040 We we've known since the '60s that to have a computer react to us fast enough that it feels instant, it needs to react in about 100 milliseconds, a tenth of a second. s49 00:05:51.360 --> 00:05:57.280 And so of course we are always aspiring to get our products to react that fast. s50 00:05:57.280 --> 00:06:04.520 And sometimes we can achieve that, but with networking and everything it can be challenging depending on what work needs to get done. s51 00:06:04.520 --> 00:06:08.080 And so sometimes we might flex up to a thousand milliseconds. s52 00:06:08.080 --> 00:06:08.560 Right? s53 00:06:08.560 --> 00:06:09.760 You get a full second. s54 00:06:09.760 --> 00:06:12.680 It's about the limit before people start to lose their train of thought. s55 00:06:12.680 --> 00:06:19.000 They ask Siri to do something, takes more than a second, we're off to something else mentally, right? s56 00:06:19.000 --> 00:06:30.440 And so we're always in that play in between those two kind of human limits of trying to respond within a second, ideally within 100 milliseconds. s57 00:06:30.840 --> 00:06:41.760 But if we want to have a seamless voice conversation with it an AI or another human, the limit is much more aggressive. s58 00:06:41.760 --> 00:06:56.120 We need to have a 200 millisecond latency or less if we want to have a fully conversive people are verbalizing something and they are interrupting or interjecting or agreeing s59 00:06:56.120 --> 00:07:03.160 and forming that sort of connection for a full voice in voice out uh conversation. s60 00:07:03.160 --> 00:07:13.919 Um and so imagine trying to do that where you have a network request, uh maybe you take turns speech to text, uh and then you do some sort of model inference on that text, s61 00:07:13.919 --> 00:07:15.280 and then send it back on the network. s62 00:07:15.280 --> 00:07:20.640 It's like a ridiculous uh amount of of work to do in 200 milliseconds. s63 00:07:20.640 --> 00:07:26.120 Um there are some clever approaches of trying to work around that. s64 00:07:26.120 --> 00:07:39.800 Uh Thinking Machines and Neolab has uh a really thoughtful architecture where they were demoing recently, um just a few weeks ago, a they've time sliced into 200 millisecond chunks like that 200 millisecond s65 00:07:39.800 --> 00:07:46.960 goal, and and has a model that's doing continuous inference in and out at 200 millisecond slices. s66 00:07:46.960 --> 00:07:57.360 Um so there are ways around it for voice in voice out, but there's also just we don't need to wait for novel architectures. s67 00:07:57.360 --> 00:08:09.240 We can just switch to having voice in visuals out and then we benefit from the more forgiving visual response envelope that people have. s68 00:08:09.240 --> 00:08:20.560 That if you have something that appears on screen within a second of what you've you've said, it feels like it has met that it's keeping with within your attention span. s69 00:08:20.560 --> 00:08:24.840 It's reacting to you in a way that feels seamless. s70 00:08:24.840 --> 00:08:30.440 Um and so we've been building for that over the last few months and it's a lot of fun and we've learned a lot doing that. s71 00:08:30.440 --> 00:08:43.280 And so I want to finish up just by sharing three of the things that we found um to be really important for getting within that latency envelope and making the experience feel really delightful s72 00:08:43.280 --> 00:08:46.960 um for for people who are using it. s73 00:08:46.960 --> 00:08:57.760 So the first one is that in order to respond in a way that feels seamless, you have to use a really fast model. s74 00:08:57.760 --> 00:09:03.640 Cuz that's not just a model that is so small um that it can respond fast enough in a few hundred milliseconds. s75 00:09:03.640 --> 00:09:05.839 Of course it it does need to be. s76 00:09:05.839 --> 00:09:10.920 Um but it also has to be on an inference platform that prioritizes latency. s77 00:09:10.920 --> 00:09:20.280 So when GPT-5 mini came out, we were excited that um that okay, well, yeah, we get this more intelligent model and presumably it'll be fast. s78 00:09:20.280 --> 00:09:29.640 But then in practice we were seeing latencies of 5,000 milliseconds, 7,000 P95, sometimes 10,000 millisecond latency for this small model. s79 00:09:29.640 --> 00:09:38.240 It was cheaper than the bigger model, but we weren't consistently or even ever seeing it respond fast enough. s80 00:09:38.240 --> 00:09:43.840 Um Haiku is much better um in terms of of that that P95 latency. s81 00:09:43.840 --> 00:09:54.000 So, you you really need a a kind of a Haiku class model as the one that's responding in this real-time and and or one of the smaller open-source ones. s82 00:09:54.000 --> 00:10:03.640 And then if there is a larger chunk of work that needs to get done, then that model then hands off or sends off an asynchronous message to a larger model that can think, s83 00:10:03.640 --> 00:10:13.920 and then the continuous real-time model or soft real-time model that's responding quickly then can re- interleave in those responses if it's doing something heavier. s84 00:10:13.920 --> 00:10:16.040 Um so, that's number one you need to have a fast model. s85 00:10:16.040 --> 00:10:22.600 Obviously, with a a short enough context that you're providing it that it can respond in a few hundred milliseconds. s86 00:10:22.600 --> 00:10:35.800 Um the second thing that's really critical if we're going to have this feeling of it being really responsive is to have short intervals of how quickly we're sending out for inference. s87 00:10:35.800 --> 00:10:46.760 So, traditionally for like a voice application, you might listen to the user for a few seconds and then they stop, you listen for a second of silence, and then you have s88 00:10:46.760 --> 00:10:55.000 uh some sort of inference goes on, and then you're you're blown your budget by a pretty wide margin just for waiting for silence. s89 00:10:55.000 --> 00:11:04.640 So, if we're going to have our something up on screen within a second, um then when we want to have our inference pretty eagerly responding as the person is talking, s90 00:11:04.640 --> 00:11:20.440 even if we're not entirely sure they've stopped talking, being willing to send inference every 1 or 2 seconds as they speak cuz you know, we sometimes have a sentence where, you know, we ask, "Oh, hey, we're going to want to change this and actually also list to this other thing." s91 00:11:20.440 --> 00:11:24.120 It feels much more seamless if you're doing those things as people talk. s92 00:11:24.120 --> 00:11:31.920 Um so, you have to have a a model and an an infrastructure where you're able to get these fast turns going. s93 00:11:31.920 --> 00:11:39.880 And then finally, in order to actually make all of that work, you need to have a stable caching regimen. s94 00:11:39.880 --> 00:11:56.440 So, we have this huge improvement in the way that we work with LLMs over the last year where we have on the different platforms different ideas of this prefix caching where if the beginning of the context you send to the model is the same each time, s95 00:11:56.440 --> 00:12:02.240 then you can get up to 90% cheaper, faster inference um depending on the conditions. s96 00:12:02.240 --> 00:12:14.440 And so, you need to lean heavily into this architecture really I think for for most applications we're moving towards this, whether it's like a long-running agent or it's a frequently running agent, the same principle applies s97 00:12:14.440 --> 00:12:27.320 that we want to the first 90% of the context window if we can to be the same from request to request, and then just use that final 10%, and then also, of course, minimize the number of output tokens s98 00:12:27.320 --> 00:12:35.640 so that you can get these really fast and relatively affordable inference turns to create that that delightful experience. s99 00:12:35.920 --> 00:12:40.839 Um So, these are some of the the techniques we've found really useful. s100 00:12:40.839 --> 00:12:53.760 Um I would love to hear from anybody who um has been exploring and experimenting, whether it's with real-time or any way that people are pushing the boundaries of creating delightful experiences s101 00:12:53.760 --> 00:12:55.280 uh with these models. s102 00:12:55.280 --> 00:12:57.839 Uh I'd love to chat, share what we've been learning. s103 00:12:57.839 --> 00:13:04.080 Uh and I hope some of this inspires some of you to go out and uh build something great. s104 00:13:04.080 --> 00:13:06.080 Thanks.