WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=OpY6MmZFeHo NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/OpY6MmZFeHo.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:13.599 --> 00:00:18.960 So, my name is Charlie uh and I work on the developer experience team at OpenAI. s3 00:00:18.960 --> 00:00:27.439 And part of my job is talking to developers to understand and see, you know, what and how they're building with our models. s4 00:00:27.439 --> 00:00:31.359 um whether that's text, image or audio. s5 00:00:31.359 --> 00:00:46.320 And lately I've been thinking about a misconception that I have seen or maybe it's just a misunderstanding and it's the idea that voice agents have to talk back. s6 00:00:48.480 --> 00:00:51.360 And to some of you that might sound, you know, absurd. s7 00:00:51.360 --> 00:00:52.160 It's a voice agent. s8 00:00:52.160 --> 00:00:54.079 What do you mean it's not supposed to talk? s9 00:00:54.079 --> 00:01:07.439 Uh, but I think if there's one thing that you take away from this presentation, I would like it to be the idea that speech is not the only way that a voice model s10 00:01:07.439 --> 00:01:09.840 has to respond. s11 00:01:12.240 --> 00:01:21.840 And I think models are getting intelligent enough and capable enough that they're starting to open up some uh new modes of design. s12 00:01:21.840 --> 00:01:32.320 I mean there's actually three kind of modes that that you know I kind of see emerging these days right uh speech to speech speech to action and event to speech s13 00:01:32.320 --> 00:01:37.439 and there's a couple of things I think worth pointing out about um these three categories. s14 00:01:37.439 --> 00:01:41.040 The first is that they're not new, right? s15 00:01:41.040 --> 00:01:50.240 I think as we've seen from from previous talks, even just today, um there's a long history of building these types of systems in and around voice. s16 00:01:50.240 --> 00:01:59.040 Um if you squint, you could make the argument that the movie phone hotline where you called in to get showtimes was an example of a speech-to-pech system. s17 00:01:59.040 --> 00:02:09.920 Uh and I think you could pretty reasonably make the argument that you know GPS navigation in your car which has existed since I was a kid is an example of an event to speech system. s18 00:02:09.920 --> 00:02:17.200 So, it's not that they are brand new, but I think it is that we are able to do some some much more interesting things with them uh now that s19 00:02:17.200 --> 00:02:18.400 uh we're in this era, right? s20 00:02:18.400 --> 00:02:24.480 And I think, you know, the other thing I would mention here is that um they're they're remixable, right? s21 00:02:24.480 --> 00:02:26.319 They're not meant to be mutually exclusive. s22 00:02:26.319 --> 00:02:32.959 Um and I think as we'll see in a little bit, the best products uh exist in a way that combines all of these modes. s23 00:02:32.959 --> 00:02:35.360 So, speechto um everybody knows it. s24 00:02:35.360 --> 00:02:36.480 Hopefully, everybody loves it. s25 00:02:36.480 --> 00:02:47.200 the user talks uh and then the model talks back and I think there are a few examples that that I can give um for this type of use case right you've got things like s26 00:02:47.200 --> 00:02:57.280 live practice and coaching especially around language learning right I think the ability to um hear a lot of emphasis or emotion and give people that feedback is really powerful s27 00:02:57.280 --> 00:03:14.640 um I think you can have you know what I am sort of cheekily calling concierge experiences which I think is just another way to say customer support plus+ Um the first voice tutorial that most people you know try to build when they have access to this technology is some sort of customer support chatbot for very good reasons. s28 00:03:14.640 --> 00:03:28.000 But I think with you know when we add a richness and a depth to the voice models as has been happening in recent months in recent years um we can build something that is like much more enjoyable to use than like talking your way through a phone tree. s29 00:03:28.000 --> 00:03:42.959 Um, and so, you know, I have the hope that like soon, if not like, you know, now, we're capable of building support experiences with agents that actually feel much more enjoyable to talk to than like arguably like the median human support agent. s30 00:03:42.959 --> 00:03:47.360 Um, and as we just saw if you hear the last talk, uh, live translation, right? s31 00:03:47.360 --> 00:03:55.040 The models have gotten good enough and fast enough that we can just dynamically translate content on the fly, um, with like little to no latency. s32 00:03:55.040 --> 00:04:04.000 It wouldn't shock me if at next year's keynote um you know they live streamed it from the main stage but also uh dubbed it in real time across multiple languages. s33 00:04:05.200 --> 00:04:07.519 The second category is speech to action. s34 00:04:07.519 --> 00:04:15.280 Uh these are talks and the model uses tools and I think this is one of the most underexplored areas that we have. s35 00:04:15.280 --> 00:04:27.360 Um I actually almost titled this talk uh voice is the next capability overhang because I think there is just a vast vast amount of stuff um that we could be doing in this category that we are not currently doing. s36 00:04:27.360 --> 00:04:36.639 Uh for example um there's a broad spectrum I don't have you know there's way too many examples even fit on this slide but three categories that that I find particularly interesting. s37 00:04:36.639 --> 00:04:42.720 Uh first is form filling right um so much of the internet is just filling out forms. s38 00:04:42.720 --> 00:04:46.720 Um, and there is, you know, today no reason why you shouldn't be able to just talk. s39 00:04:46.720 --> 00:04:57.840 You know, I would love it if instead of spending an hour filling out a government document, I could just talk for five minutes and it would get 90% of it for me and I would do a quick check, you know, just to make sure that everything looked good, right? s40 00:04:57.840 --> 00:05:06.320 That is a vastly superior experience than like having to type in every single name and address that I've lived in the last 5 years and, you know, all of my previous identities. s41 00:05:06.320 --> 00:05:15.440 Um and so I think I think that one is though it may seem boring you know affects a a significant GDP of the internet right the next category is creative tools s42 00:05:15.440 --> 00:05:30.160 uh where I am privileged enough that I can speak the language of software and so I can tell codeex you know here's exactly what I want you to build and I can articulate it in a way that um I get much more leverage than sort of just like cludily trying to iterate one thing at a time but s43 00:05:30.160 --> 00:05:40.400 I can't do that when it comes to you know using making music or painting um and so if I don't have the ability to articulate um the exact aesthetic that I'm looking for. s44 00:05:40.400 --> 00:05:47.280 Um and if I don't know how to use Photoshop or Ableton, I'm left in this state where, you know, my my taste exceeds my capability. s45 00:05:47.280 --> 00:05:59.840 Um and so I'm really looking forward to integrating voice into creative tools so that I can just sort of cludgy go along and, you know, vibe create, vibe compose, vibe paint, um and make something that that's really beautiful to me. s46 00:05:59.840 --> 00:06:05.199 And I think the generalizable um category here, right, then just starts to become computer use. s47 00:06:05.199 --> 00:06:07.600 And we've already seen some companies start to do this. s48 00:06:07.600 --> 00:06:16.560 Um, you know, it raises the question of like, look, if the models are just getting good enough to do everything on a computer that a human can do, like why am I talking to s49 00:06:16.560 --> 00:06:17.199 an app? s50 00:06:17.199 --> 00:06:18.560 Why am I talking to a terminal? s51 00:06:18.560 --> 00:06:22.000 Why am I not just talking to the entire computer? s52 00:06:22.000 --> 00:06:25.919 Uh, and so I think that's sort of a really interesting uh way to start exploring. s53 00:06:25.919 --> 00:06:28.000 But if you're a developer today, right? s54 00:06:28.000 --> 00:06:28.240 Whoops. s55 00:06:28.240 --> 00:06:32.240 If you're a developer today, um, what does that mean for building your own software, right? s56 00:06:32.240 --> 00:06:40.319 And I think it is like much easier than you think to start adding audio as an intelligence layer to the intelligence layer to the apps that you already have. s57 00:06:40.319 --> 00:06:46.400 Um, if you're building a modern web application, you already expose so much of it as like action as nouns and verbs, right? s58 00:06:46.400 --> 00:06:52.240 And if you think about all the verbs that you have, you have uh API endpoints, you have, you know, React hooks. s59 00:06:52.240 --> 00:07:00.960 Each of those things can like pretty relatively easily be converted into a tool that you expose to a model and then you can give the user the ability to just drive your existing software s60 00:07:00.960 --> 00:07:02.160 um with their voice, right? s61 00:07:02.160 --> 00:07:04.479 And yes, you still need guardrails, you still need safety checks. s62 00:07:04.479 --> 00:07:15.680 Like many of the talks today are going to talk about securing and you know productizing this, but um for this I just want you to think about you know what would it mean to take your existing software and just talk to it. s63 00:07:16.560 --> 00:07:18.319 Um and to go back to that misconception, right? s64 00:07:18.319 --> 00:07:29.120 I think there are a lot of um you know like if you're talking to the software maybe it can talk back but we've been developing other ways of communicating with the user for decades right we know these things we know we can show notifications s65 00:07:29.120 --> 00:07:43.440 and popups we can change state like the color of a button or a drop shadow we can highlight text um if you've used computer use in the codeex app you know there's this amazing like little ghost cursor animation that goes around and clicks things for you so we don't have to use words to actually tell the user s66 00:07:43.440 --> 00:07:50.800 what is happening on screen with their software Uh and then the last bucket here is event to speech, right? s67 00:07:50.800 --> 00:07:53.440 Um the model receives an event and talks to the user. s68 00:07:53.440 --> 00:07:56.080 Um and sort of the counterpoint from speech to action. s69 00:07:56.080 --> 00:07:58.879 I think this one is still very very exploratory, right? s70 00:07:58.879 --> 00:08:03.759 Um you know, if you saw Quinn's talk, I think there's a lot of uh space here of like things we can do. s71 00:08:03.759 --> 00:08:09.280 Um and to me, we haven't quite seen what AI native really looks like in this vein yet. s72 00:08:09.280 --> 00:08:13.680 But um of the things that I've seen, I think there's a couple of through lines that I tend to notice, right? s73 00:08:13.680 --> 00:08:17.280 The first is hands-free or screen-free experiences. s74 00:08:17.280 --> 00:08:25.280 There might be times where uh I need to interact with software, interact with objects and I can't use my hands or more importantly my attention is diverted elsewhere. s75 00:08:25.280 --> 00:08:29.120 Um that might be something you know like uh a recipe app. s76 00:08:29.120 --> 00:08:33.599 Maybe I'm cooking and I need to just say like what's going on and and have something else have something happen. s77 00:08:33.599 --> 00:08:36.800 Um the other category is proactive outreach, right? s78 00:08:36.800 --> 00:08:42.240 Where you the model needs to be able to tell you something or get your attention in a way um that you might not be looking at, right? s79 00:08:42.240 --> 00:08:48.720 I think every developer um has uh an endless amount of notifications and events happening in their software. s80 00:08:48.720 --> 00:08:52.480 But um no developer in their right mind would sort of say I should show all of these logs. s81 00:08:52.480 --> 00:08:55.040 Nor you know would they say I should speak all of these logs. s82 00:08:55.040 --> 00:09:07.120 But we can start to conceive of voice as this like upper level in this escalatory path of like okay maybe you animate something and then maybe you pop something up and then if that doesn't work maybe you talk to the user to get their attention. s83 00:09:07.839 --> 00:09:13.120 And underlying both of these categories and I think this this you know this whole presentation is this broader theme of accessibility. s84 00:09:13.120 --> 00:09:27.440 Um on a personal personal note, I know like multiple developers who um over the course of their careers lost mobility in their hands, lost dexterity in their fingers and for many of them, they thought their career as a programmer was more or less over. s85 00:09:27.440 --> 00:09:30.160 Um and then came large language models, right? s86 00:09:30.160 --> 00:09:39.040 Then came coding agents and voice agents and now they generate orders of magnitude more code than they like previously did um you know on a given given day or month. s87 00:09:39.040 --> 00:09:44.959 Um, and so I think there's there's a lot that we can unlock here uh for the broader world as well. s88 00:09:45.040 --> 00:09:52.080 Um, to go back to like I said, you know, I think like when it comes to these three modalities, you can mix and match them and we already have some, you know, rudimentary s89 00:09:52.080 --> 00:09:53.120 ways that we're seeing this. s90 00:09:53.120 --> 00:10:00.640 I think there's things like, you know, all of these pieces for in-car assistants exist, though nothing has quite like combined them into this seamless way. s91 00:10:00.640 --> 00:10:03.440 You can talk to like the CarPlay dashboard. s92 00:10:03.440 --> 00:10:06.399 Um, you can tell it, "Hey, go play some Spotify music for me." s93 00:10:06.399 --> 00:10:11.120 Um, and then it can come back and tell you, Google Maps can come back and tell you, hey, like, you know, there's traffic on this route. s94 00:10:11.120 --> 00:10:12.240 We're going to reroute you. s95 00:10:12.240 --> 00:10:18.327 But like we can now start to think about what does it mean to combine that into like a single voice agent across multiple modes. s96 00:10:18.327 --> 00:10:18.399 [snorts] s97 00:10:18.399 --> 00:10:22.880 Um, similarly, you know, there's a lot of experimentation in the game space with multimodality. s98 00:10:22.880 --> 00:10:29.440 Um, you can think about a real life character where you're talking to it to, you know, mine information about the game, about the world. s99 00:10:29.440 --> 00:10:39.920 um you can talk to it to execute actions on your behalf and then it can react to like world events right that are happening and then give that information to you rather than just like a simple notification s100 00:10:41.279 --> 00:10:52.640 um and I think the question you know behind the question here right is like I mentioned we've had all these things for a while people have been prototyping them for a while why focus on them now why think about building with them now um and I think that brings me to s101 00:10:52.640 --> 00:10:59.920 uh a little bit of context here right as as I'm hopefully most of you know traditionally voice agents are built in this chain Ed model, right? s102 00:10:59.920 --> 00:11:03.440 You uh talk, you transcribe, you send that to a language model. s103 00:11:03.440 --> 00:11:04.240 It calls tools. s104 00:11:04.240 --> 00:11:06.240 Hopefully, it doesn't take too long to respond. s105 00:11:06.240 --> 00:11:08.320 Um it then generates text output. s106 00:11:08.320 --> 00:11:12.160 You make that into audio and then you play that back to the user. s107 00:11:12.160 --> 00:11:16.000 And some time ago, um OpenAI, you know, decided on a different approach, right? s108 00:11:16.000 --> 00:11:20.560 The real-time model family does not do any transcription behind the scenes. s109 00:11:20.560 --> 00:11:23.519 It is trained on native audio as tokens. s110 00:11:23.519 --> 00:11:27.360 So, you send audio in and you get audio back out. s111 00:11:27.360 --> 00:11:30.800 And the industry I think in general has been, you know, trending more in this direction. s112 00:11:30.800 --> 00:11:36.399 Um, and not just making it native audio, but even just letting go of the turnbased abstraction that we've had, right? s113 00:11:36.399 --> 00:11:41.680 Um, and so making it that it's just continuous streaming audio in and out. s114 00:11:42.000 --> 00:11:50.320 And the reason that opening I did this was, you know, turns out there's a lot of stuff that you lose when you transcribe speech and when you transcribe audio, right? s115 00:11:50.320 --> 00:12:01.360 Um there's the old saying that when humans communicate face to face, 55% of the information is in body language, another 38% is in your tone of voice and the last like 7% is the actual words you are saying. s116 00:12:01.360 --> 00:12:13.680 Um and so when you transcribe, you lose tone and cadence and emotional uh you know impact, you lose like whether they're trying to interrupt you, you lose background noise, all of this stuff which is really important context for the model to understand. s117 00:12:13.680 --> 00:12:21.200 Um a much more quantitative reason to do it is that you know the first two uh voice modes in chat GBT were built with this chained approach. s118 00:12:21.200 --> 00:12:28.959 Um and as you can see had you know significantly higher latency than using uh the native approach with advanced voice mode. s119 00:12:29.040 --> 00:12:36.000 And that brings me to GPT realtime 2. Um and this is going to be the one part of the talk where you know I make my shameless plug. s120 00:12:36.000 --> 00:12:39.600 Um real time 2 is the the latest model in the real time family. s121 00:12:39.600 --> 00:12:41.440 We released it a couple of months ago. s122 00:12:41.440 --> 00:12:47.760 Um, and the the really cool thing about this model is that it brings reasoning to the audio medium. s123 00:12:47.760 --> 00:12:51.920 Um, and so much like our text models, it can now think before it speaks. s124 00:12:51.920 --> 00:12:57.680 Um, I'm sure many of us have seen some demos of voice models saying things that are a little bit less than intelligent. s125 00:12:57.680 --> 00:13:04.079 Um, and so you can now, you know, try to ensure that you give it more reasoning budget uh to come up with a good answer. s126 00:13:04.079 --> 00:13:07.760 Part of why that's also useful is that we introduced tool calling a little while ago. s127 00:13:07.760 --> 00:13:11.760 And so the model in addition to thinking it can also delegate parallel tool calls. s128 00:13:11.760 --> 00:13:13.839 Um you can start to bring these together. s129 00:13:13.839 --> 00:13:16.000 Um though of course that adds latency, right? s130 00:13:16.000 --> 00:13:18.399 And you know that's why we also added preamles. s131 00:13:18.399 --> 00:13:25.839 Um preamles are a way that you can prompt the model to uh give the user a heads up if it's going to be thinking or if it's going to be calling tools. s132 00:13:25.839 --> 00:13:29.760 Um you know if you think about the scenario of a travel agent, right? s133 00:13:29.760 --> 00:13:36.320 If I called the travel agent on the phone, uh you would want the travel agent to say to say, "Hey, like I'm going to go check flight prices, right? s134 00:13:36.320 --> 00:13:38.000 give me a couple seconds to do that. s135 00:13:38.000 --> 00:13:47.040 Um, and now with an AI travel agent, um, and preamles, you can actually have it communicate that to the user while it's performing actions in the background. s136 00:13:47.040 --> 00:13:48.560 Uh, there's a few other things here, right? s137 00:13:48.560 --> 00:13:53.760 It's got longer context, better domain understanding, um, more natural voices, and it's much more steerable. s138 00:13:53.760 --> 00:14:00.880 Um, there's some really cool features, uh, that it can do when it comes to like wake words and just waiting for you to to tell it. s139 00:14:00.880 --> 00:14:02.079 You you can give it a name. s140 00:14:02.079 --> 00:14:05.600 Uh, you can say like, you know, hey, Marin, do you want to say hi to the room? s141 00:14:05.600 --> 00:14:11.279 Um, and if you've like prompted that into the model, then it'll, you know, go ahead and and respond to you, right? s142 00:14:11.279 --> 00:14:18.959 Uh, and of course, you know, uh, obligatory benchmark slide, uh, it does pretty well on the the latest audio benchmarks, too. s143 00:14:18.959 --> 00:14:21.519 So, TLDDR, uh, is a pretty good model. s144 00:14:21.519 --> 00:14:37.839 Um, but I think the the kind of final thing that you know I want to leave you with here is um when building voice agents uh not to start with the question of like what kind of voice agent am I trying to build, right? s145 00:14:37.839 --> 00:14:45.760 I think the thing I want to leave you with is start with the question of like what is the role of voice and audio in this interaction? s146 00:14:45.760 --> 00:14:48.320 Um and then how do I move forward from there, right? s147 00:14:48.320 --> 00:14:48.480 Right? s148 00:14:48.480 --> 00:14:52.639 And often when I ask that question, it leads to a bunch more questions after that. s149 00:14:52.639 --> 00:14:54.880 Things like what can the model perceive? s150 00:14:54.880 --> 00:14:56.399 What context does it have? s151 00:14:56.399 --> 00:14:56.720 Right? s152 00:14:56.720 --> 00:15:03.839 Um what tools are available to it and which of those tools should it be, you know, uh executing safely and correctly? s153 00:15:03.839 --> 00:15:05.839 Um should it communicate now? s154 00:15:05.839 --> 00:15:06.880 Should it wait? s155 00:15:06.880 --> 00:15:08.720 Uh you know, how should it communicate? s156 00:15:08.720 --> 00:15:12.639 Should it be sending visual notifications or using audio? s157 00:15:12.639 --> 00:15:19.120 Um, and so taken together, yeah, I hope everybody in here can can start to build some much richer experiences with voice. s158 00:15:19.120 --> 00:15:25.680 Um, because like others have said, uh, I do believe that AGI will be spoken, not typed. s159 00:15:26.320 --> 00:15:26.959 Thank you very much. s160 00:15:26.959 --> 00:15:29.839 I'll be at the OpenAI booth, uh, for any Q&A after. s161 00:15:29.839 --> 00:15:32.959 Um, yeah, have a good event. s162 00:15:46.485 --> 00:15:48.485 [music]