Voice Agents Can Just Do Things — Charlie Guo, OpenAI

AI Engineer · 15 min · 162 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13So, my name is Charlie uh and I work on the developer experience team at OpenAI.
00:18And part of my job is talking to developers to understand and see, you know, what and how they're building with our models.
00:27um whether that's text, image or audio.
00:31And lately I've been thinking about a misconception that I have seen or maybe it's just a misunderstanding and it's the idea that voice agents have to talk back.
00:48And to some of you that might sound, you know, absurd.
00:51It's a voice agent.
00:52What do you mean it's not supposed to talk?
00:54Uh, but I think if there's one thing that you take away from this presentation, I would like it to be the idea that speech is not the only way that a voice model
01:07has to respond.
01:12And I think models are getting intelligent enough and capable enough that they're starting to open up some uh new modes of design.
01:21I mean there's actually three kind of modes that that you know I kind of see emerging these days right uh speech to speech speech to action and event to speech
01:32and there's a couple of things I think worth pointing out about um these three categories.
01:37The first is that they're not new, right?
01:41I think as we've seen from from previous talks, even just today, um there's a long history of building these types of systems in and around voice.
01:50Um if you squint, you could make the argument that the movie phone hotline where you called in to get showtimes was an example of a speech-to-pech system.
01:59Uh and I think you could pretty reasonably make the argument that you know GPS navigation in your car which has existed since I was a kid is an example of an event to speech system.
02:09So, it's not that they are brand new, but I think it is that we are able to do some some much more interesting things with them uh now that
02:17uh we're in this era, right?
02:18And I think, you know, the other thing I would mention here is that um they're they're remixable, right?
02:24They're not meant to be mutually exclusive.
02:26Um and I think as we'll see in a little bit, the best products uh exist in a way that combines all of these modes.
02:32So, speechto um everybody knows it.
02:35Hopefully, everybody loves it.
02:36the user talks uh and then the model talks back and I think there are a few examples that that I can give um for this type of use case right you've got things like
02:47live practice and coaching especially around language learning right I think the ability to um hear a lot of emphasis or emotion and give people that feedback is really powerful
02:57um I think you can have you know what I am sort of cheekily calling concierge experiences which I think is just another way to say customer support plus+ Um the first voice tutorial that most people you know try to build when they have access to this technology is some sort of customer support chatbot for very good reasons.
03:14But I think with you know when we add a richness and a depth to the voice models as has been happening in recent months in recent years um we can build something that is like much more enjoyable to use than like talking your way through a phone tree.
03:28Um, and so, you know, I have the hope that like soon, if not like, you know, now, we're capable of building support experiences with agents that actually feel much more enjoyable to talk to than like arguably like the median human support agent.
03:42Um, and as we just saw if you hear the last talk, uh, live translation, right?
03:47The models have gotten good enough and fast enough that we can just dynamically translate content on the fly, um, with like little to no latency.
03:55It wouldn't shock me if at next year's keynote um you know they live streamed it from the main stage but also uh dubbed it in real time across multiple languages.
04:05The second category is speech to action.
04:07Uh these are talks and the model uses tools and I think this is one of the most underexplored areas that we have.
04:15Um I actually almost titled this talk uh voice is the next capability overhang because I think there is just a vast vast amount of stuff um that we could be doing in this category that we are not currently doing.
04:27Uh for example um there's a broad spectrum I don't have you know there's way too many examples even fit on this slide but three categories that that I find particularly interesting.
04:36Uh first is form filling right um so much of the internet is just filling out forms.
04:42Um, and there is, you know, today no reason why you shouldn't be able to just talk.
04:46You know, I would love it if instead of spending an hour filling out a government document, I could just talk for five minutes and it would get 90% of it for me and I would do a quick check, you know, just to make sure that everything looked good, right?
04:57That is a vastly superior experience than like having to type in every single name and address that I've lived in the last 5 years and, you know, all of my previous identities.
05:06Um and so I think I think that one is though it may seem boring you know affects a a significant GDP of the internet right the next category is creative tools
05:15uh where I am privileged enough that I can speak the language of software and so I can tell codeex you know here's exactly what I want you to build and I can articulate it in a way that um I get much more leverage than sort of just like cludily trying to iterate one thing at a time but
05:30I can't do that when it comes to you know using making music or painting um and so if I don't have the ability to articulate um the exact aesthetic that I'm looking for.
05:40Um and if I don't know how to use Photoshop or Ableton, I'm left in this state where, you know, my my taste exceeds my capability.
05:47Um and so I'm really looking forward to integrating voice into creative tools so that I can just sort of cludgy go along and, you know, vibe create, vibe compose, vibe paint, um and make something that that's really beautiful to me.
05:59And I think the generalizable um category here, right, then just starts to become computer use.
06:05And we've already seen some companies start to do this.
06:07Um, you know, it raises the question of like, look, if the models are just getting good enough to do everything on a computer that a human can do, like why am I talking to
06:16an app?
06:17Why am I talking to a terminal?
06:18Why am I not just talking to the entire computer?
06:22Uh, and so I think that's sort of a really interesting uh way to start exploring.
06:25But if you're a developer today, right?
06:28Whoops.
06:28If you're a developer today, um, what does that mean for building your own software, right?
06:32And I think it is like much easier than you think to start adding audio as an intelligence layer to the intelligence layer to the apps that you already have.
06:40Um, if you're building a modern web application, you already expose so much of it as like action as nouns and verbs, right?
06:46And if you think about all the verbs that you have, you have uh API endpoints, you have, you know, React hooks.
06:52Each of those things can like pretty relatively easily be converted into a tool that you expose to a model and then you can give the user the ability to just drive your existing software
07:00um with their voice, right?
07:02And yes, you still need guardrails, you still need safety checks.
07:04Like many of the talks today are going to talk about securing and you know productizing this, but um for this I just want you to think about you know what would it mean to take your existing software and just talk to it.
07:16Um and to go back to that misconception, right?
07:18I think there are a lot of um you know like if you're talking to the software maybe it can talk back but we've been developing other ways of communicating with the user for decades right we know these things we know we can show notifications
07:29and popups we can change state like the color of a button or a drop shadow we can highlight text um if you've used computer use in the codeex app you know there's this amazing like little ghost cursor animation that goes around and clicks things for you so we don't have to use words to actually tell the user
07:43what is happening on screen with their software Uh and then the last bucket here is event to speech, right?
07:50Um the model receives an event and talks to the user.
07:53Um and sort of the counterpoint from speech to action.
07:56I think this one is still very very exploratory, right?
07:58Um you know, if you saw Quinn's talk, I think there's a lot of uh space here of like things we can do.
08:03Um and to me, we haven't quite seen what AI native really looks like in this vein yet.
08:09But um of the things that I've seen, I think there's a couple of through lines that I tend to notice, right?
08:13The first is hands-free or screen-free experiences.
08:17There might be times where uh I need to interact with software, interact with objects and I can't use my hands or more importantly my attention is diverted elsewhere.
08:25Um that might be something you know like uh a recipe app.
08:29Maybe I'm cooking and I need to just say like what's going on and and have something else have something happen.
08:33Um the other category is proactive outreach, right?
08:36Where you the model needs to be able to tell you something or get your attention in a way um that you might not be looking at, right?
08:42I think every developer um has uh an endless amount of notifications and events happening in their software.
08:48But um no developer in their right mind would sort of say I should show all of these logs.
08:52Nor you know would they say I should speak all of these logs.
08:55But we can start to conceive of voice as this like upper level in this escalatory path of like okay maybe you animate something and then maybe you pop something up and then if that doesn't work maybe you talk to the user to get their attention.
09:07And underlying both of these categories and I think this this you know this whole presentation is this broader theme of accessibility.
09:13Um on a personal personal note, I know like multiple developers who um over the course of their careers lost mobility in their hands, lost dexterity in their fingers and for many of them, they thought their career as a programmer was more or less over.
09:27Um and then came large language models, right?
09:30Then came coding agents and voice agents and now they generate orders of magnitude more code than they like previously did um you know on a given given day or month.
09:39Um, and so I think there's there's a lot that we can unlock here uh for the broader world as well.
09:45Um, to go back to like I said, you know, I think like when it comes to these three modalities, you can mix and match them and we already have some, you know, rudimentary
09:52ways that we're seeing this.
09:53I think there's things like, you know, all of these pieces for in-car assistants exist, though nothing has quite like combined them into this seamless way.
10:00You can talk to like the CarPlay dashboard.
10:03Um, you can tell it, "Hey, go play some Spotify music for me."
10:06Um, and then it can come back and tell you, Google Maps can come back and tell you, hey, like, you know, there's traffic on this route.
10:11We're going to reroute you.
10:12But like we can now start to think about what does it mean to combine that into like a single voice agent across multiple modes.
10:18[snorts]
10:18Um, similarly, you know, there's a lot of experimentation in the game space with multimodality.
10:22Um, you can think about a real life character where you're talking to it to, you know, mine information about the game, about the world.
10:29um you can talk to it to execute actions on your behalf and then it can react to like world events right that are happening and then give that information to you rather than just like a simple notification
10:41um and I think the question you know behind the question here right is like I mentioned we've had all these things for a while people have been prototyping them for a while why focus on them now why think about building with them now um and I think that brings me to
10:52uh a little bit of context here right as as I'm hopefully most of you know traditionally voice agents are built in this chain Ed model, right?
10:59You uh talk, you transcribe, you send that to a language model.
11:03It calls tools.
11:04Hopefully, it doesn't take too long to respond.
11:06Um it then generates text output.
11:08You make that into audio and then you play that back to the user.
11:12And some time ago, um OpenAI, you know, decided on a different approach, right?
11:16The real-time model family does not do any transcription behind the scenes.
11:20It is trained on native audio as tokens.
11:23So, you send audio in and you get audio back out.
11:27And the industry I think in general has been, you know, trending more in this direction.
11:30Um, and not just making it native audio, but even just letting go of the turnbased abstraction that we've had, right?
11:36Um, and so making it that it's just continuous streaming audio in and out.
11:42And the reason that opening I did this was, you know, turns out there's a lot of stuff that you lose when you transcribe speech and when you transcribe audio, right?
11:50Um there's the old saying that when humans communicate face to face, 55% of the information is in body language, another 38% is in your tone of voice and the last like 7% is the actual words you are saying.
12:01Um and so when you transcribe, you lose tone and cadence and emotional uh you know impact, you lose like whether they're trying to interrupt you, you lose background noise, all of this stuff which is really important context for the model to understand.
12:13Um a much more quantitative reason to do it is that you know the first two uh voice modes in chat GBT were built with this chained approach.
12:21Um and as you can see had you know significantly higher latency than using uh the native approach with advanced voice mode.
12:29And that brings me to GPT realtime 2. Um and this is going to be the one part of the talk where you know I make my shameless plug.
12:36Um real time 2 is the the latest model in the real time family.
12:39We released it a couple of months ago.
12:41Um, and the the really cool thing about this model is that it brings reasoning to the audio medium.
12:47Um, and so much like our text models, it can now think before it speaks.
12:51Um, I'm sure many of us have seen some demos of voice models saying things that are a little bit less than intelligent.
12:57Um, and so you can now, you know, try to ensure that you give it more reasoning budget uh to come up with a good answer.
13:04Part of why that's also useful is that we introduced tool calling a little while ago.
13:07And so the model in addition to thinking it can also delegate parallel tool calls.
13:11Um you can start to bring these together.
13:13Um though of course that adds latency, right?
13:16And you know that's why we also added preamles.
13:18Um preamles are a way that you can prompt the model to uh give the user a heads up if it's going to be thinking or if it's going to be calling tools.
13:25Um you know if you think about the scenario of a travel agent, right?
13:29If I called the travel agent on the phone, uh you would want the travel agent to say to say, "Hey, like I'm going to go check flight prices, right?
13:36give me a couple seconds to do that.
13:38Um, and now with an AI travel agent, um, and preamles, you can actually have it communicate that to the user while it's performing actions in the background.
13:47Uh, there's a few other things here, right?
13:48It's got longer context, better domain understanding, um, more natural voices, and it's much more steerable.
13:53Um, there's some really cool features, uh, that it can do when it comes to like wake words and just waiting for you to to tell it.
14:00You you can give it a name.
14:02Uh, you can say like, you know, hey, Marin, do you want to say hi to the room?
14:05Um, and if you've like prompted that into the model, then it'll, you know, go ahead and and respond to you, right?
14:11Uh, and of course, you know, uh, obligatory benchmark slide, uh, it does pretty well on the the latest audio benchmarks, too.
14:18So, TLDDR, uh, is a pretty good model.
14:21Um, but I think the the kind of final thing that you know I want to leave you with here is um when building voice agents uh not to start with the question of like what kind of voice agent am I trying to build, right?
14:37I think the thing I want to leave you with is start with the question of like what is the role of voice and audio in this interaction?
14:45Um and then how do I move forward from there, right?
14:48Right?
14:48And often when I ask that question, it leads to a bunch more questions after that.
14:52Things like what can the model perceive?
14:54What context does it have?
14:56Right?
14:56Um what tools are available to it and which of those tools should it be, you know, uh executing safely and correctly?
15:03Um should it communicate now?
15:05Should it wait?
15:06Uh you know, how should it communicate?
15:08Should it be sending visual notifications or using audio?
15:12Um, and so taken together, yeah, I hope everybody in here can can start to build some much richer experiences with voice.
15:19Um, because like others have said, uh, I do believe that AGI will be spoken, not typed.
15:26Thank you very much.
15:26I'll be at the OpenAI booth, uh, for any Q&A after.
15:29Um, yeah, have a good event.
15:46[music]