The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents, Arize

AI Engineer · 18 min · 220 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Folks, thanks for coming in out.
00:15I know it's lunch time, so I know a lot of you folks are trying to eat as well.
00:18Appreciate you making the time and for everyone listening online, thanks for tuning in as well.
00:22Uh my name's Fuad.
00:24I'm a product manager here at Arize.
00:26Uh I've been leading up kind of the experimentation side of the house and one thing that I've been really interested in is voice agents.
00:33Uh quick show of hands here, maybe online too.
00:36Who here is building a voice agent?
00:39Anyone?
00:40Got Okay, we got a couple.
00:42Uh I think it's a really cool space.
00:43I'll be talking a little bit about voice traces, session views of voices, voice evals, and some of the stuff that's new in Arize.
00:52But I do want to kind of mention this isn't a product pitch.
00:55This is, you know, really on why voice agents are invisible and how to see them.
01:01You can use Arize to do that, you can use another platform, but what I want you guys to take away from this is some of the failure modes and how you can actually start to see that.
01:10So, we all know voice agents are growing really, really fast.
01:15Have a few tweets up uh kind of proving this.
01:18GPT real-time just dropped.
01:20That was literally May 7th.
01:22Uh I have a screenshot from a tweet an hour ago from the LiveChat CEO talking about some of this.
01:27Uh and Vercel on June 29th, which is only a couple days ago, um on voice agents in Vercel.
01:34So, as you can tell, uh this space is exploding fast and that is going to start to matter because as the space grows really fast, it is also one of the hardest to debug categories.
01:46So, first off, it's exploding.
01:48Second off, it's pretty fragile.
01:51There's a lot of failure modes you can't really see.
01:54Um and uh things like latency, turn-taking, and transcription drift become really, really important.
02:00Uh, and the log lies.
02:01If you're just looking at transcripts, I'll play a little audio file for you guys in a second.
02:06Uh, it doesn't really capture what's actually going on under the hood.
02:10And just to give you guys some ideas of use cases, um, there's a lot of them from customer support, I think, which are the more boring ones, down through drive-thru ordering.
02:19If you've been to White Castle, they have a a bot called Linda now.
02:23Uh, Bojangles has uh, Bo Linda as well.
02:26There's McDonald's is doing a lot of these as well, which I think is pretty cool.
02:30And personally, I'd be pretty pissed off if I was at McDonald's at 2:00 a.m. and I got 600 cheeseburgers.
02:35So, I think I'll show you a little bit about the failure modes and why it actually matters.
02:41So, uh, this is an example where the transcript looks fine, but the call wasn't.
02:48And if you read the text logs, you know, user says, "I need a refund for order 14."
02:53Agent says, "Sure.
02:54Started a refund for order 40."
02:56User says, "No, 14."
02:58And it looks like the agent corrected itself.
03:00But, this wasn't actually the case, and I'm going to I'm going to play the call.
03:04It's AI-generated.
03:05Let's see if it works.
03:06I need a refund for order 14.
03:10Sure.
03:11I've started a refund for order 40.
03:13No, 14.
03:15Your refund is confirmed.
03:17So, if you notice in the transcript, if you could hear the audio there, uh, it wasn't, you know, the the the confirm- confirmation on the on the 40 wasn't actually done by the agent.
03:28It kind of interrupted.
03:30Uh, and there was a big, uh, kind of latency spike, uh, after the user asked for the initial refund.
03:35Um, so, we what actually happened was that two 2.4 seconds of dead air before the agent actually responded.
03:43The agent talking over the caller.
03:4614 misheard, and obviously, the flat robotic tone is is not a a look, either.
03:51Um, so, these are all audio specific issues.
03:53If you just read this from an LM output, you would have no idea anything is going on.
03:58So, what you need to do is you need to see the conversation beyond just the transcript.
04:04Um, so that means the audio, the transcript, and the trace all in one place.
04:09The session as a whole, so multiple turns from the user back and forth in the same in the same trace view.
04:16And then metrics on those, so what was the latency like?
04:19What was the time to first audio like?
04:21Were there any interruptions?
04:22Sentiment analysis, et cetera.
04:24And these are some of the features we've just released at the beginning of this month in Arize AIX.
04:29Uh, and really the point of this is the same workflows that you're running now to help your coding agents, uh, which are, you know, pretty complex processes now with harnesses,
04:39evals, uh, handoffs, and subagents, tool calls, you want to start to bring some of those best practices into your voice agents as well.
04:47So, first off, you got to trace it.
04:49You got to see each individual span.
04:51Um, you got to see the live audio and play it inline from your trace.
04:56Really matters to have all this data in the same place so you can start to debug things.
05:00Uh, you want to see inputs and outputs.
05:02You want to see the transcript, obviously, but you also want to see, you know, specific metrics like I mentioned, like time to first token, uh, audio token cost, interruption events.
05:11And then finally, you want to see linkage between these.
05:14You don't want to just see these as a list.
05:15You actually want to see some of the linkage between things, right?
05:20So, Arize gives you that one trace view where you can actually play the audio inline, view the transcript, view latency inline, see the conversation turn between the user, uh, and the assistant,
05:33uh, and see your evals inline as well.
05:35And this gives you a really, really powerful tool to debug.
05:40Under the hood, what this looks like is we've actually mapped, uh, the semantic conventions for audio onto open inference.
05:47So, if you guys aren't familiar, Arize is first to market with open inference in 2023.
05:52It is the first set of semantic conventions that is completely open source that maps kind of what is happening with GenAI to an OTel-compliant set of conventions.
06:03And I say this to say it's open source, it's free, and it maps pretty much any of the major foundational models or model providers.
06:11You guys can go rip this and send it to a Grafana back end.
06:14Completely free to use.
06:15Not pitching the product here, just explaining some of the real engineering work it takes to map things like web socket events to an easily understandable set of semantic conventions.
06:25So, you can start see things like the session life cycle, the audio input, the transcript, conversation items, the output, and then the response life cycle and token counts as well.
06:36And so, yeah, real engineering work went behind this.
06:38It's all open source.
06:40We've mapped all the foundational models for you.
06:43What this allows you to do is you can now be provider agnostic and have one queryable schema.
06:51So, those semantic conventions normalize sort of these disparate model providers, whether it's OpenAI, real-time, or Google with Gemini Live into a set of schema that you can query.
07:04And if you've ever tried to debug at 2:00 a.m. off of a customer call, you know how important it is to be able to query across thousands, millions, even billions of traces that are coming in.
07:15You know how important it is to actually get to the part that you need to see, to filter that, to sort by that.
07:21All that becomes incredibly important.
07:23And, you know, as new models drop, you also want to be able to swap in new providers and make sure that your systems are still functioning.
07:31So, no bespoke parsing per vendor.
07:33It's one auto instrumenter.
07:35It captures every single thing that I just mentioned.
07:40And then finally, you want to be able to replay the whole conversation.
07:43So I play I played the audio for you guys in the beginning of this, but the session ID is essentially what stitches every turn into the timeline, the full call with every websocket event that's submitted
07:54with some tags on whether it's the user or the assistant.
07:57You can actually play the audio in the UI directly from the trace.
08:01No external tooling.
08:03No exports.
08:04Just play it straight from the trace.
08:06That allows you to see who spoke when.
08:09When the agent paused, where it got cut off, where the user got cut off, what tools the agent actually went and called when it did this.
08:17And so this really starts the paradigm shift, which is you're not just reading transcripts, you're listening to audio, you're really understanding the failure cases from there with the trace open right next to it.
08:28And um I'll just show you kind of quick quick sneak preview of what this looks like in Arise.
08:34As you can see here, this is an example of like you know an agent that can call to get a specific weather event.
08:41You can actually see when the agent kicks off that tool call in line with the rest of the audio.
08:46And this allows you to debug failures of that tool call while you're looking at the session audio.
08:50So you're no longer looking at disparate events from distant systems, but you're actually looking at the entire trace tree for everything that happened in that session.
09:00So that real session, few issues that we saw, right?
09:04Turn one, "Hi, need help with the order, a refund for order 14."
09:08We saw you know kind of that time to first audio of 2.4 seconds of dead air, which really impacts the user experience.
09:15We saw in turn three that the agent actually spoke over the caller's interaction, which is not a good look for any assistant, especially one that's trying to be helpful.
09:25And then ultimately if you look at the tool call, you'll be able to also understand that the refund was processed for order 40 and not 14.
09:33Um which is a huge huge mistake and actually seeing that live in the tool call is really really important.
09:41And every single one of these things would be invisible in a text log.
09:44But with the session view you can actually scrub the audio and find those failure surfaces within the tools, within what the agent is doing as well.
09:53The next thing I want to talk about real quick is evals that are specific to audio.
09:58So I kind of mentioned a few of them.
10:00I think tonality and sentiment analysis is really important.
10:03You want to be able to classify user emotion based on the actual audio.
10:08Transcript can't do this for you.
10:10It's very very easy to look at a transcript and see that it it, you know, looks fine, but tonality really matters.
10:18Positive, neutral, negative, not just a guess.
10:21Latency is really really important in audio and it's important in a few use cases.
10:25Models are getting better.
10:26There's, you know, jet bar architectures, etc. But time to first audio SLAs are still really important for customer service use cases.
10:34And you want to be able to flag things that are breaking that.
10:38You also want to flag things like I mentioned like interruptions, inaccuracy, transcription drift, and the task success.
10:45Um And you want to be able to run these evals directly against the audio with evals that are based on audio models.
10:53So, you know, LLM as a judge evals on a transcript are only so good.
10:56But actually evaling the underlying audio is really important.
10:59And so, um we have kind of the tooling available in Arise to be able to do that and we're supporting all the major audio platforms as well as they come out with new models.
11:09So again, evals specific to audio really matter.
11:12They're not just transcription based evals.
11:15And as you start to scale up your LLM as a judge evals or even move to agent as a judge that are doing like complex tasks like, for example,
11:23doing our back checks to making sure the agent actually access the system it was supposed to have access to.
11:29Being able to run these agent as a judge workflows in Arise, starts to become really important and we're kind of seeing the industry move away from, you know, just an LM as a judge
11:39analyzing a transcript or even analyzing audio, but actually accessing tools, accessing third-party systems, and hydrating some of that evaluation information inside of it.
11:50So, really easy to get started with evals in Arise through skills.
11:54It is literally as simple as, "Hey Claude, create an audio eval for me for sentiment analysis using GPT audio and catch frustrated user tone on those audio spans."
12:05And Arise will kind of take care of all the rest for you.
12:08The CLI under the hood is able to query spans, do some trace analysis, filter for the spans that are audio specific, actually create that eval, deploy it, and put it against traces and spans in your actual system.
12:22And this is part of why Arise exists.
12:24Yes, of course, you can do this yourself, but when you're trying to query, you know, billions of spans with sub-second query latency, it becomes really hard and also deploying these live evals
12:35as they come in.
12:36So, you're scoring them while users are actually interacting with your platform is really important.
12:42And then the final thing I'll mention is you also want these evals on the spans themselves because then you start to be able to do really complex querying and filtering across
12:52your data.
12:53So, you're not just looking at a failure and looking up in an external system.
12:56You actually have the eval scores in line with the exact audio that you're looking at.
13:01And that's why some of this is really powerful in Arise.
13:04Being able to see those eval outputs directly on the spans that are relevant helps you kind of get a full understanding of what is going wrong with the user interaction.
13:15And then you can kick off workflows from that.
13:16So, evals and monitors attached to those evals can kick off investigation agents that automatically start to dive into, you know, what tool calls and misrepresenting, Um, where is my agent failing?
13:29Is it capturing, you know, maybe a product surface area that I didn't code it for?
13:34Is there a new edge case I need to code for in my agent?
13:37Maybe I need to expose a new tool.
13:39All those become um, possible once you have the results directly on the spans.
13:47And so, I'll I'll just kind of flag and and end with this thought.
13:51The same loop that you're doing to improve some of these tools like coding agents, we now need to do for voice.
13:57And the loop really follows observe, evaluate, and improve.
14:01And then from that loop, you can start to get some really really cool continuous improvement loops within the Arize platform or within another platform if you choose to use it as well.
14:11Um, and the idea is once you observe and you have the data available to you, then you evaluate and you have the scores in line with your observations.
14:20Then you can start to kick off fixes, whether they're fixes that you make or whether they're fixes that an agent is automatically making on behalf of you.
14:28And then you can start to verify and validate those fixes.
14:31So, we just released agent experiments as well in Arize.
14:34Agent experiments, you could think of it kind of like Postman with tracing.
14:38So, imagine a world where you have audio streaming in from your users, it's automatically traced, you have live evaluations in the platform that are automatically scoring those interactions.
14:49And then you have SRE agents, so not an actual engineer that you hire, but an SRE agent that's looking through the traces, understanding the failure modes, hypothesizing a fix,
15:00and then actually standing up a fix for you in dev, replaying those failed traces for you against that fixed endpoint, tracing that to validate that the latency actually decreases.
15:11And you wake up in the morning, you look at a PR, the PR has a description of a problem.
15:16Hey, I found out why the latency was high.
15:19Turns out we were not truncating our tool results, and so we were passing in way too much information to the agent.
15:25Context got overloaded.
15:27I introduced the truncation algorithm.
15:29I shipped it.
15:30I actually stood it up in dev.
15:32I tested against the traces that failed.
15:34Here's the report card.
15:36Here's how latency decreased.
15:38And you wake up in the morning and all of this is ready for you, and you just click approve or deny on the PR.
15:43That is the future that I think we want to see at Arize, where self-healing software and continuous improvement are realities for everyone who's building agents.
15:52And I think making that possible for voice is really, really important.
15:57So, start seeing your agents.
15:59Add the Open Inference Instrumenter.
16:01Open source, free to use, OTEL compliant.
16:03You can send it to any back end.
16:05Um so, use Arize, don't use Arize, start tracing your audio.
16:09The second part of this is actually starting to do that analysis, and we would recommend Arize to do that.
16:14Sign up for a free account.
16:15Uh I'll have a code up at the end of my presentation for you guys to sign up for a a free year of pro.
16:21Um but start checking time to first audio.
16:24Start checking your interruptions.
16:25And start evaluating um how these interactions are actually going and whether they're accomplishing your use cases.
16:32Whether it's a tone or latency eval, um try using the Arize skills to actually create that eval.
16:38Uh we have some docs set up for you if you want to follow along.
16:40Bunch of examples and cookbooks ready.
16:43Uh and bring voice into the same rigor as the rest of your stack.
16:47You wouldn't be okay with this happening at the coding agent level.
16:50You wouldn't be okay with it happening uh for an agent that's able to execute stocks on Robinhood.
16:55You shouldn't be okay with it, uh though the stakes might seem a little lower, for a McDonald's agent that's ordering your cheeseburger, either.
17:01Um and so I'll end with that.
17:04Please, please keep building.
17:06Please connect with me also on Twitter.
17:09I still call it Twitter, not X. Uh on Twitter or LinkedIn.
17:12Would love to kind of talk and hear about some of the use cases you guys are building towards so we can add more echos to our library and make sure this is useful for you.
17:20And if you found this talk useful and you want to start investigating your voice agents, there is a free code up on the screen.
17:25It gives you a year of Voice Pro for free.
17:28It's a i w f 2026 and you can get started and start tracing your voice agents.
17:35So thank you very much.
17:36My name is Ali.
17:37I'll be available for questions for a couple minutes after but really appreciate you guys listening.
17:42Cheers.