We Built an AI Support Agent That Resolves 80% of Tickets — AssemblyAI

AI Engineer · 16 min · 234 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00Cool.
00:01[music]
00:13All righty.
00:14Can everyone hear me?
00:15Okay.
00:16All All levels coming through good.
00:17All right.
00:18Sweet.
00:18Well, uh, how's everyone doing?
00:23How's everyone doing?
00:25Better.
00:25Okay, there we go.
00:26Cool.
00:27Well, I hope uh the conference has been good for everybody.
00:30Appreciate you all being here to to listen to a chat from uh from me.
00:33Um for anyone who hasn't met me at the at the booth yet, uh my name is Matt Lawler.
00:37I'm a forward deployed engineer at Assembly AI.
00:40Uh mostly focused on bringing in inbound customers, um helping win their business, u making sure that they have a good time building with our API.
00:47Um so, how how many other like forward deployed engineers are out in the audience right now?
00:52Any of us?
00:54Okay, a couple.
00:55Cool.
00:56Well, um, still coming through.
00:58Okay.
00:59All right.
01:00Um, yeah.
01:01So, I wanted to make sure I I gave you guys an overview of kind of some of the go to market engineering stuff that we've been doing at Assembly AI that maybe you can apply to your own FTE team or maybe you can apply to your company um, as something that you can do as an easy win
01:12for your customers.
01:13Um, so for a little bit of overview of like assembly AI in case you haven't heard of us, um, we provide voice AI infrastructure.
01:20Um, so we train our own foundation models for speech to text primarily.
01:23Um, but we're starting to branch into other parts of the voice agent space.
01:27Um, so if anyone's ever had like a Fireflies note-taker join your meeting or something like that, you've also you've already seen a transcript from Assembly AI.
01:34Um, so we train our own foundational models not only for like async use cases like meeting recordings and things like that, but also for real time uh real-time like voice agents.
01:43Um, so if you've ever used like a a phone customer support agent when you called in somewhere and it wasn't a human or like drive-through ordering, they're more than likely building with Assembly AI's models.
01:53Um, and for anyone who's building a voice agent now, you can use us for the entire stack, not just speech to text.
01:58Um, so you can use our voice agent API and we'll string together the speech to text, the LLM, and the text to speech as well.
02:04Um, as our competitors know, as we know, as anyone in the voice AI space knows, demand is absolutely insane um, for voice AI right now.
02:12Um, so assembly at our current moment, we see around a thousand API signups every single day.
02:16Um, which is somewhat of the theme of why we were focusing on automation and wanting to build kind of a new support bot for our team because at least until like two days ago, I was the only
02:27uh onboarding engineer at Assembly AI.
02:29And so you can imagine, as much as I'd want to talk to a thousand new signups every single day, that's just not feasible.
02:35Um, so we had to build something to to solve that problem.
02:39And then for anyone here who doesn't know like what a forward deployed engineer is, it's a very catch-all title.
02:43It varies a lot at every different company, but effectively we want to get deployed with our end customers.
02:48We want to understand their use case.
02:49We want to make sure that we are building exactly what they need.
02:52Um, so we frequently commit code directly to their repositories.
02:56We usually meet with them several times a month, if not a week.
02:59Um, and we really try to understand their use case and get to know them very personally.
03:03Um so like at least on the inbound side for anyone who comes into assembly who wants to build with us we are the people um that are their main point of contact for anything they need on the technical commercial side
03:13anything that they might need they get from us their FTE um and we're responsible for winning their business um and building that trust with them.
03:20So it's a great services model.
03:22It's great uh if we can actually get that kind of relationship with every customer, but again with like a thousand API signups every day, I I can't be one to many serving that many customers and getting on that many calls.
03:33Um and so ultimately this doesn't really scale super well.
03:35Uh we want to make sure that we're freeing ourselves up to be leading uh humanled deals as much as possible, but we can't do that with every deal.
03:43Not every customer win can be like handbuilt by us.
03:45Um and we can't just continue to hire more and more FTEEs if we're going to try and serve more and more customers.
03:50Um, so ultimately like for anyone who's an FTE in here, if you're looking to become an FTE, if you already have an FTE team at your company, I would encourage you to take that knowledge that you have as an FTE and try to automate yourself out of a job.
04:02I think that should be your role every single day.
04:03If you want to serve your customers better, if you want to work with them more, um, and if you want to deliver a better customer experience at scale, you can't be the bottleneck to having a good customer experience.
04:12And we learned that firsthand.
04:15So again, we were drowning in like inbound um, volume.
04:18Um, you can see like we basically had a thousand API signups up there earlier.
04:23Um, and we had the natural instinct that a lot of teams do where we were like, we just need to deflect more conversations.
04:28I just want to talk to fewer people.
04:29I want to make sure I'm freeing myself up to work on more important stuff.
04:32Um, so we basically bought this off-the-shelf support bot and we're like, let's point it at the docs.
04:36It'll answer all the docs questions for us.
04:38It's going to be great.
04:39We'll never have to answer a simple question from customers again.
04:41And it only resolved about 10% of the conversations.
04:44Um, so it helped.
04:45Um, you can imagine if we had like a thousand conversations a day, a hundred of those were now entered by a bot.
04:50But then our team still had to field 900 tickets a day.
04:53So it wasn't exactly very scalable.
04:55Um, so 10 10% uh end to end resolution rate wasn't good enough.
04:59And ultimately we couldn't iterate fast enough to be able to fix this.
05:03We didn't have access to the system prompt.
05:05We didn't have access to the tools.
05:06We didn't have access to the rag infrastructure.
05:08So anytime we needed to change something, we'd go to this vendor and they'd say, "Well, it's on the road map."
05:14and that didn't quite work for us.
05:15So, we wanted to build our own.
05:18So, we built uh another member of our team rather than hiring one and we named him Joey.
05:23Um so, anyone now who reaches out to us, whether it's on our chat widget on our site, whether you reach out to contact sales, whether you reach out to our support alias,
05:31you're going to meet Joey instead of a human.
05:33Um he's the first line of defense now for every single conversation that happens in bounded assembly.
05:37And he's built to be the most infinitely scalable FTE on our team.
05:41Um, so he remembers past conversations with you.
05:44Um, we track all of his metrics via Pylon because he's still a member of our support team and so he has access to Pylon to manage conversations and remember who you are.
05:52Um, and he's built on top of the cloud agent SDK.
05:54And so the cool things that he's able to do is he can manage his own infrastructure.
05:58He has a file system.
05:59He can write and debug code for you.
06:01He can call tools.
06:03He can give himself new abilities.
06:04Um, and so he's actually able to operate more like an FTE and less like just a standard chatbot that you'd be talking with, you know, on on most major sites nowadays.
06:14So, um, if you want to know a little bit of like how to build your own Joey, um, the architecture kind of has like four major parts.
06:21Um, so as everyone might know from working with Claude, Markdown is your friend.
06:25You should be giving Markdown files to Claude so that he understands everything about your business.
06:29And so we wanted him to have access to all of our documentation and be trained up on it all the time.
06:33So, we checked all of our docs out as local markdown and gave it to Joey.
06:37Um, and so anytime that we push an update to our documentation system, he's able to get all of those files and he has them locally in his file system.
06:44So, actually what's really cool is if our entire doc site was down right now, you could go to Joey and still get live answers on all the newest stuff that we've shipped without our team having to actually get involved.
06:54um for retrieval so we can give you a good answer.
06:56We partner with Voyage so we use their embeddings um so that he gets the most relevant documents up front so he doesn't have to spend a lot of time searching and he can get you a faster answer.
07:04And then since he has all these docs as local files if he gets confused or he needs to string together multiple resources to create a response he can agentically search the file system that he lives on so that he can go and find those resources for you.
07:16Um so he can kind of go into this like deep thinking mode where he's able to search across all these resources and then we shipped it on railway.
07:23We didn't want to deal with infrastructure.
07:24We didn't want to spin up an EC2.
07:25We didn't want to have to change instance types.
07:27So, we deployed it all on Railway.
07:29Their service is pretty incredible.
07:31Basically, we can just go ahead and see if he has a bad conversation with someone in Slack.
07:35We can write a PR real quick and we can deploy it and within 30 seconds there's a new live version of Joey.
07:40So, we're able to iterate really, really quickly.
07:42Uh we've actually had times where we've actually monitored a live conversation, caught a bug, deployed a fix, and then it's been proof for the rest of the session, and that customer doesn't even have to know that we we shipped that fix behind the scenes.
07:53Um so Joey is able to to basically get updated whenever we want.
07:56Um so we actually have full control over everything that that he does.
08:01Um and so the results were pretty remarkable from us building this.
08:04And it's another reason why and wanted to encourage everyone who's here to try and build your own Joey.
08:09So, we went from 10% to 80% endto-end resolution rate in just the first week of deploying this this build with a pretty naive implementation.
08:16And we did that all for around $700 a month in both token and inference uh or infrastructure costs.
08:22So, we resolved 80 80% of our our tickets end to end with no human in the loop.
08:27And some people like to game these metrics.
08:29They'll say like, oh, we resolved 80% but only on like a subset of types of tickets.
08:34Not the case here.
08:35you cannot reach a human directly unless you go through Joey and you tell him to escalate.
08:38So he actually does handle 100% of these inbound tickets for us now.
08:42Um and he only escalates 20% of them to a human for the right reasons.
08:45Maybe it's a rate change.
08:47Maybe it's a data opt out.
08:48Maybe you need to sign an agreement with us or you have needs that only an FTE or someone from legal needs to be involved in.
08:54These are things that actually still need a human but eventually we can automate.
08:57Um and so the best part is whatever Joey can't do yet gives us a very clear list of what we need to do to continue to scale our team.
09:04Um, if he's not able to send them a BAA, well, okay, maybe we should give him a link that he can reference as a resource.
09:09Um, if he's not able to talk to customers about pricing, well, you can negotiate with Joey.
09:14Um, if you go on the site and you test him out afterwards, you can tell him exactly how many hours you have.
09:18Um, and he'll actually quote you a rate and then you can negotiate with him.
09:22Um, so we tried to automate a lot of these other things that he was still escalating and there's still there's still plenty more to come.
09:29But the real thing that, you know, we wanted to test out by building Joey was we wanted to dog food our own product, too.
09:34We didn't want to just like build something that deflects more stuff from our customers.
09:38We didn't want to build something that just um is kind of this general support bot that anyone can build.
09:43We work in voice and our customers build voice agents.
09:46So, I wanted to build a voice agent and so I wanted to give a voice to Joey.
09:49So, now rather than just chatting with him via text, you can actually chat with him using your voice as well.
09:55Um, so we have a voice agent API as I mentioned earlier.
09:57Um, and we integrated this into Joey this week.
09:59So now if you go onto our site, you can switch to voice mode and you can talk to Joey using just your voice.
10:04Um, because I ultimately think for any FTE here, no matter how much you've been deployed with your customers, no matter how embedded you are with them, the best way to understand what your customers are building is to actually build their same product yourself.
10:15And that's exactly what we wanted to do with this.
10:19So we put our voice agent API into Joey.
10:22Um, it's all speech in, speech out, one websocket connection where we string together our speechto text model, an LLM and text to speech, all optimized for voice.
10:30It's real-time and low latency.
10:31It handles all the pauses, interruption handling, barge, all of that kind of stuff for you.
10:36U, we bring our own voices, so you don't have to find another provider.
10:39And it's all the same tools that he already had if you were communicating with him via text.
10:42It's just now you can actually talk to him like you were a human.
10:45Um, and hopefully later this week we can hook them up to a phone number.
10:47So now we'll have an Assembly AI phone number so you can get real-time support from Joey whenever you need it.
10:54So let's go ahead and talk to Joey a little bit and we can walk through some of the architecture live.
10:58Um so for anyone who wants to know a little bit more about how this was actually built um we basically have all of the the docs checked out locally in Markdown.
11:08Um so every single time someone puts an update out to our docs, these files also get locally checked out into this repo.
11:14So they stay perfectly in sync with any new features that we're shipping, our change log, our pricing, all of these major uh pages from our website get converted to markdown and get given to Joey.
11:24Um so he sees all of these in markdown format and he's able to see what the URL is so he can site his sources to the customer in case they want to double check his work or if they just want more information.
11:33Um at the same time, um all of it is provided as like Claude MD instructions.
11:37I think it's like a 30,000 line Claude MD of all these guard rails and and advice on how he should be operating with customers.
11:44Um, and this is what we majorly update whenever we see that he had a bad experience with a customer or gave a wrong answer, we ship an update to this cloud MD and now for every conversation that he has going forward, he's able to be better.
11:54Um, and it's all contained as a Docker file.
11:56Um, and so it's really easy to go and just containerize and deploy him on Railway.
12:00Um, and so again, every time we ship an update to this repo, a new version's already live on Railway and no one had to manually monitor the deployment.
12:07Um, so let's go ahead and uh I'll fire up Joey here.
12:11we can ask him a pretty typical question so you can see what our voice agent API is like.
12:18So I'm going to simulate this locally because I I know probably many of you are going to test Joey after this and I don't want to add to the queue for our actual support team.
12:25Um so I'll I'll go ahead and just do a local simulation of it.
12:28But typically, yeah, you would go into here, you'd meet Joey, you'd want to have a typical question.
12:32You might talk to him via text, but let's go ahead and talk to him with voice.
12:37Hi, this is Joey at Assembly AI.
12:39How can I help you today?
12:41Hey Joey, I'm looking to build an ambient medical scribe with Assembly AI, but before I can test, I need to assign a BAA.
12:47Is that something that you can do?
12:51One sec.
12:52Let me check that for you.
13:06Yes, you can execute the BAA yourself right away.
13:09Just complete our standard HIPPA business associate agreement, but note that you'll need a paid account with a card on file first.
13:17And signing the BAA automatically opts you out of model training.
13:21The relevant docs are linked in the chat.
13:23Cool.
13:24Thanks, Joey.
13:24Well, I got to I got to get back to my demo.
13:26I'd love to chat more, but uh thanks for all the help.
13:32You are very welcome.
13:33Good luck with your demo and feel free to reach out if you need anything else later.
13:37Have a great day.
13:39Isn't he so nice?
13:40[laughter]
13:41I enjoy talking to Joey.
13:43I think our customers will enjoy talking to Joey.
13:45But this is a great example of like two years ago when I first joined assembly on our support team.
13:49This is a conversation that would have to be handled by a human.
13:52This is something that if you roached out in in chat and a customer uh or a support engineer wasn't able to uh answer you immediately.
13:59They'd have to wait.
14:00They'd have to get a response from us.
14:01It was a canned response.
14:02Um it was just not a great customer experience.
14:04And now you have an FTE that you can talk to 24/7 for these basic questions.
14:08Um and you can do it through voice.
14:10So you can actually test our product at the same time.
14:12Um so there's there's kind of a meta layer there of where if you're talking to Joy about how to build a voice agent API, you're already using it.
14:18Um, and you're already seeing exactly what your product experience could look like.
14:25So, if there's anything that you should get out of this talk, it's not just that you should build your own Joey.
14:29Um, but I think that it's like if you're if you're an FTE at your company, you need to be dog fooding your own product all the time.
14:34It's not enough to just be teaching your customers how to get started with it.
14:37any new feature that you launch, any new product that you offer, you need to be teaching your customers, um, with all the knowledge that you've built from actually using and building and trying to ship the same product.
14:48Um, so I had to run through all the same walls.
14:49I had to figure out how to deal with latency.
14:51I had to figure out how to deal with turn taking or bargain or interruption handling.
14:55Um, and I think I have a lot better empathy of how to like work with customers that you want to use our voice agent API.
15:00Um, because now I've actually used it.
15:02I've used the same API.
15:03It's the exact one that my customers build on and I can give them better advice because I've actually shipped something with it.
15:10So go build go build something.
15:12Go build whether it's a your own Joey, go build whether it's a voice agent, especially if it's a voice agent, build it on Assembly AI.
15:19Um, but if you're if you're actually working in an FTE capacity, try to build the exact product that your customers are building and do it on your own.
15:25Don't just wait for them to ask you for help.
15:27Don't just wait to remove all their blockers.
15:29You should be going out and building the exact same thing.
15:31And at the same time, you might actually be able to make your customer experience better simultaneously.
15:35So, um, if anyone has like questions on how to build Joey, if you want to check them out at the booth afterwards, our booth is just right over here.
15:42Um, if anyone has feedback on Joey since this just shipped this week, if you have a weird conversation with him, I'd love to know about it so we can keep making fixes and I can show you how we deploy that live.
15:51U, but overall, really excited to uh, be here at the conference and talk to more people about voice agents.
15:56So, thanks for uh, coming and listen to this talk.
16:00[applause]
16:16[music]