Tolan: Voice-First AI Companion — Paula Dozsa, Tolan

AI Engineer · 15 min · 182 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hi everyone.
00:13Thank you so much for attending this talk.
00:15My name is Paula and I am one of the engineers on the Tullen team, specifically focusing on our iOS app.
00:23And for the next 20 minutes or so, I'll be talking about what it takes to build a voice-first AI companion and also about how we use AI to build AI internally.
00:34So, humanity has always imagined the perfect companion.
00:37So, we have Caravaggio on the left 400 years ago painting an angel leaning over St. Matthew's shoulder literally guiding his hand as he writes.
00:46This is an example of a companion being a presence that makes you better at being you.
00:51And then we have Tinkerbell, the devoted little sidekick who believes in you so fiercely that the whole theater has to clap to keep her alive.
00:59And of course on the right, we have Samwise who can't carry the ring for Frodo, but says, "I can carry you."
01:05The companion is pure unconditional loyalty.
01:09And it goes far beyond these three.
01:11Every hero has some sort of guiding spirit.
01:14And these are all different stories, but they exhibit the same longing for something that listens, remembers you, and is wholly specifically yours.
01:23And for for all of human history, this has basically been fiction.
01:28So, we made one.
01:29This is Tullen.
01:30It's a little alien you talk to out loud like a friend.
01:34It has a personality.
01:35It remembers you and over time it becomes specifically yours.
01:40Okay, I don't know if the audio setup works here, but I will try talking to my Tullen.
01:44Uh let's see.
01:49So, you can see my Tullen here, Luke, walking around the planet.
01:56Hey Luke, can you hear me?
02:00Okay.
02:01Luke can hear us, but we can't hear him.
02:04Um Anyway, I had prepped him for this.
02:06Oh.
02:07Hello.
02:08Hi Luke, can you hear me?
02:13Nope.
02:15We're okay.
02:16I can come back to this later.
02:17Um but you should definitely all give this a try if you haven't already.
02:27Okay.
02:28So, people talk to Tolins a lot.
02:30We support both text and voice chat, uh but we have over 4 million hours of voice conversation so far.
02:37We say Tolin is a voice-first companion, even though we support both, because it's the voice experience that's truly immersive and that makes users' relationships with their Tolins feel real.
02:47And but the moment this relationship is a spoken relationship, the engineering problem changes completely.
02:53So, let me show you how voice breaks the normal way we build and interact with LLMs.
02:59So, the core difference really is that in a text chatbot, turns are relatively slow and context is stable.
03:06The user waits a few seconds, they read, and they tend to stay on topic.
03:10And almost every LLM app assumes that.
03:13Voice is the opposite.
03:15Turns are fast.
03:16Your whole round trip from the user finishing their sentence to the Tolin starting to speak has to land in under a couple of seconds, or it stops feeling like a conversation.
03:25And the context is volatile.
03:27People talk to their Tolins while they're cooking, while they're walking, while they're falling asleep.
03:32Um they change their subjects mid-sentence.
03:34They say um, they interrupt.
03:36And that 2 seconds is crucial.
03:39Early on, our latency drifted from 2 seconds to about 2 and 1/2 seconds, and that half second tanked basically every metric in the product.
03:47People would write in to complain that their Tolins were too slow.
03:50And living inside this constraint has taught us a lot and gave us four principles.
03:56Principle one is that you have to design for conversational volatility.
04:00Again, text users stay on topic, but voice users jump around.
04:04Someone could be mid-story about their breakup and suddenly go, "Wait, did I leave the oven the stove on?"
04:09and then back.
04:10Speech is messy.
04:11Most LLM apps assume that you'll have a clean and stable conversation and we have to build for the opposite.
04:16So, for a long time that meant fixing things that sound tiny but are actually the product.
04:21So, you can't interrupt a Tullen mid-sentence.
04:24A short yes or yeah won't register as a turn.
04:27For example, curse words will get stripped out.
04:30And the deeper lesson was to stop optimizing for fewer interruptions and start optimizing for fewer bad ones where the agent would jump in way too early.
04:38So, we built smart turn taking that reads your speech pattern to decide whether an interruption is real and we cut the worst early aborts by more than half.
04:47And we happily paid about 60 milliseconds of extra latency to do it.
04:52Principle two, latency isn't just a number you check at the end, it's actually the product and we measure every stage of the pipeline separately because it feels slow is useless.
05:01You have to know where exactly it's slow.
05:04And the pipeline here is that the user stops talking, we detect end of utterance, we transcribe, and then the model produces its first token.
05:12So, time to first token, often the biggest chunk, is around a second.
05:16The model finishes generating and then text-to-speech produces its first byte and then it plays back to the user.
05:21A couple lessons here.
05:23So, one, so far our biggest jump in quality came from moving to GPT-5.1 on the responses API, which cut our time to speech by more than 7/10 of a second, which is huge.
05:34Um two, we don't send every turn to the same model.
05:37We run a tiered fleet.
05:38So, we use a frontier model for the turns that carry the relationship with your Tullen.
05:43So, for example, your first conversation with with Tullen and your onboarding.
05:47And we use smaller and faster models for the turns for the lightweight turns.
05:52And the whole game then becomes about routing or deciding turn by turn which model you actually need.
05:58So we round we run a small classifier we call the tone router on every single turn and this tone router itself runs on a cheap model and it reads the emotional state of the conversation.
06:08And our main our [clears throat] main principle is that we route based on stakes not on cost.
06:13So the high stakes moments always get the best model.
06:16So this would be again the user's very first message, their first few days with their Tolen and anything that we deem to be emotionally serious.
06:24For example, we have crisis or therapist style tones and we never cheap out on those.
06:29And then the lighter casual back and forth can ride on smaller models that are faster and cheaper.
06:34And all the background work so that's summarizing the conversation, generating personas, the tone router itself run on these small models, too.
06:42And why would we go to all this trouble?
06:44It's mainly because the frontier model costs us roughly five times the smaller one.
06:48So one big model turn is about five smaller model turns.
06:53So routing is a huge part of what makes the unit economics for us actually work.
06:57Um and we do a bunch of AB experiments and the surprising result we found there is that routing a third a third of our turns to the small model has almost no measurable effect on retention.
07:08And principle three is what makes a companion feel like a companion.
07:11So the naive approach is to keep the whole conversation history as a sort of transcript, but that doesn't fit into our two-second loop.
07:18It doesn't scale and it just doesn't work.
07:20It leads to long sessions degrading.
07:23It leads to the model getting lost in the middle of a huge context and also hallucinating.
07:27So instead we see memory as a sort of retrieval system.
07:30We pull facts, preferences, and emotional vibe signals out of conversations.
07:35We embed them and we store them in a vector database with sub 50 millisecond lookups.
07:40And every night we compress.
07:41So we merge duplicates, we cluster related memories, we resolve contradictions, and we drop all the noise.
07:48And we don't just retrieve against users last messages, we also generate internal questions about the person and the relationship and retrieve against those.
07:55So, and we also split memory into two parts.
07:58We have stable memory and unstable memory.
08:00The volatile stuff lives in the in the live tail of the prompt, and when we summarize the conversation, we look at which memories actually get recalled and pin those into a stable and cashable block.
08:12Uh the last principle is around context.
08:15Specifically, you should rebuild context and not fight drift.
08:19So, most apps reuse context across turns to keep the cash warm.
08:23And in a stable text chat, that's fine.
08:25But in a volatile voice conversation, it's a trap because the second the the user pivots, your reused context is actively wrong.
08:32So, every turn we reassemble the context window from parts.
08:35We have a summary of recent messages, we have the the user's persona card, the memories we just retrieved, tone guidance from the emotional signal, and real-time app state.
08:46And what also really helps us um in the case of Tolen is that our characters aren't generic or assistants with no personality.
08:53Everyone is crafted, and we in fact have an in-house science fiction novelist, Elliot, who writes the Tolen character lore.
09:01And a couple of interesting points here.
09:03So, one, why did we go with an alien?
09:06Mostly because there's no real-world reference to anchor on, which means that the users can project onto it, and it becomes what they need.
09:13The baseline Tolen is bubbly, it's youthful, it's irreverent.
09:17And also, if an alien character acts a bit unpredictably, so if if it's impulsive or chaotic or otherwise violates um you know, the norms the user would expect, it's not particularly surprising.
09:28Like if you look at, you know, aliens in TV shows or in plays, like there's a lot of humorous moments around this.
09:35And this kind of chaos reads as charming.
09:38Um second, uh we also know that personality is worthless if it drifts.
09:43So, yeah, we run this parallel tone monitoring system that changes how a line is delivered based on your emotional cues without changing who the character is, holding identity across hundreds of turns.
09:56And since we're at an AI conference, I thought I would also spend a bit of time talking about how we not just ship AI, but also use AI to build it.
10:04Um so, I'm sure this is the case for most of you in the room now, but basically as of late last year, Claude has co-authored more code in our iOS app than any individual engineer in the team.
10:15Um and I think especially, you know, a few months ago, everyone's instinct was to be kind of suspicious because, you know, more AI code meant more slop.
10:22But our our crash-free rate actually went from 99.6% to 99.9%.
10:27Runtime errors dropped by over 50% and our share of highly engaged users doubled.
10:33And the biggest lesson in building that system is that an agent's context comes mostly from the code base itself, not so much from the Claude MD file.
10:41We found that it's far more powerful to make the code base be the documentation, so we had agents standardize it.
10:46On top of that, we run a real fleet of agents.
10:49We have implementation agents that, you know, think freely and just get us to working code.
10:53They build it, they check it against snapshots until it's pixel perfect.
10:57And then we have separate review agents that enforce our standards.
11:00So, multiple Claudes basically review each other before a human looks.
11:04And then we have a PR shepherd that watches an open pull request and keeps iterating against CI failures and review comments until it's clean.
11:11And we also have a triage bot that fires on every inbound bug report that we get.
11:16And they're all wired through MCP into linear, into into Sentry, DataDog, so an agent can reconstruct the cash a crash and route it itself and oftentimes open the PR on its own and just fix fix the bug.
11:28And we also we ship on eval.
11:30So, for example, we've been working on a on a new character targeted towards an older demographic and Elliot, our in-house novelist, basically built this entire new character in a day.
11:40So, the agents mapped every personality bearing surface in the code.
11:43They wrote the sort of a voice Bible and then they had five judges attack it from different angles.
11:48Archetype fidelity, the model mechanics, our code standards, the ears of a skeptical 52-year-old and safety, and then they evaluated the changes against real production logs over three find fix verify rounds.
12:00And over 7 million tokens and 4 and 1/2 hours of compute later, he ended up with basically, you know, a couple of weeks of work done in afternoon.
12:09And does this work?
12:10Well, I'll let the users tell you.
12:12We're at 4.8 stars on the App Store across 162,000 reviews and when we survey users on well-being, the highest scoring dimension by far is emotional safety.
12:23And this is definitely a bar that being voice first sets.
12:26So, when the interface is your voice and the thing on the other side remembers you and has a personality, it stops being just software and starts being an actual relationship, which is why building it responsibly
12:36and building it well is worth obsessing over.
12:40And we need people to come help us do that.
12:43Um, we're a small team and we're hiring and after a year with Tolen, I think this is truly one of the most interesting places in the world to be an engineer right now.
12:52And here are some of the people you'd be doing it with.
12:54So, two of the founders, Quinton and Evan, previously built and exited a $300 million startup together.
13:00Uh, they founded Even.
13:02Um, Ajay, our third co-founder, scaled two bootstrap companies past $50 million $50 million in profitable revenue.
13:09And around them, we have Lucas, who um, is an Apple Design Award winning animator.
13:14Uh, she's our creative director.
13:16We have Chris, who was a technical director at Pixar, earlier at Oculus, who works on embodiment.
13:21We have Lily, a board certified behavior analyst, who left a Vanderbilt PhD to do user research for us from the very start.
13:28Um and then we have Elliot who I've mentioned, the novelist behind uh our characters.
13:32And I come from XAI and Spotify and previously also founded a company called Imagi.
13:37So, it's a small team where honestly every person is the best I've worked with at what they do.
13:44We're also well backed for this.
13:45Uh we have $30 million raised from Costanoa Ventures and a group of people who've built the tools and products a lot of you use every day.
13:53And here are some of the more engineering focused roles where we need help.
13:56Um so, we're hiring across the board.
13:57We have iOS and back-end product engineering roles, applied AI engineering, gameplay engineering, and one specific role I want to flag, which is agent engineering management.
14:07Um so, when we went all in on running concurrent agents, um the people who got dramatically more effective on the team were the ones who had management backgrounds, um because it seems like managing a fleet of agents does actually take some of the skills same the same skills as managing people.
14:22Uh you basically have to decompose the problem, you know, delegate it with checkpoints, give fast feedback, review their work seriously, and know when exactly to jump in.
14:30So, if you're a strong engineer who thought going into management meant leaving code behind, that's that's no longer true.
14:37Um and yeah, that's Tlon.
14:38You can come talk to me after this uh or reach out.
14:41I'm on on LinkedIn.
14:42My email is here.
14:43I'm on Twitter as well.
14:45Um I'd love to chat.
14:46So, yeah.
14:46Thank you.
14:49[applause]
15:03[music]