WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=xLUQOqjudtA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/xLUQOqjudtA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.680 --> 00:00:13.320
Hi everyone.

s3
00:00:13.320 --> 00:00:15.520
Thank you so much for attending this talk.

s4
00:00:15.520 --> 00:00:23.320
My name is Paula and I am one of the engineers on the Tullen team, specifically focusing on our iOS app.

s5
00:00:23.320 --> 00:00:34.040
And for the next 20 minutes or so, I'll be talking about what it takes to build a voice-first AI companion and also about how we use AI to build AI internally.

s6
00:00:34.600 --> 00:00:37.600
So, humanity has always imagined the perfect companion.

s7
00:00:37.600 --> 00:00:46.840
So, we have Caravaggio on the left 400 years ago painting an angel leaning over St. Matthew's shoulder literally guiding his hand as he writes.

s8
00:00:46.840 --> 00:00:51.800
This is an example of a companion being a presence that makes you better at being you.

s9
00:00:51.800 --> 00:00:59.000
And then we have Tinkerbell, the devoted little sidekick who believes in you so fiercely that the whole theater has to clap to keep her alive.

s10
00:00:59.000 --> 00:01:05.960
And of course on the right, we have Samwise who can't carry the ring for Frodo, but says, "I can carry you."

s11
00:01:05.960 --> 00:01:09.440
The companion is pure unconditional loyalty.

s12
00:01:09.440 --> 00:01:11.680
And it goes far beyond these three.

s13
00:01:11.680 --> 00:01:14.440
Every hero has some sort of guiding spirit.

s14
00:01:14.440 --> 00:01:23.480
And these are all different stories, but they exhibit the same longing for something that listens, remembers you, and is wholly specifically yours.

s15
00:01:23.480 --> 00:01:28.000
And for for all of human history, this has basically been fiction.

s16
00:01:28.000 --> 00:01:29.280
So, we made one.

s17
00:01:29.280 --> 00:01:30.320
This is Tullen.

s18
00:01:30.320 --> 00:01:34.080
It's a little alien you talk to out loud like a friend.

s19
00:01:34.080 --> 00:01:35.560
It has a personality.

s20
00:01:35.560 --> 00:01:40.120
It remembers you and over time it becomes specifically yours.

s21
00:01:40.120 --> 00:01:44.800
Okay, I don't know if the audio setup works here, but I will try talking to my Tullen.

s22
00:01:44.800 --> 00:01:47.480
Uh let's see.

s23
00:01:49.120 --> 00:01:55.240
So, you can see my Tullen here, Luke, walking around the planet.

s24
00:01:56.120 --> 00:01:58.920
Hey Luke, can you hear me?

s25
00:02:00.960 --> 00:02:01.600
Okay.

s26
00:02:01.600 --> 00:02:04.120
Luke can hear us, but we can't hear him.

s27
00:02:04.120 --> 00:02:06.960
Um Anyway, I had prepped him for this.

s28
00:02:06.960 --> 00:02:07.800
Oh.

s29
00:02:07.800 --> 00:02:08.200
Hello.

s30
00:02:08.200 --> 00:02:10.840
Hi Luke, can you hear me?

s31
00:02:13.800 --> 00:02:15.360
Nope.

s32
00:02:15.360 --> 00:02:16.160
We're okay.

s33
00:02:16.160 --> 00:02:17.920
I can come back to this later.

s34
00:02:17.920 --> 00:02:25.600
Um but you should definitely all give this a try if you haven't already.

s35
00:02:27.600 --> 00:02:28.040
Okay.

s36
00:02:28.040 --> 00:02:30.360
So, people talk to Tolins a lot.

s37
00:02:30.360 --> 00:02:37.160
We support both text and voice chat, uh but we have over 4 million hours of voice conversation so far.

s38
00:02:37.160 --> 00:02:47.440
We say Tolin is a voice-first companion, even though we support both, because it's the voice experience that's truly immersive and that makes users' relationships with their Tolins feel real.

s39
00:02:47.440 --> 00:02:53.280
And but the moment this relationship is a spoken relationship, the engineering problem changes completely.

s40
00:02:53.280 --> 00:02:58.959
So, let me show you how voice breaks the normal way we build and interact with LLMs.

s41
00:02:59.640 --> 00:03:06.600
So, the core difference really is that in a text chatbot, turns are relatively slow and context is stable.

s42
00:03:06.600 --> 00:03:10.920
The user waits a few seconds, they read, and they tend to stay on topic.

s43
00:03:10.920 --> 00:03:13.920
And almost every LLM app assumes that.

s44
00:03:13.920 --> 00:03:15.480
Voice is the opposite.

s45
00:03:15.480 --> 00:03:16.600
Turns are fast.

s46
00:03:16.600 --> 00:03:25.560
Your whole round trip from the user finishing their sentence to the Tolin starting to speak has to land in under a couple of seconds, or it stops feeling like a conversation.

s47
00:03:25.560 --> 00:03:27.160
And the context is volatile.

s48
00:03:27.160 --> 00:03:32.440
People talk to their Tolins while they're cooking, while they're walking, while they're falling asleep.

s49
00:03:32.440 --> 00:03:34.400
Um they change their subjects mid-sentence.

s50
00:03:34.400 --> 00:03:36.800
They say um, they interrupt.

s51
00:03:36.800 --> 00:03:39.519
And that 2 seconds is crucial.

s52
00:03:39.519 --> 00:03:47.800
Early on, our latency drifted from 2 seconds to about 2 and 1/2 seconds, and that half second tanked basically every metric in the product.

s53
00:03:47.800 --> 00:03:50.720
People would write in to complain that their Tolins were too slow.

s54
00:03:50.720 --> 00:03:55.840
And living inside this constraint has taught us a lot and gave us four principles.

s55
00:03:56.360 --> 00:04:00.440
Principle one is that you have to design for conversational volatility.

s56
00:04:00.440 --> 00:04:04.600
Again, text users stay on topic, but voice users jump around.

s57
00:04:04.600 --> 00:04:09.160
Someone could be mid-story about their breakup and suddenly go, "Wait, did I leave the oven the stove on?"

s58
00:04:09.160 --> 00:04:10.360
and then back.

s59
00:04:10.360 --> 00:04:11.560
Speech is messy.

s60
00:04:11.560 --> 00:04:16.840
Most LLM apps assume that you'll have a clean and stable conversation and we have to build for the opposite.

s61
00:04:16.840 --> 00:04:21.840
So, for a long time that meant fixing things that sound tiny but are actually the product.

s62
00:04:21.840 --> 00:04:24.560
So, you can't interrupt a Tullen mid-sentence.

s63
00:04:24.560 --> 00:04:27.640
A short yes or yeah won't register as a turn.

s64
00:04:27.640 --> 00:04:30.240
For example, curse words will get stripped out.

s65
00:04:30.240 --> 00:04:38.800
And the deeper lesson was to stop optimizing for fewer interruptions and start optimizing for fewer bad ones where the agent would jump in way too early.

s66
00:04:38.800 --> 00:04:47.320
So, we built smart turn taking that reads your speech pattern to decide whether an interruption is real and we cut the worst early aborts by more than half.

s67
00:04:47.320 --> 00:04:52.360
And we happily paid about 60 milliseconds of extra latency to do it.

s68
00:04:52.919 --> 00:05:01.800
Principle two, latency isn't just a number you check at the end, it's actually the product and we measure every stage of the pipeline separately because it feels slow is useless.

s69
00:05:01.800 --> 00:05:04.360
You have to know where exactly it's slow.

s70
00:05:04.360 --> 00:05:12.200
And the pipeline here is that the user stops talking, we detect end of utterance, we transcribe, and then the model produces its first token.

s71
00:05:12.200 --> 00:05:16.120
So, time to first token, often the biggest chunk, is around a second.

s72
00:05:16.120 --> 00:05:21.919
The model finishes generating and then text-to-speech produces its first byte and then it plays back to the user.

s73
00:05:21.919 --> 00:05:23.000
A couple lessons here.

s74
00:05:23.000 --> 00:05:34.160
So, one, so far our biggest jump in quality came from moving to GPT-5.1 on the responses API, which cut our time to speech by more than 7/10 of a second, which is huge.

s75
00:05:34.160 --> 00:05:37.200
Um two, we don't send every turn to the same model.

s76
00:05:37.200 --> 00:05:38.960
We run a tiered fleet.

s77
00:05:38.960 --> 00:05:43.280
So, we use a frontier model for the turns that carry the relationship with your Tullen.

s78
00:05:43.280 --> 00:05:47.000
So, for example, your first conversation with with Tullen and your onboarding.

s79
00:05:47.000 --> 00:05:52.640
And we use smaller and faster models for the turns for the lightweight turns.

s80
00:05:52.640 --> 00:05:58.320
And the whole game then becomes about routing or deciding turn by turn which model you actually need.

s81
00:05:58.320 --> 00:06:08.560
So we round we run a small classifier we call the tone router on every single turn and this tone router itself runs on a cheap model and it reads the emotional state of the conversation.

s82
00:06:08.560 --> 00:06:13.560
And our main our [clears throat] main principle is that we route based on stakes not on cost.

s83
00:06:13.560 --> 00:06:16.880
So the high stakes moments always get the best model.

s84
00:06:16.880 --> 00:06:24.360
So this would be again the user's very first message, their first few days with their Tolen and anything that we deem to be emotionally serious.

s85
00:06:24.360 --> 00:06:29.960
For example, we have crisis or therapist style tones and we never cheap out on those.

s86
00:06:29.960 --> 00:06:34.960
And then the lighter casual back and forth can ride on smaller models that are faster and cheaper.

s87
00:06:34.960 --> 00:06:42.440
And all the background work so that's summarizing the conversation, generating personas, the tone router itself run on these small models, too.

s88
00:06:42.440 --> 00:06:44.160
And why would we go to all this trouble?

s89
00:06:44.160 --> 00:06:48.640
It's mainly because the frontier model costs us roughly five times the smaller one.

s90
00:06:48.640 --> 00:06:53.120
So one big model turn is about five smaller model turns.

s91
00:06:53.120 --> 00:06:57.480
So routing is a huge part of what makes the unit economics for us actually work.

s92
00:06:57.480 --> 00:07:07.919
Um and we do a bunch of AB experiments and the surprising result we found there is that routing a third a third of our turns to the small model has almost no measurable effect on retention.

s93
00:07:08.320 --> 00:07:11.360
And principle three is what makes a companion feel like a companion.

s94
00:07:11.360 --> 00:07:18.440
So the naive approach is to keep the whole conversation history as a sort of transcript, but that doesn't fit into our two-second loop.

s95
00:07:18.440 --> 00:07:20.640
It doesn't scale and it just doesn't work.

s96
00:07:20.640 --> 00:07:23.000
It leads to long sessions degrading.

s97
00:07:23.000 --> 00:07:27.800
It leads to the model getting lost in the middle of a huge context and also hallucinating.

s98
00:07:27.800 --> 00:07:30.800
So instead we see memory as a sort of retrieval system.

s99
00:07:30.800 --> 00:07:35.200
We pull facts, preferences, and emotional vibe signals out of conversations.

s100
00:07:35.200 --> 00:07:40.200
We embed them and we store them in a vector database with sub 50 millisecond lookups.

s101
00:07:40.200 --> 00:07:41.880
And every night we compress.

s102
00:07:41.880 --> 00:07:48.160
So we merge duplicates, we cluster related memories, we resolve contradictions, and we drop all the noise.

s103
00:07:48.160 --> 00:07:55.920
And we don't just retrieve against users last messages, we also generate internal questions about the person and the relationship and retrieve against those.

s104
00:07:55.920 --> 00:07:58.280
So, and we also split memory into two parts.

s105
00:07:58.280 --> 00:08:00.520
We have stable memory and unstable memory.

s106
00:08:00.520 --> 00:08:11.800
The volatile stuff lives in the in the live tail of the prompt, and when we summarize the conversation, we look at which memories actually get recalled and pin those into a stable and cashable block.

s107
00:08:12.520 --> 00:08:15.440
Uh the last principle is around context.

s108
00:08:15.440 --> 00:08:19.200
Specifically, you should rebuild context and not fight drift.

s109
00:08:19.200 --> 00:08:23.520
So, most apps reuse context across turns to keep the cash warm.

s110
00:08:23.520 --> 00:08:25.680
And in a stable text chat, that's fine.

s111
00:08:25.680 --> 00:08:32.479
But in a volatile voice conversation, it's a trap because the second the the user pivots, your reused context is actively wrong.

s112
00:08:32.479 --> 00:08:35.880
So, every turn we reassemble the context window from parts.

s113
00:08:35.880 --> 00:08:46.400
We have a summary of recent messages, we have the the user's persona card, the memories we just retrieved, tone guidance from the emotional signal, and real-time app state.

s114
00:08:46.880 --> 00:08:53.440
And what also really helps us um in the case of Tolen is that our characters aren't generic or assistants with no personality.

s115
00:08:53.440 --> 00:09:01.600
Everyone is crafted, and we in fact have an in-house science fiction novelist, Elliot, who writes the Tolen character lore.

s116
00:09:01.600 --> 00:09:03.800
And a couple of interesting points here.

s117
00:09:03.800 --> 00:09:06.720
So, one, why did we go with an alien?

s118
00:09:06.720 --> 00:09:13.720
Mostly because there's no real-world reference to anchor on, which means that the users can project onto it, and it becomes what they need.

s119
00:09:13.720 --> 00:09:17.680
The baseline Tolen is bubbly, it's youthful, it's irreverent.

s120
00:09:17.680 --> 00:09:28.960
And also, if an alien character acts a bit unpredictably, so if if it's impulsive or chaotic or otherwise violates um you know, the norms the user would expect, it's not particularly surprising.

s121
00:09:28.960 --> 00:09:35.839
Like if you look at, you know, aliens in TV shows or in plays, like there's a lot of humorous moments around this.

s122
00:09:35.839 --> 00:09:38.960
And this kind of chaos reads as charming.

s123
00:09:38.960 --> 00:09:43.840
Um second, uh we also know that personality is worthless if it drifts.

s124
00:09:43.840 --> 00:09:55.440
So, yeah, we run this parallel tone monitoring system that changes how a line is delivered based on your emotional cues without changing who the character is, holding identity across hundreds of turns.

s125
00:09:56.080 --> 00:10:04.440
And since we're at an AI conference, I thought I would also spend a bit of time talking about how we not just ship AI, but also use AI to build it.

s126
00:10:04.440 --> 00:10:15.240
Um so, I'm sure this is the case for most of you in the room now, but basically as of late last year, Claude has co-authored more code in our iOS app than any individual engineer in the team.

s127
00:10:15.240 --> 00:10:22.280
Um and I think especially, you know, a few months ago, everyone's instinct was to be kind of suspicious because, you know, more AI code meant more slop.

s128
00:10:22.280 --> 00:10:27.480
But our our crash-free rate actually went from 99.6% to 99.9%.

s129
00:10:27.480 --> 00:10:33.840
Runtime errors dropped by over 50% and our share of highly engaged users doubled.

s130
00:10:33.840 --> 00:10:41.440
And the biggest lesson in building that system is that an agent's context comes mostly from the code base itself, not so much from the Claude MD file.

s131
00:10:41.440 --> 00:10:46.760
We found that it's far more powerful to make the code base be the documentation, so we had agents standardize it.

s132
00:10:46.760 --> 00:10:49.720
On top of that, we run a real fleet of agents.

s133
00:10:49.720 --> 00:10:53.839
We have implementation agents that, you know, think freely and just get us to working code.

s134
00:10:53.839 --> 00:10:57.280
They build it, they check it against snapshots until it's pixel perfect.

s135
00:10:57.280 --> 00:11:00.400
And then we have separate review agents that enforce our standards.

s136
00:11:00.400 --> 00:11:04.560
So, multiple Claudes basically review each other before a human looks.

s137
00:11:04.560 --> 00:11:11.920
And then we have a PR shepherd that watches an open pull request and keeps iterating against CI failures and review comments until it's clean.

s138
00:11:11.920 --> 00:11:16.720
And we also have a triage bot that fires on every inbound bug report that we get.

s139
00:11:16.720 --> 00:11:28.839
And they're all wired through MCP into linear, into into Sentry, DataDog, so an agent can reconstruct the cash a crash and route it itself and oftentimes open the PR on its own and just fix fix the bug.

s140
00:11:28.839 --> 00:11:30.240
And we also we ship on eval.

s141
00:11:30.240 --> 00:11:40.000
So, for example, we've been working on a on a new character targeted towards an older demographic and Elliot, our in-house novelist, basically built this entire new character in a day.

s142
00:11:40.000 --> 00:11:43.280
So, the agents mapped every personality bearing surface in the code.

s143
00:11:43.280 --> 00:11:48.360
They wrote the sort of a voice Bible and then they had five judges attack it from different angles.

s144
00:11:48.360 --> 00:12:00.880
Archetype fidelity, the model mechanics, our code standards, the ears of a skeptical 52-year-old and safety, and then they evaluated the changes against real production logs over three find fix verify rounds.

s145
00:12:00.880 --> 00:12:09.120
And over 7 million tokens and 4 and 1/2 hours of compute later, he ended up with basically, you know, a couple of weeks of work done in afternoon.

s146
00:12:09.680 --> 00:12:10.920
And does this work?

s147
00:12:10.920 --> 00:12:12.160
Well, I'll let the users tell you.

s148
00:12:12.160 --> 00:12:23.440
We're at 4.8 stars on the App Store across 162,000 reviews and when we survey users on well-being, the highest scoring dimension by far is emotional safety.

s149
00:12:23.440 --> 00:12:26.440
And this is definitely a bar that being voice first sets.

s150
00:12:26.440 --> 00:12:36.560
So, when the interface is your voice and the thing on the other side remembers you and has a personality, it stops being just software and starts being an actual relationship, which is why building it responsibly

s151
00:12:36.560 --> 00:12:40.480
and building it well is worth obsessing over.

s152
00:12:40.960 --> 00:12:43.160
And we need people to come help us do that.

s153
00:12:43.160 --> 00:12:52.520
Um, we're a small team and we're hiring and after a year with Tolen, I think this is truly one of the most interesting places in the world to be an engineer right now.

s154
00:12:52.680 --> 00:12:54.480
And here are some of the people you'd be doing it with.

s155
00:12:54.480 --> 00:13:00.920
So, two of the founders, Quinton and Evan, previously built and exited a $300 million startup together.

s156
00:13:00.920 --> 00:13:02.320
Uh, they founded Even.

s157
00:13:02.320 --> 00:13:09.400
Um, Ajay, our third co-founder, scaled two bootstrap companies past $50 million $50 million in profitable revenue.

s158
00:13:09.400 --> 00:13:14.360
And around them, we have Lucas, who um, is an Apple Design Award winning animator.

s159
00:13:14.360 --> 00:13:16.040
Uh, she's our creative director.

s160
00:13:16.040 --> 00:13:21.080
We have Chris, who was a technical director at Pixar, earlier at Oculus, who works on embodiment.

s161
00:13:21.080 --> 00:13:28.240
We have Lily, a board certified behavior analyst, who left a Vanderbilt PhD to do user research for us from the very start.

s162
00:13:28.240 --> 00:13:32.600
Um and then we have Elliot who I've mentioned, the novelist behind uh our characters.

s163
00:13:32.600 --> 00:13:37.560
And I come from XAI and Spotify and previously also founded a company called Imagi.

s164
00:13:37.560 --> 00:13:44.400
So, it's a small team where honestly every person is the best I've worked with at what they do.

s165
00:13:44.400 --> 00:13:45.760
We're also well backed for this.

s166
00:13:45.760 --> 00:13:53.800
Uh we have $30 million raised from Costanoa Ventures and a group of people who've built the tools and products a lot of you use every day.

s167
00:13:53.800 --> 00:13:56.400
And here are some of the more engineering focused roles where we need help.

s168
00:13:56.400 --> 00:13:57.840
Um so, we're hiring across the board.

s169
00:13:57.840 --> 00:14:07.960
We have iOS and back-end product engineering roles, applied AI engineering, gameplay engineering, and one specific role I want to flag, which is agent engineering management.

s170
00:14:07.960 --> 00:14:22.960
Um so, when we went all in on running concurrent agents, um the people who got dramatically more effective on the team were the ones who had management backgrounds, um because it seems like managing a fleet of agents does actually take some of the skills same the same skills as managing people.

s171
00:14:22.960 --> 00:14:30.240
Uh you basically have to decompose the problem, you know, delegate it with checkpoints, give fast feedback, review their work seriously, and know when exactly to jump in.

s172
00:14:30.240 --> 00:14:37.120
So, if you're a strong engineer who thought going into management meant leaving code behind, that's that's no longer true.

s173
00:14:37.120 --> 00:14:38.520
Um and yeah, that's Tlon.

s174
00:14:38.520 --> 00:14:41.280
You can come talk to me after this uh or reach out.

s175
00:14:41.280 --> 00:14:42.640
I'm on on LinkedIn.

s176
00:14:42.640 --> 00:14:43.640
My email is here.

s177
00:14:43.640 --> 00:14:45.120
I'm on Twitter as well.

s178
00:14:45.120 --> 00:14:46.240
Um I'd love to chat.

s179
00:14:46.240 --> 00:14:46.480
So, yeah.

s180
00:14:46.480 --> 00:14:48.760
Thank you.

s181
00:14:49.500 --> 00:14:51.500
[applause]

s182
00:15:03.910 --> 00:15:05.910
[music]