WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=5ZGyKWjQDr0

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/5ZGyKWjQDr0.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.960 --> 00:00:13.519
Okay.

s2
00:00:13.519 --> 00:00:15.599
Uh, hi everyone.

s3
00:00:15.599 --> 00:00:23.359
I'm Schllo and I've spent the past year studying different memory systems.

s4
00:00:23.680 --> 00:00:32.160
Now before I get started, one thing I've realized speaking to people over the last two days is that memory is a very overloaded term now.

s5
00:00:32.160 --> 00:00:34.160
It can mean a lot of different things.

s6
00:00:34.160 --> 00:00:42.399
So when I talk about memory today, it is going to be in the context of personalization, especially for consumer AI applications.

s7
00:00:43.040 --> 00:00:44.640
Now a little bit about me.

s8
00:00:44.640 --> 00:00:55.440
My claim to fame, the reason I get to speak to you here is that I've spent the past year trying to reverse engineer how products like ChatGpt, Claude, Gemini, and Poke

s9
00:00:55.440 --> 00:00:58.320
implement their memory systems.

s10
00:00:58.320 --> 00:01:05.440
And I've then worked with multiple teams across different domains in helping them design their memory.

s11
00:01:05.680 --> 00:01:08.400
I'm going to break the talk down into two parts.

s12
00:01:08.400 --> 00:01:13.840
Uh first, we're going to look at how memory has evolved over the past 3 years.

s13
00:01:13.840 --> 00:01:17.840
especially in the context of chat, GPT, and Claude.

s14
00:01:17.840 --> 00:01:27.759
And then in part two, I'm going to discuss some of the lessons I've learned, maybe a rant, and uh where I think all of this is going.

s15
00:01:28.240 --> 00:01:36.320
To kick things off, we go back to ancient times, uh which in our industry is 2023.

s16
00:01:36.880 --> 00:01:41.759
This is, uh, Chat GPT just after the launch of GPD4.

s17
00:01:41.840 --> 00:01:54.799
Now you could have back and forth conversations within a single thread and context was maintained inside that thread but as soon as you started a new conversation nothing was carried over.

s18
00:01:54.799 --> 00:01:57.280
Now for early adopters this wasn't a problem.

s19
00:01:57.280 --> 00:02:05.520
GPT4 was such an amazing model that if we ever had the need to carry context we would do so by hand.

s20
00:02:05.520 --> 00:02:17.440
But as chat GPT started becoming more popular, as regular people started using it for things like learning, cooking, uh as a companion, the need for some sort of memory system became

s21
00:02:17.440 --> 00:02:19.840
really apparent.

s22
00:02:20.480 --> 00:02:26.560
So in February of 2024, we got Chad GPT memory v1.

s23
00:02:26.640 --> 00:02:32.160
And what you could do is you could ask Chad GPT to remember things about you.

s24
00:02:32.160 --> 00:02:35.920
So you could say things like, "Hey, remember that I'm vegetarian."

s25
00:02:35.920 --> 00:02:50.480
And Chad GPD would extract what it thought was a fact, which is that the user is vegetarian, store it in a list of memories, and this list was then added to the context window for every single conversation.

s26
00:02:51.440 --> 00:02:55.360
You could also then go into settings and view this list of memories.

s27
00:02:55.360 --> 00:03:00.400
And if you thought that something didn't apply anymore, you could delete a memory.

s28
00:03:00.400 --> 00:03:11.120
Now, as the first serious memory implementation within our industry, I think this was a really decent effort, but there were also some fundamental flaws with it.

s29
00:03:11.120 --> 00:03:22.239
The biggest one was that as a user, because you could see every time a memory was created, it felt like you were responsible for both creating memories while you were just trying to have a conversation.

s30
00:03:22.239 --> 00:03:26.480
So, the burden of memory management fell to the user.

s31
00:03:26.480 --> 00:03:36.799
Also, if you notice this list of memories here, these held true at the time they were being created, but that doesn't necessarily hold true over time.

s32
00:03:36.799 --> 00:03:40.159
For example, it says that Schllo is going to Bengaluru.

s33
00:03:40.159 --> 00:03:42.319
Now, I obviously I'm an SF right now.

s34
00:03:42.319 --> 00:03:49.120
I am not going to Bengaluru, but this fact, this memory is still added to my context window today.

s35
00:03:49.120 --> 00:03:55.280
So, staleness was another huge problem with this version of chat GPT's memory.

s36
00:03:56.560 --> 00:04:02.480
A little more than a year later, April of 2025, Chad GPT released V2 of its memory.

s37
00:04:02.480 --> 00:04:05.439
And this was a little more sophisticated.

s38
00:04:06.000 --> 00:04:10.879
The most important addition was this thing called user knowledge memories.

s39
00:04:10.879 --> 00:04:14.720
Uh I'm just going to call it a running profile for the rest of this talk.

s40
00:04:14.720 --> 00:04:22.240
And what a running profile really is is that every few days, Chad GPD looks at all the conversations you've had with it.

s41
00:04:22.240 --> 00:04:30.000
It extracts anything it thinks is important for it to know about you and it updates this profile that it maintains on you.

s42
00:04:30.000 --> 00:04:36.479
Now this updation process is also what a bunch of folks called dream now called dreaming.

s43
00:04:36.479 --> 00:04:40.880
Uh how many of you all were there for Lance Martin's talk yesterday?

s44
00:04:41.040 --> 00:04:45.120
Okay, not many but he did a great talk on this.

s45
00:04:45.120 --> 00:04:57.840
So every few days, Chat GP looks at the new conversations you're having, uh, updates your profile, and then this updated profile is added to the context window for every single new conversation.

s46
00:04:58.080 --> 00:05:01.600
These are two excerpts from my running profile.

s47
00:05:01.600 --> 00:05:04.080
I want you to notice a few things.

s48
00:05:04.080 --> 00:05:06.800
First, these are extremely dense memories.

s49
00:05:06.800 --> 00:05:13.919
So ChatG tries to pack in as much context as it can within every single memory.

s50
00:05:13.919 --> 00:05:25.759
What's what's essentially happening here is that they're trying to put in keywords almost like clues and because LLM especially the frontier models today are so good at inferring context from limited information.

s51
00:05:25.759 --> 00:05:30.160
When you're having a conversation, it connects these clues to what you're talking about.

s52
00:05:30.160 --> 00:05:33.759
Also, these are just two of 16 different sections in my profile.

s53
00:05:33.759 --> 00:05:37.600
Other sections include my personal life, things I'm working on.

s54
00:05:37.600 --> 00:05:41.520
Uh in total, my profile is almost 4,000 tokens long.

s55
00:05:41.520 --> 00:05:46.080
And because these updates are happening happening asynchronously, they're happening in the background.

s56
00:05:46.080 --> 00:05:52.240
This new version does away with the flaw we discussed in V1, which is the burden of user management was taken up.

s57
00:05:52.240 --> 00:05:56.160
The burden of memory management was taken away from the user.

s58
00:05:56.160 --> 00:05:59.520
But I want you to notice the highlighted memory.

s59
00:05:59.520 --> 00:06:02.960
This is about places I traveled to in 2025.

s60
00:06:02.960 --> 00:06:08.319
But if you pay attention, it says Thailand and Turkey, but the dates are overlapping.

s61
00:06:08.319 --> 00:06:16.960
And that's because the source of this memory was conversations I was having with Chad GPT deciding between where to go among these two places.

s62
00:06:16.960 --> 00:06:18.800
Now I did end up going to Thailand.

s63
00:06:18.800 --> 00:06:24.240
I've never been to Turkey, but Chad GPT still says that I've been to Turkey in 2025.

s64
00:06:24.240 --> 00:06:28.880
So the stainless problem with V2 didn't completely go away.

s65
00:06:29.360 --> 00:06:36.319
Another very important thing is that if you go to your settings, chat GPT doesn't let you view this raw profile.

s66
00:06:36.319 --> 00:06:41.759
So you could view your memories from v1 your raw profile is not visible to you.

s67
00:06:41.759 --> 00:06:50.240
Now you may ask sllo how did you see your profile then that's because this prompt works really well if you want to jailbreak chat gpt uh and view your raw profile

s68
00:06:50.240 --> 00:06:57.600
you might have to attempt a few times try different thinking modes but pro enough and you shall receive.

s69
00:06:59.199 --> 00:07:04.639
August of 2025 uh claude released its first version of memory.

s70
00:07:04.639 --> 00:07:10.800
This surprised me a bit because if you compare chat GPT and claude, they are very similar applications, right?

s71
00:07:10.800 --> 00:07:18.000
So you have a chat box, you have back and forth conversations, you have a list of previous conversations, you can start a new conversation.

s72
00:07:18.000 --> 00:07:24.880
My assumption going into study claude was that the memory systems would also be similarly designed.

s73
00:07:24.880 --> 00:07:27.360
Not the case at least for V1.

s74
00:07:27.360 --> 00:07:30.400
So in V1 of Claude, you had no user profile.

s75
00:07:30.400 --> 00:07:32.479
You had no list of facts.

s76
00:07:32.479 --> 00:07:35.360
Instead, the model was given two tools.

s77
00:07:35.360 --> 00:07:42.720
It was given a tool to search over previous conversations by keyword or topic and it was given another tool to search over conversations by time period.

s78
00:07:42.720 --> 00:07:49.520
So queries like what did we discuss last week or what did we discuss at the start of November of 2025.

s79
00:07:49.680 --> 00:08:03.120
So in van every single context every single conversation starts fresh with no context on the user and when the model thinks that it needs to retrieve something it can do so on demand.

s80
00:08:03.919 --> 00:08:11.280
Uh on September 11 of last year, I released a blog post saying Claude's memory architecture is the opposite of chat GPTs.

s81
00:08:11.280 --> 00:08:13.759
This hit the hacker news front page.

s82
00:08:13.759 --> 00:08:27.840
Finally, on that very day, Claude released V2 of its memory and uh they added a running profile similar to chat GPT, but with a few differences.

s83
00:08:27.840 --> 00:08:30.639
First, Claude made this profile visible to users.

s84
00:08:30.639 --> 00:08:34.320
So you could go to settings and you could view your raw profile.

s85
00:08:34.320 --> 00:08:36.959
Second, this profile was a,000 tokens.

s86
00:08:36.959 --> 00:08:40.159
So it was much smaller than chat GPT's 4,000 tokens.

s87
00:08:40.159 --> 00:08:45.360
And also if you notice, these are complete sentences rather than a dense keyword approach of chat GPT.

s88
00:08:45.360 --> 00:08:49.040
So less dense and smaller.

s89
00:08:49.360 --> 00:08:51.600
Claude's profile updates every 24 hours.

s90
00:08:51.600 --> 00:08:53.920
For chat GPT, it's every few days.

s91
00:08:53.920 --> 00:08:58.399
And Claude also let you let the user make explicit edits to this profile.

s92
00:08:58.399 --> 00:09:04.399
So you could request for an edit and that edit would lead to a reynthesis of the profile.

s93
00:09:04.399 --> 00:09:09.760
It gave you an interface to manage previous edits and you could delete the things that no longer held true.

s94
00:09:09.760 --> 00:09:12.720
And this is how Claude's memory works even today.

s95
00:09:12.720 --> 00:09:16.640
So it hasn't changed since September of last year.

s96
00:09:16.959 --> 00:09:21.680
We have seen two updates within chat GPT's memory this year though.

s97
00:09:21.680 --> 00:09:26.800
The first was it added a tool to look over past conversations like we just saw with Claude.

s98
00:09:26.800 --> 00:09:42.080
uh so the model can retrieve summarized context based on queries it makes and then a month ago uh start of June chat GPT finally made user profile visible to them

s99
00:09:42.080 --> 00:09:55.680
somewhat so what you can see is a LLM generated summary of your profile which is weird because your profile is already an LLM generated summary of your conversations uh it's all a bit confusing I've written about it

s100
00:09:55.680 --> 00:09:58.720
but it is visible in some sense.

s101
00:09:58.720 --> 00:10:02.720
You could also request explicit edits to your profile.

s102
00:10:02.720 --> 00:10:09.279
And with this update, Chad GPT deprecated B1 the fact list from its memory system.

s103
00:10:09.279 --> 00:10:17.440
So what we've seen here is a convergence after 3 years of each of these products evolving independently where they both now have a running profile.

s104
00:10:17.440 --> 00:10:25.760
This profile is visible and editable again somewhat uh and the model has tools to look over past conversations.

s105
00:10:26.240 --> 00:10:26.720
Okay.

s106
00:10:26.720 --> 00:10:32.160
So what can we learn from this evolution and where are things going?

s107
00:10:32.640 --> 00:10:38.640
I think the biggest lesson for me is that there is no single way to do memory.

s108
00:10:39.120 --> 00:10:50.399
It wasn't too long ago that everyone including me assumed that rag was the way to go go about memory where you would take conversations you would chunk them uh

s109
00:10:50.399 --> 00:11:02.000
create embeddings put them in a vector store and then as user queries came in do some sort of semantic search but as we saw neither chat gi nor claude really do this instead they both evolved

s110
00:11:02.000 --> 00:11:12.079
independently using different approaches and while the general architectures have converged the spec specific implementation details are still very different.

s111
00:11:12.079 --> 00:11:18.880
And then if you look at Gemini, it also has a running profile, but each memory comes with detailed timing logs.

s112
00:11:18.880 --> 00:11:21.200
So when was it created, when was it last updated?

s113
00:11:21.200 --> 00:11:26.880
And then if you look at agents like cloud code, openclaw, Hermes, they have completely different memory systems, right?

s114
00:11:26.880 --> 00:11:31.200
With markdown files, heartbeat, knowledge bases, skills.

s115
00:11:31.360 --> 00:11:34.399
The point being that there is no one way to do memory.

s116
00:11:34.399 --> 00:11:37.279
The implications of this is that memory cannot be outsourced.

s117
00:11:37.279 --> 00:11:40.560
If you're a serious team, you do not outsource memory.

s118
00:11:40.560 --> 00:11:43.200
It is something that you build alongside your product.

s119
00:11:43.200 --> 00:11:50.320
Your memory system evolves with your product and it cannot be thought of uh as an afterthought.

s120
00:11:50.399 --> 00:11:52.000
And there is plenty of evidence for this.

s121
00:11:52.000 --> 00:12:01.680
So if you look at all of the top consumer products today across different categories, each of these has some form of memory.

s122
00:12:01.680 --> 00:12:03.760
Yet none of them outsource it.

s123
00:12:03.760 --> 00:12:06.800
All of them build memory inhouse.

s124
00:12:08.639 --> 00:12:11.360
Lesson two, memory is a function of compute.

s125
00:12:11.360 --> 00:12:13.360
What does that mean?

s126
00:12:13.360 --> 00:12:16.880
Let's look at the costs associated with a running profile.

s127
00:12:16.880 --> 00:12:18.720
So there are two types of costs.

s128
00:12:18.720 --> 00:12:20.880
There is a cost to maintain a profile.

s129
00:12:20.880 --> 00:12:26.880
Now that depends on how frequently you update it and how much compute you apply to each update.

s130
00:12:26.880 --> 00:12:37.519
And then because these profiles are part of the context window for every single conversation, there's a cost of serving which is the longer the profile the more it costs to serve.

s131
00:12:37.519 --> 00:12:38.800
Now thought experiment.

s132
00:12:38.800 --> 00:12:44.800
If you were to design the ideal memory system with no restraints, what would you do?

s133
00:12:44.800 --> 00:12:52.959
You might want to update um your profile every hour or maybe after every conversation.

s134
00:12:53.040 --> 00:12:59.040
uh you might want to task Fable with a bunch of Opus sub agents for the update itself.

s135
00:12:59.040 --> 00:13:01.920
Uh and why stop at 4,000 tokens?

s136
00:13:01.920 --> 00:13:04.560
Why not make it 400,000 tokens?

s137
00:13:04.560 --> 00:13:08.399
Store every single thing you would want about the user.

s138
00:13:08.399 --> 00:13:15.440
Unfortunately, we live we live in a GPU constrainted world and trade-offs have to be made and you can see that happening here.

s139
00:13:15.440 --> 00:13:18.240
So, chat GPT the profile length is 4,000 tokens.

s140
00:13:18.240 --> 00:13:20.240
It updates every few days.

s141
00:13:20.240 --> 00:13:27.600
So they have a higher serving cost for a lower update cost and for claude it's a thousand tokens updates every 24 hours.

s142
00:13:27.600 --> 00:13:32.320
So they make the exact opposite tradeoff and this is what I mean by memory is a function of compute.

s143
00:13:32.320 --> 00:13:36.639
You have to really think about how much compute you want to put into memory.

s144
00:13:37.120 --> 00:13:40.560
Third we had a bunch of talks about continual learning today.

s145
00:13:40.560 --> 00:13:47.440
I'm not an expert here but what I would say is that continual learning is already here.

s146
00:13:47.440 --> 00:13:50.800
Going back to running profiles, what exactly is happening here?

s147
00:13:50.800 --> 00:13:55.839
Your running profile starts with something that the model knows about you.

s148
00:13:55.839 --> 00:13:59.680
This is then applied to every single conversation.

s149
00:14:00.800 --> 00:14:04.639
Each of these conversations bring in new information.

s150
00:14:04.639 --> 00:14:12.399
This new information is then synthesized through the streaming process back into the profile and then this profile dictates for the conversations.

s151
00:14:12.399 --> 00:14:15.519
And this loop keeps repeating itself again and again and again.

s152
00:14:15.519 --> 00:14:18.480
And what you have is a continual learning process.

s153
00:14:18.480 --> 00:14:32.160
Now obviously this learning loop is happening outside the weights and a big question particularly for consumer AI is will this process ever make its way into the weights.

s154
00:14:32.160 --> 00:14:39.360
Now at an obviously updating weights um training models is an expensive process.

s155
00:14:39.360 --> 00:14:52.560
uh continuous learning does make sense at an enterprise level because the costs of these models are amotized across different employees different customers but that's not the case and at an individual level

s156
00:14:52.560 --> 00:15:05.360
right so big big open questions that I don't know don't yet know the answers to which is will each of us get our own self learning model what data do we need to kick the CL process off

s157
00:15:05.360 --> 00:15:11.920
and how do we generate it and finally who's going to pay for this How about the economics for this work?

s158
00:15:12.320 --> 00:15:22.079
Uh Guan recently wrote an essay called Guardian Angels where he explores this topic in beautiful detail.

s159
00:15:22.079 --> 00:15:30.639
And if you're interested in what the future for one model a person looks like, I would recommend reading this.

s160
00:15:32.560 --> 00:15:38.880
Finally, uh my rant is that we have a massive context problem.

s161
00:15:39.440 --> 00:15:42.959
You could have the best memory architecture in the world.

s162
00:15:42.959 --> 00:15:46.560
You could pour infinite amounts of compute into it.

s163
00:15:46.560 --> 00:15:57.279
You could have continuous learning working at an individual level where the every single data point you bring up is somehow perfectly integrated into the model weights.

s164
00:15:57.279 --> 00:16:03.600
Yet your memory system is capped by how much context it can gather about you.

s165
00:16:03.600 --> 00:16:07.440
Let's go back to the example we discussed earlier.

s166
00:16:07.440 --> 00:16:13.120
which was the conflict between where I traveled to uh in the summer of 2025.

s167
00:16:13.120 --> 00:16:15.040
These are the two source conversations.

s168
00:16:15.040 --> 00:16:21.040
Again, I was trying to use chat GPT to decide between which of these two countries to go to.

s169
00:16:21.040 --> 00:16:30.959
Now, the decision to go to Thailand was actually made in a conversation I had with my partner in person and chat GPT could couldn't reason over this or couldn't listen to this.

s170
00:16:30.959 --> 00:16:38.160
But there were also traces of this conversation in my emails because I I had flight and hotel bookings for Thailand.

s171
00:16:38.160 --> 00:16:50.079
But because even if chat GPD is connected to my email, it doesn't reason over my email and it doesn't update my profile over my email, it couldn't resolve this conflict.

s172
00:16:50.959 --> 00:16:52.480
And I think that's okay.

s173
00:16:52.480 --> 00:16:54.720
Uh it's understandable.

s174
00:16:54.720 --> 00:17:04.720
But what really bothers me is that chat GPD today doesn't realize that there is a conflict.

s175
00:17:04.720 --> 00:17:10.799
It's not curious about trying to fill in gaps in the information it knows about me.

s176
00:17:10.799 --> 00:17:18.640
And this is particularly interesting and also infuriating because the tech it's not a technology problem.

s177
00:17:18.640 --> 00:17:19.760
It's a product problem.

s178
00:17:19.760 --> 00:17:26.959
There is no fundamental reason from an LLM level that these things can't be solved.

s179
00:17:26.959 --> 00:17:34.160
Uh it's just that a products today are not designed to help us with this.

s180
00:17:34.799 --> 00:17:45.200
So my personal stack today is a bunch of chat bots, assistants, vertical specific applications, agents, and even hardware devices.

s181
00:17:45.280 --> 00:17:49.919
Each of these products is trying to build its own memory of me.

s182
00:17:49.919 --> 00:17:52.799
None of these memories are shared with each other.

s183
00:17:52.799 --> 00:17:57.440
So I have to rebuild context within every single product from scratch.

s184
00:17:57.440 --> 00:18:04.160
Every time when something in my life changes, I have to individually update all of them.

s185
00:18:04.160 --> 00:18:09.120
And then I have a bunch of very rich existing context sources, right?

s186
00:18:09.120 --> 00:18:12.640
my email, calendar, uh, photos.

s187
00:18:12.640 --> 00:18:20.240
None of these products are able to reason over my existing very rich context sources.

s188
00:18:21.039 --> 00:18:24.240
So for me, none of this feels like 2026.

s189
00:18:24.240 --> 00:18:31.440
And what I keep asking myself every day is when will personal AI feel like personal AI?

s190
00:18:33.600 --> 00:18:38.559
All of that frustration aside, I still think we're very early.

s191
00:18:38.559 --> 00:18:42.720
Memory for AI is just a three-year-old field.

s192
00:18:42.720 --> 00:18:46.799
Memory is also foundational to how humans interact with AI.

s193
00:18:46.799 --> 00:18:58.160
And because I hope to be talking to AIS all my life, and I know that's going to be the case for every single one of us here today, memory is something that's going to be important for the rest of human history.

s194
00:18:58.160 --> 00:19:02.480
And uh there's so much left to build.

s195
00:19:02.480 --> 00:19:03.760
That's it from me.

s196
00:19:03.760 --> 00:19:05.600
Uh thank you so much.

s197
00:19:05.600 --> 00:19:07.120
You can find my website.

s198
00:19:07.120 --> 00:19:09.039
You can find me on Twitter.

s199
00:19:09.039 --> 00:19:10.848
Have a great rest of the conference.

s200
00:19:10.848 --> 00:19:12.848
[applause]

s201
00:19:28.533 --> 00:19:30.533
[music]