Lessons from Studying Every Memory System — Shlok Khemani, Independent

AI Engineer · 19 min · 201 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Okay.
00:13Uh, hi everyone.
00:15I'm Schllo and I've spent the past year studying different memory systems.
00:23Now before I get started, one thing I've realized speaking to people over the last two days is that memory is a very overloaded term now.
00:32It can mean a lot of different things.
00:34So when I talk about memory today, it is going to be in the context of personalization, especially for consumer AI applications.
00:43Now a little bit about me.
00:44My claim to fame, the reason I get to speak to you here is that I've spent the past year trying to reverse engineer how products like ChatGpt, Claude, Gemini, and Poke
00:55implement their memory systems.
00:58And I've then worked with multiple teams across different domains in helping them design their memory.
01:05I'm going to break the talk down into two parts.
01:08Uh first, we're going to look at how memory has evolved over the past 3 years.
01:13especially in the context of chat, GPT, and Claude.
01:17And then in part two, I'm going to discuss some of the lessons I've learned, maybe a rant, and uh where I think all of this is going.
01:28To kick things off, we go back to ancient times, uh which in our industry is 2023.
01:36This is, uh, Chat GPT just after the launch of GPD4.
01:41Now you could have back and forth conversations within a single thread and context was maintained inside that thread but as soon as you started a new conversation nothing was carried over.
01:54Now for early adopters this wasn't a problem.
01:57GPT4 was such an amazing model that if we ever had the need to carry context we would do so by hand.
02:05But as chat GPT started becoming more popular, as regular people started using it for things like learning, cooking, uh as a companion, the need for some sort of memory system became
02:17really apparent.
02:20So in February of 2024, we got Chad GPT memory v1.
02:26And what you could do is you could ask Chad GPT to remember things about you.
02:32So you could say things like, "Hey, remember that I'm vegetarian."
02:35And Chad GPD would extract what it thought was a fact, which is that the user is vegetarian, store it in a list of memories, and this list was then added to the context window for every single conversation.
02:51You could also then go into settings and view this list of memories.
02:55And if you thought that something didn't apply anymore, you could delete a memory.
03:00Now, as the first serious memory implementation within our industry, I think this was a really decent effort, but there were also some fundamental flaws with it.
03:11The biggest one was that as a user, because you could see every time a memory was created, it felt like you were responsible for both creating memories while you were just trying to have a conversation.
03:22So, the burden of memory management fell to the user.
03:26Also, if you notice this list of memories here, these held true at the time they were being created, but that doesn't necessarily hold true over time.
03:36For example, it says that Schllo is going to Bengaluru.
03:40Now, I obviously I'm an SF right now.
03:42I am not going to Bengaluru, but this fact, this memory is still added to my context window today.
03:49So, staleness was another huge problem with this version of chat GPT's memory.
03:56A little more than a year later, April of 2025, Chad GPT released V2 of its memory.
04:02And this was a little more sophisticated.
04:06The most important addition was this thing called user knowledge memories.
04:10Uh I'm just going to call it a running profile for the rest of this talk.
04:14And what a running profile really is is that every few days, Chad GPD looks at all the conversations you've had with it.
04:22It extracts anything it thinks is important for it to know about you and it updates this profile that it maintains on you.
04:30Now this updation process is also what a bunch of folks called dream now called dreaming.
04:36Uh how many of you all were there for Lance Martin's talk yesterday?
04:41Okay, not many but he did a great talk on this.
04:45So every few days, Chat GP looks at the new conversations you're having, uh, updates your profile, and then this updated profile is added to the context window for every single new conversation.
04:58These are two excerpts from my running profile.
05:01I want you to notice a few things.
05:04First, these are extremely dense memories.
05:06So ChatG tries to pack in as much context as it can within every single memory.
05:13What's what's essentially happening here is that they're trying to put in keywords almost like clues and because LLM especially the frontier models today are so good at inferring context from limited information.
05:25When you're having a conversation, it connects these clues to what you're talking about.
05:30Also, these are just two of 16 different sections in my profile.
05:33Other sections include my personal life, things I'm working on.
05:37Uh in total, my profile is almost 4,000 tokens long.
05:41And because these updates are happening happening asynchronously, they're happening in the background.
05:46This new version does away with the flaw we discussed in V1, which is the burden of user management was taken up.
05:52The burden of memory management was taken away from the user.
05:56But I want you to notice the highlighted memory.
05:59This is about places I traveled to in 2025.
06:02But if you pay attention, it says Thailand and Turkey, but the dates are overlapping.
06:08And that's because the source of this memory was conversations I was having with Chad GPT deciding between where to go among these two places.
06:16Now I did end up going to Thailand.
06:18I've never been to Turkey, but Chad GPT still says that I've been to Turkey in 2025.
06:24So the stainless problem with V2 didn't completely go away.
06:29Another very important thing is that if you go to your settings, chat GPT doesn't let you view this raw profile.
06:36So you could view your memories from v1 your raw profile is not visible to you.
06:41Now you may ask sllo how did you see your profile then that's because this prompt works really well if you want to jailbreak chat gpt uh and view your raw profile
06:50you might have to attempt a few times try different thinking modes but pro enough and you shall receive.
06:59August of 2025 uh claude released its first version of memory.
07:04This surprised me a bit because if you compare chat GPT and claude, they are very similar applications, right?
07:10So you have a chat box, you have back and forth conversations, you have a list of previous conversations, you can start a new conversation.
07:18My assumption going into study claude was that the memory systems would also be similarly designed.
07:24Not the case at least for V1.
07:27So in V1 of Claude, you had no user profile.
07:30You had no list of facts.
07:32Instead, the model was given two tools.
07:35It was given a tool to search over previous conversations by keyword or topic and it was given another tool to search over conversations by time period.
07:42So queries like what did we discuss last week or what did we discuss at the start of November of 2025.
07:49So in van every single context every single conversation starts fresh with no context on the user and when the model thinks that it needs to retrieve something it can do so on demand.
08:03Uh on September 11 of last year, I released a blog post saying Claude's memory architecture is the opposite of chat GPTs.
08:11This hit the hacker news front page.
08:13Finally, on that very day, Claude released V2 of its memory and uh they added a running profile similar to chat GPT, but with a few differences.
08:27First, Claude made this profile visible to users.
08:30So you could go to settings and you could view your raw profile.
08:34Second, this profile was a,000 tokens.
08:36So it was much smaller than chat GPT's 4,000 tokens.
08:40And also if you notice, these are complete sentences rather than a dense keyword approach of chat GPT.
08:45So less dense and smaller.
08:49Claude's profile updates every 24 hours.
08:51For chat GPT, it's every few days.
08:53And Claude also let you let the user make explicit edits to this profile.
08:58So you could request for an edit and that edit would lead to a reynthesis of the profile.
09:04It gave you an interface to manage previous edits and you could delete the things that no longer held true.
09:09And this is how Claude's memory works even today.
09:12So it hasn't changed since September of last year.
09:16We have seen two updates within chat GPT's memory this year though.
09:21The first was it added a tool to look over past conversations like we just saw with Claude.
09:26uh so the model can retrieve summarized context based on queries it makes and then a month ago uh start of June chat GPT finally made user profile visible to them
09:42somewhat so what you can see is a LLM generated summary of your profile which is weird because your profile is already an LLM generated summary of your conversations uh it's all a bit confusing I've written about it
09:55but it is visible in some sense.
09:58You could also request explicit edits to your profile.
10:02And with this update, Chad GPT deprecated B1 the fact list from its memory system.
10:09So what we've seen here is a convergence after 3 years of each of these products evolving independently where they both now have a running profile.
10:17This profile is visible and editable again somewhat uh and the model has tools to look over past conversations.
10:26Okay.
10:26So what can we learn from this evolution and where are things going?
10:32I think the biggest lesson for me is that there is no single way to do memory.
10:39It wasn't too long ago that everyone including me assumed that rag was the way to go go about memory where you would take conversations you would chunk them uh
10:50create embeddings put them in a vector store and then as user queries came in do some sort of semantic search but as we saw neither chat gi nor claude really do this instead they both evolved
11:02independently using different approaches and while the general architectures have converged the spec specific implementation details are still very different.
11:12And then if you look at Gemini, it also has a running profile, but each memory comes with detailed timing logs.
11:18So when was it created, when was it last updated?
11:21And then if you look at agents like cloud code, openclaw, Hermes, they have completely different memory systems, right?
11:26With markdown files, heartbeat, knowledge bases, skills.
11:31The point being that there is no one way to do memory.
11:34The implications of this is that memory cannot be outsourced.
11:37If you're a serious team, you do not outsource memory.
11:40It is something that you build alongside your product.
11:43Your memory system evolves with your product and it cannot be thought of uh as an afterthought.
11:50And there is plenty of evidence for this.
11:52So if you look at all of the top consumer products today across different categories, each of these has some form of memory.
12:01Yet none of them outsource it.
12:03All of them build memory inhouse.
12:08Lesson two, memory is a function of compute.
12:11What does that mean?
12:13Let's look at the costs associated with a running profile.
12:16So there are two types of costs.
12:18There is a cost to maintain a profile.
12:20Now that depends on how frequently you update it and how much compute you apply to each update.
12:26And then because these profiles are part of the context window for every single conversation, there's a cost of serving which is the longer the profile the more it costs to serve.
12:37Now thought experiment.
12:38If you were to design the ideal memory system with no restraints, what would you do?
12:44You might want to update um your profile every hour or maybe after every conversation.
12:53uh you might want to task Fable with a bunch of Opus sub agents for the update itself.
12:59Uh and why stop at 4,000 tokens?
13:01Why not make it 400,000 tokens?
13:04Store every single thing you would want about the user.
13:08Unfortunately, we live we live in a GPU constrainted world and trade-offs have to be made and you can see that happening here.
13:15So, chat GPT the profile length is 4,000 tokens.
13:18It updates every few days.
13:20So they have a higher serving cost for a lower update cost and for claude it's a thousand tokens updates every 24 hours.
13:27So they make the exact opposite tradeoff and this is what I mean by memory is a function of compute.
13:32You have to really think about how much compute you want to put into memory.
13:37Third we had a bunch of talks about continual learning today.
13:40I'm not an expert here but what I would say is that continual learning is already here.
13:47Going back to running profiles, what exactly is happening here?
13:50Your running profile starts with something that the model knows about you.
13:55This is then applied to every single conversation.
14:00Each of these conversations bring in new information.
14:04This new information is then synthesized through the streaming process back into the profile and then this profile dictates for the conversations.
14:12And this loop keeps repeating itself again and again and again.
14:15And what you have is a continual learning process.
14:18Now obviously this learning loop is happening outside the weights and a big question particularly for consumer AI is will this process ever make its way into the weights.
14:32Now at an obviously updating weights um training models is an expensive process.
14:39uh continuous learning does make sense at an enterprise level because the costs of these models are amotized across different employees different customers but that's not the case and at an individual level
14:52right so big big open questions that I don't know don't yet know the answers to which is will each of us get our own self learning model what data do we need to kick the CL process off
15:05and how do we generate it and finally who's going to pay for this How about the economics for this work?
15:12Uh Guan recently wrote an essay called Guardian Angels where he explores this topic in beautiful detail.
15:22And if you're interested in what the future for one model a person looks like, I would recommend reading this.
15:32Finally, uh my rant is that we have a massive context problem.
15:39You could have the best memory architecture in the world.
15:42You could pour infinite amounts of compute into it.
15:46You could have continuous learning working at an individual level where the every single data point you bring up is somehow perfectly integrated into the model weights.
15:57Yet your memory system is capped by how much context it can gather about you.
16:03Let's go back to the example we discussed earlier.
16:07which was the conflict between where I traveled to uh in the summer of 2025.
16:13These are the two source conversations.
16:15Again, I was trying to use chat GPT to decide between which of these two countries to go to.
16:21Now, the decision to go to Thailand was actually made in a conversation I had with my partner in person and chat GPT could couldn't reason over this or couldn't listen to this.
16:30But there were also traces of this conversation in my emails because I I had flight and hotel bookings for Thailand.
16:38But because even if chat GPD is connected to my email, it doesn't reason over my email and it doesn't update my profile over my email, it couldn't resolve this conflict.
16:50And I think that's okay.
16:52Uh it's understandable.
16:54But what really bothers me is that chat GPD today doesn't realize that there is a conflict.
17:04It's not curious about trying to fill in gaps in the information it knows about me.
17:10And this is particularly interesting and also infuriating because the tech it's not a technology problem.
17:18It's a product problem.
17:19There is no fundamental reason from an LLM level that these things can't be solved.
17:26Uh it's just that a products today are not designed to help us with this.
17:34So my personal stack today is a bunch of chat bots, assistants, vertical specific applications, agents, and even hardware devices.
17:45Each of these products is trying to build its own memory of me.
17:49None of these memories are shared with each other.
17:52So I have to rebuild context within every single product from scratch.
17:57Every time when something in my life changes, I have to individually update all of them.
18:04And then I have a bunch of very rich existing context sources, right?
18:09my email, calendar, uh, photos.
18:12None of these products are able to reason over my existing very rich context sources.
18:21So for me, none of this feels like 2026.
18:24And what I keep asking myself every day is when will personal AI feel like personal AI?
18:33All of that frustration aside, I still think we're very early.
18:38Memory for AI is just a three-year-old field.
18:42Memory is also foundational to how humans interact with AI.
18:46And because I hope to be talking to AIS all my life, and I know that's going to be the case for every single one of us here today, memory is something that's going to be important for the rest of human history.
18:58And uh there's so much left to build.
19:02That's it from me.
19:03Uh thank you so much.
19:05You can find my website.
19:07You can find me on Twitter.
19:09Have a great rest of the conference.
19:10[applause]
19:28[music]