Agent Memory Is Solved. Agent Learning Isn't. — Karthik Ranganathan, Yugabyte

AI Engineer · 20 min · 298 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13Good to meet you folks.
00:14Thanks for everyone who's here.
00:16Um, today we're going to talk about sharing context, right?
00:20Everything in AI is about the data you give it.
00:22Bad data, bad outputs, right?
00:25Agent memory is a solved problem, but agent learning is not.
00:29And we're going to talk about not just like you know what we're building etc. It's about how why we built it and how we're using it for ourselves.
00:37Okay.
00:38So as we said agent memory works you know there's a bunch of stuff you feed it conversations etc. And you're able to extract memories from the conversation and agents are able to reuse them.
00:50It's per agent.
00:51However learning is never done in isolation.
00:55It's when a group of agents come together and need to work with each other, they have to now start learning and sharing, right?
01:03And that is an unsolved problem because we're infrastructure people and infrastructure built for a single agent to do memory is often not the same infrastructure that is needed in order to learn, right?
01:16We're going to talk a little bit more about that.
01:19Okay, so by way of background, I'm Karthik, one of the co-founders and a CEO of a database company, Yugabyte DB, right?
01:27So what the hell are we doing here?
01:28Why are we building a aentic data infrastructure?
01:32What are we doing doing that?
01:33Right?
01:34We actually found this the hard way.
01:36We run agents for ourselves.
01:37We run two agents internal to the company.
01:40Hagen for customer support and growth vector for a coordinated go to market between sales and marketing.
01:46And we run two agents externally for our customers and users.
01:50The first one is Perf Advisor, which is an agentic database admin, right?
01:55So it runs databases for our customers as a database company.
01:58Pretty useful.
01:59And Voyager, which helps them migrate and modernize from other databases to a cloudnative one, right?
02:05So four agents.
02:06We failed to ship anything and make it work at all.
02:10It was really hard.
02:11We've been trying for almost a year before we realized we were failing at the data problem right and if a database company is failing at a data problem be it for agentic it becomes personal right so yes it became personal we're like what the hell's going on and when we then realized that a lot of people were
02:28asking us like yes we can do scalable vectors we can do scalable graph we can do scalable relational or people came to us and said more scale more performance less cost we're like wait a minute at some point this is ridiculous right it's breaking the laws physics.
02:41So what is the real failure?
02:44Right?
02:44When you cut everything o apart and just go to the core, it's because agent A, my agent is doing some work and I need to work with say one of you guys, say maybe Heather, right?
02:55My who's going to be showing a demo and I am not able to give that context.
03:00I'm only able to give the output.
03:03The output being an MD file, a PDF, a document, a something.
03:08But I'm not able to give the reasoning behind it.
03:11Right?
03:11So what does this lose?
03:12This loses reasoning.
03:14This loses the deadends, the number of failures you've had before you actually succeeded.
03:19And it loses the whole context.
03:21Right?
03:22That means Heather's agent is going to be rederiving all of this stuff.
03:26And it's not for free.
03:27You're burning tokens and paying somebody else for it.
03:30This is the same as a group of humans working together.
03:33When a new person joins the team, they're going to be told, "We're going to do X." And they come up with 50 reasons why we should do something else.
03:42And each time, yes, we considered that two weeks ago, two months ago.
03:46That's not how humans operate.
03:47We actually bring the person in and says, "Okay, look, this is the context.
03:51This is the history of the project.
03:52This is how it's done.
03:53Now, let's go from here."
03:55That context is missing for agents.
03:58So, five things, misconceptions.
04:00Okay, really quick.
04:02More memory.
04:03Agentic memory is not learning.
04:05It's just more memory, right?
04:06Like it's more expensive.
04:08Shared state is not shared knowledge.
04:11Just because you can share something with somebody else, like an MD file or something, doesn't mean you have shared knowledge.
04:17Rag transcripts, let me rag everything, is not equal to learning and quality.
04:22It just means there's a whole bunch of garbage there and now LLMs get to burn even more tokens to give you even worse results.
04:29Fine-tuning is not scalable.
04:31You cannot, you know, save and persist lessons that agents have come up with through fine-tuning and building custom infrastructure.
04:39And a bigger context window is the shest way to quickest failure in an expensive manner, right?
04:44Because it doesn't do anything.
04:45Your context rots.
04:47Every single one of these is a architecture gap, right?
04:51So this is what we realized we needed to plug for ourselves, right?
04:56Right.
04:56And we've been finding a lot of users are are really enjoying and really liking this approach of thinking about context.
05:03Firstly, if there's a context that agent A can share with agent B, it closes the lost context problem.
05:09They're working on the same context.
05:12Second thing is this context can degrade.
05:15That means not everybody can contribute to that context.
05:18It's just like code.
05:19Not anyone can contribute.
05:21You need to go through a process of supervised promotion.
05:24And only then does it make it durable.
05:27And lastly, you need to be able to ask your the question, where the hell did this piece of information come from?
05:32It makes no sense at all.
05:33And be able to trace it back to something, a data source, a person, a conversation from which it was extracted.
05:40And that's the first step to improving your context.
05:42So we started building this by assuming how hard can it be.
05:49So the first thing we did was it's just a knowledge base.
05:52We throw everything in the knowledge base.
05:53is going to be great, right?
05:55And how wrong we were.
05:56We built a whole distributed rag pipeline, extremely scalable, very elastic, amazing.
06:01It was just amazing, right?
06:02And it didn't do anything good for us.
06:04And then we said, you know what?
06:05This whole shared stuff doesn't work.
06:07We'll let every agent spin up a tiny database.
06:09We're a database company.
06:10How hard can it be?
06:11Tiny Postgress, make it efficient.
06:13Well, we realized that every repeatable workflow was being reinvented on the fly.
06:19Didn't work at all.
06:20And then we said, "Okay, fine.
06:21we'll just have some backend that does a whole bunch of processing and you know we'll just you know not even bother with the UI cuz agents don't need UI and then we realized we couldn't tell what was going on.
06:31No human in the loop equals really bad results.
06:34Okay.
06:35And then we thought okay at least we have the rag pipeline.
06:38Well, it turned out even that was shitty because, you know, we took all the best components from the, you know, whatever the academia academia says and we put it all in there and we got poor numbers.
06:4914% faithfulness using an MD benchmark, public benchmark, 20% on a PDF, 57% context precision using all of this other stuff.
06:59And when we actually started looking and tracing and doing our traces and tuning and experimenting, we realized a lot of things were off.
07:07And as we improved them one variable at a time, we were able to jump to 65 74 82.
07:13Right?
07:13And here's the kicker.
07:15Our context chunks, which is the context size, went down from over 7,000 chunks to about a,000 chunks.
07:23So 17th the context size, a lot higher precision and a lot less money burned on tokens, right?
07:31So that's where Mako comes in.
07:34What is Mako?
07:35It's a agentnative persistence layer that takes all the mistakes we made and all the things we got right and puts it into one box.
07:41Right?
07:42And the way Mako works, it exposes a data pack, right?
07:46As a database company, we don't call it a database because it has multiple types of databases.
07:51Vector, graph, relational, no SQL, whatever, right?
07:54And what it does is stores aspects of the context, knowledge base, memory, conversations, observability, a lot more, right?
08:03And what it enables is it enables agents to come in and push pieces of their context into a private space and promote that to a shared space, right?
08:14And then other agents or humans that come in can now go against that promoted highquality context and quickly retrieve stuff and you can now continue to add more people and more agents and do the handoff efficiently.
08:28Right?
08:28Hopefully that makes sense.
08:30Um the idea is take the durable repeatable workflows and make it infrastructure.
08:37Okay, before we go into a demo, right, a confession.
08:42This slide deck that you're looking at was built using Makeo as the context persistence layer, right?
08:49The way it worked was Heather, who's going to give you show you the demo, actually wrote up an outline and created a data pack in Mako and simply sh sent sent me a share request,
08:59right?
09:00And she said, "I'm doing the demo.
09:02Here's the outline of my demo.
09:03Here's the rough talk track that needs to lead up to it because that's what makes sense."
09:07I took it and I'm I'm sure she used some agent harness.
09:10I don't know what it is.
09:11I really don't know.
09:12I didn't ask her.
09:13I didn't even have a meeting with her.
09:14She just sent me this.
09:15I plugged it into my cloud desktop and said, "Take this outline and break it down for me."
09:21And it gave me a few bullet points.
09:22This is what you're talking about.
09:24I'm like, "Okay, great.
09:25I want to change a couple of things."
09:26Changed a couple of things and said, "Here's the new outline.
09:29Can you build me my slide deck based on this?"
09:31And it did.
09:32And that diagram on the left, it built a really crappy diagram, right?
09:36It didn't look good at all.
09:37So, I said, like, you know what?
09:38Everyone's saying this uh cloud design is the rage, so why don't I give that thing a go?
09:43And it actually did a better job.
09:44So, you know, there you go.
09:45That image came from cloud design.
09:47Everything else came from cloud desktop.
09:49Shot it back to Heather and, you know, Heather took the final pass over it to get hook it up with the demo and here we are.
09:56So, we barely talked, right?
09:58We didn't need to.
09:59We just shared context and our agents did the work and uh you know, you can go you can watch it.
10:05I think maybe I'll invite Heather to talk to this slide and and the thing after.
10:08I hope the demo's good because I haven't seen it myself.
10:11So, yeah.
10:20The idea is to watch collective knowledge grow over time, right?
10:25What Nico provides uh that I was able to use for this is a resumability.
10:29So resumability I'm going to show you is being able to start a session and then kill it and then start it again with the same agent.
10:37Right now if you do it with something like cloud desktop, they have projects for this, right?
10:42Right?
10:42So if you do a cloud project, everything's together or if you turn on memory.
10:46But in this case, I guess like two months ago, I turned off all the memory on all of my idees because it doesn't matter what if I use something while I'm on the go, like a mobile app.
10:55Maybe I use chat EBT that isn't as sophisticated, right, as some of my coding idees.
11:00And so maybe I want to continue working on that topic.
11:03I can do that because all of the context is shared in our cloud now.
11:08And by sharing I mean that obviously chatbt is a different um agent but it's still my account so I should be able to get to it right but because we're security focused and security minded we don't want like just any agent to get to it so I have to promote the kind of memory that that agent can
11:25also retrieve in like a collective pool and because you do this the cost itself goes down tremendously because the first time I figure stuff out I'm going back and forth with my LLM
11:36I'm burning lots of tokens to get to an end goal, whether that is code or it's just a knowledge work.
11:42But the next time that I want to retrieve it from another agent or another session, it doesn't need to burn through that.
11:48It just calls to the Miko memory and then it brings it right back with far less token.
11:53So, we're already saving a lot of token spend.
11:56The alternative to this would be to just again keep things in empty files or keep things where you have to constantly traverse with every single agent's LLM and burn through tokens again and again in order to retrieve.
12:09Does that make sense?
12:10The difference here is just an MCP tool call.
12:13So to kind of prove that out because we have sensational internet here, I have recorded this for you at a little bit of a faster speed hopefully.
12:23All right.
12:24See this is resumability.
12:25So here I'm saying that I want to store um the conversation in a certain data pack, right?
12:32Let's call it the early access launch and I want to put a piece of information in it.
12:36In this case, um our signup emails failed.
12:39True story because we actually only used the SCS's sandbox instead of the prod.
12:44So uh we kind of went over the limit there when we were testing things.
12:47And that's a good piece of information for maybe our our ops team, right?
12:53So in this process, it's calling data pack create uh because I've connected the MCP server to clot here and I've created a skill that will tell it what to do.
13:05You can decide how often you want it to call or if you just want to do it judiciously here.
13:10And so now it's storing the incident fact as a searchable memory with memory add.
13:15And then it's done now.
13:16So the data pack has been created.
13:18there's a conversation in there and it'll be retrievable in the in the future sessions.
13:23That sounds great.
13:24Awesome.
13:25Now I've killed that session and notice I'm not in projects.
13:28This is just a new chat window.
13:31Now I'm going to ask the same question so I can resume.
13:35Before I had to again have to make sure I turn on cloud memory or turn or put it inside of a project, but it's not in either one of these.
13:42I could be logged in with the same account in like claw.ai, which would be technically a different agent.
13:48So I'm trying to prove that you can at least resume your sessions after maybe days with the same agent.
13:53So the fail result there was actually that it tried to search first instead of creating uh a conversation a new conversation here.
14:03We want to save everyone that we do.
14:05So it's the MCP server instructing to create a new conversation for it so we can store this.
14:11And see now they searched Mo and it found that hey this is what happened back in May and it was able to pull things back out.
14:22All right for sharability.
14:24Let's go ahead and get this done.
14:26If you notice this is a completely different IDE.
14:29In this case it's codeex.
14:31Um so I've asked it questions like okay um do you have any known cause for the signup emails that are failing?
14:38And so it's gonna look and it's gonna look and it says, "Nope, I don't see anything.
14:43I don't see anything.
14:44I'm looking."
14:45Um, it did say that it saw that there was a brand new uh data pack that it's connected to, but it can't actually see anything.
14:54And why is that?
14:55Because I actually haven't shared that private memory from my claw desktop yet uh into a shared space.
15:02So right now, those memories are still private.
15:04If you notice the early access data pack is seen because I have connected it but I just haven't promoted it yet.
15:12So in that case I'm going to go to sign out.
15:15I'm going to sign in with my other account here to show you that yes I do have access to this data pack and I need to promote that piece of memory
15:24into a shared knowledge space.
15:27Allow it to go ahead and and go through and then I it says it doesn't see it but I said well same question.
15:34Let's try it again.
15:37So, now that I've promoted it into that space, sped up for your pleasure here
15:44[laughter]
15:44because we all know how long-winded our LLMs can get.
15:47Um, now it can find that piece of memory.
15:51It can find the piece of memory because I've allowed that one piece to go.
15:54It's very fine grained.
15:55It doesn't have to be.
15:56You can open the whole data pack if you want as a maintainer to it, but this is much more ideal if you're just trying to give bespoke pieces of information, maybe best practices, maybe something that is the ongoing
16:09process that you want to keep going.
16:14And then like the last part of this is the cost.
16:16How do you even know that it saves you tokens?
16:20If you go into your data pack, you can look at the conversations that are stored in there and how they were done because we give you embedded language traces for free in this.
16:30And you can actually look at the decision trace all the way down to the tool call and prove whether or not it called as much and burned as many tokens.
16:43Right?
16:48So, human promotions today kind of equal the training signal for self-improving orchestration tomorrow.
16:54Every human that says this is worth keeping is kind of a labelled example of what good promotion looks like.
16:59So, you can actually start training maybe in the future an agent that's trained on your taste and your preferences long term.
17:06But right now, we're keeping the human in the loop.
17:09Those calls are just a signal, but this is actually how it starts to learn.
17:13The biggest lift that I saw is that when something goes wrong, maybe in a support situation, it's hard to constantly have calls call the same server that's down all the time and then it's tails all this time.
17:29Hey, do you know if this is down?
17:30Is that down?
17:31Slack starts blowing up.
17:32How do your agents handle that?
17:34They're all also wasting tokens calling the same thing.
17:37If they had access to the data pack, the first agent would hit it and then tell everybody that it happened and all the other agents would know that it already happened.
17:52Yeah.
17:53So, yeah.
17:55So, thank you, Heather.
17:56Like, I think you know, pretty awesome demo.
17:58I think uh if you folks were following along what happened, Heather was was in charge of the early access and launch of Mako, right?
18:06and obviously used a data pack which is Mako to launch Mak Makeo right recursively and uh you know the first few people had a lot of trouble accessing their data packs on Mako and you know we you know she created a data pack for troubleshooting
18:20issues with Mako a different data pack and so that's the thing that she was talking about uh a lot of us in the company you know it's shared with us so we're all able to look at it load it into our agents and able to iterate with it the folks that are in engineering they load it straight
18:35into their you know cloud cloud code or codeex or like you know whatever the coding agents are like I I myself more of a cloud desktop kind of user I use it for other things like you know some a user that wanted to get access couldn't get access apology email like go through cloud help me design it
18:50have a I have a data pack actually that is my voice of how I speak and same thing for posting on LinkedIn that thing can get tiresome if you want to keep writing it in your voice you know sometimes training a data pack for the context to do so will help so a lot of these things get
19:05automated straight away and it gets you know makes it much much quicker to assemble right and obviously if a new person comes into the team very easy to share this context with them have them get up to speed have them contribute to it either willingly or unwillingly right sometimes they're just doing things they don't even know and
19:21you extract patterns out of it oh this is really great we should make this a thing right so so like um yeah so I think these are some of the things anyways I think the bottom line is we're trying to get it closer and closer to how humans work which is how can you capture tribal knowledge how
19:34can you share that How can you give controls over sharing and how can you actually tell what's going on?
19:39Right.
19:40Yeah.
19:40Um Yeah.
19:41So
19:41you should you should try it for free.
19:43If you want.
19:44Yeah.
19:44Try it for free.
19:45Yeah.
19:45I think the left side is you know the website makeata.ai which will take you to a login.
19:51There's a free tier.
19:52You can sign up and just hook it up to one of your agents and you know get your agents to start learning right right away.
19:57And the right side is Discord.
19:59We'd love to hear from you if you do try it out.
20:00You know do drop us a note.
20:02We're working on improvements.
20:03That's how we communicate what we're building.
20:05So do join us on Discord, right?
20:07We'd love to see you guys there.
20:08So I think that brings us Yeah.
20:10Thank you.
20:10Thanks.
20:11Yeah, if we have Yeah, we're right on time actually.
20:13Yeah.
20:13So yeah, thank you.
20:14If you guys have questions, please feel free to connect with us on the site.
20:17Thank you.
20:33[music]