Memory Harnesses for Long-Running Research Agents — Stefania Druga, Sakana.ai

AI Engineer · 13 min · 107 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hello, welcome.
00:14Uh, this is a big room, so you're if you're in the back, don't hesitate to come closer.
00:19Um, my name is Stefania Dug.
00:21I'm a research scientist at Sakana AI in Tokyo.
00:25Uh I used to be based here and AI engineering uh is home community for me before being the hyperloop.
00:32So it's very good to be back and today I'm going to talk to you about memory harnesses for longunning research agents on device.
00:42So if you work with long horizon tasks, you probably run into this issue of context blow, right?
00:49Like when the model starts contradicting itself or it has to redo the work because it forgot it did that task in the first place or it starts to drift from your questions
01:01because it forgot them.
01:03And this this matters now more than ever because from this recent projections from meter we see that the trend is to solve longer and longer uh horizon tasks and also that we're getting fewer and fewer model releases.
01:20So at some point later this year we're going to have this convergence right where we'll get many more long-term horizon tasks and fewer model releases.
01:29So that makes this issue of dealing with context rot a priority.
01:36And why did I wanted to to tackle this problem on local models and with a local harness?
01:42Uh maybe some of you have seen this tweet.
01:44It's only two days old.
01:46Uh the CEO of Coinbase actually shared how their company managed to reduce their AI spent while actually increasing uh the AI usage.
01:57And the way they did that was by transitioning to use many more local models but also having better practices like using better routing, better caching, keeping the context clean
02:10and then having better visibility for what people are using and for what uh what kind of task.
02:17So we are seeing the local models like crossing the line, right?
02:21Like GLM is on everyone's minds like especially with Fable going away.
02:26uh DeepSync V4 flash can now be run on uh M3 Ultra and there's still a bottleneck for RAM.
02:35It's tricky, but these local models are starting to be useful for agentic tasks and for tool use.
02:43So, I wanted to show you what has been my setup for the experiments I'm going to share with you today.
02:48Uh this this is my Mac.
02:51It's still running evaluations right now uh back in my desk in Tokyo and I'm controlling it from my phone.
02:57Um and after running EVAs non-stop for a couple of days, it started to get hot.
03:03So I had my husband put fans around it.
03:06Um we're running out of fans, but the the machine is still running and the valves are still giving results.
03:13Um, on this M3 Ultra with 96 gigabytes and 28 core CPUs, I'm using two models.
03:21I'm using a Quen 27B quantise at 4 and the DC V4 flash.
03:29And before I show you how I built the memory harness on this machine, I wanted to tell you what this little what is this an example of, right?
03:38like memory when we design a harness for memory this is the mental model I want you to have in mind um you can think of memory as a write manage read loop
03:48so it's not just the database store it's actually this control loop around the model more concretely how did I take that loop and customized it so this is my harness design like I started with research agents that are the small agents because they have zero durable memory
04:05and I wanted all the memory to come from the harness And then um in the middle I have a core which is always shown to to the agent um of traces.
04:16And then I have a recall block where I'm testing different modes and an archival block where I'm keep keeping track of information across different um sessions.
04:28And in that recall block I'm actually going through a ladder of modes that I'm testing.
04:34The baseline is like not to use memory at all.
04:37no recall at all.
04:38So I'm I'm testing for that.
04:40Uh next is to use rag vector vector rag um just to see whatever like the harness would pull in terms of similarity.
04:50Then is to use a decisions uh ledger where I actually keep track of what decisions are being made for every turn and then I can prioritize them.
05:00And last but not least and this piece is very important.
05:03I have a what I call an oracle, but basically this is the ground truth.
05:08So this is like telling the harness for every loop what the correct memory that needs to be retrieved is.
05:18And the model is fixed across all the different tasks.
05:22So the only things that I'm changing is like these different variables in the recall block.
05:27And I wanted to to give you an example of a first task that I tested.
05:31So I wanted to see if I give the agent a task of doing literature review and I'm including a lot of papers in the corpus where there was a big scientific claim like this is actually a nature paper
05:46where they said they discovered 742,000 promising materials like it was a very big claim which got retracted later but the retraction it's a much smaller like haystack back needle
06:02in that corpus than the headlines and the citations.
06:05So I wanted to see if if the system can retrieve the right answer for these type of questions.
06:14And what I found was because like for these tasks all the papers and all the information fit into the context the memory actually didn't add more capability.
06:26It was the same performance with memory and without memory and it only added more cost.
06:32So when your task fits in context, the harness doesn't add much.
06:39However, if I start to run tasks that are longer term horizon and the entire task and the relevant context doesn't uh fit, then having a good memory harness really starts to pay off.
06:55So this is another example of a task that I ran.
06:57This is actually from an established benchmark for a long horizon uh tasks memory.
07:03It's called Xbench.
07:04And this is an example of a question, right?
07:07So I'm asking a question and then like the right answer is in a like step 124.
07:17But the moment when I ask the question, I'm asking it like at step 500.
07:22So it's completely outside of the context window and the model needs to use the memory harness to retrieve the specific answer from the right step.
07:34So I'm testing this by uh changing the different policy ladder that I explained before with memory off uh by deploying recall different types of recall and by using the oracle as a reference.
07:49And what I found was that with the ranked recall, the model gets the right answer um more frequently than without.
07:59And here is a breakdown of the decomposition of performance on this Xbench tasks.
08:05So I ran over uh 68 questions and for each of these questions there were like multiple um cells and lots of different seeds.
08:17And what I found was that the rank only ledger performed the best and it performed better than like just gating the harness by saying do you need to use memory or do you not need to use memory
08:32and you're probably going to ask like why is the oracle not hitting like the max and I'm going to explain that too.
08:40So the oracle what it does it provides the right information the right memory to the model but it doesn't force it to use it.
08:48So the model can get the right memory but still retrieve the wrong information or choose to ignore it or be confused.
08:55So that's why the oracle in this case doesn't hit the max performance.
08:59And I've done lots of ablations on these tasks to see like what happens if I give arbitrary um examples.
09:08What happens if I give it the wrong step?
09:11What happens if I give it the most recent step?
09:14And I still found that the best performing condition was the one with the ranked policy for recall.
09:22And this actually works on several models, not only on the Quen 27B, but also on the DS4 flash.
09:31And it also works across different benchmarks.
09:33I also tried it on the Spider V2 benchmark.
09:37And it's not just that it gives you better recall, it actually costs less.
09:42So maybe a good heristic to have here is that bad memory is expensive because it spends more token and it can send agent the wrong way.
09:52But having like a good structural policy for recall can save you a lot of tokens and uh budget.
10:02So one thing that I want to encourage you from this experiment is to consider the recall policy as a first class metric and to start to think about how you might use it in your systems.
10:15Like what are the type of memories that you want to store?
10:20What how do you rank them?
10:21Like how do you design your recall function?
10:25And then um what are the type what survives when you run this over and over and over?
10:30and um multiple sessions, multiple runs and this is just a simple first kind of experiment.
10:39Um but the memory technique landscape is very rich.
10:43Um so there's over 30 runnable cookbooks that are shared in this open-source repository from um Diamond and memory is complex.
10:55We have short-term long-term different cognitive techniques.
10:59uh we can use start to use evaluation results as well.
11:02Um and right now there's actually a a pretty broad landscape of solutions, right?
11:08So going from simple file system retrieval to training memory models.
11:15Um there's there's a wide spectrum of solutions from less structural to completely structured.
11:21Um so I think there's a lot of research we're going to see in this space.
11:26uh it's important and it becomes more and more relevant and for me it's been super fun to to test this on local models um because I got to control everything.
11:38I got to control the data I was using the entire traces of compute and evaluations and um yeah I I see that as an example of sovereignty and it comes at a cost.
11:51Uh I didn't tell you that these local models I can only what uh run them in serial like they don't support batch querying for the DeepSc V4 flash.
12:00So that's why I am still running evaluations back on my computer in Tokyo or I I was doing it on the flight on my way here because it takes a long time.
12:09Um, but I still think it's very powerful and it's a very good test for what memory can do when you can control every single step of the pipeline.
12:20And this sovereign capability is part of a bigger ecosystem that is very important for us at Sakanai in Japan.
12:27Um, we believe in the importance of sovereign AI today more than ever.
12:32And we are also hiring.
12:34So, if you're interested and want to hear more about this and if you want to come join us in Japan, come talk to me.
12:40Uh, thank you very much.
12:43[applause]