WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=R3-anFK1YM8

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/R3-anFK1YM8.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.719 --> 00:00:14.639
Hello, welcome.

s3
00:00:14.639 --> 00:00:19.680
Uh, this is a big room, so you're if you're in the back, don't hesitate to come closer.

s4
00:00:19.680 --> 00:00:21.600
Um, my name is Stefania Dug.

s5
00:00:21.600 --> 00:00:25.840
I'm a research scientist at Sakana AI in Tokyo.

s6
00:00:25.840 --> 00:00:32.559
Uh I used to be based here and AI engineering uh is home community for me before being the hyperloop.

s7
00:00:32.559 --> 00:00:42.640
So it's very good to be back and today I'm going to talk to you about memory harnesses for longunning research agents on device.

s8
00:00:42.640 --> 00:00:49.920
So if you work with long horizon tasks, you probably run into this issue of context blow, right?

s9
00:00:49.920 --> 00:01:01.359
Like when the model starts contradicting itself or it has to redo the work because it forgot it did that task in the first place or it starts to drift from your questions

s10
00:01:01.359 --> 00:01:03.280
because it forgot them.

s11
00:01:03.280 --> 00:01:20.000
And this this matters now more than ever because from this recent projections from meter we see that the trend is to solve longer and longer uh horizon tasks and also that we're getting fewer and fewer model releases.

s12
00:01:20.000 --> 00:01:29.280
So at some point later this year we're going to have this convergence right where we'll get many more long-term horizon tasks and fewer model releases.

s13
00:01:29.280 --> 00:01:36.479
So that makes this issue of dealing with context rot a priority.

s14
00:01:36.880 --> 00:01:42.880
And why did I wanted to to tackle this problem on local models and with a local harness?

s15
00:01:42.880 --> 00:01:44.960
Uh maybe some of you have seen this tweet.

s16
00:01:44.960 --> 00:01:46.399
It's only two days old.

s17
00:01:46.399 --> 00:01:57.040
Uh the CEO of Coinbase actually shared how their company managed to reduce their AI spent while actually increasing uh the AI usage.

s18
00:01:57.040 --> 00:02:10.399
And the way they did that was by transitioning to use many more local models but also having better practices like using better routing, better caching, keeping the context clean

s19
00:02:10.399 --> 00:02:17.360
and then having better visibility for what people are using and for what uh what kind of task.

s20
00:02:17.360 --> 00:02:21.360
So we are seeing the local models like crossing the line, right?

s21
00:02:21.360 --> 00:02:26.160
Like GLM is on everyone's minds like especially with Fable going away.

s22
00:02:26.160 --> 00:02:35.440
uh DeepSync V4 flash can now be run on uh M3 Ultra and there's still a bottleneck for RAM.

s23
00:02:35.440 --> 00:02:43.840
It's tricky, but these local models are starting to be useful for agentic tasks and for tool use.

s24
00:02:43.840 --> 00:02:48.959
So, I wanted to show you what has been my setup for the experiments I'm going to share with you today.

s25
00:02:48.959 --> 00:02:51.440
Uh this this is my Mac.

s26
00:02:51.440 --> 00:02:57.440
It's still running evaluations right now uh back in my desk in Tokyo and I'm controlling it from my phone.

s27
00:02:57.440 --> 00:03:03.760
Um and after running EVAs non-stop for a couple of days, it started to get hot.

s28
00:03:03.760 --> 00:03:06.640
So I had my husband put fans around it.

s29
00:03:06.640 --> 00:03:13.360
Um we're running out of fans, but the the machine is still running and the valves are still giving results.

s30
00:03:13.360 --> 00:03:21.440
Um, on this M3 Ultra with 96 gigabytes and 28 core CPUs, I'm using two models.

s31
00:03:21.440 --> 00:03:29.440
I'm using a Quen 27B quantise at 4 and the DC V4 flash.

s32
00:03:29.760 --> 00:03:38.480
And before I show you how I built the memory harness on this machine, I wanted to tell you what this little what is this an example of, right?

s33
00:03:38.480 --> 00:03:48.560
like memory when we design a harness for memory this is the mental model I want you to have in mind um you can think of memory as a write manage read loop

s34
00:03:48.560 --> 00:04:05.920
so it's not just the database store it's actually this control loop around the model more concretely how did I take that loop and customized it so this is my harness design like I started with research agents that are the small agents because they have zero durable memory

s35
00:04:05.920 --> 00:04:16.079
and I wanted all the memory to come from the harness And then um in the middle I have a core which is always shown to to the agent um of traces.

s36
00:04:16.079 --> 00:04:28.639
And then I have a recall block where I'm testing different modes and an archival block where I'm keep keeping track of information across different um sessions.

s37
00:04:28.639 --> 00:04:34.720
And in that recall block I'm actually going through a ladder of modes that I'm testing.

s38
00:04:34.720 --> 00:04:37.120
The baseline is like not to use memory at all.

s39
00:04:37.120 --> 00:04:38.080
no recall at all.

s40
00:04:38.080 --> 00:04:40.320
So I'm I'm testing for that.

s41
00:04:40.320 --> 00:04:50.960
Uh next is to use rag vector vector rag um just to see whatever like the harness would pull in terms of similarity.

s42
00:04:50.960 --> 00:05:00.960
Then is to use a decisions uh ledger where I actually keep track of what decisions are being made for every turn and then I can prioritize them.

s43
00:05:00.960 --> 00:05:03.759
And last but not least and this piece is very important.

s44
00:05:03.759 --> 00:05:08.479
I have a what I call an oracle, but basically this is the ground truth.

s45
00:05:08.479 --> 00:05:18.560
So this is like telling the harness for every loop what the correct memory that needs to be retrieved is.

s46
00:05:18.639 --> 00:05:22.000
And the model is fixed across all the different tasks.

s47
00:05:22.000 --> 00:05:27.520
So the only things that I'm changing is like these different variables in the recall block.

s48
00:05:27.520 --> 00:05:31.759
And I wanted to to give you an example of a first task that I tested.

s49
00:05:31.759 --> 00:05:46.160
So I wanted to see if I give the agent a task of doing literature review and I'm including a lot of papers in the corpus where there was a big scientific claim like this is actually a nature paper

s50
00:05:46.160 --> 00:06:02.400
where they said they discovered 742,000 promising materials like it was a very big claim which got retracted later but the retraction it's a much smaller like haystack back needle

s51
00:06:02.400 --> 00:06:05.759
in that corpus than the headlines and the citations.

s52
00:06:05.759 --> 00:06:14.160
So I wanted to see if if the system can retrieve the right answer for these type of questions.

s53
00:06:14.160 --> 00:06:26.960
And what I found was because like for these tasks all the papers and all the information fit into the context the memory actually didn't add more capability.

s54
00:06:26.960 --> 00:06:32.479
It was the same performance with memory and without memory and it only added more cost.

s55
00:06:32.479 --> 00:06:39.520
So when your task fits in context, the harness doesn't add much.

s56
00:06:39.520 --> 00:06:55.280
However, if I start to run tasks that are longer term horizon and the entire task and the relevant context doesn't uh fit, then having a good memory harness really starts to pay off.

s57
00:06:55.280 --> 00:06:57.680
So this is another example of a task that I ran.

s58
00:06:57.680 --> 00:07:03.120
This is actually from an established benchmark for a long horizon uh tasks memory.

s59
00:07:03.120 --> 00:07:04.960
It's called Xbench.

s60
00:07:04.960 --> 00:07:07.360
And this is an example of a question, right?

s61
00:07:07.360 --> 00:07:17.599
So I'm asking a question and then like the right answer is in a like step 124.

s62
00:07:17.599 --> 00:07:22.880
But the moment when I ask the question, I'm asking it like at step 500.

s63
00:07:22.880 --> 00:07:34.160
So it's completely outside of the context window and the model needs to use the memory harness to retrieve the specific answer from the right step.

s64
00:07:34.160 --> 00:07:49.199
So I'm testing this by uh changing the different policy ladder that I explained before with memory off uh by deploying recall different types of recall and by using the oracle as a reference.

s65
00:07:49.199 --> 00:07:59.440
And what I found was that with the ranked recall, the model gets the right answer um more frequently than without.

s66
00:07:59.440 --> 00:08:05.520
And here is a breakdown of the decomposition of performance on this Xbench tasks.

s67
00:08:05.520 --> 00:08:17.360
So I ran over uh 68 questions and for each of these questions there were like multiple um cells and lots of different seeds.

s68
00:08:17.360 --> 00:08:32.880
And what I found was that the rank only ledger performed the best and it performed better than like just gating the harness by saying do you need to use memory or do you not need to use memory

s69
00:08:32.880 --> 00:08:40.080
and you're probably going to ask like why is the oracle not hitting like the max and I'm going to explain that too.

s70
00:08:40.080 --> 00:08:48.240
So the oracle what it does it provides the right information the right memory to the model but it doesn't force it to use it.

s71
00:08:48.240 --> 00:08:55.279
So the model can get the right memory but still retrieve the wrong information or choose to ignore it or be confused.

s72
00:08:55.279 --> 00:08:59.680
So that's why the oracle in this case doesn't hit the max performance.

s73
00:08:59.680 --> 00:09:08.959
And I've done lots of ablations on these tasks to see like what happens if I give arbitrary um examples.

s74
00:09:08.959 --> 00:09:11.519
What happens if I give it the wrong step?

s75
00:09:11.519 --> 00:09:14.240
What happens if I give it the most recent step?

s76
00:09:14.240 --> 00:09:22.959
And I still found that the best performing condition was the one with the ranked policy for recall.

s77
00:09:22.959 --> 00:09:31.040
And this actually works on several models, not only on the Quen 27B, but also on the DS4 flash.

s78
00:09:31.040 --> 00:09:33.200
And it also works across different benchmarks.

s79
00:09:33.200 --> 00:09:37.040
I also tried it on the Spider V2 benchmark.

s80
00:09:37.040 --> 00:09:42.160
And it's not just that it gives you better recall, it actually costs less.

s81
00:09:42.160 --> 00:09:52.880
So maybe a good heristic to have here is that bad memory is expensive because it spends more token and it can send agent the wrong way.

s82
00:09:52.880 --> 00:10:02.800
But having like a good structural policy for recall can save you a lot of tokens and uh budget.

s83
00:10:02.800 --> 00:10:15.839
So one thing that I want to encourage you from this experiment is to consider the recall policy as a first class metric and to start to think about how you might use it in your systems.

s84
00:10:15.839 --> 00:10:20.480
Like what are the type of memories that you want to store?

s85
00:10:20.480 --> 00:10:21.839
What how do you rank them?

s86
00:10:21.839 --> 00:10:25.360
Like how do you design your recall function?

s87
00:10:25.360 --> 00:10:30.720
And then um what are the type what survives when you run this over and over and over?

s88
00:10:30.720 --> 00:10:39.440
and um multiple sessions, multiple runs and this is just a simple first kind of experiment.

s89
00:10:39.440 --> 00:10:43.440
Um but the memory technique landscape is very rich.

s90
00:10:43.440 --> 00:10:55.760
Um so there's over 30 runnable cookbooks that are shared in this open-source repository from um Diamond and memory is complex.

s91
00:10:55.760 --> 00:10:59.519
We have short-term long-term different cognitive techniques.

s92
00:10:59.519 --> 00:11:02.880
uh we can use start to use evaluation results as well.

s93
00:11:02.880 --> 00:11:08.320
Um and right now there's actually a a pretty broad landscape of solutions, right?

s94
00:11:08.320 --> 00:11:15.200
So going from simple file system retrieval to training memory models.

s95
00:11:15.200 --> 00:11:21.760
Um there's there's a wide spectrum of solutions from less structural to completely structured.

s96
00:11:21.760 --> 00:11:26.160
Um so I think there's a lot of research we're going to see in this space.

s97
00:11:26.160 --> 00:11:38.480
uh it's important and it becomes more and more relevant and for me it's been super fun to to test this on local models um because I got to control everything.

s98
00:11:38.480 --> 00:11:51.839
I got to control the data I was using the entire traces of compute and evaluations and um yeah I I see that as an example of sovereignty and it comes at a cost.

s99
00:11:51.839 --> 00:12:00.880
Uh I didn't tell you that these local models I can only what uh run them in serial like they don't support batch querying for the DeepSc V4 flash.

s100
00:12:00.880 --> 00:12:09.839
So that's why I am still running evaluations back on my computer in Tokyo or I I was doing it on the flight on my way here because it takes a long time.

s101
00:12:09.839 --> 00:12:20.399
Um, but I still think it's very powerful and it's a very good test for what memory can do when you can control every single step of the pipeline.

s102
00:12:20.399 --> 00:12:27.519
And this sovereign capability is part of a bigger ecosystem that is very important for us at Sakanai in Japan.

s103
00:12:27.519 --> 00:12:32.639
Um, we believe in the importance of sovereign AI today more than ever.

s104
00:12:32.639 --> 00:12:34.560
And we are also hiring.

s105
00:12:34.560 --> 00:12:40.959
So, if you're interested and want to hear more about this and if you want to come join us in Japan, come talk to me.

s106
00:12:40.959 --> 00:12:43.083
Uh, thank you very much.

s107
00:12:43.083 --> 00:12:45.083
[applause]