WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=r9OwPx_HoV0

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/r9OwPx_HoV0.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.040 --> 00:00:14.160
Hi everybody.

s3
00:00:14.160 --> 00:00:17.039
Uh thank you for coming today.

s4
00:00:17.039 --> 00:00:20.480
Uh welcome to a talk about nothing.

s5
00:00:20.480 --> 00:00:22.560
Sorry, a talk about retrieval.

s6
00:00:22.560 --> 00:00:24.400
Uh my name is Yuval.

s7
00:00:24.400 --> 00:00:29.199
I work at AI21 which is essentially an AI research lab.

s8
00:00:29.199 --> 00:00:37.680
And today I want to talk to you about something that most people don't want to talk about which is chunking.

s9
00:00:37.680 --> 00:00:45.200
And I hope to convince you by the end that chunking isn't dead and there is something to do with that.

s10
00:00:45.200 --> 00:00:58.399
And really if you are at Axe LinkedIn wherever you probably seen that rug is dead right I think people also killed MCP lately and Rag is dead again.

s11
00:00:58.399 --> 00:01:13.600
Long live identic retrieval, identic search and there is come a time where you have to ask yourself how many times can Rag die right and even when someone says well ra isn't dead like

s12
00:01:13.600 --> 00:01:32.000
Jerry the CEO of Llama index they still have to kill something and apparently this something is chunking like don't invest in it don't do it and this is the reason that People said that chunking is dead because everybody's using agentic search now, right?

s13
00:01:32.000 --> 00:01:35.360
You have gs, you have ls, you have finds.

s14
00:01:35.360 --> 00:01:45.759
All of these are great, but these are still not enough if you have a lot of data and you have various amount of queries.

s15
00:01:46.640 --> 00:01:47.520
Just the second.

s16
00:01:47.520 --> 00:01:48.079
Okay.

s17
00:01:48.079 --> 00:01:56.079
And I think that the main reason that a lot of people don't like to talk about chunking, it's because it's not the fun part, right?

s18
00:01:56.079 --> 00:02:02.159
In every rag or files system we have two stages.

s19
00:02:02.159 --> 00:02:11.440
The first stage is the like the boring one as you may the one you do in the beginning you have a lot of data you have to pre-process it you have to decide

s20
00:02:11.440 --> 00:02:30.239
on the chunk size and then you have to store everything in a vector DB the other part is the retrieval part essentially the the one that happens per query this is something which is much easier to do right it's much easier to optimize you can use all your queries and then you can play with the max k

s21
00:02:30.239 --> 00:02:46.400
uh top k sorry you can uh play with a hybrid search maybe those kind of things much more fun to do retrieval tuning right h so I will claim that if we have to kill something if something has to be dead

s22
00:02:46.400 --> 00:03:04.800
then it's probably retrieval tuning and yes agentic search probably killed that and but still agentic search even if we can accept the fact that it killed retrieval tuning it's still not good enough when you have a lot of right scale a lot of data

s23
00:03:04.800 --> 00:03:16.239
it costs a lot of money I don't think I have to mention that anymore token maxing is like something that everybody's talking about and the thing underneath which is

s24
00:03:16.239 --> 00:03:30.400
if the data itself is not ordered in a right way in your folders in your directories you still get something which is inefficient so let's try to think of a like a timely example

s25
00:03:30.400 --> 00:03:33.280
right the FIFA World Cup is now.

s26
00:03:33.280 --> 00:03:38.080
And let's imagine that we have a data set that contains of all the FIFA World Cup.

s27
00:03:38.080 --> 00:03:44.879
So every directory is the let's say the 98 one, the 2002 one and so on and so on.

s28
00:03:44.879 --> 00:03:54.720
But if your query asks how which team won the most World Cups, you can't just go to a folder and ask that.

s29
00:03:54.720 --> 00:04:00.640
You have to go to every folder, see who won, and then aggregate this together, which is very inefficient.

s30
00:04:00.640 --> 00:04:02.480
The answer by the way is Brazil.

s31
00:04:02.480 --> 00:04:10.720
I hope at least according to yeah according to the time that this uh conversation is happening.

s32
00:04:10.720 --> 00:04:12.799
So retrieval didn't actually die.

s33
00:04:12.799 --> 00:04:15.760
Okay, we're not killing anything in this lecture.

s34
00:04:15.760 --> 00:04:18.799
It is got devoted into plumbing.

s35
00:04:18.799 --> 00:04:24.639
And I think that everybody who worked on any rug system know the feeling.

s36
00:04:24.639 --> 00:04:29.680
day one or week one or maybe even month one if you're very thorough.

s37
00:04:29.680 --> 00:04:32.000
You're picking some sort of a chunk size.

s38
00:04:32.000 --> 00:04:42.320
Let's say 512 and maybe you're probably putting some overlap right 10 20% so on indexing everything and forget all about it.

s39
00:04:42.320 --> 00:04:57.600
And you can right we talk a lot about the fixed chunking strategies where if your chunk something which is too big right so you get the whole picture which is nice but you're losing a lot of the nuance and all the chunks will not get meaningful embeddings

s40
00:04:57.600 --> 00:05:05.600
where if you will choose your chunks to be too small you're getting the big picture lost and really it won't be as efficient.

s41
00:05:05.600 --> 00:05:11.840
So what this tells us is that chunking is essentially a lossy compression.

s42
00:05:11.840 --> 00:05:15.600
No matter what we're doing, we're losing something.

s43
00:05:15.600 --> 00:05:21.120
And I will I will claim that there is no right chunk size.

s44
00:05:21.120 --> 00:05:25.360
And a lot of you who worked on data will say, "No, but we have this corpus.

s45
00:05:25.360 --> 00:05:32.560
We have this data set and we really used and we optimized our system to work very very well on this data."

s46
00:05:32.560 --> 00:05:33.919
And we thought so too.

s47
00:05:33.919 --> 00:05:43.199
We had a lot of experience with it with a lot of different types of agents and systems and workflows that you can really and right you think about benchmarks

s48
00:05:43.199 --> 00:05:57.680
how easy it is to overfit your model to a benchmark but not with rug it doesn't happen there and you cannot really optimize it per data set and I will claim that it is query dependent

s49
00:05:57.680 --> 00:06:05.919
and how can I be so sure how can I claim such a thing because we ran experiments and we tested And now I'm going to present it to you.

s50
00:06:05.919 --> 00:06:19.840
So what we did instead of saying what is the best chunk size per data let's find out let's let's actually take a data set and duplicate this data set several times.

s51
00:06:19.840 --> 00:06:25.759
In this case six times in every duplication in every instance the chunk size is different.

s52
00:06:25.759 --> 00:06:34.960
So we have a database with a chunk size of 2,00 a database with a chunk size of 1,000 and so on and so on.

s53
00:06:34.960 --> 00:06:36.800
And we did it with several data sets.

s54
00:06:36.800 --> 00:06:47.199
So QM sum which is a meeting transcript data set, narrative QA which is question answering on novels and Seinfeld data set which is a trivia about nothing.

s55
00:06:47.199 --> 00:06:47.840
Not really.

s56
00:06:47.840 --> 00:06:52.400
It's a trivia trivia questions about the transcripts of Seinfeld.

s57
00:06:52.400 --> 00:06:56.479
It's a kind of a trolling data set that we built uh in-house.

s58
00:06:56.479 --> 00:07:00.080
We also published it if anybody wants the link at the end.

s59
00:07:00.080 --> 00:07:04.000
And we tested on all of them to see what happens.

s60
00:07:04.000 --> 00:07:10.000
And first of all, we just wanted to see for every data set which chunk size is the best.

s61
00:07:10.000 --> 00:07:22.479
And what we're seeing here is an example from the Seinfold data set where essentially two queries which are different by nature get different results uh based on the chunk size.

s62
00:07:22.479 --> 00:07:25.680
So the first question what is the name for Jerry's favorite church?

s63
00:07:25.680 --> 00:07:28.880
You can see this is a very focused question very specific question.

s64
00:07:28.880 --> 00:07:34.880
The answer to it is probably very contained and this is something that a smaller chunk size will do best in.

s65
00:07:34.880 --> 00:07:39.680
And you can see uh rank one versus rank below 50.

s66
00:07:39.680 --> 00:07:53.199
uh between 100 tokens fixed at chunk size to 100 whereas a question like who does Jerry describe as his nemesis and pure evil which I'm not even that big of a Seinfeld

s67
00:07:53.199 --> 00:08:07.280
fan and I know it's Newman but if you look at the transcript it's not something you can find that easily and you can see that it really changes right if you use small chunk size you will not get the answer

s68
00:08:07.520 --> 00:08:17.759
and what we did to really after we ran all of these things and we've noticed that we said what if we had an oracle or a genie if you want

s69
00:08:17.759 --> 00:08:32.240
that can tell us for every query what is the best chunk size to do retrieval for this essentially is the Oracle experiment this is what we wanted to know to see the potential this is not right we already have the answers

s70
00:08:32.240 --> 00:08:38.880
so we're not actually building a system here we just want to see what is the potential that we have here and what you can see here.

s71
00:08:38.880 --> 00:08:43.839
Okay, in this uh graph all the blue, first of all, the yaxis is the recall.

s72
00:08:43.839 --> 00:08:45.600
Higher is better.

s73
00:08:45.600 --> 00:08:48.720
The x-axis is the number of retrieved chunks.

s74
00:08:48.720 --> 00:09:02.720
So, it's recall at K versus K. You can see all the blue lines probably indistinguishable, but each of them is the performance for a fixed chunk size, whereas the orange one is the oracle line.

s75
00:09:02.720 --> 00:09:07.920
This is for every query, we took the best one out of these.

s76
00:09:07.920 --> 00:09:11.839
And you can see it happens across several data sets.

s77
00:09:11.839 --> 00:09:16.959
In a lot of them, you can actually see that the blue lines inter intersect with each other.

s78
00:09:16.959 --> 00:09:23.519
Meaning that indeed for a lot of the data sets, no chunk size actually dominates.

s79
00:09:23.519 --> 00:09:27.600
And what's more interesting is that there's a lot of potential.

s80
00:09:27.600 --> 00:09:33.519
The gap which you can see between the orange line and all the blue lines is big.

s81
00:09:33.519 --> 00:09:44.399
And when I say big, it's something like 20 to 40% just from doing strategy on chunking and very simple strategy may I add.

s82
00:09:44.399 --> 00:09:53.040
And this is like that this gap this is what the choice of 512 or a thousand or whatever right this number is just arbitrary.

s83
00:09:53.040 --> 00:09:55.920
this is what it costs you.

s84
00:09:55.920 --> 00:10:09.040
And I think that the problem here is like it's a bit tricky because it's kind of like an information problem that we don't have the information that we need at every stage.

s85
00:10:09.040 --> 00:10:10.320
And what do I mean by that?

s86
00:10:10.320 --> 00:10:18.240
If I'm looking at the indexing part where I do have control over the chunk size, I don't know what the queries will be.

s87
00:10:18.240 --> 00:10:24.480
I can guess, I can maybe estimate, I I can try but I don't know what the queries will be.

s88
00:10:24.480 --> 00:10:28.800
So I cannot adjust my chunk size accordingly.

s89
00:10:28.800 --> 00:10:34.880
And the retrieval part where I do have my queries, I cannot control the chunk size, right?

s90
00:10:34.880 --> 00:10:41.600
It's already fixed and I obviously will not do the entire process per query from the beginning.

s91
00:10:41.600 --> 00:10:56.720
So we looked at prior works such as notably entropic contextual retrieval where they en enrich every chunk and others that essentially try to improve the latent space of every chunk.

s92
00:10:56.720 --> 00:10:59.200
But this is not the direction that we went.

s93
00:10:59.200 --> 00:11:05.120
All of them just stayed in the model of let's work with a fixed chunk size.

s94
00:11:05.120 --> 00:11:18.640
Whereas we took a different approach and we said why commit to one where we can commit to several and we call it the multiscale indexing essentially we're just doing what we've seen before.

s95
00:11:18.640 --> 00:11:20.880
So we're checking the database.

s96
00:11:20.880 --> 00:11:36.720
We duplicate it and chunk it uh with several chunk chunk sizes or window sizes and then sorry um and then this is what happens at the indexing and then at retrieval time we are querying all of them.

s97
00:11:36.720 --> 00:11:47.360
So if we had n duplicates of database n window sizes we now have to run six different retrieval calls per query.

s98
00:11:47.360 --> 00:11:51.440
Sorry, six is n. And how do we combine them?

s99
00:11:51.440 --> 00:11:53.760
We obviously cannot use the oracle, right?

s100
00:11:53.760 --> 00:11:57.200
The oracle is something that we have just for potential.

s101
00:11:57.200 --> 00:11:59.600
In real life, we don't know the answer.

s102
00:11:59.600 --> 00:12:05.440
Uh but what we can do is to find some sort of merging algorithm.

s103
00:12:05.440 --> 00:12:10.000
Now you would say when we look at it like this, what can be the issue?

s104
00:12:10.000 --> 00:12:26.880
the fact that we have n ranking but the rankings are for chunks and chunks with different sizes are not really comparable right so instead we opted to do something which is pretty popular these days and a lot of the rug systems

s105
00:12:26.880 --> 00:12:36.880
actually work like this that instead of just retrieving the chunk when we're getting a chunk we're retrieving the entire document right when context window grows we want to give more and more context

s106
00:12:36.880 --> 00:12:51.920
and now in this case we have n right n uh rankings of the same documents because there are not chunks anymore and this we can compare and in this case you can think of retrieval as essentially just voting

s107
00:12:51.920 --> 00:13:05.600
right so it's not pure ranking we don't have round ranking and then we're doing it rerank we're having n different ranks of the relevant documents and we want to aggregate them all into one

s108
00:13:05.600 --> 00:13:13.440
that's why we are using something called rf receive reciprocal rank fusion, okay, which is pretty much a simple formula.

s109
00:13:13.440 --> 00:13:14.959
We tried several things.

s110
00:13:14.959 --> 00:13:16.560
This worked the best.

s111
00:13:16.560 --> 00:13:18.399
And as you can see, it's not a model.

s112
00:13:18.399 --> 00:13:26.880
It's not something that you have to do specifically like especially this is just a simple script that takes really no time.

s113
00:13:26.880 --> 00:13:31.519
And this is how the full uh how the full system looks like.

s114
00:13:31.519 --> 00:13:34.160
So we have the indexing end times.

s115
00:13:34.160 --> 00:13:48.959
Then uh we query each query from every database and we're using RRF to combine them all and the results you can guess that they're good otherwise I would not uh standing here and uh

s116
00:13:48.959 --> 00:14:06.800
being way too much uh confident right but you can see we tested across several data sets QM sam narrative QA Seinfeld and also Finance Bench we took all of them and it matches the best or bits the best fixed size.

s117
00:14:06.800 --> 00:14:08.320
Let's see it in a graph.

s118
00:14:08.320 --> 00:14:12.320
It's a bit hard to see here, so I'll walk it slowly.

s119
00:14:12.320 --> 00:14:15.360
Every row here is a chunk size.

s120
00:14:15.360 --> 00:14:18.160
So you can see 50, 100, and so on.

s121
00:14:18.160 --> 00:14:20.560
The bottom row is our method.

s122
00:14:20.560 --> 00:14:25.040
This one, the one that you do from all of them and then combine.

s123
00:14:25.040 --> 00:14:30.480
And the every column is recall at something.

s124
00:14:30.480 --> 00:14:34.079
So recall at one two three up until 10.

s125
00:14:34.079 --> 00:14:37.040
What you can see here is that two things right.

s126
00:14:37.040 --> 00:14:52.000
First of all that across like recall at whatever h our method still wins which you can think is very easy but the fact that you have to combine all of them is not very it's not something which is very trivial

s127
00:14:52.000 --> 00:15:03.839
and also you can see that the quality actually increases the heat map where you can see it become much greener and again this was just something that I wanted to show in large here you can see all four of the data sets

s128
00:15:03.839 --> 00:15:17.360
where we do achieve better results results u really quite like 20 30 40% even in a lot of the things h also there are results that I did not show you here uh which

s129
00:15:17.360 --> 00:15:30.160
are on MTB um you can see in our blog I will put the link later we're getting there also a lot of improvements somewhere between 10 to 40% depending on the data set

s130
00:15:30.160 --> 00:15:37.279
now I'm I'm not naive I'm not going to claim here that this costs nothing obviously There is a cost, right?

s131
00:15:37.279 --> 00:15:38.639
No free lunches.

s132
00:15:38.639 --> 00:15:41.279
Everything has to come with something.

s133
00:15:41.279 --> 00:15:44.000
And yes, this costs with extra memory.

s134
00:15:44.000 --> 00:15:48.480
It costs something between two to five to all of one, right?

s135
00:15:48.480 --> 00:15:56.000
A constant of additional memory where you have to keep all of those uh all those copies of the database.

s136
00:15:56.000 --> 00:16:10.320
However, if you think about it latency wise, it doesn't really affect that because you can do all the retrieval partly and also the RRF part doesn't really take a lot of time.

s137
00:16:10.720 --> 00:16:17.120
I will say that this was a very nice research project that we did and we got really really cool results.

s138
00:16:17.120 --> 00:16:19.839
There are things to do right there are places to improve.

s139
00:16:19.839 --> 00:16:22.399
There are future work to do.

s140
00:16:22.399 --> 00:16:35.519
More precisely, we want to understand how many chunk sizes do we want and and which right the fact that we worked with 50, 100, 200 and so on was

s141
00:16:35.519 --> 00:16:37.839
pretty arbitrary to be honest.

s142
00:16:37.839 --> 00:16:44.079
So, we do need to figure out how to compute this and how to know how many copies exactly do you need.

s143
00:16:44.079 --> 00:16:46.079
Also go beyond RF, right?

s144
00:16:46.079 --> 00:16:54.880
the fact that we're using RF is because it worked the best from the methods that we used, but it doesn't mean that there is no better method.

s145
00:16:54.880 --> 00:17:01.440
And if I need to leave you with something, I would say that agents didn't kill retrieval.

s146
00:17:01.440 --> 00:17:02.240
Nothing died.

s147
00:17:02.240 --> 00:17:03.279
Come on.

s148
00:17:03.279 --> 00:17:05.280
It's just infrastructure.

s149
00:17:05.280 --> 00:17:10.720
And the part the bad bad part is that it's infrastructure from 2022.

s150
00:17:10.720 --> 00:17:25.520
And with really simple simple methods, you can take your rag system or anything that has to do with storing data and then retrieve it with 20 to 40% again without any

s151
00:17:25.520 --> 00:17:28.079
something too sophisticated.

s152
00:17:28.079 --> 00:17:33.600
So if you uh want to hear more about read more about it, you can read the blog.

s153
00:17:33.600 --> 00:17:37.280
There is also an example code there and the Seinfeld data set.

s154
00:17:37.280 --> 00:17:38.240
And that's it.

s155
00:17:38.240 --> 00:17:39.120
I'm Yuval.

s156
00:17:39.120 --> 00:17:40.400
Thank you so much.

s157
00:17:40.400 --> 00:17:42.422
May here.

s158
00:17:42.422 --> 00:17:44.422
[applause]

s159
00:17:58.797 --> 00:18:00.797
[music]