WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=V-isu4eTHgw

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/V-isu4eTHgw.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.040 --> 00:00:14.160
Okay.

s3
00:00:14.160 --> 00:00:15.280
Hi everyone.

s4
00:00:15.280 --> 00:00:22.480
Welcome um to this 20 minute talk about connecting AI to loads of legal documents.

s5
00:00:22.480 --> 00:00:23.840
My name is Jacob.

s6
00:00:23.840 --> 00:00:26.160
I'm an engineer at Legora.

s7
00:00:26.160 --> 00:00:26.480
Yeah.

s8
00:00:26.480 --> 00:00:28.000
And I got to step into frame here.

s9
00:00:28.000 --> 00:00:28.800
Uh I'm Simon.

s10
00:00:28.800 --> 00:00:35.680
I'm the CEO and uh CEO and co-founder of Turopuffer, a search engine that we work with Legora and others on.

s11
00:00:35.680 --> 00:00:36.160
Neat.

s12
00:00:36.160 --> 00:00:41.360
Um super quickly um introduction to Legora.

s13
00:00:41.360 --> 00:00:45.360
We're a collaborative AI platform for legal work.

s14
00:00:45.360 --> 00:00:50.320
And so that means we have law firms that are clients and we have uh in-house legal teams that are clients.

s15
00:00:50.320 --> 00:00:54.239
And they use Legora to do reviews of contracts.

s16
00:00:54.239 --> 00:00:58.399
They use it to go through an absurd amount of contracts and make sure that they all look good.

s17
00:00:58.399 --> 00:01:00.960
They use them to create new contracts.

s18
00:01:00.960 --> 00:01:07.600
They do legal research which means um looking over all potential law and they collaborate inside Lor.

s19
00:01:07.600 --> 00:01:14.240
So you can think of Lorra sort of as a linearfigmas notion GitHub for legal work.

s20
00:01:14.240 --> 00:01:16.640
It's a lot.

s21
00:01:16.640 --> 00:01:20.159
We um are one of the fastest growing companies right now.

s22
00:01:20.159 --> 00:01:21.920
We grow extremely fast.

s23
00:01:21.920 --> 00:01:24.159
Yeah, tons of numbers on the screen.

s24
00:01:24.159 --> 00:01:26.560
I'll just skip through that.

s25
00:01:26.560 --> 00:01:29.439
What we really want to talk about is search today.

s26
00:01:29.439 --> 00:01:33.040
So at Lor there's two types of search that we do.

s27
00:01:33.040 --> 00:01:35.680
There is project search and legal research.

s28
00:01:35.680 --> 00:01:41.840
Project search is um basically projects in Lor is like the the unit of work that you have.

s29
00:01:41.840 --> 00:01:50.240
So if um let's say you are SpaceX and you want to acquire Cursor, then that would be one project with your law firm.

s30
00:01:50.240 --> 00:01:59.439
H and so they would go into Lor and they would upload all these documents and the law firm that helped you would go through all of the employment agreements, all of the contracts with suppliers.

s31
00:01:59.439 --> 00:02:04.000
Um I know cursor is using Turbo Puffer so maybe there's a contract there they'd look at.

s32
00:02:04.000 --> 00:02:11.360
Um but basically you do the search confined to a project and projects can be tens of documents to millions of documents.

s33
00:02:11.360 --> 00:02:13.360
The other use case is legal research.

s34
00:02:13.360 --> 00:02:27.280
And legal research is sort of a deep research style workload where we'll search across tons of laws, previous cases, regulations, etc., etc. And people use this to answer questions such as like how do I handle this specific thing?

s35
00:02:27.280 --> 00:02:29.280
And they'll also use it for litigation.

s36
00:02:29.280 --> 00:02:35.840
Maybe they want to sue someone or maybe they are getting sued and they'll use legal research to support and help their case.

s37
00:02:36.560 --> 00:02:43.760
So if we start at number one, project search, we've been through a little bit of a ride here on how we do search.

s38
00:02:43.760 --> 00:02:49.760
Um, starting at, you know, hundreds of thousands of documents all the way into two billions of documents.

s39
00:02:49.760 --> 00:02:52.720
And we've tried a lot of different things.

s40
00:02:52.720 --> 00:02:59.120
So first we start started with the very very simple one, which is just a single elastic search cluster for all of our search workloads.

s41
00:02:59.120 --> 00:03:02.080
Um, that worked relatively well.

s42
00:03:02.080 --> 00:03:03.840
It was sort of a simple setup.

s43
00:03:03.840 --> 00:03:15.519
All of the tenants, you know, our clients, our users would be on one big blob storage where we store the raw documents and on one big elastic search where we would do all of the searching, the indexing and the searching.

s44
00:03:15.519 --> 00:03:19.599
Super simple, worked relatively well initially.

s45
00:03:19.840 --> 00:03:24.480
Then we wanted to enter the land of the free and uh we got some new requirements.

s46
00:03:24.480 --> 00:03:32.879
They you know Americans only want processing to happen within the US and Europeans only want it to happen within the EU and Australians only want it to happen within Australia.

s47
00:03:32.879 --> 00:03:37.280
And so we had to basically um well here's a scaling graph.

s48
00:03:37.280 --> 00:03:39.120
We had to move to multiple elastic searches.

s49
00:03:39.120 --> 00:03:47.360
And what we actually did was we took the entire setup and we just basically iterated over the set that is EU, US and Asia Pacific.

s50
00:03:47.360 --> 00:03:50.720
And so we just had this like multiplied by three or four.

s51
00:03:50.720 --> 00:03:56.560
Kind of annoying, lots of overhead, but it uh got us to where we needed to be.

s52
00:03:57.760 --> 00:04:01.040
Then the next iteration of the story is enterprise.

s53
00:04:01.040 --> 00:04:07.200
So really big banks, the biggest law firms in the world, they have really annoying requirements.

s54
00:04:07.200 --> 00:04:12.720
And number one they have is they'll ask for full physical isolation of all of their data.

s55
00:04:12.720 --> 00:04:19.840
There's probably a little bit of a war on what physical, you know, isolation actually means, but essentially means they want their own database.

s56
00:04:19.840 --> 00:04:22.720
They also want customer managed encryption keys.

s57
00:04:22.720 --> 00:04:30.960
And what that means is they basically have a key volt thing where they have an encryption key and they give us access to read the key and we then use that key to

s58
00:04:30.960 --> 00:04:33.600
encrypt and decrypt all of their data at rest.

s59
00:04:33.600 --> 00:04:39.360
And what that gives them is they can just revoke our access to their key and then we can't decrypt their data anymore and so it's safe.

s60
00:04:39.360 --> 00:04:47.199
And so in a way that gives enterprises a lot of control over all of their data because they control you know the key to to reading it.

s61
00:04:48.080 --> 00:04:54.960
So we moved from elastic search to Postgress and I imagine a bunch of you guys are like why would you ever put your vectors into Postgress.

s62
00:04:54.960 --> 00:05:06.160
Um it actually worked surprisingly well and the reason that we did this was we were already using Postgress for OOLTP workloads and uh so we sort of already had to do this split

s63
00:05:06.160 --> 00:05:15.600
of multiple Postgresses and multiple blobs and so it was really easy for us to try to shift all of our search into Postgress as well because then we only have one system.

s64
00:05:15.600 --> 00:05:20.880
So the setup here was PG vector uh specifically disk NNN TS vector for the search.

s65
00:05:20.880 --> 00:05:25.360
So not BM25 which was you know we lost a little bit of retrieval performance there.

s66
00:05:25.360 --> 00:05:29.199
And what we do is we would partition the table where we would store all the document chunks.

s67
00:05:29.199 --> 00:05:38.240
We'd partition it super aggressively like 4,000 partitions and then each project we'd basically have the project key and we'd binack them into the partitions.

s68
00:05:38.240 --> 00:05:45.199
that actually worked relatively well, but it was expensive and search performance wasn't super super good.

s69
00:05:45.199 --> 00:05:51.600
And what happened was when we scaled a lot, uh everything just broke and exploded.

s70
00:05:51.600 --> 00:06:01.120
And so what happened was the basically you can imagine like you have a bunch of projects and some of them you spin up a project, you work on it and then you close it and you basically never go back to it again.

s71
00:06:01.120 --> 00:06:06.960
And we have a bunch of those where like they never get queried and we have a bunch that get queried all the time because they're super active projects.

s72
00:06:06.960 --> 00:06:13.120
And when we pack them into partitions, the cold ones and the hot ones would land on the same ones and the partitions would get really really big.

s73
00:06:13.120 --> 00:06:23.840
And so when we query them, they would Postgress would pull the partition, put it into memory, we do the stuff and then we create another partition and another partition and it would essentially thrash the cache all the time.

s74
00:06:23.840 --> 00:06:26.240
And what that meant was our latencies would spike.

s75
00:06:26.240 --> 00:06:37.120
So we went from like search and ingestion P99 of 100 milliseconds into 20 seconds, which you can imagine is a really bad user experience.

s76
00:06:37.759 --> 00:06:50.400
So then we went to turbopuffer in about you know when we're about I think 400 million documents something like that and what we did with turbopuffer was we did one name space per project

s77
00:06:50.400 --> 00:07:03.280
and the advantages of moving to turbuffer is we got BM25 real BM25 much better uh relevancy much better latencies and it was extremely simple to operate because we could just have a single

s78
00:07:03.280 --> 00:07:12.160
turbo puffer cluster you know we didn't have to have a bunch of different ones like with Postgress and it would since it's blob based it could just query the blobs that we had anyway

s79
00:07:12.160 --> 00:07:26.560
and so much lower cost and it was extremely simple to operate and we didn't have this problem with the partitions because if a project's not used it's just in blob and so it's really easy and Simon can talk a bit more about why that works so well.

s80
00:07:28.639 --> 00:07:29.039
Yeah.

s81
00:07:29.039 --> 00:07:39.599
So, Legora has some of and and legal in general has, by the way, if if uh Jake and I have similar accents and maybe even look a bit similar, it's because we're both Danish.

s82
00:07:39.599 --> 00:07:46.639
Um, Turboper has a very has a particular architecture that supports these kinds of very regulated environments really really well.

s83
00:07:46.639 --> 00:07:50.479
But in order to understand that, we have to understand what kind of search engine is Turboper.

s84
00:07:50.479 --> 00:07:54.000
Why is it different than the ones that they used in the past?

s85
00:07:54.000 --> 00:07:58.240
Since the very beginning of Turbopuffer, um the design has more or less been the same.

s86
00:07:58.240 --> 00:08:03.039
Uh there may be changes in the future, but the design has stood the test of time.

s87
00:08:03.039 --> 00:08:07.919
When you do a write to Turbopuffer, we write directly to obic storage.

s88
00:08:07.919 --> 00:08:10.000
There is no like disk replication.

s89
00:08:10.000 --> 00:08:11.039
There's no paxas.

s90
00:08:11.039 --> 00:08:12.639
There's there's none of that.

s91
00:08:12.639 --> 00:08:14.560
Direct to S3.

s92
00:08:14.560 --> 00:08:17.520
That's the fundamental trade-off in Turbopuffer, right?

s93
00:08:17.520 --> 00:08:18.479
Hundreds of milliseconds.

s94
00:08:18.479 --> 00:08:29.919
If you're like Shopify and doing inventory uh reservations for a Kylie Jenner flash sale not going to work very very good for search because generally when you're doing search doing a slow write is fine

s95
00:08:29.919 --> 00:08:32.640
um as long as the read performance is is adaptable and good.

s96
00:08:32.640 --> 00:08:38.479
So that's what happens on write it just goes into write ahead log you can imagine you write 1.json 2.json 3.json JSON.

s97
00:08:38.479 --> 00:08:43.599
Obviously, it's a it's a database, so it's not JSON, but for illustrative purposes, that's what happens.

s98
00:08:43.599 --> 00:08:53.120
And in the background, we build the vector indexes, the text indexes, the colmer indexes, and so on to satisfy the queries that that Jake and other customers have.

s99
00:08:53.120 --> 00:08:59.200
Um, so then at query time, we can go in and then um the query reaches some namespace.

s100
00:08:59.200 --> 00:09:01.760
The namespace is kind of our concept of a of a table.

s101
00:09:01.760 --> 00:09:06.560
Um, you can think of it as a directory on S3 that's isolated from everything else.

s102
00:09:06.560 --> 00:09:08.640
We go to the node that is most likely to have it.

s103
00:09:08.640 --> 00:09:10.000
It could go to any node, right?

s104
00:09:10.000 --> 00:09:16.399
It could go to every single node and they're all read replicas, but we go with some affinity to the node that has the highest probability of having it in cache.

s105
00:09:16.399 --> 00:09:21.519
We check the memory cache for any objects, NVME SSD cache, and then finally to object storage.

s106
00:09:21.519 --> 00:09:26.720
Everything in Turbopuffer is optimized around doing as much work in as few round trips as possible.

s107
00:09:26.720 --> 00:09:26.959
Right?

s108
00:09:26.959 --> 00:09:32.640
S3 has a P99 um on a like 1 megaby blob size of around uh 200 milliseconds.

s109
00:09:32.640 --> 00:09:34.880
So, you want to do as few round trips as possible, right?

s110
00:09:34.880 --> 00:09:36.399
Ideally, you do around three.

s111
00:09:36.399 --> 00:09:42.000
And everything in Turbopuffer, the database is Oh, I'm going to need your fingerprint.

s112
00:09:42.000 --> 00:09:42.480
Got it.

s113
00:09:42.480 --> 00:09:47.600
Um, everything in in Turbopuffer is designed around minimizing the number of round trips.

s114
00:09:47.600 --> 00:09:49.600
This is also amazing for modern D discs.

s115
00:09:49.600 --> 00:09:53.440
If you do a lot of concurrency and few round trips, you utilize them optimally.

s116
00:09:53.440 --> 00:09:56.320
And everything in Turbopuffer is designed around this.

s117
00:09:56.320 --> 00:09:59.600
So, why is this so good for a company like Lora?

s118
00:09:59.600 --> 00:10:09.279
Well, object storage native if you design it around the atomic unit of separation being the namespace or the table, every single table could be encrypted with a different key.

s119
00:10:09.279 --> 00:10:11.839
Every single namespace could be in a different bucket.

s120
00:10:11.839 --> 00:10:24.000
We have customers that have um thousands of buckets that they have namespaces in so that their customers get the warm IT fuzzies of having the bucket in their own cloud account.

s121
00:10:24.000 --> 00:10:26.000
They can also be encrypted with their own keys.

s122
00:10:26.000 --> 00:10:27.120
You can share buckets.

s123
00:10:27.120 --> 00:10:31.120
You can you can do whatever configuration that you need at the namespace level.

s124
00:10:31.120 --> 00:10:32.640
You can re-encrypt with different keys.

s125
00:10:32.640 --> 00:10:33.760
You can move them around.

s126
00:10:33.760 --> 00:10:36.240
Um and you can re-encrypt with other keys.

s127
00:10:36.240 --> 00:10:41.200
Um for Lora in particular, this was really important for this full physical separation, right?

s128
00:10:41.200 --> 00:10:42.720
An encryption separation.

s129
00:10:42.720 --> 00:10:48.880
All of the namespaces needed to be physically at rest with different keys and as separated as possible.

s130
00:10:48.880 --> 00:11:00.480
S3, GCS, Azure BOP stories, they pass that um and the other parts of the hierarchy also except the NVME SSD cache because in the SSD cache we consider that to be volatile like memory

s131
00:11:00.480 --> 00:11:03.440
um but your customers did not.

s132
00:11:03.440 --> 00:11:18.399
So in the uh so what we did was that we thought we were going to implement encryption into the disc cache but instead we just disabled the disc cache and saw how it fared and the performance of turbopuffer even without the disc cache with just the memory cache was so good that we just kept it that way

s133
00:11:18.399 --> 00:11:29.600
for some of the lora workloads where we couldn't have the disc cache for multi-tenency turbuffer will support that in the future but it just goes to show the um natural point where turbopuffer

s134
00:11:29.600 --> 00:11:36.000
allows this encryption and storage and separation to become fully multi-tenency uh native.

s135
00:11:36.000 --> 00:11:39.200
I'll hand it back to you on what happened then.

s136
00:11:39.200 --> 00:11:41.279
And then, drum roll please.

s137
00:11:41.279 --> 00:11:44.160
Latencies look like this.

s138
00:11:44.160 --> 00:11:47.040
Um, is my mic working?

s139
00:11:47.040 --> 00:11:50.959
No. Could I or I'll start screaming really loudly.

s140
00:11:50.959 --> 00:11:53.680
Um, it speaks for itself if you can't hear me.

s141
00:11:53.680 --> 00:11:54.480
Okay.

s142
00:11:54.480 --> 00:11:57.200
Um, latency is improved in order magnitude basically.

s143
00:11:57.200 --> 00:11:58.959
And these are median latencies.

s144
00:11:58.959 --> 00:12:01.120
So, P99 were even better.

s145
00:12:01.120 --> 00:12:12.240
Um, so obviously this is a huge thing when you're doing I mean one thing is if you're doing a single sort of rag style thing but if you have an agent that does 20 queries 100 queries these really really add up.

s146
00:12:12.240 --> 00:12:18.079
So that was on the project side and then a more recent thing is legal research.

s147
00:12:18.160 --> 00:12:26.800
So legal research um is a kind of a difficult problem and the reason it's difficult is that um the corpus is extremely big.

s148
00:12:26.800 --> 00:12:32.240
So we're racing towards 10 billion vectors and we're growing extremely fast.

s149
00:12:32.240 --> 00:12:34.639
We also have quite high read.

s150
00:12:34.639 --> 00:12:38.560
Um so QPS can spike a lot because we do a lot of fan out.

s151
00:12:38.560 --> 00:12:44.399
Like if you do a a sort of legal research query, we will fan it out to a bunch of different queries and we'll keep going.

s152
00:12:44.399 --> 00:12:51.600
And the reason we do that is we need this um heavy filtering because essentially it's a kind of like a graph for a few different reasons.

s153
00:12:51.600 --> 00:12:53.839
Firstly, it's um hierarchical.

s154
00:12:53.839 --> 00:12:59.760
you know, you have cities and you have counties and you have states and you have federal law and it's the same all around the world.

s155
00:12:59.760 --> 00:13:03.360
Um, and so you need to respect that authoritative sort of hierarchy.

s156
00:13:03.360 --> 00:13:05.760
There's also some temporal validity.

s157
00:13:05.760 --> 00:13:11.600
So, one judge might overrule a decision that's been made somewhere else and you need to also respect that and figure that out.

s158
00:13:11.600 --> 00:13:18.000
And then sometimes there's even like a new regulation that has exemptions or special cases of an old regulation.

s159
00:13:18.000 --> 00:13:20.959
And so if you're finding this one, you need to find all the other ones as well.

s160
00:13:20.959 --> 00:13:35.440
So you can imagine that it sort of explodes the search and so we started on elastic search for this but also moving to turbuffer um elastic search just got extremely expensive because we have to have everything there

s161
00:13:35.440 --> 00:13:52.240
but um with turbopuffer we can basically uh take different jurisdictions and we can make them name spaces in turbo buffer and that means some of them here's an example where like you have the EU that gets quered all the time that's super hot and some of them let's say Danish law cuz we're Danish, no one cares really.

s162
00:13:52.240 --> 00:13:56.720
It's such a small country, so like it doesn't really get queried and so that can just stay on blob and that's fine.

s163
00:13:56.720 --> 00:14:05.279
Um, and because it's sort of a deep research style workload, if there's 500 milliseconds latency to fetch that cold blob, that's okay.

s164
00:14:05.279 --> 00:14:05.760
That's fine.

s165
00:14:05.760 --> 00:14:07.279
It's not really a big problem.

s166
00:14:07.279 --> 00:14:15.920
So, the way that Turbo is designed lends itself super well to this super long scale of like cold weird name spaces and a few that are really, really hot.

s167
00:14:15.920 --> 00:14:17.680
Um, yeah.

s168
00:14:17.680 --> 00:14:20.079
And Simon wants to talk more about that.

s169
00:14:20.079 --> 00:14:20.639
Yeah.

s170
00:14:20.639 --> 00:14:32.720
So, um um I was I was talking about why the company is called Turbopuffer at another uh talk here earlier today, but uh one of the other explanations of the name of Turbopuffer

s171
00:14:32.720 --> 00:14:39.440
is that it's about puffing into the different memory hierarchies and really mastering when data should be in particular memory hierarchies.

s172
00:14:39.440 --> 00:14:48.000
So, you can think about it here, right, of something like the EU law might be more or less part of almost every one of the legal research uh queries, right?

s173
00:14:48.000 --> 00:14:52.240
So that probably sits closer to NVME SSDs in memory, right?

s174
00:14:52.240 --> 00:14:55.839
The economics kind of change as you move up and down this hierarchy.

s175
00:14:55.839 --> 00:15:00.880
Um in in memory, you want things that are queried a lot, right?

s176
00:15:00.880 --> 00:15:02.399
Then the economics of memory are great.

s177
00:15:02.399 --> 00:15:14.240
NVME SSDs can you can do a lot of things directly on them, but the economics change as you move up and down this boundary, the latency changes and the way that the database is architected to take advantage of it in terms of round trips versus random versus sequential

s178
00:15:14.240 --> 00:15:16.800
all changes as you navigate this hierarchy.

s179
00:15:16.800 --> 00:15:26.079
Turbopuffer is a database that is really designed around the memory hierarchy and all of the smarts in Turbopuffer is that all of these namespaces are puffed in and out um of the cache.

s180
00:15:26.079 --> 00:15:35.360
You can think of this as we want to spend as much time have as much data pushed as far down in this hierarchy as possible to get the the best um performance cost ratios.

s181
00:15:35.360 --> 00:15:38.000
So how does that apply to search?

s182
00:15:38.000 --> 00:15:42.959
Well, for something like vector search for example, there's two fundamental ways to do vector search.

s183
00:15:42.959 --> 00:15:46.000
One is to navigate it basically design a graph.

s184
00:15:46.000 --> 00:15:53.120
The problem with a graph on something like object storage or disk, again, we want to have things as far down that memory hierarchy as possible.

s185
00:15:53.120 --> 00:15:56.000
The problem with a graph, this is not a graph, this is a tree.

s186
00:15:56.000 --> 00:15:59.759
Um, but in a graph, you have to navigate from the center of the graph.

s187
00:15:59.759 --> 00:16:06.320
So, and then every time you navigate through these nodes, you're doing 200 millisecond P99 to S3, right?

s188
00:16:06.320 --> 00:16:08.399
And so, you're trying to shrink the diameter of the graph.

s189
00:16:08.399 --> 00:16:18.079
You're trying to do all these tricks to make the graph, but fundamentally you're at odds with the fact that graph is about a random sequential trade-off that you have in memory and in registers,

s190
00:16:18.079 --> 00:16:20.480
but not further down the memory hierarchy.

s191
00:16:20.480 --> 00:16:23.279
The way Turbopuffer does it is organize it into clusters, right?

s192
00:16:23.279 --> 00:16:29.199
Vectors you can think of in two dimensions just as point is a massive coordinate system and we can organize them into clusters.

s193
00:16:29.199 --> 00:16:38.160
Turbopuffer then creates clusters of clusters and clusters of clusters of clusters to essentially organize all of the vector data in a tree.

s194
00:16:38.160 --> 00:16:43.040
You can basically think of turbopuffer as a very very complicated B tree, right?

s195
00:16:43.040 --> 00:16:48.880
Because it's a tree on this geometry of this entire space and the clustering of it in an approximate way.

s196
00:16:48.880 --> 00:16:55.120
Now the root centroidids further up the tree you can imagine are part of every single time you search right there.

s197
00:16:55.120 --> 00:16:59.759
We're always trying to figure out which clusters that we're in and we're always looking at the upper levels of the tree.

s198
00:16:59.759 --> 00:17:02.079
So they're going to be further of the memory hierarchy, right?

s199
00:17:02.079 --> 00:17:05.199
Closer to the registers almost all in DRAM.

s200
00:17:05.199 --> 00:17:15.120
Now the leaves that have all of the actual legal cases or whatever long document it could be could be images all of that is probably going to be on SSDs with that single 1 millisecond roundtrip at the end.

s201
00:17:15.120 --> 00:17:18.160
It doesn't make sense to have all that puffed into DRAM.

s202
00:17:18.160 --> 00:17:24.160
This is fundamentally the cheapest way that you can run a database period.

s203
00:17:24.160 --> 00:17:33.039
So for something like Lora or even web search which is in hundreds of billion or tens of billions depending on how much of the web you've scraped this is fundamentally the cheapest way to do it.

s204
00:17:33.039 --> 00:17:41.360
And we have customers that are indexing massive parts of the entire web into turbopuffer which is really also a part of what legal research is.

s205
00:17:41.360 --> 00:17:46.160
Full text is also also really respectful of the memory hierarchies.

s206
00:17:46.160 --> 00:17:49.679
The way the text search works is essentially you can think of it as a hashmap.

s207
00:17:49.679 --> 00:17:55.360
You have a big document and then you take every single one of the tokens and you put them into the key in the hashmap.

s208
00:17:55.360 --> 00:18:00.080
The value in the hashmap is some set with all of the document IDs that has that term.

s209
00:18:00.080 --> 00:18:08.720
So then if you search for New York population, you're finding those three places in the hashmap and then you're taking the three sets and doing an intersect on the intersect on the sets.

s210
00:18:08.720 --> 00:18:12.480
While you're intersecting, you're also trying to do some kind of scoring, right?

s211
00:18:12.480 --> 00:18:20.559
A document that has York in it is probably more valuable than a document that has new in it because York is a more rare word.

s212
00:18:20.559 --> 00:18:25.120
When people say BM25, this is the scoring that they're referring to.

s213
00:18:25.120 --> 00:18:29.280
The art of fulltech search is one, we want to minimize the number of round trips.

s214
00:18:29.280 --> 00:18:31.840
So first you download the parts of the dictionary that are relevant.

s215
00:18:31.840 --> 00:18:37.760
Round trip one maybe a round trip one before that to index into where the parts of the term terms are.

s216
00:18:37.760 --> 00:18:41.520
And then the second round trip is to get these massive lists.

s217
00:18:41.520 --> 00:18:44.320
Try to make the list as small as possible by compressing them.

s218
00:18:44.320 --> 00:18:51.280
But also while you're doing the text search you're trying to minimize the amount again of memory bandwidth that you want to intersect these list.

s219
00:18:51.280 --> 00:19:01.919
You can probably imagine that at some point there's a point where you've seen so many documents with population in York that have much higher scores that documents that just have new in it are irrelevant anymore.

s220
00:19:01.919 --> 00:19:04.880
This is like a mega crash course in how text search works.

s221
00:19:04.880 --> 00:19:13.200
And counterintuitively to most people, text search at web scale is more difficult and more computationally expensive than doing vector search.

s222
00:19:13.200 --> 00:19:16.000
I'll hand it over to you.

s223
00:19:16.799 --> 00:19:17.679
Cool.

s224
00:19:17.679 --> 00:19:24.720
So key learnings from um what you heard today, retrieval is extremely important to legora.

s225
00:19:24.720 --> 00:19:27.520
Um it's key to legal reasoning.

s226
00:19:27.520 --> 00:19:34.160
Turbopuffer really excels when uh for us because it makes it extremely easy to operate.

s227
00:19:34.160 --> 00:19:35.840
We have 70 plus tenants.

s228
00:19:35.840 --> 00:19:37.440
We have 100, we have 200 tenants.

s229
00:19:37.440 --> 00:19:41.840
You know, if we had to have separate elastic search databases for each of these, it would be just hell.

s230
00:19:41.840 --> 00:19:51.919
Um but we can do this natively with turboper with data residency and cmech, etc., etc. And then it's extremely cost efficient generally when you have these types of workflows

s231
00:19:51.919 --> 00:20:02.480
um or workloads that we do where there's a long tail of cold indices basically that you don't need to query so much and you don't you're sort of you're okay paying the small latency cost for it.

s232
00:20:02.480 --> 00:20:14.880
So now with um turbo buffer in 4 seconds to go now we can focus on making lora we can focus on the product making it really really great and not on scalability and infra and also David our CFO is really happy about the cost.

s233
00:20:14.880 --> 00:20:15.679
So, it's great.

s234
00:20:15.679 --> 00:20:17.658
Thanks everyone.

s235
00:20:17.658 --> 00:20:19.658
[applause]

s236
00:20:34.034 --> 00:20:36.034
[music]