WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=u_jvuCjuvEk

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/u_jvuCjuvEk.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.720 --> 00:00:13.240
All right.

s3
00:00:13.240 --> 00:00:14.440
Hello, everyone.

s4
00:00:14.440 --> 00:00:15.520
Thank you all for coming.

s5
00:00:15.520 --> 00:00:18.200
I'm Nick Hill, CTO of Turbo Puffer.

s6
00:00:18.200 --> 00:00:23.160
And I'm Aaron Linsky, the head of product engineering at Bridgewater Associates.

s7
00:00:23.160 --> 00:00:28.600
And we're here to talk to you today about how you build an AI analyst that can keep a secret.

s8
00:00:28.600 --> 00:00:31.920
And I'll explain a bit more about what that means.

s9
00:00:31.920 --> 00:00:32.880
Here.

s10
00:00:32.880 --> 00:00:41.760
Now, this is a very oversimplified view of how you would go about building an AI analyst, but it is meant to capture an important dimension around permissions.

s11
00:00:41.760 --> 00:00:52.000
So, in this diagram, you're user interacting with this AI agent, you start with a question, maybe something like, "Why have token prices come down so much in the last month?"

s12
00:00:52.000 --> 00:00:57.440
And you feed that question into an LLM, which will begin thinking about that question, investigating, researching.

s13
00:00:57.440 --> 00:01:05.440
And as it does that investigation, it's going to search for facts to ground its information, to lead its investigation.

s14
00:01:05.440 --> 00:01:13.240
Some of those facts might come from the public web, but other facts might come from private documents in your organization.

s15
00:01:13.240 --> 00:01:19.600
Now, the public web is on one dimension much easier to search than the information that's private to your organization.

s16
00:01:19.600 --> 00:01:21.240
By definition, the public web is public.

s17
00:01:21.240 --> 00:01:24.720
Everyone in your organization can see exactly the same documents.

s18
00:01:24.720 --> 00:01:26.440
So, there's no permission issue there.

s19
00:01:26.440 --> 00:01:32.880
Every public web document can be immediately fed into the AI analyst, no matter who's asking the question.

s20
00:01:32.880 --> 00:01:36.680
But the information internal to your organization looks very different.

s21
00:01:36.680 --> 00:01:45.320
Every email, every meeting note, every report might have a completely different set of permissions attached to it, belonging to different groups, maybe individual permissions assignments on those docs.

s22
00:01:45.320 --> 00:01:52.920
And you can see here, we've got different colors here reflecting the fact that every document in that private data set has a different set of permissions.

s23
00:01:52.920 --> 00:02:00.600
And this is a classically hard problem in information retrieval, and it's something we explicitly designed Turbo Puffer to solve.

s24
00:02:00.600 --> 00:02:07.640
But before I get into that, I'm going to hand it over to Aaron to talk about why this private information is so important to Bridgewater's AI analyst.

s25
00:02:07.640 --> 00:02:08.039
Great.

s26
00:02:08.039 --> 00:02:08.960
Thank you.

s27
00:02:08.960 --> 00:02:10.039
Oh.

s28
00:02:10.039 --> 00:02:10.840
There we go.

s29
00:02:10.840 --> 00:02:22.440
So, first I'm going to go talk a little bit about what who Bridgewater is, what we do, and then we get into how how we're using Turbo Puffer and and the needs there.

s30
00:02:22.440 --> 00:02:31.160
So, for the past 50 years, we at Bridgewater have pursued the best understanding of how markets and economies work.

s31
00:02:31.160 --> 00:02:38.400
We then encode that systematically with logic into a system that we can then stress test over time.

s32
00:02:38.400 --> 00:02:45.240
What you see here are on the on the left with the yellow legal pad, that is our bond system from 1980.

s33
00:02:45.240 --> 00:02:50.920
It's a set of if-then statements that we used to trade bonds back then.

s34
00:02:50.920 --> 00:03:08.080
Over the next few decades, we've evolved that process from a yellow legal pad to Excel to eventually Scala and now Python um to to a different learning process with humans at the center.

s35
00:03:08.080 --> 00:03:12.720
This five This research circle, as what we call it, has five steps.

s36
00:03:12.720 --> 00:03:16.360
Perception, that's seeing what's going on the outside world.

s37
00:03:16.360 --> 00:03:21.320
Hypothesis, which is coming up with ideas of why that might be true.

s38
00:03:21.320 --> 00:03:25.200
Investigation, which is the research of seeing is that thing true.

s39
00:03:25.200 --> 00:03:31.600
If it is true, we then systematize and integrate it with our system and industrialize and run that every day and trade on that.

s40
00:03:31.600 --> 00:03:33.040
That's been our learning process.

s41
00:03:33.040 --> 00:03:34.640
I've been at Bridgewater for about 20 years.

s42
00:03:34.640 --> 00:03:38.320
That's been what we've been doing doing for 20 years with humans at the center.

s43
00:03:38.320 --> 00:03:43.959
We're now moving towards putting machines at the center and running that process.

s44
00:03:43.959 --> 00:03:52.000
In order for even you know, for both humans and machines, in order for that to work well, we need access to high-quality data.

s45
00:03:52.000 --> 00:04:04.320
We need both time series data, which we spent decades uh decades creating a great bi-temporal time macro time series database, which we call Delorean.

s46
00:04:04.320 --> 00:04:11.720
And more recently, we've been creating a unstructured database, which we call the Vault.

s47
00:04:11.920 --> 00:04:14.080
And so, what is the Vault?

s48
00:04:14.080 --> 00:04:28.240
Um today, it's about 100 million documents, which we index, organize, and secure that we think represents some of the best third-party and extra third-party external research, and also our internal research.

s49
00:04:28.240 --> 00:04:38.520
So, we've sourced with 135 different external sources, which is a huge deal for both legal and our data team to actually nego- negotiate and get them in.

s50
00:04:38.520 --> 00:04:46.320
And all of all of those sources have um some sorts of permissions around it where it might not be open to everyone at Bridgewater.

s51
00:04:46.320 --> 00:04:50.320
And then, of course, we also have our inter- internal sources of unstructured data.

s52
00:04:50.320 --> 00:04:55.520
That's our own writing, our meeting recordings, and and our internal reports, as well.

s53
00:04:55.520 --> 00:05:02.240
And both that those all go into the Vault, and then both humans investors and our AI agents read out of the Vault.

s54
00:05:02.240 --> 00:05:04.880
What I haven't shown here is actually it circles back.

s55
00:05:04.880 --> 00:05:12.560
So, the investors and and the agents will create new new content that goes back into the Vault.

s56
00:05:12.560 --> 00:05:14.600
So, it's constantly improving.

s57
00:05:14.600 --> 00:05:20.840
And so, here's a very high-level view of what permissions in the Vault look like.

s58
00:05:20.840 --> 00:05:25.480
So, I have a user we'll just say their ID is user one.

s59
00:05:25.480 --> 00:05:30.480
They're in groups B and C. This is incre- incredibly simplified, unfortunately.

s60
00:05:30.480 --> 00:05:42.160
Um and there's a set of all of our documents where it could be either um uh permission to them via groups or directly.

s61
00:05:42.160 --> 00:05:54.720
And then, here we have the actual code of us going ahead and creating a query to try to look for some data with whatever that query vector was that matches

s62
00:05:54.720 --> 00:05:56.520
the viewers and groups.

s63
00:05:56.520 --> 00:06:01.320
And so, this is simple in one way.

s64
00:06:01.320 --> 00:06:06.640
And actually, when we first implemented this without Turbo Buffer, we thought it worked fine.

s65
00:06:06.640 --> 00:06:14.160
Um but then, pretty quickly, as we started to add more and more documents to the vault, it fell over um tremendously.

s66
00:06:14.160 --> 00:06:18.720
And so, what I did then was I looked for other possible providers.

s67
00:06:18.720 --> 00:06:25.560
We found Turbo Buffer, tested them out, and they had a solution which Nikhil is going to go over right now.

s68
00:06:25.560 --> 00:06:25.919
Awesome.

s69
00:06:25.919 --> 00:06:28.200
Thank you, Aaron.

s70
00:06:29.600 --> 00:06:33.000
Let me very quickly talk about what Turbo Buffer is.

s71
00:06:33.000 --> 00:06:40.040
We are an extremely fast, extremely scalable, extremely cheap search engine built entirely on top of object storage.

s72
00:06:40.040 --> 00:06:47.200
And what that means is we write directly to S3 or GCS or Azure Blob storage, whatever your hyperscaler provides.

s73
00:06:47.200 --> 00:06:50.040
And that gives us S3 economics.

s74
00:06:50.040 --> 00:06:53.720
So, by default, your data is inert, cold on S3.

s75
00:06:53.720 --> 00:07:00.040
Only when you start querying it do we pull it into more expensive storage mediums, like SSD or memory.

s76
00:07:00.040 --> 00:07:08.800
And this is really powerful because once a query is hot, once the data query is accessing is in the cache, it's fast.

s77
00:07:08.800 --> 00:07:17.560
But if you're not accessing a particular namespace or a particular region of a namespace, that data can just hang out on object storage where you pay rock-bottom prices for it.

s78
00:07:17.560 --> 00:07:28.600
So, this architecture allows Turbo Buffer to scale cheaply to terabytes of data, which would cost tens of millions of dollars in in other systems, and we can often do it at a tenth of the cost

s79
00:07:28.600 --> 00:07:35.840
at performance that's comparable once you get the data hot uh to what you get with traditional disk-based search systems.

s80
00:07:35.840 --> 00:07:37.919
And we support all types of search.

s81
00:07:37.919 --> 00:07:39.200
We support keyword search.

s82
00:07:39.200 --> 00:07:40.600
We support regex search.

s83
00:07:40.600 --> 00:07:44.040
And what Aaron was showing earlier, vector search.

s84
00:07:44.040 --> 00:07:49.280
Um so we're really not uh opinionated about how you search your data in Turbo Puffer.

s85
00:07:49.280 --> 00:07:52.240
We want to do search generically.

s86
00:07:52.320 --> 00:08:01.320
But for the rest of this talk, I'm going to focus on vector search because uh it's it's good to zoom in and go deep on one particular type of search.

s87
00:08:01.320 --> 00:08:04.040
Let's do a little bit of napkin math.

s88
00:08:04.040 --> 00:08:06.360
That's something we love to do at Turbo Puffer.

s89
00:08:06.360 --> 00:08:09.720
It helps us anchor in the simplest possible solution.

s90
00:08:09.720 --> 00:08:13.760
We never want to introduce complexity to Turbo Puffer that's not justified.

s91
00:08:13.760 --> 00:08:18.040
One of the justifications for taking on complexity is if you need it for performance.

s92
00:08:18.040 --> 00:08:25.600
The absolute simplest way to do search is to put everything in a list and every time somebody shows up saying, "I want queries that match this.

s93
00:08:25.600 --> 00:08:27.720
I want documents that match this query."

s94
00:08:27.720 --> 00:08:30.640
You look at every single piece of data.

s95
00:08:30.640 --> 00:08:36.159
Let's conservatively say that every document that you store in Turbo Puffer is 1 kilobyte.

s96
00:08:36.159 --> 00:08:38.000
This is incredibly conservative.

s97
00:08:38.000 --> 00:08:41.240
A vector embedding is often 1,024 dimensions.

s98
00:08:41.240 --> 00:08:45.440
So if you're using just a byte per dimension, that's a kilobyte right there.

s99
00:08:45.440 --> 00:08:47.680
Uh in practice, documents are often larger than this.

s100
00:08:47.680 --> 00:08:51.120
But for the sake of example, let's say it's a kilobyte per doc.

s101
00:08:51.120 --> 00:09:00.000
Even if you had all of that data resident in memory and you were ripping through it as fast as you possibly could, which on modern hardware is about 20 gigabytes a second.

s102
00:09:00.000 --> 00:09:03.920
Once you get up to 100 million docs, that's going to take 5 seconds.

s103
00:09:03.920 --> 00:09:06.680
And for most search workloads, that's way too slow.

s104
00:09:06.680 --> 00:09:10.206
What we see in practice is most people want their search results to be about 100 milliseconds.

s105
00:09:10.206 --> 00:09:13.600
[clears throat] So the naive solution here does not work.

s106
00:09:13.600 --> 00:09:15.480
If you've only got 10K docs, great.

s107
00:09:15.480 --> 00:09:16.760
That's going to take half a millisecond.

s108
00:09:16.760 --> 00:09:19.560
Turbo Puffer can do literally anything and it will be fast enough.

s109
00:09:19.560 --> 00:09:28.960
But as you get to scale, once you're at 10 million or 100 million, you really need to start thinking about how you represent your data so that you're not crunching through all of that.

s110
00:09:28.960 --> 00:09:31.480
And of course, I gave you the the numbers for memory here.

s111
00:09:31.480 --> 00:09:33.000
That's only if the cache is hot.

s112
00:09:33.000 --> 00:09:37.840
If that data's on S3, you're looking at a minute or more to pull that data off S3.

s113
00:09:37.840 --> 00:09:42.240
So, this just isn't viable at the scale that Bridgewater operates.

s114
00:09:43.080 --> 00:09:47.440
So, step one in Turbo Puffer is to build a vector index.

s115
00:09:47.440 --> 00:09:49.800
And this is something that we handle under the hood for you.

s116
00:09:49.800 --> 00:09:57.920
When you store vectors in Turbo Puffer, we automatically build this index behind the scenes once you get enough data to to justify it.

s117
00:09:57.920 --> 00:10:05.920
Vector embeddings, if you're not familiar, are the internal numeric representation of concepts in LLM space.

s118
00:10:05.920 --> 00:10:09.760
And we like to use this example of red shoes and maroon sandals.

s119
00:10:09.760 --> 00:10:16.120
You as a human know that these are very similar concepts and much more similar to one another than concrete block is.

s120
00:10:16.120 --> 00:10:20.920
And maybe you're thinking about shape and size and texture and and weight.

s121
00:10:20.920 --> 00:10:24.280
The LLM can translate that into this numeric vector.

s122
00:10:24.280 --> 00:10:31.560
Here, I've only got two dimensions shown so that we can think two-dimensionally, but in practice, there's often thousands of dimension.

s123
00:10:31.560 --> 00:10:34.320
1024 dimensions is a very common size.

s124
00:10:34.320 --> 00:10:36.720
Thinking multi-dimensionally like that is very hard.

s125
00:10:36.720 --> 00:10:40.880
So, for the sake of example, we're just going to imagine that there are two dimensions here.

s126
00:10:40.880 --> 00:10:45.760
Now, we don't actually know what those dimensions represent, but again, you can think of it as like size and color.

s127
00:10:45.760 --> 00:10:48.440
And then you can plot that on a coordinate system.

s128
00:10:48.440 --> 00:10:56.317
So, red shoes might be at the top left along with maroon sandals, and then concrete block is going to be down at the bottom right.

s129
00:10:56.317 --> 00:10:56.600
[snorts]

s130
00:10:56.600 --> 00:11:04.880
And I'm going to simplify wildly here, but a very workable vector index in two dimensions is just to plot things on a graph.

s131
00:11:04.880 --> 00:11:06.080
And then you can zoom in.

s132
00:11:06.080 --> 00:11:16.680
If you're looking for red shoes, that's the Q on this diagram, you plot that on this two-dimensional graph, and you zoom in, and you look for the vectors around it.

s133
00:11:16.680 --> 00:11:25.920
And because of the way these vectors are created by the LLM, vectors that are near each other represent similar concepts, represent good search results to return.

s134
00:11:25.920 --> 00:11:28.960
So, if we were just doing similarity search, great.

s135
00:11:28.960 --> 00:11:36.400
We look in our two-dimensional coordinate grid, we find some nearby vectors, we return them, and life is good.

s136
00:11:36.400 --> 00:11:37.760
And this saves us a lot of time.

s137
00:11:37.760 --> 00:11:39.400
We don't have to search the whole vector space.

s138
00:11:39.400 --> 00:11:42.880
We can zoom right in to the documents that are going to be relevant to the query.

s139
00:11:42.880 --> 00:11:45.960
So, that napkin math from earlier, we can we can ignore.

s140
00:11:45.960 --> 00:11:50.683
We're maybe only looking at 10,000 vectors here, and we know we can do that in a millisecond or two.

s141
00:11:50.683 --> 00:11:50.800
[snorts]

s142
00:11:50.800 --> 00:11:59.560
The problem is when you add in these permissions, not all of the documents that are most relevant for a given query are going to match those permissions.

s143
00:11:59.560 --> 00:12:08.240
And you can see in this diagram here, the minuses don't match the permissions, while the pluses do match the permissions.

s144
00:12:08.560 --> 00:12:11.920
So, our vector index is not immediately helpful to us.

s145
00:12:11.920 --> 00:12:18.560
And we've invested a lot of time and energy at Turbo Puffer into solving this problem.

s146
00:12:19.320 --> 00:12:24.600
Naively, you might try a post-filtering strategy.

s147
00:12:24.600 --> 00:12:33.880
And the way that works is the query comes in, you get maybe the 10,000 closest vectors, and then you throw away all the ones that don't match.

s148
00:12:33.880 --> 00:12:40.680
This does not work very well because it doesn't provide the results that you're looking for.

s149
00:12:40.680 --> 00:12:50.360
If I'm the user that only has access to that data in the bottom right, and I search for Q, I still want to see the documents that were closest

s150
00:12:50.360 --> 00:12:54.480
to Q from my perspective, that include the results that I can see.

s151
00:12:54.480 --> 00:13:03.920
But when you do post-filtering, you kind of spiral outwards, and then at some point you have to give up because it's been a couple seconds and you haven't found any docs that match, or you haven't found

s152
00:13:03.920 --> 00:13:05.200
the top 100 docs that match.

s153
00:13:05.200 --> 00:13:06.880
Maybe you were only able to find 40.

s154
00:13:06.880 --> 00:13:15.760
And this makes users unhappy because they might know that a doc exists with maroon sandals, and they were searching for red shoes, and you don't turn it up.

s155
00:13:15.760 --> 00:13:18.560
So, you have very poor recall.

s156
00:13:20.000 --> 00:13:26.800
So, instead of starting with the vector index, what if we start with the filters?

s157
00:13:26.880 --> 00:13:31.040
Turbo Puffer builds an index of all of the documents that match the filters.

s158
00:13:31.040 --> 00:13:37.120
So, Aaron can show up and say, "Hey, show me all of the documents that I have access to."

s159
00:13:37.360 --> 00:13:42.760
And then Turbopuffer can scan over all of those documents and see which of them are closest to the query vector.

s160
00:13:42.760 --> 00:13:48.480
But what if Aaron has access to 50 million out of the 100 million documents in the vault?

s161
00:13:48.480 --> 00:13:49.880
Now we're back to square one.

s162
00:13:49.880 --> 00:13:56.640
We're back in that napkin math problem where it's going to take multiple seconds, if not minutes, to get through all of that data.

s163
00:13:56.640 --> 00:13:58.680
We're not actually using the vector index.

s164
00:13:58.680 --> 00:14:04.600
If you look here, you can see we're just finding all the docs that match the filters and then scanning through them sequentially.

s165
00:14:04.600 --> 00:14:08.000
So, this also doesn't work very well.

s166
00:14:08.960 --> 00:14:14.560
And the trick to doing well on this problem is to do both.

s167
00:14:14.560 --> 00:14:18.880
So, Turbopuffer's vector index automatically clusters similar vectors.

s168
00:14:18.880 --> 00:14:25.040
So, you can see there's a group of documents here that are similar labeled five and another group labeled zero and another group labeled one.

s169
00:14:25.040 --> 00:14:40.440
And when we build these permissions indexes, we say, "Well, Aaron has access to documents in cluster five and maybe cluster two, but he doesn't have access to any documents in cluster three or any documents in cluster zero."

s170
00:14:40.440 --> 00:14:42.280
So, we can just throw those clusters out.

s171
00:14:42.280 --> 00:14:45.880
We don't have to consider them at all.

s172
00:14:45.880 --> 00:14:57.160
So, first we find all of the clusters that Aaron has access to, uh i.e. he has access to at least one doc in those clusters, and then we search the closest clusters,

s173
00:14:57.200 --> 00:14:59.839
and then we just scan sequentially inside those clusters.

s174
00:14:59.839 --> 00:15:02.880
And in Turbopuffer right now, the clusters have about 200 documents.

s175
00:15:02.880 --> 00:15:04.520
This is something we're always tuning.

s176
00:15:04.520 --> 00:15:08.520
Over time, we'll probably increase that number to to maximize throughput.

s177
00:15:08.520 --> 00:15:15.920
But this basic strategy fundamentally makes search at scale with permissions possible.

s178
00:15:15.920 --> 00:15:22.920
And the trick is to combine pre- and post-filtering into what we we native filtering.

s179
00:15:23.680 --> 00:15:25.960
And this is the result.

s180
00:15:25.960 --> 00:15:35.560
Across 450 million vectors, those 100 million documents turn into 400 million vectors, TurboPuffer is able to get 120 millisecond searches.

s181
00:15:35.560 --> 00:15:39.040
That's the P90 latency across all of those namespaces.

s182
00:15:39.040 --> 00:15:44.360
By employing this sophisticated native filtering strategy.

s183
00:15:47.520 --> 00:16:00.240
So, with TurboPuffer, you show up with your permissions logic and your conceptual model, and you do what I think is a very simple translation of those permissions into TurboPuffer filter language.

s184
00:16:00.240 --> 00:16:08.960
And hopefully, you can directly map this code sample here, viewers contains ones viewers contains one groups contains any BC.

s185
00:16:08.960 --> 00:16:14.320
Hopefully, you can see how you would project that onto your organization's own permission structure.

s186
00:16:14.320 --> 00:16:21.920
And that turns into this very complicated systems engineering challenge under the hood, but TurboPuffer hides that all away from you.

s187
00:16:21.920 --> 00:16:22.960
So that you don't have to think about it.

s188
00:16:22.960 --> 00:16:27.600
You can just express the logical filters that represent your permissions in a fairly direct way.

s189
00:16:27.600 --> 00:16:38.400
And we handle the complicated query planning under the hood to make that search come back in 1 to 200 milliseconds even at 100 million documents scale.

s190
00:16:38.400 --> 00:16:41.760
Now, we're engineers at TurboPuffer, so there are some limitations here.

s191
00:16:41.760 --> 00:16:44.960
In particular, when you have nested groups, things get tricky.

s192
00:16:44.960 --> 00:16:50.960
I'm not going to have time to go into that in too much detail, but if you have problems that look like that, please come talk to us.

s193
00:16:50.960 --> 00:16:59.160
Uh we have a lot of experience working through these, and we're always looking for ways to improve the product so that stuff is handled more automatically.

s194
00:17:00.440 --> 00:17:04.880
So, permission filters are really just the tip of the iceberg in TurboPuffer.

s195
00:17:04.880 --> 00:17:07.680
We're trying to build an isolation native search engine.

s196
00:17:07.680 --> 00:17:21.800
That means whether you have fine-grained permissions within namespaces, whether you have security, regulatory compliance obligations that force you to regionally lock data, keep that data in your own VPC,

s197
00:17:21.800 --> 00:17:27.360
encrypt that data potentially not just with an encryption key you control, but that your own customers control.

s198
00:17:27.360 --> 00:17:31.160
Turbo Buffer has solutions for all of those levels of isolation.

s199
00:17:31.160 --> 00:17:39.880
So, permission filtering is just one piece of the puzzle in building isolation-aware search in Turbo Buffer.

s200
00:17:40.440 --> 00:17:41.600
So, that's all we got.

s201
00:17:41.600 --> 00:17:48.480
That's how Turbo Buffer and Bridgewater work together to do fast isolation-native search at 100 million scale.

s202
00:17:48.480 --> 00:17:49.627
Thank you, everyone.

s203
00:17:49.627 --> 00:17:51.627
[applause]