Can Your AI Analyst Keep a Secret? - Aaron Linsky, Bridgewater & Nikhil Benesch, turbopuffer

AI Engineer · 17 min · 203 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12All right.
00:13Hello, everyone.
00:14Thank you all for coming.
00:15I'm Nick Hill, CTO of Turbo Puffer.
00:18And I'm Aaron Linsky, the head of product engineering at Bridgewater Associates.
00:23And we're here to talk to you today about how you build an AI analyst that can keep a secret.
00:28And I'll explain a bit more about what that means.
00:31Here.
00:32Now, this is a very oversimplified view of how you would go about building an AI analyst, but it is meant to capture an important dimension around permissions.
00:41So, in this diagram, you're user interacting with this AI agent, you start with a question, maybe something like, "Why have token prices come down so much in the last month?"
00:52And you feed that question into an LLM, which will begin thinking about that question, investigating, researching.
00:57And as it does that investigation, it's going to search for facts to ground its information, to lead its investigation.
01:05Some of those facts might come from the public web, but other facts might come from private documents in your organization.
01:13Now, the public web is on one dimension much easier to search than the information that's private to your organization.
01:19By definition, the public web is public.
01:21Everyone in your organization can see exactly the same documents.
01:24So, there's no permission issue there.
01:26Every public web document can be immediately fed into the AI analyst, no matter who's asking the question.
01:32But the information internal to your organization looks very different.
01:36Every email, every meeting note, every report might have a completely different set of permissions attached to it, belonging to different groups, maybe individual permissions assignments on those docs.
01:45And you can see here, we've got different colors here reflecting the fact that every document in that private data set has a different set of permissions.
01:52And this is a classically hard problem in information retrieval, and it's something we explicitly designed Turbo Puffer to solve.
02:00But before I get into that, I'm going to hand it over to Aaron to talk about why this private information is so important to Bridgewater's AI analyst.
02:07Great.
02:08Thank you.
02:08Oh.
02:10There we go.
02:10So, first I'm going to go talk a little bit about what who Bridgewater is, what we do, and then we get into how how we're using Turbo Puffer and and the needs there.
02:22So, for the past 50 years, we at Bridgewater have pursued the best understanding of how markets and economies work.
02:31We then encode that systematically with logic into a system that we can then stress test over time.
02:38What you see here are on the on the left with the yellow legal pad, that is our bond system from 1980.
02:45It's a set of if-then statements that we used to trade bonds back then.
02:50Over the next few decades, we've evolved that process from a yellow legal pad to Excel to eventually Scala and now Python um to to a different learning process with humans at the center.
03:08This five This research circle, as what we call it, has five steps.
03:12Perception, that's seeing what's going on the outside world.
03:16Hypothesis, which is coming up with ideas of why that might be true.
03:21Investigation, which is the research of seeing is that thing true.
03:25If it is true, we then systematize and integrate it with our system and industrialize and run that every day and trade on that.
03:31That's been our learning process.
03:33I've been at Bridgewater for about 20 years.
03:34That's been what we've been doing doing for 20 years with humans at the center.
03:38We're now moving towards putting machines at the center and running that process.
03:43In order for even you know, for both humans and machines, in order for that to work well, we need access to high-quality data.
03:52We need both time series data, which we spent decades uh decades creating a great bi-temporal time macro time series database, which we call Delorean.
04:04And more recently, we've been creating a unstructured database, which we call the Vault.
04:11And so, what is the Vault?
04:14Um today, it's about 100 million documents, which we index, organize, and secure that we think represents some of the best third-party and extra third-party external research, and also our internal research.
04:28So, we've sourced with 135 different external sources, which is a huge deal for both legal and our data team to actually nego- negotiate and get them in.
04:38And all of all of those sources have um some sorts of permissions around it where it might not be open to everyone at Bridgewater.
04:46And then, of course, we also have our inter- internal sources of unstructured data.
04:50That's our own writing, our meeting recordings, and and our internal reports, as well.
04:55And both that those all go into the Vault, and then both humans investors and our AI agents read out of the Vault.
05:02What I haven't shown here is actually it circles back.
05:04So, the investors and and the agents will create new new content that goes back into the Vault.
05:12So, it's constantly improving.
05:14And so, here's a very high-level view of what permissions in the Vault look like.
05:20So, I have a user we'll just say their ID is user one.
05:25They're in groups B and C. This is incre- incredibly simplified, unfortunately.
05:30Um and there's a set of all of our documents where it could be either um uh permission to them via groups or directly.
05:42And then, here we have the actual code of us going ahead and creating a query to try to look for some data with whatever that query vector was that matches
05:54the viewers and groups.
05:56And so, this is simple in one way.
06:01And actually, when we first implemented this without Turbo Buffer, we thought it worked fine.
06:06Um but then, pretty quickly, as we started to add more and more documents to the vault, it fell over um tremendously.
06:14And so, what I did then was I looked for other possible providers.
06:18We found Turbo Buffer, tested them out, and they had a solution which Nikhil is going to go over right now.
06:25Awesome.
06:25Thank you, Aaron.
06:29Let me very quickly talk about what Turbo Buffer is.
06:33We are an extremely fast, extremely scalable, extremely cheap search engine built entirely on top of object storage.
06:40And what that means is we write directly to S3 or GCS or Azure Blob storage, whatever your hyperscaler provides.
06:47And that gives us S3 economics.
06:50So, by default, your data is inert, cold on S3.
06:53Only when you start querying it do we pull it into more expensive storage mediums, like SSD or memory.
07:00And this is really powerful because once a query is hot, once the data query is accessing is in the cache, it's fast.
07:08But if you're not accessing a particular namespace or a particular region of a namespace, that data can just hang out on object storage where you pay rock-bottom prices for it.
07:17So, this architecture allows Turbo Buffer to scale cheaply to terabytes of data, which would cost tens of millions of dollars in in other systems, and we can often do it at a tenth of the cost
07:28at performance that's comparable once you get the data hot uh to what you get with traditional disk-based search systems.
07:35And we support all types of search.
07:37We support keyword search.
07:39We support regex search.
07:40And what Aaron was showing earlier, vector search.
07:44Um so we're really not uh opinionated about how you search your data in Turbo Puffer.
07:49We want to do search generically.
07:52But for the rest of this talk, I'm going to focus on vector search because uh it's it's good to zoom in and go deep on one particular type of search.
08:01Let's do a little bit of napkin math.
08:04That's something we love to do at Turbo Puffer.
08:06It helps us anchor in the simplest possible solution.
08:09We never want to introduce complexity to Turbo Puffer that's not justified.
08:13One of the justifications for taking on complexity is if you need it for performance.
08:18The absolute simplest way to do search is to put everything in a list and every time somebody shows up saying, "I want queries that match this.
08:25I want documents that match this query."
08:27You look at every single piece of data.
08:30Let's conservatively say that every document that you store in Turbo Puffer is 1 kilobyte.
08:36This is incredibly conservative.
08:38A vector embedding is often 1,024 dimensions.
08:41So if you're using just a byte per dimension, that's a kilobyte right there.
08:45Uh in practice, documents are often larger than this.
08:47But for the sake of example, let's say it's a kilobyte per doc.
08:51Even if you had all of that data resident in memory and you were ripping through it as fast as you possibly could, which on modern hardware is about 20 gigabytes a second.
09:00Once you get up to 100 million docs, that's going to take 5 seconds.
09:03And for most search workloads, that's way too slow.
09:06What we see in practice is most people want their search results to be about 100 milliseconds.
09:10[clears throat] So the naive solution here does not work.
09:13If you've only got 10K docs, great.
09:15That's going to take half a millisecond.
09:16Turbo Puffer can do literally anything and it will be fast enough.
09:19But as you get to scale, once you're at 10 million or 100 million, you really need to start thinking about how you represent your data so that you're not crunching through all of that.
09:28And of course, I gave you the the numbers for memory here.
09:31That's only if the cache is hot.
09:33If that data's on S3, you're looking at a minute or more to pull that data off S3.
09:37So, this just isn't viable at the scale that Bridgewater operates.
09:43So, step one in Turbo Puffer is to build a vector index.
09:47And this is something that we handle under the hood for you.
09:49When you store vectors in Turbo Puffer, we automatically build this index behind the scenes once you get enough data to to justify it.
09:57Vector embeddings, if you're not familiar, are the internal numeric representation of concepts in LLM space.
10:05And we like to use this example of red shoes and maroon sandals.
10:09You as a human know that these are very similar concepts and much more similar to one another than concrete block is.
10:16And maybe you're thinking about shape and size and texture and and weight.
10:20The LLM can translate that into this numeric vector.
10:24Here, I've only got two dimensions shown so that we can think two-dimensionally, but in practice, there's often thousands of dimension.
10:311024 dimensions is a very common size.
10:34Thinking multi-dimensionally like that is very hard.
10:36So, for the sake of example, we're just going to imagine that there are two dimensions here.
10:40Now, we don't actually know what those dimensions represent, but again, you can think of it as like size and color.
10:45And then you can plot that on a coordinate system.
10:48So, red shoes might be at the top left along with maroon sandals, and then concrete block is going to be down at the bottom right.
10:56[snorts]
10:56And I'm going to simplify wildly here, but a very workable vector index in two dimensions is just to plot things on a graph.
11:04And then you can zoom in.
11:06If you're looking for red shoes, that's the Q on this diagram, you plot that on this two-dimensional graph, and you zoom in, and you look for the vectors around it.
11:16And because of the way these vectors are created by the LLM, vectors that are near each other represent similar concepts, represent good search results to return.
11:25So, if we were just doing similarity search, great.
11:28We look in our two-dimensional coordinate grid, we find some nearby vectors, we return them, and life is good.
11:36And this saves us a lot of time.
11:37We don't have to search the whole vector space.
11:39We can zoom right in to the documents that are going to be relevant to the query.
11:42So, that napkin math from earlier, we can we can ignore.
11:45We're maybe only looking at 10,000 vectors here, and we know we can do that in a millisecond or two.
11:50[snorts]
11:50The problem is when you add in these permissions, not all of the documents that are most relevant for a given query are going to match those permissions.
11:59And you can see in this diagram here, the minuses don't match the permissions, while the pluses do match the permissions.
12:08So, our vector index is not immediately helpful to us.
12:11And we've invested a lot of time and energy at Turbo Puffer into solving this problem.
12:19Naively, you might try a post-filtering strategy.
12:24And the way that works is the query comes in, you get maybe the 10,000 closest vectors, and then you throw away all the ones that don't match.
12:33This does not work very well because it doesn't provide the results that you're looking for.
12:40If I'm the user that only has access to that data in the bottom right, and I search for Q, I still want to see the documents that were closest
12:50to Q from my perspective, that include the results that I can see.
12:54But when you do post-filtering, you kind of spiral outwards, and then at some point you have to give up because it's been a couple seconds and you haven't found any docs that match, or you haven't found
13:03the top 100 docs that match.
13:05Maybe you were only able to find 40.
13:06And this makes users unhappy because they might know that a doc exists with maroon sandals, and they were searching for red shoes, and you don't turn it up.
13:15So, you have very poor recall.
13:20So, instead of starting with the vector index, what if we start with the filters?
13:26Turbo Puffer builds an index of all of the documents that match the filters.
13:31So, Aaron can show up and say, "Hey, show me all of the documents that I have access to."
13:37And then Turbopuffer can scan over all of those documents and see which of them are closest to the query vector.
13:42But what if Aaron has access to 50 million out of the 100 million documents in the vault?
13:48Now we're back to square one.
13:49We're back in that napkin math problem where it's going to take multiple seconds, if not minutes, to get through all of that data.
13:56We're not actually using the vector index.
13:58If you look here, you can see we're just finding all the docs that match the filters and then scanning through them sequentially.
14:04So, this also doesn't work very well.
14:08And the trick to doing well on this problem is to do both.
14:14So, Turbopuffer's vector index automatically clusters similar vectors.
14:18So, you can see there's a group of documents here that are similar labeled five and another group labeled zero and another group labeled one.
14:25And when we build these permissions indexes, we say, "Well, Aaron has access to documents in cluster five and maybe cluster two, but he doesn't have access to any documents in cluster three or any documents in cluster zero."
14:40So, we can just throw those clusters out.
14:42We don't have to consider them at all.
14:45So, first we find all of the clusters that Aaron has access to, uh i.e. he has access to at least one doc in those clusters, and then we search the closest clusters,
14:57and then we just scan sequentially inside those clusters.
14:59And in Turbopuffer right now, the clusters have about 200 documents.
15:02This is something we're always tuning.
15:04Over time, we'll probably increase that number to to maximize throughput.
15:08But this basic strategy fundamentally makes search at scale with permissions possible.
15:15And the trick is to combine pre- and post-filtering into what we we native filtering.
15:23And this is the result.
15:25Across 450 million vectors, those 100 million documents turn into 400 million vectors, TurboPuffer is able to get 120 millisecond searches.
15:35That's the P90 latency across all of those namespaces.
15:39By employing this sophisticated native filtering strategy.
15:47So, with TurboPuffer, you show up with your permissions logic and your conceptual model, and you do what I think is a very simple translation of those permissions into TurboPuffer filter language.
16:00And hopefully, you can directly map this code sample here, viewers contains ones viewers contains one groups contains any BC.
16:08Hopefully, you can see how you would project that onto your organization's own permission structure.
16:14And that turns into this very complicated systems engineering challenge under the hood, but TurboPuffer hides that all away from you.
16:21So that you don't have to think about it.
16:22You can just express the logical filters that represent your permissions in a fairly direct way.
16:27And we handle the complicated query planning under the hood to make that search come back in 1 to 200 milliseconds even at 100 million documents scale.
16:38Now, we're engineers at TurboPuffer, so there are some limitations here.
16:41In particular, when you have nested groups, things get tricky.
16:44I'm not going to have time to go into that in too much detail, but if you have problems that look like that, please come talk to us.
16:50Uh we have a lot of experience working through these, and we're always looking for ways to improve the product so that stuff is handled more automatically.
17:00So, permission filters are really just the tip of the iceberg in TurboPuffer.
17:04We're trying to build an isolation native search engine.
17:07That means whether you have fine-grained permissions within namespaces, whether you have security, regulatory compliance obligations that force you to regionally lock data, keep that data in your own VPC,
17:21encrypt that data potentially not just with an encryption key you control, but that your own customers control.
17:27Turbo Buffer has solutions for all of those levels of isolation.
17:31So, permission filtering is just one piece of the puzzle in building isolation-aware search in Turbo Buffer.
17:40So, that's all we got.
17:41That's how Turbo Buffer and Bridgewater work together to do fast isolation-native search at 100 million scale.
17:48Thank you, everyone.
17:49[applause]