Can Your AI Analyst Keep a Secret? - Aaron Linsky, Bridgewater & Nikhil Benesch, turbopuffer
https://www.youtube.com/watch?v=u_jvuCjuvEk
AI Engineer · 17 min · 203 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] All right.
[00:13] Hello, everyone.
[00:14] Thank you all for coming.
[00:15] I'm Nick Hill, CTO of Turbo Puffer.
[00:18] And I'm Aaron Linsky, the head of product engineering at Bridgewater Associates.
[00:23] And we're here to talk to you today about how you build an AI analyst that can keep a secret.
[00:28] And I'll explain a bit more about what that means.
[00:31] Here.
[00:32] Now, this is a very oversimplified view of how you would go about building an AI analyst, but it is meant to capture an important dimension around permissions.
[00:41] So, in this diagram, you're user interacting with this AI agent, you start with a question, maybe something like, "Why have token prices come down so much in the last month?"
[00:52] And you feed that question into an LLM, which will begin thinking about that question, investigating, researching.
[00:57] And as it does that investigation, it's going to search for facts to ground its information, to lead its investigation.
[01:05] Some of those facts might come from the public web, but other facts might come from private documents in your organization.
[01:13] Now, the public web is on one dimension much easier to search than the information that's private to your organization.
[01:19] By definition, the public web is public.
[01:21] Everyone in your organization can see exactly the same documents.
[01:24] So, there's no permission issue there.
[01:26] Every public web document can be immediately fed into the AI analyst, no matter who's asking the question.
[01:32] But the information internal to your organization looks very different.
[01:36] Every email, every meeting note, every report might have a completely different set of permissions attached to it, belonging to different groups, maybe individual permissions assignments on those docs.
[01:45] And you can see here, we've got different colors here reflecting the fact that every document in that private data set has a different set of permissions.
[01:52] And this is a classically hard problem in information retrieval, and it's something we explicitly designed Turbo Puffer to solve.
[02:00] But before I get into that, I'm going to hand it over to Aaron to talk about why this private information is so important to Bridgewater's AI analyst.
[02:07] Great.
[02:08] Thank you.
[02:08] Oh.
[02:10] There we go.
[02:10] So, first I'm going to go talk a little bit about what who Bridgewater is, what we do, and then we get into how how we're using Turbo Puffer and and the needs there.
[02:22] So, for the past 50 years, we at Bridgewater have pursued the best understanding of how markets and economies work.
[02:31] We then encode that systematically with logic into a system that we can then stress test over time.
[02:38] What you see here are on the on the left with the yellow legal pad, that is our bond system from 1980.
[02:45] It's a set of if-then statements that we used to trade bonds back then.
[02:50] Over the next few decades, we've evolved that process from a yellow legal pad to Excel to eventually Scala and now Python um to to a different learning process with humans at the center.
[03:08] This five This research circle, as what we call it, has five steps.
[03:12] Perception, that's seeing what's going on the outside world.
[03:16] Hypothesis, which is coming up with ideas of why that might be true.
[03:21] Investigation, which is the research of seeing is that thing true.
[03:25] If it is true, we then systematize and integrate it with our system and industrialize and run that every day and trade on that.
[03:31] That's been our learning process.
[03:33] I've been at Bridgewater for about 20 years.
[03:34] That's been what we've been doing doing for 20 years with humans at the center.
[03:38] We're now moving towards putting machines at the center and running that process.
[03:43] In order for even you know, for both humans and machines, in order for that to work well, we need access to high-quality data.
[03:52] We need both time series data, which we spent decades uh decades creating a great bi-temporal time macro time series database, which we call Delorean.
[04:04] And more recently, we've been creating a unstructured database, which we call the Vault.
[04:11] And so, what is the Vault?
[04:14] Um today, it's about 100 million documents, which we index, organize, and secure that we think represents some of the best third-party and extra third-party external research, and also our internal research.
[04:28] So, we've sourced with 135 different external sources, which is a huge deal for both legal and our data team to actually nego- negotiate and get them in.
[04:38] And all of all of those sources have um some sorts of permissions around it where it might not be open to everyone at Bridgewater.
[04:46] And then, of course, we also have our inter- internal sources of unstructured data.
[04:50] That's our own writing, our meeting recordings, and and our internal reports, as well.
[04:55] And both that those all go into the Vault, and then both humans investors and our AI agents read out of the Vault.
[05:02] What I haven't shown here is actually it circles back.
[05:04] So, the investors and and the agents will create new new content that goes back into the Vault.
[05:12] So, it's constantly improving.
[05:14] And so, here's a very high-level view of what permissions in the Vault look like.
[05:20] So, I have a user we'll just say their ID is user one.
[05:25] They're in groups B and C. This is incre- incredibly simplified, unfortunately.
[05:30] Um and there's a set of all of our documents where it could be either um uh permission to them via groups or directly.
[05:42] And then, here we have the actual code of us going ahead and creating a query to try to look for some data with whatever that query vector was that matches
[05:54] the viewers and groups.
[05:56] And so, this is simple in one way.
[06:01] And actually, when we first implemented this without Turbo Buffer, we thought it worked fine.
[06:06] Um but then, pretty quickly, as we started to add more and more documents to the vault, it fell over um tremendously.
[06:14] And so, what I did then was I looked for other possible providers.
[06:18] We found Turbo Buffer, tested them out, and they had a solution which Nikhil is going to go over right now.
[06:25] Awesome.
[06:25] Thank you, Aaron.
[06:29] Let me very quickly talk about what Turbo Buffer is.
[06:33] We are an extremely fast, extremely scalable, extremely cheap search engine built entirely on top of object storage.
[06:40] And what that means is we write directly to S3 or GCS or Azure Blob storage, whatever your hyperscaler provides.
[06:47] And that gives us S3 economics.
[06:50] So, by default, your data is inert, cold on S3.
[06:53] Only when you start querying it do we pull it into more expensive storage mediums, like SSD or memory.
[07:00] And this is really powerful because once a query is hot, once the data query is accessing is in the cache, it's fast.
[07:08] But if you're not accessing a particular namespace or a particular region of a namespace, that data can just hang out on object storage where you pay rock-bottom prices for it.
[07:17] So, this architecture allows Turbo Buffer to scale cheaply to terabytes of data, which would cost tens of millions of dollars in in other systems, and we can often do it at a tenth of the cost
[07:28] at performance that's comparable once you get the data hot uh to what you get with traditional disk-based search systems.
[07:35] And we support all types of search.
[07:37] We support keyword search.
[07:39] We support regex search.
[07:40] And what Aaron was showing earlier, vector search.
[07:44] Um so we're really not uh opinionated about how you search your data in Turbo Puffer.
[07:49] We want to do search generically.
[07:52] But for the rest of this talk, I'm going to focus on vector search because uh it's it's good to zoom in and go deep on one particular type of search.
[08:01] Let's do a little bit of napkin math.
[08:04] That's something we love to do at Turbo Puffer.
[08:06] It helps us anchor in the simplest possible solution.
[08:09] We never want to introduce complexity to Turbo Puffer that's not justified.
[08:13] One of the justifications for taking on complexity is if you need it for performance.
[08:18] The absolute simplest way to do search is to put everything in a list and every time somebody shows up saying, "I want queries that match this.
[08:25] I want documents that match this query."
[08:27] You look at every single piece of data.
[08:30] Let's conservatively say that every document that you store in Turbo Puffer is 1 kilobyte.
[08:36] This is incredibly conservative.
[08:38] A vector embedding is often 1,024 dimensions.
[08:41] So if you're using just a byte per dimension, that's a kilobyte right there.
[08:45] Uh in practice, documents are often larger than this.
[08:47] But for the sake of example, let's say it's a kilobyte per doc.
[08:51] Even if you had all of that data resident in memory and you were ripping through it as fast as you possibly could, which on modern hardware is about 20 gigabytes a second.
[09:00] Once you get up to 100 million docs, that's going to take 5 seconds.
[09:03] And for most search workloads, that's way too slow.
[09:06] What we see in practice is most people want their search results to be about 100 milliseconds.
[09:10] [clears throat] So the naive solution here does not work.
[09:13] If you've only got 10K docs, great.
[09:15] That's going to take half a millisecond.
[09:16] Turbo Puffer can do literally anything and it will be fast enough.
[09:19] But as you get to scale, once you're at 10 million or 100 million, you really need to start thinking about how you represent your data so that you're not crunching through all of that.
[09:28] And of course, I gave you the the numbers for memory here.
[09:31] That's only if the cache is hot.
[09:33] If that data's on S3, you're looking at a minute or more to pull that data off S3.
[09:37] So, this just isn't viable at the scale that Bridgewater operates.
[09:43] So, step one in Turbo Puffer is to build a vector index.
[09:47] And this is something that we handle under the hood for you.
[09:49] When you store vectors in Turbo Puffer, we automatically build this index behind the scenes once you get enough data to to justify it.
[09:57] Vector embeddings, if you're not familiar, are the internal numeric representation of concepts in LLM space.
[10:05] And we like to use this example of red shoes and maroon sandals.
[10:09] You as a human know that these are very similar concepts and much more similar to one another than concrete block is.
[10:16] And maybe you're thinking about shape and size and texture and and weight.
[10:20] The LLM can translate that into this numeric vector.
[10:24] Here, I've only got two dimensions shown so that we can think two-dimensionally, but in practice, there's often thousands of dimension.
[10:31] 1024 dimensions is a very common size.
[10:34] Thinking multi-dimensionally like that is very hard.
[10:36] So, for the sake of example, we're just going to imagine that there are two dimensions here.
[10:40] Now, we don't actually know what those dimensions represent, but again, you can think of it as like size and color.
[10:45] And then you can plot that on a coordinate system.
[10:48] So, red shoes might be at the top left along with maroon sandals, and then concrete block is going to be down at the bottom right.
[10:56] [snorts]
[10:56] And I'm going to simplify wildly here, but a very workable vector index in two dimensions is just to plot things on a graph.
[11:04] And then you can zoom in.
[11:06] If you're looking for red shoes, that's the Q on this diagram, you plot that on this two-dimensional graph, and you zoom in, and you look for the vectors around it.
[11:16] And because of the way these vectors are created by the LLM, vectors that are near each other represent similar concepts, represent good search results to return.
[11:25] So, if we were just doing similarity search, great.
[11:28] We look in our two-dimensional coordinate grid, we find some nearby vectors, we return them, and life is good.
[11:36] And this saves us a lot of time.
[11:37] We don't have to search the whole vector space.
[11:39] We can zoom right in to the documents that are going to be relevant to the query.
[11:42] So, that napkin math from earlier, we can we can ignore.
[11:45] We're maybe only looking at 10,000 vectors here, and we know we can do that in a millisecond or two.
[11:50] [snorts]
[11:50] The problem is when you add in these permissions, not all of the documents that are most relevant for a given query are going to match those permissions.
[11:59] And you can see in this diagram here, the minuses don't match the permissions, while the pluses do match the permissions.
[12:08] So, our vector index is not immediately helpful to us.
[12:11] And we've invested a lot of time and energy at Turbo Puffer into solving this problem.
[12:19] Naively, you might try a post-filtering strategy.
[12:24] And the way that works is the query comes in, you get maybe the 10,000 closest vectors, and then you throw away all the ones that don't match.
[12:33] This does not work very well because it doesn't provide the results that you're looking for.
[12:40] If I'm the user that only has access to that data in the bottom right, and I search for Q, I still want to see the documents that were closest
[12:50] to Q from my perspective, that include the results that I can see.
[12:54] But when you do post-filtering, you kind of spiral outwards, and then at some point you have to give up because it's been a couple seconds and you haven't found any docs that match, or you haven't found
[13:03] the top 100 docs that match.
[13:05] Maybe you were only able to find 40.
[13:06] And this makes users unhappy because they might know that a doc exists with maroon sandals, and they were searching for red shoes, and you don't turn it up.
[13:15] So, you have very poor recall.
[13:20] So, instead of starting with the vector index, what if we start with the filters?
[13:26] Turbo Puffer builds an index of all of the documents that match the filters.
[13:31] So, Aaron can show up and say, "Hey, show me all of the documents that I have access to."
[13:37] And then Turbopuffer can scan over all of those documents and see which of them are closest to the query vector.
[13:42] But what if Aaron has access to 50 million out of the 100 million documents in the vault?
[13:48] Now we're back to square one.
[13:49] We're back in that napkin math problem where it's going to take multiple seconds, if not minutes, to get through all of that data.
[13:56] We're not actually using the vector index.
[13:58] If you look here, you can see we're just finding all the docs that match the filters and then scanning through them sequentially.
[14:04] So, this also doesn't work very well.
[14:08] And the trick to doing well on this problem is to do both.
[14:14] So, Turbopuffer's vector index automatically clusters similar vectors.
[14:18] So, you can see there's a group of documents here that are similar labeled five and another group labeled zero and another group labeled one.
[14:25] And when we build these permissions indexes, we say, "Well, Aaron has access to documents in cluster five and maybe cluster two, but he doesn't have access to any documents in cluster three or any documents in cluster zero."
[14:40] So, we can just throw those clusters out.
[14:42] We don't have to consider them at all.
[14:45] So, first we find all of the clusters that Aaron has access to, uh i.e. he has access to at least one doc in those clusters, and then we search the closest clusters,
[14:57] and then we just scan sequentially inside those clusters.
[14:59] And in Turbopuffer right now, the clusters have about 200 documents.
[15:02] This is something we're always tuning.
[15:04] Over time, we'll probably increase that number to to maximize throughput.
[15:08] But this basic strategy fundamentally makes search at scale with permissions possible.
[15:15] And the trick is to combine pre- and post-filtering into what we we native filtering.
[15:23] And this is the result.
[15:25] Across 450 million vectors, those 100 million documents turn into 400 million vectors, TurboPuffer is able to get 120 millisecond searches.
[15:35] That's the P90 latency across all of those namespaces.
[15:39] By employing this sophisticated native filtering strategy.
[15:47] So, with TurboPuffer, you show up with your permissions logic and your conceptual model, and you do what I think is a very simple translation of those permissions into TurboPuffer filter language.
[16:00] And hopefully, you can directly map this code sample here, viewers contains ones viewers contains one groups contains any BC.
[16:08] Hopefully, you can see how you would project that onto your organization's own permission structure.
[16:14] And that turns into this very complicated systems engineering challenge under the hood, but TurboPuffer hides that all away from you.
[16:21] So that you don't have to think about it.
[16:22] You can just express the logical filters that represent your permissions in a fairly direct way.
[16:27] And we handle the complicated query planning under the hood to make that search come back in 1 to 200 milliseconds even at 100 million documents scale.
[16:38] Now, we're engineers at TurboPuffer, so there are some limitations here.
[16:41] In particular, when you have nested groups, things get tricky.
[16:44] I'm not going to have time to go into that in too much detail, but if you have problems that look like that, please come talk to us.
[16:50] Uh we have a lot of experience working through these, and we're always looking for ways to improve the product so that stuff is handled more automatically.
[17:00] So, permission filters are really just the tip of the iceberg in TurboPuffer.
[17:04] We're trying to build an isolation native search engine.
[17:07] That means whether you have fine-grained permissions within namespaces, whether you have security, regulatory compliance obligations that force you to regionally lock data, keep that data in your own VPC,
[17:21] encrypt that data potentially not just with an encryption key you control, but that your own customers control.
[17:27] Turbo Buffer has solutions for all of those levels of isolation.
[17:31] So, permission filtering is just one piece of the puzzle in building isolation-aware search in Turbo Buffer.
[17:40] So, that's all we got.
[17:41] That's how Turbo Buffer and Bridgewater work together to do fast isolation-native search at 100 million scale.
[17:48] Thank you, everyone.
[17:49] [applause]