{ "video": { "id": "u_jvuCjuvEk", "title": "Can Your AI Analyst Keep a Secret? - Aaron Linsky, Bridgewater & Nikhil Benesch, turbopuffer", "duration": 1077, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "u_jvuCjuvEk", "title": "Can Your AI Analyst Keep a Secret? - Aaron Linsky, Bridgewater & Nikhil Benesch, turbopuffer", "one_liner": "Permission-aware vector search over Bridgewater's 100M-document 'Vault' works only when you combine pre- and post-filtering. turbopuffer calls this 'native filtering': it skips vector clusters that hold no documents the user can see, then scans only the nearest permitted clusters, and gets 120ms P90 across roughly 400–450M vectors.", "summary": "An AI analyst that pulls in private company documents has to respect a different set of permissions on every document, which is a classically hard information-retrieval problem. Bridgewater's first permission-filtered vector query worked at first, then 'fell over tremendously' as the Vault grew, so they moved to turbopuffer. Nikhil Benesch uses napkin math to show why brute-force scanning, post-filtering and pre-filtering each fail at 100M-document scale. He then explains how turbopuffer's native filtering combines its cluster-based vector index with permission indexes to keep searches in the 100–200ms range.", "key_points": [ "Bridgewater's research loop has five steps (perception, hypothesis, investigation, systematize, industrialize) and has always run with humans at the center. The firm is now moving to put machines at the center. Its rules started as a 1980 bond system of if-then statements on a yellow legal pad and moved through Excel and Scala to Python.", "Bridgewater has two core data stores. Delorean is a bitemporal macro time-series database built over decades. The Vault is a newer unstructured store of about 100M documents drawn from 135 external sources plus internal writing, meeting recordings and reports, and it has per-source permissions. Agents and investors also write new content back into the Vault.", "The permission model, heavily simplified: a user (e.g. user 1 in groups B and C) can reach a document either through a group or directly. In turbopuffer's filter language this is 'viewers contains 1' or 'groups contains any [B, C]'.", "Napkin math: at 1KB per document (a 1,024-dimension embedding at one byte per dimension), scanning 100M docs from memory at about 20GB/s takes 5 seconds, against a typical target of about 100ms. Pulling the data cold from S3 takes a minute or more. 10K docs takes half a millisecond.", "Post-filtering (take the ~10,000 nearest vectors, then drop the ones the user can't see) gives poor recall. The search spirals outward and gives up, for example returning 40 results when 100 were wanted.", "Pre-filtering (list every document the user can access, then scan them) fails when a user can see something like 50M of the 100M documents, because you are back to a brute-force scan.", "Native filtering: turbopuffer's index clusters similar vectors (currently about 200 documents per cluster, and still being tuned upward). Permission indexes record which clusters contain at least one document the user can access. Search drops the other clusters, picks the closest permitted ones and scans inside them sequentially. Result: 120ms P90 across roughly 400–450M vectors.", "turbopuffer is built on object storage (S3, GCS, Azure Blob). Data sits cold in object storage and is pulled into SSD or memory only when queried. They claim about a tenth of the cost of other systems once the data is hot. It supports keyword, regex and vector search. Nested groups are a known limitation, and the wider aim is 'isolation-native' search: regional data locking, deployment in your own VPC, and customer-controlled encryption keys." ], "takeaways": [ "Do napkin math (document size × count ÷ memory bandwidth) before choosing a search architecture. Brute force is fine around 10K docs and hopeless at 10–100M.", "Don't build permission-aware RAG on plain post-filtering: recall collapses for users who can see only a small slice of the corpus. Pre-filtering alone fails for users who can see most of it.", "Express document ACLs as simple filter predicates (direct viewers OR group membership) and let the search engine combine those filters with the vector index, rather than handling them in application code.", "Test permission-filtered search at realistic corpus size early. Bridgewater's first implementation seemed fine and then failed as documents were added.", "Expect nested group hierarchies to be the hard case, and plan for them explicitly." ], "topics": [ "rag", "vector search", "permissions", "access control", "information retrieval", "ai agents", "search infrastructure", "finance" ], "tools": [ "turbopuffer", "Bridgewater Associates", "Delorean", "the Vault", "Amazon S3", "Google Cloud Storage", "Azure Blob Storage", "Excel", "Scala", "Python" ], "quotes": [ { "text": "when we first implemented this without Turbo Buffer, we thought it worked fine. Um but then, pretty quickly, as we started to add more and more documents to the vault, it fell over um tremendously.", "at": "06:02", "url": "https://www.youtube.com/watch?v=u_jvuCjuvEk&t=362s" }, { "text": "Once you get up to 100 million docs, that's going to take 5 seconds. And for most search workloads, that's way too slow.", "at": "09:00", "url": "https://www.youtube.com/watch?v=u_jvuCjuvEk&t=540s" }, { "text": "And the trick is to combine pre- and post-filtering into what we we native filtering.", "at": "15:15", "url": "https://www.youtube.com/watch?v=u_jvuCjuvEk&t=915s" }, { "text": "We're trying to build an isolation native search engine.", "at": "17:03", "url": "https://www.youtube.com/watch?v=u_jvuCjuvEk&t=1023s" } ], "words": 3571 }, "summary_url": "/#u_jvuCjuvEk", "transcript": { "html": "/transcripts/u_jvuCjuvEk.html", "txt": "/transcripts/u_jvuCjuvEk.txt", "vtt": "/transcripts/u_jvuCjuvEk.vtt" } }