{
  "video": {
    "id": "V-isu4eTHgw",
    "title": "Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora",
    "duration": 1236,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "V-isu4eTHgw",
    "title": "Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora",
    "one_liner": "Legora went from one Elasticsearch cluster to per-region clusters to sharded Postgres/pgvector and finally to turbopuffer, where one namespace per project cut median search latency by an order of magnitude and made per-tenant encryption keys and data residency a config detail rather than a fleet of databases.",
    "summary": "Jacob Lauritzen walks through four generations of search architecture at Legora, an AI platform for legal work, as it scaled from hundreds of thousands to billions of documents under enterprise constraints: regional data residency, full physical isolation, and customer-managed encryption keys. Postgres with pgvector/DiskANN and 4,000 aggressive partitions worked until hot and cold projects landed in the same partitions and thrashed the cache, blowing P99 from 100ms to 20 seconds. Moving to turbopuffer with one namespace per project restored real BM25, collapsed the operational sprawl to a single cluster, and let cold projects simply sit in object storage. Simon Eskildsen explains the underlying design: writes go straight to S3, namespaces are the atomic unit of separation and encryption, and vector data is organised as a tree of clusters rather than a graph so it can live far down the memory hierarchy.",
    "key_points": [
      "Legora has two distinct search workloads: project search (a deal or matter scoped to one project, from tens to millions of documents) and legal research (a deep-research-style fan-out across laws, cases and regulations).",
      "Architecture history: single Elasticsearch cluster → one Elasticsearch per region (EU/US/APAC) for data residency → Postgres with pgvector, DiskANN and ts_vector (not BM25) → turbopuffer at roughly 400 million documents.",
      "Enterprise clients (big banks, the largest law firms) demand full physical isolation — effectively their own database — plus customer-managed encryption keys they can revoke to cut off decryption of their data at rest.",
      "The Postgres setup partitioned document chunks into ~4,000 partitions keyed by project; hot and cold projects landed in the same partitions, which grew large and thrashed the cache, pushing search and ingestion P99 from 100ms to 20 seconds.",
      "turbopuffer with one namespace per project gave real BM25, better relevancy, order-of-magnitude better median latencies (P99 even better), a single cluster to operate, and much lower cost because unused projects just sit in blob storage.",
      "turbopuffer writes go directly to object storage as a write-ahead log — no disk replication, no Paxos — trading hundreds of milliseconds of write latency for cheap, adaptable reads; indexes are built in the background.",
      "Because the namespace is the atomic unit of separation, each can have its own bucket and encryption key; some customers run thousands of buckets. For Legora, turbopuffer disabled the NVMe SSD cache entirely rather than building encryption into it, and memory-cache-only performance was good enough to keep it that way.",
      "Vector search in turbopuffer uses clusters of clusters (a tree, effectively a complicated B-tree over the vector space) instead of a graph, because graph traversal means repeated ~200ms S3 round trips; root centroids stay in DRAM while leaves sit on SSD with a single ~1ms round trip.",
      "Legal research is graph-like and needs heavy filtering — jurisdictional hierarchy (city/county/state/federal), temporal validity when one judge overrules another, and new regulations that carve exemptions from old ones — so one query fans out into many.",
      "Eskildsen's counterintuitive claim: at web scale, text search is more difficult and more computationally expensive than vector search.",
      "Legora is racing towards 10 billion vectors, runs 70+ (heading to 100–200) tenants, and treats the long tail of cold indices as acceptable at ~500ms cold-blob latency for deep-research workloads."
    ],
    "takeaways": [
      "Make the tenant/project the atomic unit of storage separation (a namespace or table), not a row key inside a shared partition — that's what makes per-customer encryption keys, buckets and data residency configuration rather than fleet duplication.",
      "Beware partitioning schemes that co-locate hot and cold tenants: bin-packing projects into fixed partitions caused Legora's 100ms → 20s P99 blowup when Postgres pulled whole partitions into memory.",
      "Object-storage-native search is the right trade when writes can be slow but reads must be cheap and elastic — cold namespaces cost almost nothing at rest, and a ~500ms cold fetch is fine for deep-research workloads.",
      "Latency compounds with agents: a single RAG call hides a slow query, but an agent issuing 20–100 queries does not, so treat median and P99 retrieval latency as an agent-scale budget.",
      "Putting vectors in Postgres (pgvector + DiskANN) is a defensible interim step if you already run Postgres for OLTP — but expect to lose BM25 quality (ts_vector instead) and to hit a cost and scaling wall."
    ],
    "topics": [
      "vector-search",
      "rag",
      "retrieval",
      "database-architecture",
      "multi-tenancy",
      "object-storage",
      "legaltech",
      "bm25"
    ],
    "tools": [
      "turbopuffer",
      "Legora",
      "Elasticsearch",
      "PostgreSQL",
      "pgvector",
      "DiskANN",
      "ts_vector",
      "BM25",
      "Amazon S3",
      "GCS",
      "Azure Blob Storage",
      "Shopify",
      "Cursor",
      "SpaceX"
    ],
    "quotes": [
      {
        "text": "When you do a write to Turbopuffer, we write directly to object storage. There is no like disk replication. There's no Paxos. There's none of that. Direct to S3.",
        "at": "08:01",
        "url": "https://www.youtube.com/watch?v=V-isu4eTHgw&t=481s"
      },
      {
        "text": "We went from like search and ingestion P99 of 100 milliseconds into 20 seconds, which you can imagine is a really bad user experience.",
        "at": "06:28",
        "url": "https://www.youtube.com/watch?v=V-isu4eTHgw&t=388s"
      },
      {
        "text": "We thought we were going to implement encryption into the disc cache but instead we just disabled the disc cache and saw how it fared and the performance of turbopuffer even without the disc cache with just the memory cache was so good that we just kept it that way.",
        "at": "11:05",
        "url": "https://www.youtube.com/watch?v=V-isu4eTHgw&t=665s"
      },
      {
        "text": "Counterintuitively to most people, text search at web scale is more difficult and more computationally expensive than doing vector search.",
        "at": "19:04",
        "url": "https://www.youtube.com/watch?v=V-isu4eTHgw&t=1144s"
      }
    ],
    "words": 4508
  },
  "summary_url": "/#V-isu4eTHgw",
  "transcript": {
    "html": "/transcripts/V-isu4eTHgw.html",
    "txt": "/transcripts/V-isu4eTHgw.txt",
    "vtt": "/transcripts/V-isu4eTHgw.vtt"
  }
}