{
  "video": {
    "id": "lyL5QhgIOxc",
    "title": "Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub — Arek Borucki, Hugging Face",
    "duration": 1299,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "lyL5QhgIOxc",
    "title": "Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub — Arek Borucki, Hugging Face",
    "one_liner": "Arek Borucki, ML platform/database engineer at Hugging Face, walks through the concrete infrastructure behind the Hub at 3 million models and 14 million users — precomputed search tokens plus Atlas Search on Lucene replacing $regex, a 7-node MongoDB replica set with read offloading and a hidden analytics node, and two-layer Kubernetes autoscaling — with sharding as the next step.",
    "summary": "Hugging Face went from 20,000 public models a few years ago to 3 million today (~150x), plus 1 million datasets and 14 million users, and the talk is about the architectural decisions that kept the Hub fast through that growth. The core argument is that approaches which work at small scale — regex search, everything on the primary, fixed capacity — break outright at Hub scale, so the team denormalized a read-optimized MongoDB collection, moved tokenization to insert time, switched full-text search to Atlas Search (Apache Lucene via the mongot process), pushed every query that can leave the primary onto secondaries or a hidden analytics node, and layered Cast AI node autoscaling beneath Kubernetes HPA. Borucki stresses P99 over P50 because 1% of 14 million users is 140,000 people hitting a slow search. Next steps are MongoDB sharding and migrating HPA to KEDA.",
    "key_points": [
      "Scale numbers: 14M+ users, 3M public models, 1M datasets, 50,000 organizations, and more than 30% of the Fortune 500 using Hugging Face in AI workflows; models went from 20K to 3M in a few years (~150x), datasets from 10K in 2022 to 100K in 2024 to 500K less than a year ago to 1M today. Every big release like Llama or DeepSeek spawns thousands of derivative models.",
      "MongoDB Atlas is the source of truth for metadata only — users, repositories, models, datasets, buckets, spaces info, configuration, billing, access control — while the actual model artifacts, tokenizer files, card assets and config files live in cloud object storage such as AWS S3, so metadata, binary storage and compute scale independently.",
      "Search reads do not hit the main repo collection: a separate denormalized read-only copy serves queries and listings. Model names are tokenized at insert time, not query time — 'meta-llama/Llama-3.1-8B' is split into tokens like 'meta', 'llama', '3.1', '8b' stored in an array on the document — and Atlas Search autocomplete matches against that array.",
      "The old implementation was a classical find() with the $regex operator over the search_tokens array, sorted by a trending score recomputed every 5 minutes from downloads and likes over roughly the last 7 days. Regex didn't scale with the dataset and latency became a problem; switching to an $search aggregation stage against the 'model_search' Lucene index removed the search-bar latency issues.",
      "MongoDB does not do full-text search in the core server — a separate mongot process wraps Apache Lucene, and it is transparent to the user through the unified MongoDB query API; using $search in an aggregation pipeline tells MongoDB to hit the Lucene index.",
      "Hundreds of Hugging Face services share a 7-node MongoDB cluster. All inserts/updates/deletes go to the single primary; reads are spread across secondaries — anything not needing the latest data, complex aggregations that scan/sort/group large data, and change streams (used for cache invalidation, syncing to stores like AWS Redshift, and event-driven workloads). One hidden analytics node replicates but is invisible to the driver and takes ad hoc, reporting and experimental queries, isolated from production traffic.",
      "Kubernetes has two autoscaling layers: the horizontal pod autoscaler takes the hub deployment from 10 to 500 pods on CPU/memory thresholds, and Cast AI adds nodes when pods are pending because the scheduler has no capacity — avoiding both manual intervention and overprovisioning.",
      "Planned work: MongoDB sharding to cut the dataset across shards (each with its own primary/secondaries, balanced by the MongoDB balancer) to scale CPU, memory, storage, reads and writes horizontally; and migrating from HPA to KEDA so scaling responds to real application metrics like requests per second or event loop utilization rather than resource utilization alone — a pod can have low CPU but a high request queue, which KEDA sees and HPA does not."
    ],
    "takeaways": [
      "Optimize for P99, not P50 — at 14 million users, the 1% tail is 140,000 people experiencing a slow product and leaving for somewhere else.",
      "Move work to insert time: precompute and store search tokens in an array on the document instead of paying tokenization and regex cost on every query.",
      "Stop using $regex for text search as the dataset grows; use Atlas Search / Apache Lucene with an autocomplete index via the $search aggregation stage.",
      "Keep a denormalized, read-optimized copy of your data for listings and search rather than querying the main write collection.",
      "Treat the primary as scarce: only strongly-consistent queries stay there. Push stale-tolerant reads, heavy aggregations and change streams to secondaries, and ad hoc/reporting queries to a hidden replica-set member that the driver never routes to.",
      "Separate metadata storage from binary artifact storage so each can be scaled and optimized for its own workload."
    ],
    "topics": [
      "infrastructure",
      "scaling",
      "mongodb",
      "search",
      "kubernetes",
      "autoscaling",
      "sharding",
      "databases"
    ],
    "tools": [
      "Hugging Face Hub",
      "MongoDB",
      "MongoDB Atlas",
      "Atlas Search",
      "mongot",
      "Apache Lucene",
      "Kubernetes",
      "Horizontal Pod Autoscaler (HPA)",
      "KEDA",
      "Cast AI",
      "AWS S3",
      "AWS Redshift",
      "Llama / Meta Llama 3.1 8B",
      "DeepSeek"
    ],
    "quotes": [
      {
        "text": "you would just leave and go somewhere else. And this is also what user are doing. They expect fast instant results.",
        "at": "04:30",
        "url": "https://www.youtube.com/watch?v=lyL5QhgIOxc&t=270s"
      },
      {
        "text": "P99 is much more important than P50 and we are paying a lots of attention to P99",
        "at": "04:58",
        "url": "https://www.youtube.com/watch?v=lyL5QhgIOxc&t=298s"
      },
      {
        "text": "We tokenize model names on insert time, not at query time.",
        "at": "08:40",
        "url": "https://www.youtube.com/watch?v=lyL5QhgIOxc&t=520s"
      },
      {
        "text": "The pattern is simple. Primary should focus on what only primary can do. Anything else can be pushed to different machines.",
        "at": "16:23",
        "url": "https://www.youtube.com/watch?v=lyL5QhgIOxc&t=983s"
      }
    ],
    "words": 2335
  },
  "summary_url": "/#lyL5QhgIOxc",
  "transcript": {
    "html": "/transcripts/lyL5QhgIOxc.html",
    "txt": "/transcripts/lyL5QhgIOxc.txt",
    "vtt": "/transcripts/lyL5QhgIOxc.vtt"
  }
}