{
  "video": {
    "id": "ACPEpji5NV4",
    "title": "Distill the LLM, Don't Serve It: Search & Personalization at DoorDash — Raghav Saboo, DoorDash",
    "duration": 1331,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "ACPEpji5NV4",
    "title": "Distill the LLM, Don't Serve It: Search & Personalization at DoorDash — Raghav Saboo, DoorDash",
    "one_liner": "DoorDash does its expensive LLM reasoning once, offline. It turns that reasoning into graded relevance labels, semantic IDs and consumer memory, then trains smaller, faster production retrieval and ranking models on those outputs, because 'the online LLM call is often not the product architecture you need.'",
    "summary": "Raghav Saboo argues that the real bottleneck in marketplace discovery is semantic understanding: knowing what items mean and what a shopper intends, not only optimizing engagement. He describes four shared primitives that let DoorDash use LLMs across search and recommendations: LLM supervision (graded relevance labels), catalog semantics (semantic IDs), semantic personalization (consumer memory) and steerable content generation (personalized collections). In each case the LLM work happens offline and is distilled into existing retrieval and ranking systems, not served online, and he shows the metric gains from each.",
    "key_points": [
      "Supervision pipeline: they start from a human-labelled seed set of query-item pairs on a 0/1/2 relevance scale. Suspicious cases, such as an item labelled irrelevant that performs well on add-to-cart or conversion, go to a stronger LLM with more granular prompts. Labels are also reconciled against the query-to-taxonomy/category models. The resulting high-precision golden set is used to fine-tune a lightweight LLM (e.g. GPT-4o mini), which then labels graded query-item pairs across the full catalog offline. Those labels are one shared target for both retrieval and ranking.",
      "Retrieval uses two-stage contrastive training on the graded labels. Stage 1 ('mining') shapes the global geometry with two-tower encoders and a multi-level supervised contrastive loss, then uses that base model to mine hard negatives and under-ranked positives. Stage 2 is curriculum training on those relabelled hard examples. The result was +2.3% NDCG on relevance, plus downstream business-metric gains.",
      "Ranking adds an ordinal relevance tower, trained on the LLM-graded labels, alongside the existing click, add-to-cart and conversion towers. The towers share bottom layers, so relevance is distilled in the same backward pass. A value function on top blends engagement and relevance differently for each surface.",
      "Semantic IDs: the catalog has a few billion store-level items, and SKU IDs carry no meaning while the human taxonomy is too coarse and rigid. Each item gets a learned hierarchical code whose prefix captures broad neighbourhoods and whose later tokens capture finer distinctions. For example, hot sauces split into Mexican, Caribbean and Korean with zero labels.",
      "Semantic IDs enable cross-category relationships (chips, salsa and guacamole share a neighbourhood), cold start for new items (n-gram and byte-pair encoding of the tokens lets them be added as sparse ID features), tail coverage, and reverse audits of human taxonomy labels. As a ranker feature they improved MRR by 4-5%, with conversion wins. Catalog-grounded query reformulation (Sriracha → chili garlic sauce, sambal oelek) gave large MRR gains.",
      "Consumer memory works on three time scales: long-term memory (orders, searches, browsing, support interactions), real-time in-session context (cart state, active searches), and stated preferences from agentic interactions such as Ask DoorDash. Memory is stored as extensible memory blocks (dietary, dining, substitution preferences) and materialized as human-readable text, latent embeddings, and graph/hierarchical forms.",
      "A context graph links consumers to extracted memory concepts. For retrieval, graph-based embeddings outperform the existing taxonomy-based embeddings, especially at fine-grained taxonomy levels. Memory is used in personalized collections, Ask DoorDash personalization, and as features in retrieval and ranking models.",
      "Steerable content generation: an offline batch LLM process takes consumer memory and semantic IDs and generates personalized store-page collections, from title and subtitle to the items (e.g. plant-based pantry rows, cat dry food rows). The existing retrieval and ranking stack still handles item hydration and collection ranking. In the pets vertical this drove close to a 1% order rate increase and a 6% increase in active users."
    ],
    "takeaways": [
      "Treat discovery as a semantic understanding problem, not only engagement optimization. Engagement signals are biased by exposure, position, price, promotions and previous model choices.",
      "Use expensive LLM reasoning once, offline: audit and expand a human seed set, fine-tune a cheap LLM labeller, and label the whole catalog. Then distill those labels into production retrieval and ranking models; don't replace those models with an online LLM call.",
      "Add LLM-derived relevance as its own tower beside the engagement towers, sharing bottom layers, and tune the relevance/engagement trade-off per surface with a value function.",
      "Build shared representations (graded labels, semantic IDs, memory blocks) that every downstream model, whether an LLM, a small language model or a traditional recommender, can consume. Each one then powers many use cases.",
      "Represent users in LLM-native, inspectable forms (text memory blocks) as well as embeddings, since LLMs can't readily use opaque user embeddings."
    ],
    "topics": [
      "search",
      "recommendations",
      "distillation",
      "llm-as-judge",
      "semantic-ids",
      "personalization",
      "retrieval",
      "ranking"
    ],
    "tools": [
      "DoorDash",
      "GPT-4o mini",
      "Ask DoorDash"
    ],
    "quotes": [
      {
        "text": "the real bottleneck is semantic understanding",
        "at": "00:55",
        "url": "https://www.youtube.com/watch?v=ACPEpji5NV4&t=55s"
      },
      {
        "text": "expensive reasoning once offline then distill it into models that can you know be served cheaply and quickly",
        "at": "06:10",
        "url": "https://www.youtube.com/watch?v=ACPEpji5NV4&t=370s"
      },
      {
        "text": "We're not replacing the retrieval and ranking system with an LLM, but rather distilling the uh reasoning and understanding into some production ranking architecture.",
        "at": "09:07",
        "url": "https://www.youtube.com/watch?v=ACPEpji5NV4&t=547s"
      },
      {
        "text": "The online LLM call is often not the product architecture you need.",
        "at": "21:29",
        "url": "https://www.youtube.com/watch?v=ACPEpji5NV4&t=1289s"
      }
    ],
    "words": 3515
  },
  "summary_url": "/#ACPEpji5NV4",
  "transcript": {
    "html": "/transcripts/ACPEpji5NV4.html",
    "txt": "/transcripts/ACPEpji5NV4.txt",
    "vtt": "/transcripts/ACPEpji5NV4.vtt"
  }
}