{
  "video": {
    "id": "ewtOo0scUh0",
    "title": "Data and Environment Curation for Post-Training LLMs — Mahesh Sathiamoorthy, Bespoke Labs",
    "duration": 1152,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer"
  },
  "analysis": {
    "video_id": "ewtOo0scUh0",
    "title": "Data and Environment Curation for Post-Training LLMs — Mahesh Sathiamoorthy, Bespoke Labs",
    "one_liner": "The Bespoke Labs CEO argues that for post-training LLMs and agents, data and RL environments — not compute, models or infra — are the bottleneck, and walks through the Open Thoughts curation recipes plus their counterintuitive findings (sample many answers per question; stronger models aren't always better teachers).",
    "summary": "Mahesh Sathiamoorthy (co-founder/CEO, Bespoke Labs; ex-Google DeepMind) frames the shift from evaluating what models know to whether agents can act autonomously for hours or days, where the blocker is reliability and post-training is a primary lever. He argues compute, base models and post-training infra (fireworks, tinker, slime) are well-defined, so the real gap for enterprises and frontier labs is high-quality data and RL environments. He walks through the Open Thoughts and Open Thoughts Agents curation pipelines — source questions, mixing, filtering, teacher-model answer generation, answer filtering — established via stage-by-stage ablations that produced a scaling curve, and shares counterintuitive lessons. He closes with an Intuit Credit Karma production case study and a reference stack for building RL environments and post-training agents.",
    "key_points": [
      "Bespoke shipped Curator (synthetic data curation for SFT), then Bespoke Stratos after DeepSeek landed, which grew into the Open Thoughts consortium with Stanford, UC Berkeley and UDub; they're also core contributors to Terminal-Bench.",
      "The Open Thoughts paper's main figure is a scaling curve: with their curation recipe, benchmark metrics (AIME, LiveCodeBench) keep improving as dataset size scales — evidence the recipe itself is scalable, not just a one-off dataset.",
      "The curation pipeline has explicit knobs at each stage — source question selection, how to mix questions across datasets, LLM-based filtering for quality/hardness, teacher model choice (DeepSeek, Qwen-based, Gemini), one vs. many answers per question, answer filtering — and the recipe was derived by running ablations stage by stage.",
      "Counterintuitive finding: sampling multiple answers per question (e.g. one question answered 16 times) beats collecting many more questions answered once; the hypothesis is that diversity in the reasoning traces used during fine-tuning is what helps.",
      "Stronger models are not always better teachers — this held in both Open Thoughts and Open Thoughts Agents, where he says some Qwen models were better teachers than Claude models.",
      "Things that didn't work: answer filtering (while synthetic question generation/answering did work), and for agents, synthetic rewriting and task augmentation.",
      "In Open Thoughts Agents, SFT still contributed most of the gains; RL is very compute-intensive and mainly bought the last few percentage points — in many enterprise settings SFT alone works well.",
      "Intuit Credit Karma case: explaining why a credit card was recommended requires a long list of compliance rules in the prompt, which blows up latency. Naive fine-tuning made the model hallucinate numbers (e.g. 0% APR) because the data was imbalanced; adding tags to the prompt-response pairs so the model focused on form rather than the specific numbers gave a big boost, improving compliance, latency and throughput while letting the customer own the model."
    ],
    "takeaways": [
      "Spend your ablation budget on answers-per-question, not just question count — sampling many reasoning traces per prompt gave better returns than a bigger set of single-answer questions.",
      "Don't default to the strongest available model as your teacher; benchmark cheaper/smaller teachers (e.g. Qwen-based) against frontier ones, since stronger repeatedly failed to mean better here.",
      "Try SFT before reaching for RL — in their agent work SFT drove most of the gains and RL was compute-intensive for the final few percent.",
      "When fine-tuning on data with skewed numeric distributions, structure the training pairs (e.g. tag the numbers) so the model learns the form of the response rather than memorizing and hallucinating specific values.",
      "Treat data curation as a research loop where you sit in the researcher's seat — build the recipe stage by stage with ablations and check that metrics actually move as data scales, rather than shipping a static dataset.",
      "Post-train not just for capability but for latency, cost and throughput — moving long compliance rule lists out of the prompt and into the weights was the win in the Credit Karma deployment."
    ],
    "topics": [
      "post-training",
      "data-curation",
      "rl-environments",
      "synthetic-data",
      "agents",
      "reasoning-models",
      "sft",
      "reinforcement-learning"
    ],
    "tools": [
      "Bespoke Labs",
      "Curator",
      "Bespoke Stratos",
      "Open Thoughts",
      "Open Thoughts Agents",
      "Terminal-Bench",
      "SWE-bench",
      "AIME",
      "LiveCodeBench",
      "DeepSeek",
      "Qwen",
      "Gemini",
      "Claude",
      "Fireworks",
      "Tinker",
      "slime",
      "Hugging Face",
      "Google DeepMind",
      "Thinking Machines",
      "Microsoft",
      "Intuit Credit Karma",
      "Stack Exchange",
      "GEPA (transcribed as 'Japa')"
    ],
    "quotes": [
      {
        "text": "we have moved on from knowing to doing right so that's the idea of agents",
        "at": "03:04",
        "url": "https://www.youtube.com/watch?v=ewtOo0scUh0&t=184s"
      },
      {
        "text": "ultimately for post training be it SFT or or uh reinforcement learning data is the bottleneck",
        "at": "05:02",
        "url": "https://www.youtube.com/watch?v=ewtOo0scUh0&t=302s"
      },
      {
        "text": "the other thing we saw is like the stronger teachers are not always the best uh uh stronger models are not always the better teachers",
        "at": "11:33",
        "url": "https://www.youtube.com/watch?v=ewtOo0scUh0&t=693s"
      },
      {
        "text": "SFT still contributed a lot to the gains um RL was kind of you know it's very comput inensive and for for the last few few percentages it really helped",
        "at": "13:27",
        "url": "https://www.youtube.com/watch?v=ewtOo0scUh0&t=807s"
      }
    ],
    "words": 3517
  },
  "summary_url": "/#ewtOo0scUh0",
  "transcript": {
    "html": "/transcripts/ewtOo0scUh0.html",
    "txt": "/transcripts/ewtOo0scUh0.txt",
    "vtt": "/transcripts/ewtOo0scUh0.vtt"
  }
}