{
  "video": {
    "id": "zL1kLftVTlo",
    "title": "Scaling up Continual Learning — Ronak Malde, Trajectory",
    "duration": 1383,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "zL1kLftVTlo",
    "title": "Scaling up Continual Learning — Ronak Malde, Trajectory",
    "one_liner": "Ronak Malde (ex-WinSurf, now Trajectory) argues RL/GRPO is the wrong shape for continual learning and presents on-policy self-distillation (OPSD) — where a 'hinted' copy of the model acts as its own teacher — plus the fixes needed to scale it to 120B+ models and 100-tool-call trajectories.",
    "summary": "The talk frames post-training algorithms against four criteria — online task distribution, on-policy sampling, single (non-parallel) rollouts, and per-token reward — and shows SFT, DPO/RLHF and GRPO each satisfying only some of them. OPSD satisfies all four: you roll out the student, re-run the same model with privileged information (a hint) prepended as the 'teacher', and fit the student's log probs to the hinted teacher's, shifting the whole 65k-token vocabulary distribution rather than sharpening around the sampled token. It works well on short-horizon tasks — beating GRPO's saturation around Sonnet-level on LiveCodeBench and collapsing tokens-to-solve — but breaks at 120B with 50–100 tool calls, producing the 'wait' problem and hint leakage. Malde presents step-level KL divergence weighting and residual guidance as the fixes that got a 12B model working on Mercor Apex agents, surpassing RL.",
    "key_points": [
      "Four criteria for a post-training algorithm: online task distribution, on-policy sampling, parallelism of one, per-token reward. SFT gets parallelism and per-token reward; DPO/RLHF gets online task distribution; GRPO gets on-policy sampling but explodes parallelism and reverts to sequence-level reward. None gets all four.",
      "The 'trillion token problem': hundreds of trillions of tokens per day are spent on inference, generating real signal about how models fail and succeed, and none of it is being captured and trained on — while benchmarks get more expensive (4, 6, 24 hours, several days) and drift from real use.",
      "GRPO's reward is 'drinking through a straw' — like an essay graded 87/100, you need enormous numbers of examples to infer what a good essay is.",
      "OPSD's core trick: since there is no smarter teacher at the frontier, give the same model privileged information (a hint, at simplest a golden solution) in the prompt, and match the unhinted student's log probs to the hinted teacher's.",
      "Because it matches the full log-prob distribution over ~65k vocabulary tokens per position, OPSD can push probability mass toward tokens the student never sampled — shifting entire distributions rather than sharpening one, as RL does.",
      "On LiveCodeBench, GRPO saturates around Sonnet-level performance; OPSD reaches new territory. It also inverts RL's 'think longer' pathology — tokens to solve hard problems collapse, improving token efficiency.",
      "The 'wait' problem at scale: on long trajectories the student diverges, the teacher course-corrects at every opportunity, and training collapses into hedging tokens — a word cloud filling with 'wait', 'but', 'maybe' until everything becomes 'maybe'. Fix: weight each step's tokens by the student–teacher KL divergence (a multiplicative weight, not the usual KL penalty), so a divergent step gets trained on its first step then skipped, and a trajectory that gets back on track is still mildly shifted.",
      "Hint leakage is OPSD's equivalent of reward hacking: given the answer ('the last three digits are all zeros'), the rollout says it already knows the solution and back-fills a reasoning trace. Fixes: LLM-filter hints into what the model should reasonably have known (e.g. 'look through the logs' instead of 'the SSO token is expired'), or residual guidance — take a half hint and a full hint, and use a linear combination of the partial and full teachers to gauge hint strength and avoid shifting into unknown territory."
    ],
    "takeaways": [
      "If you're doing post-training on short-horizon tasks or chatbots, you can plug OPSD in today via open-source projects like open claw / open claw RL to learn from unstructured user behaviour.",
      "Audit your algorithm against the four criteria rather than the benchmark score — in particular, notice when your task distribution is an offline benchmark rather than real production traffic.",
      "Design hints as guidance, not answers: filter out anything that lets the model shortcut the reasoning it actually needs to learn, and check rollouts for back-filled reasoning traces as a leakage smell.",
      "Don't assume short-horizon results transfer — at 120B with 50–100 tool calls, expect wild eval-accuracy range, high run-to-run variance, and tool-call format errors; budget for step-level divergence weighting.",
      "Treat continual learning as still 'pseudo' — batch offline updates and model re-upload — and expect the open problem of merging 10,000 concurrent production rollouts, plus the unexplored model/harness co-evolution question."
    ],
    "topics": [
      "continual-learning",
      "post-training",
      "reinforcement-learning",
      "distillation",
      "agents",
      "tool-calling",
      "evals"
    ],
    "tools": [
      "Trajectory",
      "WinSurf",
      "SWE-1 (sui 1)",
      "DeepMind",
      "GRPO",
      "DPO",
      "RLHF",
      "SFT",
      "GPT-3.5 / ChatGPT",
      "Sonnet",
      "LiveCodeBench",
      "open claw / open claw RL",
      "Mercor Apex",
      "Meta Super Intelligence",
      "OpenAI",
      "Harvey",
      "Decagon",
      "Rogo"
    ],
    "quotes": [
      {
        "text": "we are actually spending hundreds of trillions of tokens every single day on inference uh and we're generating great amounts of data on how models in the real world are are failing, how they're doing well. And that should be signal that we should be capturing and training on.",
        "at": "01:46",
        "url": "https://www.youtube.com/watch?v=zL1kLftVTlo&t=106s"
      },
      {
        "text": "imagine you were trying to write an essay and your teacher just gave you a score of 87 out of 100. You would have to run through so many different examples to get to the idea of what a good essay is",
        "at": "06:58",
        "url": "https://www.youtube.com/watch?v=zL1kLftVTlo&t=418s"
      },
      {
        "text": "we're not just taking now a distribution like RL and slightly sharpening it, but we're instead actually shifting entire distributions",
        "at": "10:53",
        "url": "https://www.youtube.com/watch?v=zL1kLftVTlo&t=653s"
      },
      {
        "text": "with RL the number one problem that people face is reward hacking ... Well there is an equivalent for OPSD as well and that is hint leakage.",
        "at": "15:55",
        "url": "https://www.youtube.com/watch?v=zL1kLftVTlo&t=955s"
      }
    ],
    "words": 4843
  },
  "summary_url": "/#zL1kLftVTlo",
  "transcript": {
    "html": "/transcripts/zL1kLftVTlo.html",
    "txt": "/transcripts/zL1kLftVTlo.txt",
    "vtt": "/transcripts/zL1kLftVTlo.vtt"
  }
}