{
  "video": {
    "id": "cJ0EOzey--o",
    "title": "What's Next After RLHF? — Diogo Almeida, TypeSafe AI",
    "duration": 1085,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer"
  },
  "analysis": {
    "video_id": "cJ0EOzey--o",
    "title": "What's Next After RLHF? — Diogo Almeida, TypeSafe AI",
    "one_liner": "An OpenAI post-training co-author (GPT-4, ChatGPT, InstructGPT) argues that today's AI is stuck in an \"assistance era\" because RLHF literally optimizes for human preference — so LLMs are superhuman at human-in-the-loop tasks and useless at real automation, and the next era is a third post-training objective built for calibrated decision-making.",
    "summary": "Almeida maps the split between AI optimists (every benchmark crushed, autonomous operating time growing exponentially) and pessimists (bubble, circular financing, everything is just a chat app) and offers the simplest explanation for the divide: the tasks AI is superhuman at are intrinsically human-in-the-loop tasks whose goal is to please the human, while the tasks it fails at are ones where the goal is to remove the human. He traces this directly to RLHF — collect human preferences, optimize for human preferences — which by construction makes overpromising a feature and makes wrong models look right. He argues Claude Code is not the next era but the same assistance era (still RLHF, not pure RLVR), that SaaS hasn't fundamentally changed since 2019 because AI is assistance-native, and that the real next step is automation and genuinely smarter software. He closes by pitching his stealth company TypeSafe, which is building a third post-training branch optimized for calibrated decision-making rather than human preference or pure correctness.",
    "key_points": [
      "The optimist/pessimist divide has one simple explanation: tasks where AI looks superhuman (unsolved math problems, benchmarks, coding assistants) are tasks whose goal is to please a human in the loop; tasks it fails at (customer service decisions) are ones whose goal is to remove the human. That's the assistance-vs-automation divide.",
      "Roughly 100% of LLMs in usage today are trained with RLHF, which he summarizes as: collect human preferences, optimize for human preferences. So the answer to \"why do all LLMs require a human in the loop?\" is that we literally put them in the loop.",
      "Overpromising is by design, not a bug: every RLHF model has a structural gap between human preference and actual results because preference is the optimization target. His example is a tweet of someone sending ChatGPT an audio file of fart sound effects and getting back \"It's a very eerie vibe atmosphere piece.\"",
      "The business lesson everyone has learned: do not use AI for decisions with stakes to your business. The common pattern is pushing all costs onto the user — infinite customer-service docs are fine, expensive decisions are not. He calls this a horrible pattern but the state of AI.",
      "Claude Code is part of the same assistance era, not the next one — it's still RLHF, and would look very different if it were purely RLVR. The dance between agentic capability and instruction-following is a trade-off in optimization space where neither side adds automation.",
      "SaaS has basically not changed since 2019 except that a chatbot gets latched on — predictable if AI is assistance-native. Early AI pioneers (and OpenAI's charter) expected software to get smarter, not just cheaper to write; Garry Tan's \"golden age of just-in-time software\" is a double-edged sword.",
      "On the bitter lesson: he claims algorithms-over-compute holds in games but not reality; his stack is that data matters more than compute, and doing the right task matters way more than data.",
      "Each post-training branch has its own North Star: RLHF optimizes human preference, RLVR optimizes log error rates of pure correctness, and TypeSafe is doing a third thing optimized for calibrated decision-making — even the shape of the API differs across the three.",
      "He argues hallucination is intrinsic to optimizing for human preference, not a pre-training problem: an asymmetry in the reward model (analogous to GANs) encourages mode-dropping and confidence, because a reward model can easily detect and punish visible uncertainty."
    ],
    "takeaways": [
      "Diagnose your use case as assistance or automation before picking a model — RLHF-trained models are excellent when the objective is pleasing a human and structurally unsuited to running unattended in the background.",
      "Assume model confidence is uncalibrated by construction. Don't read a fluent, agreeable answer as a correct one; the reward-model asymmetry means wrong outputs will still look right.",
      "Keep stakes-bearing business decisions out of LLM hands, and be honest when a design pattern is just shifting the cost of errors onto users.",
      "Stop treating \"AI wrote the code faster\" as automation — the expressibility of the resulting software is unchanged. Ask instead what rote work could be handed off repeatedly for free.",
      "Don't assume more agentic post-training gets you to automation; RLHF and RLVR are optimizing for preference and correctness respectively, and neither North Star is calibrated decision-making."
    ],
    "topics": [
      "rlhf",
      "rlvr",
      "post-training",
      "automation",
      "agents",
      "alignment",
      "hallucination",
      "scaling-laws"
    ],
    "tools": [
      "OpenAI",
      "ChatGPT",
      "GPT-4",
      "InstructGPT",
      "Claude Code",
      "TypeSafe",
      "Twitter",
      "Garry Tan (Y Combinator)",
      "Yoshua Bengio",
      "Richard Sutton's bitter lesson"
    ],
    "quotes": [
      {
        "text": "The And the simple answer is we literally put them in the loop. The goal of the loop is to optimize for human preference. It is not to run software autonomously. It's kind of super obvious.",
        "at": "06:42",
        "url": "https://www.youtube.com/watch?v=cJ0EOzey--o&t=402s"
      },
      {
        "text": "and because of that, overpromising is a feature. This is by design.",
        "at": "07:00",
        "url": "https://www.youtube.com/watch?v=cJ0EOzey--o&t=420s"
      },
      {
        "text": "no matter how wrong the models are, they will look right because of the asymmetry within the reward model in RLHF.",
        "at": "08:30",
        "url": "https://www.youtube.com/watch?v=cJ0EOzey--o&t=510s"
      },
      {
        "text": "kind of like the craziest part of software in my opinion is that all of the SaaS basically has not changed since 2019.",
        "at": "10:15",
        "url": "https://www.youtube.com/watch?v=cJ0EOzey--o&t=615s"
      }
    ],
    "words": 3681
  },
  "summary_url": "/#cJ0EOzey--o",
  "transcript": {
    "html": "/transcripts/cJ0EOzey--o.html",
    "txt": "/transcripts/cJ0EOzey--o.txt",
    "vtt": "/transcripts/cJ0EOzey--o.vtt"
  }
}