{
  "video": {
    "id": "xbPriQWXtWM",
    "title": "The Base Model Is Dead — Varun Singh, Arcee AI",
    "duration": 1065,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "xbPriQWXtWM",
    "title": "The Base Model Is Dead — Varun Singh, Arcee AI",
    "one_liner": "Arcee AI's pre-training lead argues the classic web-text base model is being replaced by one built as a prior for large-scale RL — web text has fallen from ~85% of GPT-3's mix to ~15% in MAI Thinking 1, with code, STEM and even SFT-style data pulled back into pre-training.",
    "summary": "Varun Singh traces how pre-training changed after o1 (2024) and DeepSeek R1 (January 2025) turned RL from 'a cherry on top' into a phase that can dominate the compute budget. He shows the data-mix evidence: GPT-3 was ~85% web text with no dedicated code set, Llama 3 still ~50% general knowledge, while MAI Thinking 1 is down to 15% web text and NeMoTron 3 Ultra pulls SFT-labelled chat data into pre-training outright. He argues the useful frame is no longer pre/mid/post-training but two paradigms — supervised next-token learning and RL — where supervised learning exists to give the model the atomic skills and task shapes RL will compose, and that synthetic rephrasing (as in Arcee's Trinity and Kimi K2) is how you get those tokens at scale.",
    "key_points": [
      "Data-mix shift, with numbers: web text (incl. Wikipedia) was ~85% of GPT-3's training mix and ~50% of Llama 3's tokens as general knowledge, but only 15% in MAI Thinking 1; GPT-3 had no specific code data sets, whereas code is now the dominating subset of pre-training recipes.",
      "Two contrasting current bets: MAI Thinking 1 makes a point of using zero synthetic data (and filters web scrapes for it), while NeMoTron 3 Ultra's disclosed recipe has its three largest components prefixed 'SFT' — question-answer chat data pulled all the way back into pre-training so the model learns conversation shape from the start.",
      "Synthetic data at web scale works when it's rephrasing, not blind generation: Arcee's own Trinity took seed items and upsampled them as synthetic rephrases so the model sees the same information multiple ways; Kimi K2 applied this broadly across its whole pre-training set; Swallow Code and Swallow Math are early examples.",
      "An MoE-specific argument for early post-training data: expert load balancing breaks when the post-training distribution differs sharply from pre-training. MAI Thinking 1 reports fixing this by cranking up the load-balancing coefficient during SFT — a late intervention the speaker thinks you shouldn't need if representations were stable from early on.",
      "Mid-training — exposing the model to the RL/post-training distribution at longer context, including agentic traces — is emerging, but he argues many models already pre-train at long context so there's no reason those sets can't go into the mix from the beginning.",
      "Compute has rebalanced: the head of Xiaomi's MiMo Labs describes roughly equal pre- and post-training compute in the final model, and Composer 2.5 sank far more RL compute into the model than it ever saw in supervised learning.",
      "On how SL affects RL: the base model needs exposure to the atomic skills it will have to compose during RL, and can extrapolate from there provided the environment is sufficiently difficult. He shows the AlphaGo graph where RL overtakes SL, but doubts it fully transfers — human language is 'an insane distribution' to learn through RL alone.",
      "Some labs now inject genuinely novel shapes during SFT — reasoning traces look nothing like human output — and warm the model up to test-time-compute schemes during SFT or even pre-training (screenshots from the Ziya 1 paper)."
    ],
    "takeaways": [
      "Design the pre-training mix as a prior for RL, not as a compression of the internet: budget for code and STEM over general web text, and judge data by the downstream RL skills it seeds.",
      "Pull post-training-shaped data (SFT chat, agentic traces, long-context) into pre-training rather than saving it for later — it teaches task shape early and avoids distribution shock.",
      "If you train an MoE, treat the pre/post-training distribution gap as a load-balancing risk up front; retuning the balancing coefficient during SFT is a late patch, not the intended fix.",
      "Use synthetic data as rephrasing of seed content to upsample and clean tokens — the demonstrated-safe pattern at scale — rather than blindly tossing generated text in and risking collapse.",
      "Stop reasoning in pre/mid/post/RL stages; think in two paradigms, supervised next-token learning and RL, and ask what atomic skills the supervised phase must plant for RL to compose."
    ],
    "topics": [
      "pre-training",
      "synthetic-data",
      "reinforcement-learning",
      "data-mixtures",
      "mixture-of-experts",
      "mid-training",
      "reasoning",
      "agents"
    ],
    "tools": [
      "Arcee AI",
      "Trinity (Arcee's own model)",
      "GPT-3",
      "Llama 3",
      "GLM 4.5",
      "GLM 5",
      "OpenAI o1",
      "DeepSeek R1",
      "Claude Code",
      "MAI Thinking 1",
      "NeMoTron 3 Ultra",
      "Kimi K2",
      "Swallow Code",
      "Swallow Math",
      "Composer 2.5",
      "Xiaomi MiMo Labs",
      "Ziya 1 paper",
      "Common Crawl",
      "WebText-2",
      "Wikipedia",
      "AlphaGo",
      "AIME"
    ],
    "quotes": [
      {
        "text": "RL was mostly just a cherry on top, um shaping the, you know, flavor of the interactions more than conferring extra um knowledge or quality onto the base model itself.",
        "at": "03:22",
        "url": "https://www.youtube.com/watch?v=xbPriQWXtWM&t=202s"
      },
      {
        "text": "the main reason for that is that web text, which used to make up like up to 85% of the train data in GPT uh 3, is now all the way down at 15%.",
        "at": "07:07",
        "url": "https://www.youtube.com/watch?v=xbPriQWXtWM&t=427s"
      },
      {
        "text": "with RL dominating such a massive amount of the compute budget, uh it makes sense to view supervised learning as a way specifically to prepare the model for to build useful representations for for RL instead of it being the bulk of like um what the model would be used for like previously.",
        "at": "14:22",
        "url": "https://www.youtube.com/watch?v=xbPriQWXtWM&t=862s"
      },
      {
        "text": "base models have kind of moved from general, uh, human knowledge and world priors to reasoning and agentic behavior priors.",
        "at": "16:39",
        "url": "https://www.youtube.com/watch?v=xbPriQWXtWM&t=999s"
      }
    ],
    "words": 2961
  },
  "summary_url": "/#xbPriQWXtWM",
  "transcript": {
    "html": "/transcripts/xbPriQWXtWM.html",
    "txt": "/transcripts/xbPriQWXtWM.txt",
    "vtt": "/transcripts/xbPriQWXtWM.vtt"
  }
}