{
  "video": {
    "id": "KhYifX22yhE",
    "title": "The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside",
    "duration": 1051,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "KhYifX22yhE",
    "title": "The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside",
    "one_liner": "poolside's data and pre-training leads explain how they fixed the two things that broke when scaling Laguna from M.1 to XS.2 and the unreleased 118B Laguna S: token repetition (fixed with 13% synthetic data from modular, orchestrated pipelines) and silent numerical corruption (broken GPUs, BF16 accumulation before the LM head, and a race condition in FP8 kernels).",
    "summary": "Marah Abdin covers poolside's synthetic data strategy: they don't see it as a replacement for organic data but as a way to surface the 'implicitly hidden' rationale, planning and structure in organic data, and to solve the token-uniqueness problem that appeared when they scaled their training budget and started re-seeing high-quality data too often. She frames every synthetic pipeline as the same six modular components and four recurring 'shapes', which is what lets them build Hive — configurable queues of agents with orchestrators and a supervisor. Robert McHardy then argues data and training-code correctness must be treated holistically ('we don't trust anything'), and walks through never-before-shared failures at scale: a broken GPU causing silent data corruption, BF16 accumulation in the tensor-parallel unembedding killing convergence around 50k steps, and a race condition in open-source FP8 kernels silently corrupting ~0.5% of gradients. He closes with base-model eval previews of Laguna S — 118B total / 8B active, 30T tokens, 4,000 GPUs — showing the recipe held at scale.",
    "key_points": [
      "Three data-side changes between model generations: an 'auto mixer' that does a cheap sweep over clusters of datasets before committing to expensive experiments; resampling web data for higher recall; and leaning much harder on synthetic data.",
      "Synthetic data complements rather than replaces organic data — organic data has a lot 'implicitly hidden' and not presented optimally, so synthetic generation extracts those features and 'projects them on some new planes' (implicit rationale, planning, structure), fills gaps and regularizes how the model is taught.",
      "Synthetic data was 13% of the pre-training mix for XS.2 (pre-training only, before post-training); their synthetic corpus is now 6 trillion tokens and continuously growing.",
      "They over-indexed on quality vs quantity at small scale; when the training budget scaled up, non-optimal repetition of high-quality data saturated the model too early. Rephrasing (as in BeyondWeb) fixed the token-uniqueness problem — ablations consistently show reduced-repetition runs (green) beating seeds-with-repetition (orange).",
      "Beyond a generic scalable multi-model rephrasing pipeline, they built two specialized ones: math → code and text, and one for STEM/scan documents — because a cheap seed-heavy pipeline depends very heavily on the seed.",
      "Every synthetic pipeline decomposes into the same six components: seeds, primary inputs, metadata, secondary inputs, the generator/function (an agent with tools, or prompt templates), and supplementary functions like filters and validators. Complexity runs from cheap seed-heavy pipelines with small models to heavily orchestrated workflows reserved for data worth it, e.g. educational data.",
      "Four pipeline shapes: rephrasing; multi-stage (generate a novel's setting, character names and styles, plot and twists first, then chapters one by one — 'you will absolutely get a better novel'); cross-domain porting (translating code, converting math problems to code); and multi-turn/iterative (two agents chatting, or a judge and an evolver going back and forth for k rounds).",
      "Hive is their configurable generation infrastructure: a queue of agents each with its own prompt, parameters, model and I/O plus configurable entry/exit/frequency; orchestrators in between that police the LLMs, dynamically rewrite instructions and choose or skip the next agent; and a supervisor that polices the orchestrator with a global view.",
      "Model replica hashing: weights must be identical across DDP replicas, so they hash and periodically compare; a mismatch crashes training. This caught a broken GPU causing silent data corruption — two runs with identical config, data and implementation produced very different loss curves, the bad one spiky with huge gradient norms.",
      "Around 50,000 steps into the Laguna M.1 run the loss flattened out: activations kept growing right before the LM head/unembedding, and the tensor-parallel accumulation there ran in BF16 by default, losing numerical precision. Restarting the checkpoint with FP32 accumulation resumed convergence and flipped the gradient norm from increasing to decreasing.",
      "Laguna S preview: 118B total / 8B active parameters, 30 trillion tokens, 4,000 GPUs. Adding FP8 training on open-source Deep Chem FP8 kernels introduced a race condition surfacing as illegal memory accesses and NaNs in gradients, plus an unobservable effect — about 0.5% of the gradient silently replaced by random values. Their fix is a public, not-yet-merged PR.",
      "Hash checks have a blind spot: real training runs have no redundancy where the same weights meet the same data, so you can never verify forward/backward agree across replicas or catch a race condition there — they're building a dry-run hash checker for this.",
      "Base-model evals (caveated as only partly indicative pre-post-training): Laguna S beats both XS.2 and the much larger M.1 on coding benchmarks, beats GLM 4.5 Air and the more recent DeepSeek comparison model, is close on BigBench Hard and Eval Plus, and does much better than everything tested on SWE-bench agentless multilingual — their pre-training proxy for agentic performance. On MMLU Pro, Nemotron and DeepSeek are much better; they call this a data gap they could plug if they cared about knowledge over agentic coding."
    ],
    "takeaways": [
      "Don't push quality-over-quantity so far that scaling the training budget forces repetition of your best data — budget token uniqueness, and use rephrasing pipelines to convert repeated tokens into fresh ones.",
      "Decompose synthetic generation into the six components and pick complexity per dataset: cheap seed-heavy pipelines where the seed carries the quality, orchestrated multi-stage workflows only where the data is worth it. If a task is too hard for your teacher model, break it into simpler steps rather than accepting degraded correctness and diversity.",
      "Add invariant checks to distributed training: hash model weights across replicas and crash on mismatch — it catches broken GPUs doing silent data corruption that otherwise just looks like a spiky loss curve.",
      "Audit the numerics at the points where activations grow, especially accumulations in tensor-parallel unembedding/LM head — move them to FP32 rather than accepting BF16 defaults, and watch gradient-norm trend as the early warning.",
      "Treat third-party kernels (e.g. open-source FP8 kernels) as suspect: NaNs and illegal memory accesses can be the visible tip of silent partial gradient corruption that no replica hash check will catch.",
      "Treat data quality and training-code correctness as one problem, not two teams' problems — either one being bad makes a good model impossible."
    ],
    "topics": [
      "synthetic-data",
      "pre-training",
      "distributed-training",
      "numerical-precision",
      "data-mixing",
      "agents",
      "evals",
      "open-weights"
    ],
    "tools": [
      "poolside",
      "Laguna M.1",
      "Laguna XS.2",
      "Laguna S (unreleased preview)",
      "Hive (poolside's synthetic data generation infrastructure)",
      "Hugging Face",
      "BeyondWeb",
      "Deep Chem FP8 kernels (open source, as named in the talk)",
      "GLM 4.5 Air",
      "DeepSeek",
      "Nemotron",
      "LiveCodeBench",
      "BigCodeBench",
      "BigBench Hard",
      "Eval Plus",
      "SWE-bench agentless multilingual",
      "MMLU Pro"
    ],
    "quotes": [
      {
        "text": "if task is too hard for your model, then your model will start to fall on its face. Lose correctness, lose diversity. So break down the task, make it simpler.",
        "at": "06:11",
        "url": "https://www.youtube.com/watch?v=KhYifX22yhE&t=371s"
      },
      {
        "text": "If you've got data that sucks, you can't train a good model. If you've got a training code base that sucks, you also can't.",
        "at": "09:29",
        "url": "https://www.youtube.com/watch?v=KhYifX22yhE&t=569s"
      },
      {
        "text": "the way we we look at things in my team is uh we don't trust anything. There's so many things that can go wrong when you scale models to billions of parameters to hundreds of billions of parameters um training on thousands of GPUs",
        "at": "09:41",
        "url": "https://www.youtube.com/watch?v=KhYifX22yhE&t=581s"
      },
      {
        "text": "we noticed about 0.5% of the gradient gets silently corrupted, essentially replaced by random values.",
        "at": "14:28",
        "url": "https://www.youtube.com/watch?v=KhYifX22yhE&t=868s"
      }
    ],
    "words": 3989
  },
  "summary_url": "/#KhYifX22yhE",
  "transcript": {
    "html": "/transcripts/KhYifX22yhE.html",
    "txt": "/transcripts/KhYifX22yhE.txt",
    "vtt": "/transcripts/KhYifX22yhE.vtt"
  }
}