{
  "video": {
    "id": "_PdK6x7PQNM",
    "title": "Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI",
    "duration": 1145,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer"
  },
  "analysis": {
    "video_id": "_PdK6x7PQNM",
    "title": "Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI",
    "one_liner": "DatologyAI's CEO argues that in a compute-scarce world, better data curation is the cheapest lever there is — showing curation alone buying a 14-point VLM accuracy gain, Qwen-3.5-4B-level quality at 145x less training compute, and post-training gains that nearly triple when applied on top of a mid-trained model.",
    "summary": "Ari Morcos frames data quality as a 'compute multiplier': better data makes the learning curve steeper, so you get the same performance for a fraction of the compute — which matters now that H100 prices are back up ~40% off their lows and reasoning models are consuming 8x the tokens. He walks through DatologyAI's 'four C's' pipeline (clean, curate, create, compose) and presents results across vision-language models, multilingual text models, synthetic rephrasing (BeyondWeb), and two customer case studies (Thomson Reuters legal mid-training, Arcee's 17T-token open model). The recurring claim is that curation moves models well past the public Pareto frontier at 8x–145x less compute, and that training stages should be designed synergistically rather than handed off independently.",
    "key_points": [
      "Compute is getting scarcer, not cheaper: H100 prices reversed a multi-year decline and are ~40% up from their lows at the end of last year; reasoning models use 8x the tokens of non-reasoning models with another projected 5x in the next year; Google capped Meta's Gemini usage over inference constraints and OpenAI is effectively selling 'token futures' because frontier API access may become genuinely supply-limited.",
      "The four C's pipeline: clean (heuristic/Gopher-style filters, rigorous benchmark decontamination at a low n-gram), curate (quality classifiers, topic taxonomy and balancing, semantic redundancy reduction, up/downsampling, task distribution matching), create (synthetic rephrasing to expand size and diversity), compose (mixing and sequencing across the now-standard three-plus training phases, including continuous curricula).",
      "VLM result: starting from the ~25B-token MAmmoTH dataset used to train the fusion adapter between the text and vision models, curation alone yielded ~14 absolute percentage points of improvement holding everything else constant, and matched Qwen 3.5 4B to within about a percentage point using 145x less training compute — with no post-training at all.",
      "Curation also produced markedly more concise responses; on error vs. log flops-per-response, the curated models reached roughly Qwen 3.5-level performance with ~35x fewer flops per correct answer.",
      "Multilingual: using only 8% multilingual tokens (most languages capped around 6B tokens), curated models beat the Pareto frontier set by Qwen, Liquid and Cohere's best small multilingual model on multilingual MMLU, edging past Qwen 3 with ~8x less compute budget.",
      "Cross-lingual transfer is real: curating only the English data improved non-English accuracy, with the magnitude of transfer strongly correlated with how similar the language is to English; the reverse effect (non-English curation helping English) exists but is smaller.",
      "Scaling extrapolated cleanly — the line through two 1T-token dense llama-style curated models passed almost exactly through Arcee's 17T-token hypersparse Trinity Large despite 50x more training compute, meaning a hero run can be de-risked at 50–100x less compute if you simulate token scarcity correctly.",
      "Rephrasing synthetic data (BeyondWeb) converts a source document into one of hundreds of templates (e.g. a set of true/false questions); because all information comes from the source document there's no model collapse, and you can train students better than the rephrasing model since transforming a document is far easier than understanding the concepts. Which documents you rephrase matters — random document selection does not work.",
      "Thomson Reuters case: 100B tokens of continued pre-training/mid-training (<1% of the pre-training budget) on their proprietary legal data mixed with public data lifted LegalBench ~5 points with no catastrophic forgetting — general capabilities rose too, because the majority of the mix was data representative of the pre-training distribution. Their existing post-training harness then produced nearly 3x the delta on the mid-trained model versus the default instruction-tuned model.",
      "Arcee case: a fully open, US-made model trained on 17T tokens curated entirely from public data — no proprietary data and no closed-model usage ('no asking Claude to do this for you') — matching GLM-5 and Kimi on many tasks and beating Claude on a couple, built by a team that had never trained a model before mid-last-year, for under $20M total including salaries, compute, R&D and all failed repetitions."
    ],
    "takeaways": [
      "If you're compute-constrained, spend the next increment of effort on data rather than GPUs — optimize for maximum marginal information gain per data point, not more tokens.",
      "Stop looking for a universally good dataset: a dataset is only optimal with respect to the specific output tasks you want, so do task distribution matching (legal data for a legal model) and inject diversity deliberately, since brittleness usually traces back to insufficiently diverse training data.",
      "Decontaminate training data against all downstream benchmarks with a low n-gram threshold before believing any of your own eval numbers.",
      "When adapting a model to a domain, make the majority of the mid-training mix representative of the pre-training distribution — that's what prevents catastrophic forgetting while still gaining domain capability, and it makes downstream post-training 2–3x more effective without changing the post-training data at all.",
      "Design pre-training, mid-training and post-training synergistically instead of handing each stage to a separate team; and de-risk large runs by fitting scaling lines from small curated runs at 50–100x less compute before launching the hero run.",
      "When generating synthetic data, rephrase your highest-quality documents into many formats rather than sampling documents at random — and note that repeating high-quality data usually beats showing low-quality data, up to a threshold."
    ],
    "topics": [
      "data-quality",
      "data-curation",
      "pretraining",
      "synthetic-data",
      "scaling-laws",
      "multilingual",
      "mid-training",
      "vlms"
    ],
    "tools": [
      "DatologyAI",
      "BeyondWeb",
      "MAmmoTH dataset",
      "Gopher filters",
      "Qwen 3 / Qwen 3.5 (incl. 4B)",
      "InternVL",
      "Liquid AI models",
      "Cohere",
      "Thomson Reuters",
      "Arcee AI (Trinity Large, 17T-token open model)",
      "GLM-5",
      "Kimi",
      "Claude",
      "OpenAI",
      "Google Gemini",
      "Meta",
      "NVIDIA H100",
      "LegalBench",
      "multilingual MMLU",
      "'Beyond Scaling Laws' (NeurIPS best paper)"
    ],
    "quotes": [
      {
        "text": "Data quality is a compute multiplier because what it does is it makes the learning curve steeper.",
        "at": "02:09",
        "url": "https://www.youtube.com/watch?v=_PdK6x7PQNM&t=129s"
      },
      {
        "text": "There's no one golden data set to rule them all that's good for everything no matter what you want to do.",
        "at": "03:18",
        "url": "https://www.youtube.com/watch?v=_PdK6x7PQNM&t=198s"
      },
      {
        "text": "Even if you don't change the post-training data at all, showing your model better domain specific data can actually make post-training two to three times more effective out of the box.",
        "at": "15:50",
        "url": "https://www.youtube.com/watch?v=_PdK6x7PQNM&t=950s"
      },
      {
        "text": "So if you hear this story over and over again, oh, if I want to customize a model, it's going to cost hundreds of millions of dollars. That's just not true.",
        "at": "17:24",
        "url": "https://www.youtube.com/watch?v=_PdK6x7PQNM&t=1044s"
      }
    ],
    "words": 4709
  },
  "summary_url": "/#_PdK6x7PQNM",
  "transcript": {
    "html": "/transcripts/_PdK6x7PQNM.html",
    "txt": "/transcripts/_PdK6x7PQNM.txt",
    "vtt": "/transcripts/_PdK6x7PQNM.vtt"
  }
}