{
  "video": {
    "id": "ZyIoTOAbRfs",
    "title": "State of Data — Sean Cai, Independent / State of Data",
    "duration": 1102,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "ZyIoTOAbRfs",
    "title": "State of Data — Sean Cai, Independent / State of Data",
    "one_liner": "A data-market insider argues the frontier bottleneck isn't labeling volume but verification — and that most \"hard\" benchmarks are contrived type-two data sold alongside the data to hill-climb them.",
    "summary": "Sean Cai reframes AI data markets away from the 2019 Scale-style annotation picture (~$10–15B/year per lab, and \"the least interesting part\") toward process-based data — trajectories and reasoning traces that show how expert work actually gets done. He argues the supply chain has permanently unbundled into specialists for sourcing, environments, reward design and evals, with labs diversifying across 20–30 vendors because quality doesn't scale linearly with quantity. Using Jason Wei's Verifier's Law he breaks verifiability into three axes to predict which domains mature next, then shows private finance benchmarks where Opus 4.8 regresses against 4.7 and GPT 5.5 and Opus fail in opposite directions on the same task. He closes on the model layer — no infrastructure pioneer has historically held over 10% of the market — and announces he is building Antikythera mechanisms and RL-as-a-service for enterprises.",
    "key_points": [
      "The annotation market everyone pictures is ~$10–15B/year per lab but is the least interesting part; the scarce, badly-priced data is what moves a model from generalist competence to real expertise.",
      "Type one data is pure capture of real workflows (GitHub commits, session replays) with minimal reward shaping; type two is hiring experts in an arbitrary setting to manufacture examples. Type two was right when models read at a first-grade level; type one is what gets you from 20 to 80%. \"Everybody sells type two and bills it as type one.\"",
      "State-based data (rows in an ERP, the saved file) is the 2023 next-token-prediction picture and is mostly personal data wrapped in privacy law; what is actually available now is process-based data — the trajectory, the reasoning trace, the sequence of decisions from blank page to finished output.",
      "Verifier's Law broken into three axes: asymmetry (can the task decompose into checkable steps), veracity (is there consensus on what correct means), proliferation (how often the real world hands you fresh verified examples). Coding matured first because GitHub scored high on all three — unit tests, community agreement, and commit messages as free reasoning traces. Biology, security, finance, healthcare, law and taste score low, which predicts the maturation order: code → search → finance → healthcare/law → cyber/bio/scientific discovery → taste.",
      "The perverse benchmark recipe: hire domain experts, generate plausible tasks via chat, cherry-pick where the model diverged, package as a hard North Star benchmark — then sell the data to hill-climb that same benchmark. \"Goodhart's law with a profit motive,\" leaving labs, vendors and enterprises in a fog of war.",
      "Three real-vendor finance tasks (ARR waterfall reconciliation, LBO valuation memo, long-short pair trade) run with deterministic verifiers plus LLM-as-judge: Opus 4.8 scores worse than 4.7 on many rubrics from over-engineered self-reflection; GPT 5.5 and Opus 4.8 land within three points but fail in opposite directions — GPT nails arithmetic, Opus nails methodology and loses arithmetic. Cross-harness and cross-infrastructure differences are the primary cause of benchmark divergence, hiding high false-positive and false-negative rates.",
      "Data markets are an upstream indicator of lab products: Anthropic bought heavily on cybersecurity data in January and biological data in March–April, and Metis, Cybor and Claude bio/life sciences shipped two to three months later.",
      "Robotics is the counterexample — the modality is unsettled (ego vs teleop vs UMI) and vendors are deeply unsophisticated (\"100,000 hours of iPhone video from my friends in India\"), so vendor choice is entangled with an unsolved research question. \"Don't die on a modality hill.\"",
      "No pioneer of an infrastructure technology has held more than 10% of the market long-run; railroads got nationalized when the automobile arrived, AWS and Google never captured the app layer, and GLM 5.2 surpassing GPT on real-world rubrics is proof app-layer companies can decouple from the model layer. Successful data companies are quietly pivoting to enterprise — \"data companies all realize that they have to be neo labs.\""
    ],
    "takeaways": [
      "Stop outsourcing your definition of realism to the same vendors you buy evals and tasks from — that is letting the test writer grade the test.",
      "Never read a single benchmark number as a measurement; it is one sample from a distribution under one scaffold. Do rubric-level analysis across harnesses to see where a post-training team actually went.",
      "Score any candidate domain on asymmetry, veracity and proliferation of verification before betting on it, then apply the long-horizon bar labs use: enforced step length, heterogeneous non-interchangeable tool calls, state transitions that constrain future actions, and mandatory failure recovery.",
      "Source data from live businesses you partner with, not dead startups' code bases — a dataset is only valuable as long as the frontier hasn't moved past it.",
      "If you are building, treat the moat as the pipeline into real-world work plus the infrastructure to keep retraining on it — including managing RL datasets across base-model migration so swapping to a new open-source base reruns post-training instead of starting over."
    ],
    "topics": [
      "data-markets",
      "evals",
      "benchmarks",
      "rl-environments",
      "verification",
      "post-training",
      "enterprise-ai",
      "agents"
    ],
    "tools": [
      "Scale AI",
      "Surge",
      "Mercor (transcribed as \"Remarque\"/\"Mercari\")",
      "Anthropic",
      "OpenAI",
      "Claude Opus 4.7 / 4.8",
      "GPT 5.5",
      "GLM 5.2",
      "Claude bio / life sciences",
      "Metis",
      "Cybor",
      "GitHub",
      "Vals AI",
      "GPQA",
      "Slack",
      "Jira",
      "AWS",
      "Google",
      "UMI",
      "Substack",
      "Twitter"
    ],
    "quotes": [
      {
        "text": "The dirty secret of the industry though is that everybody sells type two and bills it as type one.",
        "at": "05:43",
        "url": "https://www.youtube.com/watch?v=ZyIoTOAbRfs&t=343s"
      },
      {
        "text": "It's Goodhart's law with a profit motive. Basically, the moment your measure becomes a target, and then the target is set by people who aren't true domain experts, it stops measuring anything real.",
        "at": "08:39",
        "url": "https://www.youtube.com/watch?v=ZyIoTOAbRfs&t=519s"
      },
      {
        "text": "A single benchmark number is a sample from a distribution nobody measured, and it's basically like taking a Swiss Army knife and using the screwdriver bit to cut cheese and concluding the knife is broken.",
        "at": "11:26",
        "url": "https://www.youtube.com/watch?v=ZyIoTOAbRfs&t=686s"
      },
      {
        "text": "If you're a builder, your moat's not the data, it's the pipeline into real-world work. Plus the infra to keep retraining on it as the models improve underneath you.",
        "at": "17:25",
        "url": "https://www.youtube.com/watch?v=ZyIoTOAbRfs&t=1045s"
      }
    ],
    "words": 3934
  },
  "summary_url": "/#ZyIoTOAbRfs",
  "transcript": {
    "html": "/transcripts/ZyIoTOAbRfs.html",
    "txt": "/transcripts/ZyIoTOAbRfs.txt",
    "vtt": "/transcripts/ZyIoTOAbRfs.vtt"
  }
}