{
  "video": {
    "id": "MpkARD5pq-M",
    "title": "Robots Will Be Trained on Video Games - Pim de Witte, General Intuition",
    "duration": 1131,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "MpkARD5pq-M",
    "title": "Robots Will Be Trained on Video Games - Pim de Witte, General Intuition",
    "one_liner": "General Intuition is betting that gameplay footage paired with controller inputs from Medal, which yields over 1.5 trillion action tokens a month, can train one real-time pixels-in, actions-out model that plays games at the highest level and controls over 10 types of robots.",
    "summary": "Pim de Witte argues that turn-based, token-predicting LLMs are a poor fit for environments that unfold in space and time, because an agent there must keep observing as it acts. Medal, the company's video game recorder, links inputs and actions to frames, and game controllers often share an action space with robot teleoperation, so game data can train vision-based action models. Just as Anthropic bet on scaling coding space, General Intuition is betting on scaling pixel space and expects people to bring more problems into it. No model demo is shown because the talk is live-streamed; early access and hackathons are promised instead.",
    "key_points": [
      "LLMs suit turn-based work such as code, but struggle to generalize to environments that include space and time. General Intuition's models instead receive frames or batches of camera or screen information continuously and predict actions in real time.",
      "The action space of video games, because of game controllers, is in many cases the same as that of teleoperation. De Witte says every robot below a certain number of degrees of freedom ships with a game controller.",
      "Most robotics companies try to predict exact poses or motor values end to end. General Intuition instead uses the controller layer that robotics companies already built, so the model outputs keyboard-and-mouse or controller actions.",
      "Medal sees about a billion videos uploaded per year, is on track for 3 billion this year, and produces over 1.5 trillion action tokens per month. De Witte claims it collects in half a day the total data frontier labs use today to train robotics models.",
      "Games are far more action-dense than teleoperation, and the dataset's high APM brings complex workloads within reach, such as a drone that needs many small adjustments. Actions are also semantically labeled, so the model understands the meaning of each key it predicts, which he says matters for interpretability.",
      "Learning in the human's native pixel modality avoids a slow pipeline of passing data through an LLM, creating a digital twin, deploying the model and running verifier checks before the robot learns. De Witte says the models will be significantly faster than any LLM at these tasks.",
      "The models are meant to sit under an orchestrator such as Gemini or Astra, which handles long-term planning. Proposed uses include exploring generated environments, testing whether a UX is intuitive, orchestrating robots in science labs, and accessibility, such as a player using only one arm controlling the mouse while the model predicts the keyboard actions.",
      "Asked about Jeppa, de Witte says it doesn't train alongside action labels and bets on inferred ones, so it needs a separate scaling law for action inference. He calls it probably more correct in the limit but less practical in the short term, and notes that action data captures what frames cannot, such as a plane's rudder movement."
    ],
    "takeaways": [
      "Check whether a workload is turn-based or unfolds as the agent acts, and consider a purpose-built pixels-to-actions model rather than an LLM for the latter.",
      "Keep an LLM as the orchestrator for long-term planning, and plug real-time action models into the workload for execution.",
      "For control tasks that need many small adjustments, such as drone flight, start from high-APM game data rather than teleoperation data, which contains far fewer actions.",
      "Join the waitlist early and attend a hackathon if you want access: access is prioritized by signup time, and the first hackathon is planned for New York City in mid-November.",
      "Learn these primitives early if you build products, since de Witte expects being early to pay off as it did with early LLMs, including businesses built on top."
    ],
    "topics": [
      "robotics",
      "world models",
      "computer use",
      "action prediction",
      "video game data",
      "real-time agents",
      "simulation",
      "accessibility"
    ],
    "tools": [
      "General Intuition",
      "Medal",
      "Anthropic",
      "Gemini",
      "Astra",
      "Jeppa",
      "Mira"
    ],
    "quotes": [
      {
        "text": "Code is turn-based.",
        "at": "03:55",
        "url": "https://www.youtube.com/watch?v=MpkARD5pq-M&t=235s"
      },
      {
        "text": "we are generally pixel-to-action converters plus sensors",
        "at": "04:55",
        "url": "https://www.youtube.com/watch?v=MpkARD5pq-M&t=295s"
      },
      {
        "text": "The total amount of data that Frontier Labs used today to train robotics models, we collect in at the moment half a day.",
        "at": "06:39",
        "url": "https://www.youtube.com/watch?v=MpkARD5pq-M&t=399s"
      },
      {
        "text": "Jeppa I think in the limit is the more correct approach, but it's just a lot less practical in the short term",
        "at": "17:26",
        "url": "https://www.youtube.com/watch?v=MpkARD5pq-M&t=1046s"
      }
    ],
    "words": 4032
  },
  "summary_url": "/#MpkARD5pq-M",
  "transcript": {
    "html": "/transcripts/MpkARD5pq-M.html",
    "txt": "/transcripts/MpkARD5pq-M.txt",
    "vtt": "/transcripts/MpkARD5pq-M.vtt"
  }
}