{
  "video": {
    "id": "oVsEddfhdxc",
    "title": "We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect",
    "duration": 1179,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "oVsEddfhdxc",
    "title": "We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect",
    "one_liner": "Prime Intellect let Claude Code (Opus 4.8) and Codex (GPT 5.5) run unattended on the NanoGPT optimizer speedrun and both beat the human record, but neither came up with a truly novel optimizer: they stacked known papers for incremental '+1' gains.",
    "summary": "Elie Bakouch argues that nobody outside the big labs can currently measure claims about recursive self-improvement, and that speedruns make a good open benchmark and training environment for automated AI research. The speedruns (Keller Jordan's modded-nanogpt and the newer optimizer speedrun) have clear rules, a verifiable target loss and short runs. Prime Intellect ran Codex and Claude Code autonomously on their Slurm cluster. Both agents stayed ahead of the human record for most of the run. They behaved very differently, and neither produced a genuinely new optimizer or mechanism. The team is now building a structured multi-model, multi-track benchmark and an AlphaEvolve-style multi-agent loop aimed at discovery.",
    "key_points": [
      "Setup: Karpathy reproduced GPT-2 validation loss in 90 minutes. The community's modded-nanogpt, led by Keller Jordan, cut that to under 2 minutes over about two years. The optimizer speedrun allows only optimizer-related changes (e.g. Adam to Muon or Shampoo) and is framed as 'more researchy'. Each run takes about 15-20 minutes.",
      "Harness: a homemade goal.md plus agents.md define the rules. The agent proposes ideas, submits sbatch jobs on preemptible Slurm nodes, reads the training logs and decides whether it has a record. A record must pass a statistical threshold so it isn't just random noise. The agent was restarted in versions V1-V3; V3 pulled in the latest human records and improved on them.",
      "Agent behaviour: Claude Code stopped every 9-10 hours, saying it could not beat the record, and sat idle about a third of the time. Codex worked almost continuously and never asked questions. Codex wrote far more to the scratch pad (in a 'super robotic' tone, while Claude used emoji), spawned more subagents and used far more tokens (around a billion in total, mostly cached input). It also compacted about 20 times an hour on a 250k context window; Claude compacted about once in the whole run.",
      "Result: against a human record of about 2,990 steps, Claude beat it by roughly 50-60 steps. For Codex the speaker said 'like 20 step above', which is ambiguous. For almost the whole run both agents were ahead of the human record progression, and Claude was very fast early on.",
      "Follow-up benchmark (not yet released): multiple seeds and identical conditions across models and harnesses. It has three tracks: model weights only (no access), arXiv papers only, and full access including the latest human records. It will cover both the NanoGPT and the optimizer speedruns.",
      "Early optimizer-speedrun results over about 5-6 days: Claude was strong and improved steadily. Kimi K2.7 Code made a step-function breakthrough on day 4 and beat Codex. The GLM run was still going. Measured by output tokens, Claude in max mode used far more than Codex or Kimi, and Kimi was very token-efficient.",
      "Claude searched papers heavily. It found one that no other model found, and that paper led to the best record.",
      "Main finding: the agents combined papers and made '+1 improvements' over existing methods, but produced no novel optimizer or mechanism, even on a problem human researchers can reach within days or weeks.",
      "Proposed discovery loop inspired by AlphaEvolve: closed and cost-effective open-source generator models suggest ideas, the speedrun supplies the reward, and a judge with 'taste' gives quality feedback. Promising methods are scaled up to more parameters and tokens, because speedrun tricks often don't work at large scale. Humans steer the process. Varying the speedrun's objectives and constraints would create more diversity."
    ],
    "takeaways": [
      "Use tasks with clear rules and a verifiable target, such as speedruns, to evaluate or train research agents. Require records to pass a statistical threshold so noise doesn't count as progress.",
      "Plan for different autonomy profiles. Claude Code may give up and go idle after hours, so it needs monitoring and nudges ('continue, explore new direction'). Codex runs continuously but uses far more tokens and compacts heavily.",
      "Compare agents on token and cost axes as well as wall-clock time. Kimi K2.7 Code looked much more competitive per output token.",
      "Don't expect today's frontier agents to invent new methods on their own. Expect them to recombine literature, and use multi-agent generator/judge loops, human steering and scale-up tests to push toward real discovery.",
      "Separate what a model knows from its weights from its ability to use literature, by running it with no access, papers only and full access."
    ],
    "topics": [
      "automated ai research",
      "agents",
      "benchmarks",
      "speedruns",
      "optimizers",
      "recursive self-improvement",
      "reinforcement learning environments",
      "open research"
    ],
    "tools": [
      "Prime Intellect",
      "Claude Code",
      "Claude Opus 4.8",
      "Codex",
      "GPT 5.5",
      "Kimi K2.7 Code",
      "GLM",
      "modded-nanogpt",
      "NanoGPT speedrun",
      "optimizer speedrun",
      "GPT-2",
      "Slurm",
      "Adam",
      "Muon",
      "Shampoo",
      "AlphaEvolve",
      "Google",
      "arXiv",
      "verifiers",
      "prime-rl"
    ],
    "quotes": [
      {
        "text": "code clothes code keep stopping every nine or 10 hours and basically said yeah I cannot improve the record it's too hard for me",
        "at": "07:49",
        "url": "https://www.youtube.com/watch?v=oVsEddfhdxc&t=469s"
      },
      {
        "text": "CL was super excited about getting new record with a bunch of emoji and so on and CEX was just like here is what I do here is the decision I take what I will do next like super robotic kind of",
        "at": "09:15",
        "url": "https://www.youtube.com/watch?v=oVsEddfhdxc&t=555s"
      },
      {
        "text": "there was really like no novel optimizer or mechanism that was uh coming from those model",
        "at": "15:08",
        "url": "https://www.youtube.com/watch?v=oVsEddfhdxc&t=908s"
      },
      {
        "text": "I think it's super important to have uh a part of like this recursive self-improvement to happen in the open",
        "at": "18:57",
        "url": "https://www.youtube.com/watch?v=oVsEddfhdxc&t=1137s"
      }
    ],
    "words": 3480
  },
  "summary_url": "/#oVsEddfhdxc",
  "transcript": {
    "html": "/transcripts/oVsEddfhdxc.html",
    "txt": "/transcripts/oVsEddfhdxc.txt",
    "vtt": "/transcripts/oVsEddfhdxc.vtt"
  }
}