{
  "video": {
    "id": "iCj_ATyThvc",
    "title": "How Autoresearch is changing ML research — Zhengyao Jiang, Weco",
    "duration": 976,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "iCj_ATyThvc",
    "title": "How Autoresearch is changing ML research — Zhengyao Jiang, Weco",
    "one_liner": "Weco's autoresearch agent Aiden entered OpenAI's 'Parameter Golf' hiring competition and out-contributed ~1,000 human ML engineers — 7 leaderboard records to the best human's 3, and the highest h-index over PRs in the community — which the speaker reads as proof that agents excel at execution while humans move up the stack to designing evals and codebase abstractions.",
    "summary": "Zhengyao Jiang argues the interesting question isn't whether autoresearch agents can climb a benchmark, but whether they can produce work a human community actually recognizes, merges and builds on. He shows Aiden, Weco's multi-agent self-improving system, doing exactly that in OpenAI's Parameter Golf competition: 7 leaderboard records in 22 days on a single H100 node, ~4% of the competition's total compute, a 28% submission hit rate (~6x the community average), and the highest h-index over PRs. Tracing where Aiden's winning ideas came from, almost all originated in human papers, other participants' PRs, or abandoned notes — so the division of labour is humans supplying creative ideas and agents supplying execution. His conclusion: doing autoresearch is like training a model, where your eval is the loss function and data and your codebase abstraction is the architecture, so creativity and the judgement to design those become exponentially more valuable.",
    "key_points": [
      "OpenAI ran a hiring competition in April called Parameter Golf — train the best language model under size and compute constraints. ~1,000 ML engineers and researchers took part, firing 2,000 submissions; only 47 passed open review onto the leaderboard, and 7 of those were from Aiden, an agent, 'more than twice what any human contributed.'",
      "Aiden ran for about 22 days and set 7 leaderboard records (each a new competition best, stamped by OpenAI); the best human made 3.",
      "On the community-recognition measure — h-index computed over PRs — Aiden scored 10 versus 7 for the next human, with many other leaderboard entries building on its work.",
      "Efficiency, not brute parallelism: ~1,300 experiments on a single H100 node, at most 4% of the competition's total compute, producing ~15% of the records; 28% of its submissions made the leaderboard, roughly 6x the community average hit rate.",
      "Idea provenance: almost all of Aiden's record PRs trace to human research papers, other Parameter Golf participants, or adjacent communities like nanoGPT — including ideas humans explicitly abandoned in notes because of implementation difficulty. Only a very small fraction were original, and those emerged from navigating the file size constraint.",
      "Worked example: Aiden took gated attention from a Qwen paper — it worked but added parameters and broke the 16 MB file size limit, so it devised a quantization mechanism to fit; combined, the score barely moved. Five days later another contributor posted a tokenizer improvement, Aiden recognised and combined all three, and the synergy produced a big jump and a leaderboard record.",
      "Weco is an autoresearch company founded ~2.5 years ago; two years ago it built AIDE, the top autoresearch agent as independently evaluated by OpenAI in the MLE-bench paper. Aiden is the next step — a multi-agent self-improving prototype that reads public papers and PRs, runs its own experiments, and submits a PR only once findings pass a quality gate.",
      "Fraud-detection case study on abstraction: with a loose API where one function processed both train and test data, scores looked great but the solution was polluted by test-set leakage; tightening to a strict API where test data couldn't reach training dropped the leakage rate to zero."
    ],
    "takeaways": [
      "Treat your eval as the loss function and the data — it's what the agent actually optimizes for, and a proprietary eval or a unique understanding of what matters in your field is where a vertical moat gets built as autoresearch gets stronger.",
      "Treat your codebase abstraction as the architecture: it biases the whole search direction. Tighten APIs so the shortcut isn't reachable (the strict train/test API that zeroed out leakage) rather than hoping the agent won't reward hack.",
      "Point agents at the noisy public channel, not just at the search space — papers, other people's PRs, and ideas humans abandoned over implementation difficulty are the richest seam, because agents are strong at finding and implementing existing ideas and at finding combinations across a huge space.",
      "Don't judge an autoresearch run only by score — measure whether other people build on the output (Aiden's PR h-index), and watch compute efficiency and submission hit rate rather than reaching for massive parallelization.",
      "Invest in the skills that get amplified: designing the competition/eval, the abstraction, and driving these systems — a skill that 'barely existed one or two years ago.'"
    ],
    "topics": [
      "autoresearch",
      "agents",
      "evals",
      "ml-engineering",
      "benchmarks",
      "reward-hacking",
      "abstractions",
      "human-ai-collaboration"
    ],
    "tools": [
      "Weco",
      "Aiden",
      "AIDE",
      "OpenAI",
      "MLE-bench",
      "Parameter Golf",
      "NVIDIA H100",
      "Qwen",
      "nanoGPT",
      "X"
    ],
    "quotes": [
      {
        "text": "The top contributor was one candidate that they couldn't hire. It wasn't a person, it's an agent we build called Aiden.",
        "at": "00:20",
        "url": "https://www.youtube.com/watch?v=iCj_ATyThvc&t=20s"
      },
      {
        "text": "So Aiden actually lifted the signal noise ratio within the whole community's public communication channel, which is a PR. It didn't win through massive paralization.",
        "at": "05:00",
        "url": "https://www.youtube.com/watch?v=iCj_ATyThvc&t=300s"
      },
      {
        "text": "Okay, maybe none of those sounds very sexy. Most of them are just a good execution. But in reality, execution is a mostly the bottleneck.",
        "at": "08:41",
        "url": "https://www.youtube.com/watch?v=iCj_ATyThvc&t=521s"
      },
      {
        "text": "So the search is automated. the human would just move up the stack not out of it.",
        "at": "15:22",
        "url": "https://www.youtube.com/watch?v=iCj_ATyThvc&t=922s"
      }
    ],
    "words": 2095
  },
  "summary_url": "/#iCj_ATyThvc",
  "transcript": {
    "html": "/transcripts/iCj_ATyThvc.html",
    "txt": "/transcripts/iCj_ATyThvc.txt",
    "vtt": "/transcripts/iCj_ATyThvc.vtt"
  }
}