{ "video": { "id": "7RFJRDYroic", "title": "Parameter Golf with AutoResearch — Vayum Arora, Zhengyao Jiang, Dixing Xu & Dhruv Srikanth, Weco AI", "duration": 2053, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "7RFJRDYroic", "title": "Parameter Golf with AutoResearch — Vayum Arora, Zhengyao Jiang, Dixing Xu & Dhruv Srikanth, Weco AI", "one_liner": "AutoResearch is a loop where an agent tries ideas against a fast, cheap, low-noise eval that returns one number, and keeps whatever improves the score. Weco's agent Aiden used it, plus human steering, to become the top contributor to OpenAI's Parameter Golf challenge.", "summary": "The Weco AI team explain AutoResearch as a trial-and-error loop: a language model is given your code and a replayable evaluation script, and it keeps only the ideas that improve the score. They show how their agent Aiden landed 7 merged leaderboard records in OpenAI's Parameter Golf challenge, which was twice as many as the next best human, while using only 4% of the community's GPU compute. They then break down what goes into a run: the source file, the eval, the metric and goal, the step count, the timeout and the guidance. They argue that production-grade AutoResearch needs tree-based search, observability and steerability, because real breakthroughs came from folding human ideas back into the loop.", "key_points": [ "AutoResearch: give an LLM your code and a replayable eval script. It keeps trying ideas, keeps the ones that improve the score and discards everything else. It went viral after Karpathy posted about it for model training. It has since been used to cut latency in Shopify's CI/CD pipeline and to improve Browser Use's agent.", "OpenAI's Parameter Golf: train the best language model you can, with weights plus training code under 16 MB and training finished within 10 minutes, scored by bits per byte (BPB) on the FineWeb validation set. Over 44 days there were about 1,000+ researchers, 248 PRs and roughly $250k of GPU compute, but only 47 PRs from 31 contributors were merged as records.", "Aiden is a self-improving multi-agent system built on the Claude and GPT model families. It pulls papers from arXiv, implementations from GitHub and its own past logs, then proposes ideas, tests them, audits them for legality and submits a PR once results clear quality gates.", "Aiden's results: 7 merged records (twice the next best human), the highest PR h-index in the competition, and 435 downstream PRs that built on its work. From 4% of the community's compute it got 15% of merged records, which they put at 3.7x more cost-effective than the field average.", "Their earlier agent AIDE (April 2024) scored about 4x the medal rate of the next best agent on OpenAI's MLE-bench.", "Anatomy of a run: a source file (the search space, here solution.py), an eval that outputs one scalar, a metric and goal (minimise validation BPB), a step count (20 for the workshop), a bounded eval timeout (210 s = 60 s training + 60 s testing + 90 s buffer for Modal GPU spin-up) and guidance in a markdown file.", "Steerability in practice: Aiden plateaued for five days (April 15–19). Then a community member shipped a clever new tokenizer, and within eight hours of it being merged Aiden fused it with its own work and shipped the biggest jump in weeks.", "For multiple objectives, combine them into one metric with weights, or treat some as constraints and optimise the rest. Parameter Golf holds model size as a constraint and optimises accuracy. AutoResearch also works on RL training scripts; they cited the Speedrunner project post-training a Qwen model to play a game." ], "takeaways": [ "Spend the most effort on the eval. It should be fast, cheap and low-noise and output a single scalar, because that sets how many steps you can afford and how confident the hill-climbing is. Harden it, because agents love to reward hack.", "Put a time limit on every eval run so stalled loops, failed hardware or hanging agent code can't freeze the search.", "Write your domain knowledge into a guidance markdown file up front: hard constraints, promising directions (for example compression, quantization, weight sharing) and things that don't work.", "Use tree-based search so ideas compound, and inspect node diffs to see why the metric moved. Inject your own ideas or outside work mid-run when the agent plateaus, rather than treating it as set-and-forget.", "Make small, focused edits from a working baseline instead of rewriting, and score on a hidden validation set so the agent can't overfit." ], "topics": [ "autoresearch", "agents", "evals", "ml-training", "reward-hacking", "human-in-the-loop", "optimization" ], "tools": [ "Weco", "Aiden", "AIDE", "OpenAI", "Parameter Golf", "MLE-bench", "Modal", "Claude", "Claude Code", "GPT", "Opus 4.8", "FineWeb", "arXiv", "GitHub", "Shopify", "Browser Use", "Qwen", "Flappy Bird", "Nintendo Switch" ], "quotes": [ { "text": "pointed at the right eval you basically let it boil the ocean for you, but otherwise you're just burning through all your RM tokens", "at": "03:13", "url": "https://www.youtube.com/watch?v=7RFJRDYroic&t=193s" }, { "text": "agents love to reward hack. So, they love to find holes in your evaluation protocol, holes in your benchmark.", "at": "12:55", "url": "https://www.youtube.com/watch?v=7RFJRDYroic&t=775s" }, { "text": "Running auto research on something this hard is less of a set it and forget it. Uh, it's a collaboration where we steer the agent.", "at": "20:25", "url": "https://www.youtube.com/watch?v=7RFJRDYroic&t=1225s" }, { "text": "At least for now, we believe it's important to find ways to fold human judgment into the system whenever it's necessary.", "at": "19:31", "url": "https://www.youtube.com/watch?v=7RFJRDYroic&t=1171s" } ], "words": 5534 }, "summary_url": "/#7RFJRDYroic", "transcript": { "html": "/transcripts/7RFJRDYroic.html", "txt": "/transcripts/7RFJRDYroic.txt", "vtt": "/transcripts/7RFJRDYroic.vtt" } }