Parameter Golf with AutoResearch — Vayum Arora, Zhengyao Jiang, Dixing Xu & Dhruv Srikanth, Weco AI

AI Engineer · 34 min · 330 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Uh, welcome.
00:14Please grab a seat if you haven't already.
00:16My name is Vam.
00:17We have Jung Yao, Droo, and Dex.
00:19Before anything else, uh, a huge thank you to Sean and the AI engineers worldfare for having us.
00:26This is generally um a great room of AI and we're really really happy to be here.
00:32Um if you haven't been able to get set up yet, no worries.
00:35Uh we're going to have ample time to get it to to it later.
00:38Um Junga will give us a brief background on the basics behind auto research.
00:52Okay.
00:53Um in case any of you still don't know uh auto research is this is just a quick brief.
01:01So auto research is basically a loop of a try and errors.
01:06You give a language model your code and a replayable evaluation script.
01:13It will keep trying ideas and test them against the eval and it keeps the ideas that improve the score and discard everything else.
01:24So some example metrics includes accuracy, speed uh and cost.
01:31Similar loop has been exist uh in academic for a while like two years but it really got viral in this year when Kapasi uh posted about it in his model training
01:48scripts and turned out works really well for him.
01:55In just a few weeks, it just the use case of auto research spread from training to everything like a backend engineering, prompt and agent engineering, speed up model inference
02:09and it also quickly show up in production.
02:13For example, it reduce the latency of Shopify's CI/CD pipeline and improve the browser use the company's browser use agent.
02:27In April, our experimental research agent Aiden even won OpenAI's hiring challenge uh called the prime golf uh which we will tell you a bit more soon and uh I also have a talk on Wednesday specifically
02:44about it.
02:46Um so okay auto research can be really powerful but it's also hard to master with a normal coding agent you basically steer um agent every single step but for auto research it's like the agent is running in a loop
03:06on its own so it's much bigger multiplier on your prompt and your setup pointed at the right eval you basically let it boil the ocean for you, but otherwise you're just
03:20burning through all your RM tokens.
03:23Um so hopefully this workshop will give you a good starting point to learn about auto research and um apply that in your own work.
03:34Okay, we'll come back to Bay.
03:40Thanks Janelle.
03:41Um so here's how we'll spend the next two hours.
03:44Um, Dex will take you inside parameter golf, uh, the challenge that we entered our agent into.
03:50Um, what that challenge was actually all about and how our agent ended up as the most influential contributor.
03:57Um, then it's your turn.
03:58Uh, we'll get you set up and kick off an auto research run on your laptop.
04:03While your auto research is running in the background, we'll break down the basics of an auto research loop and the pieces that actually shape what an agent does.
04:11Um, and we'll go deeper and do a guided demo where you're able to run, steer, and inspect what your agent is doing live.
04:19And we'll finish with a fun part, which is a live competition.
04:23Um, you'll take everything that you learned today, point auto research at a game that we've all played before, and go head-to-head to see whose agent can get the highest score
04:35uh, with the winner walking away with a really cool prize.
04:42a little bit more on who we are.
04:44Um, so we've been working on auto research for quite some time now.
04:47Before WICO, uh, we released aid in April 2024, which was an autonomous ML engineering agent.
04:55The basic idea was give it a machine learning problem, let it write code, run experiments, read the results, and keep improving.
05:04When OpenAI built MLE Bench, aid was one of the first agents they evaluated independently and on that benchmark it scored about four times the medal rate of the next best agent.
05:17More recently, as we discussed, we pointed our agent to parameter golf and across more than a thousand research it researchers, it became the number one contributor.
05:25I'm going to hand it off to Dex who's going to tell us more about the competition.
05:39Hello everyone, my name is Dex.
05:42Uh so I'm going to talk about the paramet golf.
05:45So paramet golf is open as uh model craft challenge.
05:50The goal is to train the best language model you can under two hard constraints.
05:56uh once the four artifacts including model weights plus training code has to fit in 16 megabytes and the training has to finish within 10 minutes.
06:06Submissions are scored by bits per bpb on the fine wide validation set and the low is better.
06:14Openi provided a baseline data set and evaluation scripts so participants can for the ripple improve the model and submit their result through GitHub.
06:25So everything is open sourced.
06:27They also put up a million dollar in computer credits and reviewed every submission.
06:33During the 44 day competition, there were,6 researchers entered and filed 248 PR roughly about uh 250k of GPU compute burn.
06:46The final lead board is short.
06:50Only 47 PRs merge as records from 31 contributors.
06:55And for this competition, we build Aiden, a self-improving multi-aging system based on the cloud and GPT model families that run indefinitely to help clamp on a single metric.
07:07In permit golf, Aiden automated the four research loop.
07:12It pulls public page papers from archive, strong implementations from GitHub and its own part logs.
07:19Then propose new ideas, tests them, audit them for legality, keep the ones that work, write the implementation and run the parents.
07:30Once the results, clear the quality gates, it will create and submit a PR for review.
07:38And the results are remarkable.
07:40Aiden landed seven mer leaderboard records.
07:43That's twice the next best human.
07:45It also had the highest OPR H index in the whole competition which means Aiden didn't just produce accept records.
07:53It published reusable work that other participants kept building on 435 downstream PR stated and built on Aiden's work and it got there on a fraction of the budget from just 4% of the community's GPU compute.
08:10It got 15% of total merged records.
08:13That's 3.7 times more cost effective than the field average.
08:18And this wraps up a how Aiden becomes the number one contributor to the open herring challenge.
08:24Thank you.
08:30Cool.
08:31So, we're actually going to run auto research on a version of the same parameter golf challenge that Dex just walked you through.
08:39Uh, for future reference, we will just call this mini PG, which stands for mini parameter golf.
08:45Um, as Dex discussed, the real parameter golf competition was heavy.
08:50Every single experiment was training a language model on a massive data set, which is fine when the competition is over four weeks, but in a workshop like today, you'd never actually be able to see your auto research run.
09:03So, think of mini PG as a scale model of the challenge.
09:09uh you're still going to train a real language model, score it on the same fixed evaluator, and drive down the same single number valp, which is essentially measuring how well the model predicts text it hasn't seen.
09:21But what we did is we shrank this in scale.
09:23We're using a tiny data set instead of fine web and training an eval that only takes a few minutes.
09:29So you'll actually be able to get a loop going and see several steel several real steps live.
09:37So before I get to the instructions on the next slide, just a little bit of orientation.
09:42Um you're going to run auto research yourself on mini parameter golf.
09:47You're going to sign up for two platforms if you haven't already.
09:50One is we our auto research agent that runs a search and the other is modal uh the compute your experiments train and evaluate on.
09:58Free credits.
09:59Thank you modal.
10:01Um then you will clone our parameter golf repo uh and hand it to your coding agent.
10:06tell it to kick off a run.
10:07It handles the setup and it'll start the loop.
10:10Once it's going and it's running in the background, uh you can kick it off and we will check back in in around 15 or 20 minutes to take you
10:17inside the details.
10:22Instructions are here.
10:23Um sign up for WO on dashboard.ai.
10:27Um modal for moto modto.com-signup.
10:30Uh there's a code to claim your modal credits.
10:33Um, if you already have a modal account, you can go directly to modal.com/crediteds and enter your code there.
10:39Um, and once you're done with those two steps, uh, clone the repo shown here, um, and tell your agent to set up mini PG from the readme and kick off a run.
10:50If you run into any issues or have any questions, feel free to scan the QR code and enter it there and we'll be monitoring that or uh, raise your hand and we'll come to you.
11:01check back in in around 105 minutes.
11:04Thanks, guys.
11:04Uh, all right.
11:05So, sorry about the internet problem.
11:07I think it's being worked on, but until then, I think it would be good to maybe go through uh the anatomy of an auto research run.
11:16And we're going to be doing this in the context of uh WCO as the auto research system and mini PG as the task.
11:24But really, this this kind of contract that we have here is fairly general.
11:28Most auto research systems are going to have you ask very similar questions and uh think about the same things.
11:36Okay.
11:36So the first thing is the source file.
11:39Uh this is what the agent actually has to modify.
11:43This defines the search space and where it can make changes.
11:46So this can be a single file, it can be it can be multiple files.
11:50In the case of minip it's uh this one file called solution.py and solution.py PI essentially defines the model config and the uh uh the training dynamics that's going to go on while the model is training.
12:06And then the second thing and in my opinion this is probably the most crucial part of an auto research system is the evaluation.
12:14And so you want to have a very simple way of allowing your auto research system to evaluate every solution that it generates.
12:23Um, and you want to have it output a single scalar metric.
12:28And you want the eval to be fast, you want it to be cheap, you want it to have low noise.
12:34Uh, because this essentially leads to your ability to run more steps if you have a fixed budget or a fixed time constraint.
12:42Um, and if there's low noise, then both you as well as the agent are more confident as you're kind of hill climbing on, uh, whatever task you're, uh, autoresarching
12:52on.
12:53And so, uh, one thing to be aware of though is agents love to reward hack.
12:58So, they love to find holes in your evaluation protocol, holes in your benchmark.
13:03And so, you really want to figure out how to make your eval as robust as possible.
13:10The next two things are the metric and the goal.
13:13So you really want to ask yourself two questions.
13:15Um what do I want to improve and is higher or lower better?
13:20Um and in our case for mini PG, you want to minimize the validation BP.
13:26So you essentially want to find a model that is better at predicting on unseen text.
13:35The next thing is the number of steps you actually run the auto research system for.
13:41This is a bit tricky because it's difficult to know a priority how many steps you should give your your system.
13:48We've chosen 20 in this case just because we want to give your agents enough time to kind of explore different ideas and you can you can see what the agent is actually exploring, what works, what doesn't.
14:00Um but really more ideas that gets tried uh generally leads to more breakthroughs.
14:07And so uh this goes back to what I was saying earlier where if you have a fast eval um or if you have a cheap eval then when you're under some kind of time constraint
14:17or cost constraint, you can run more steps.
14:20Um and so uh again the eval is is always super important to uh consider.
14:30All right.
14:30So you also want to bound the uh time for which your eval runs for uh because a lot of things can go wrong.
14:39So loops can stall, hardware can fail, uh sometimes the agent will produce really bad code that causes the entire system to hang.
14:48And so you want to bound the the actual evaluation running.
14:52Um in our case, we set it to be 210 seconds.
14:55And the reason for that is we give 60 seconds for uh training the model, 60 seconds for testing the model on unseen text and then 90 seconds which is a buffer for
15:08you know modal to spin up the GPUs for training, spin it up for um uh for the testing and any other kinds of uh overheads that exist.
15:21And the last thing is the guidance that you give the agent at the start of your optimization.
15:28Uh so this is how you steer the agent from the start.
15:31And you really want to think about it as you know you've probably run experiments before.
15:37You've compounded a bunch of domain expertise and human knowledge.
15:41And you might have some ideas about things that work, things that might be interesting to try and maybe things that don't work.
15:48And so this is where you can provide that kind of guidance.
15:51Um in the case of WICO, you can simply point it to a markdown file, have all of your information and context in there.
15:58Um and the agent is able to use this and uh it gives the agent the best chance at uh at finding success.
16:05Um, in the case of mini PG, this can be uh some hard constraints or different ideas that seem to work quite well like uh focusing on compression uh different quantization
16:16methods uh maybe even weight sharing across different blocks in the model.
16:31Um we are just going to demo how to get started on auto research for you considering the internet issues.
16:39Um before we get to that a little bit more about what we think um makes auto production grade auto research actually production grade.
16:50Um the first is structure.
16:52So uh we have a treeb brace search uh where the agent shows you every attempt it's run so that ideas as duv had mentioned can sort of compound on each other instead of starting from scratch.
17:04The second is observability.
17:06Um you can sort of open any attempt read the exact code change behind its score so you can see why the agent is taking the approach it is.
17:16And the third thing is steerability.
17:18So you have a sense of control.
17:20You can inject your own ideas and knowledge while the loop is running and have it try new directions in case it's stuck.
17:29A little bit more about the treebased search.
17:31Um we will shortly demo this on our auto research dashboard, but um treebased auto research essentially treats the problem as a structured optimization process over code.
17:43Uh it starts with the root of the tree.
17:45As you can see the score was 0.5 um which is just the baseline program and a measured score.
17:51Every node after that route is a modified version of the code run against your eval with a score attached.
17:58You can think of this tree as an experiment graph.
18:01As you can see the search builds on what works.
18:05We're finding good candidates and pruning bad ones.
18:07So when a branch improves a score it builds on those ideas and goes deeper next.
18:12which means that you can inspect the exact changes your auto research system is making and build off of those.
18:22So the next idea we talked about was observability.
18:26Um because you can see every one of the nodes in this treebased search, you can open any one of them and see exactly what changed.
18:35You can see your metric move in real time correlating to a movement correlating the movement to a reason.
18:41You can hover over over a node, get a summary of what changed, and you can open it and read the exact diff.
18:48So when the metric jumps, you can read the change that caused it.
18:52And typically, these runs go for hours across hundreds of experiments.
18:56So once you can see what it's doing, the next natural step is to be able to change what it does.
19:03This is essentially when you stop watching and start to take over.
19:07Um, if I have an idea I want to try in plain English, um, I take the idea, it opens a new branch and you can see it try this new idea and have
19:17new nodes develop and a new curve.
19:20I can also put a budget to each node so it spends compute where I want.
19:24And this is the part that made it usable on real problems because you might have context and intuition that an agent doesn't.
19:31At least for now, we believe it's important to find ways to fold human judgment into the system whenever it's necessary.
19:40I want to ground that point uh steerability in something real.
19:44Um because parameter because in parameter golf steerability was a big part of why we did well.
19:49Um this is an actual breakthrough from the competition.
19:52It started with our agent uh which is the line in pink.
19:56um for five days from April 15th to April 19th.
20:00Uh it kept trying new things, um new attention, compression, a richer stack, but the score essentially plateaued.
20:08There was a human in the community uh that shipped a clever new tokenizer, which is the blue line.
20:14Um, and within eight hours of emerging that change, our system picked it up, fused it with the pieces it had already been working on, and shipped the biggest jump in weeks.
20:25Running auto research on something this hard is less of a set it and forget it.
20:30Uh, it's a collaboration where we steer the agent.
20:33We pull in ideas from the people around us.
20:36And steerability is essentially what makes that loop possible.
20:39It's how real breakthroughs actually happen.
20:44Um, I'm going to hand it over to Dex and Jungao to talk a little bit more about how to get auto research set up.
20:51Um, we'll essentially mimic the process that you guys were trying to do on your laptops.
20:56Okay.
20:57Um, so what we are going to do here is we just walk you guys through the high level idea like how how does it happen?
21:08We just use our mobile hotspot.
21:11Um so we will be freestyling a little bit here.
21:15Uh hope you don't mind.
21:18Um so yeah can you just uh yeah okay maybe we can also walk through the whole doc being fed to uh claude.
21:30So
21:30yeah.
21:31So basically uh so you can go into the rebel and some like simple instruction like start the wo raw based on the readme and it should be like if the internet is good I think it should start the
21:48uh default wle run and uh we can also go over via the like the how is the wle command working.
21:58Yeah, this case you have a source file.
22:01The solution is basically the training script of mini PG.
22:06I assume like a eval command is I think Z also mentioned this is additional guidance or prompt that will guide the whole uh search process and the metric is a validation BPB.
22:21Uh this is like a hold out um uh loss like a loss only hold out set.
22:29Uh go is minimize and others are just configs we just mentioned.
22:36Oh, sorry.
22:37Eval commander is actually the the bash command that uh you're going to run the uh uh run the eval and eventually it will output a uh score like it print a score to the terminal and it's going to be passed by the agent
22:56and that's the key uh signal that will drive the whole tri and error loops.
23:05Yeah.
23:06Um in meanwhile I can also show the dashboard.
23:10So if you go into the W dashboard um and I think there I did some wrong and you you basically see a lot of things and uh after the
23:23the command like auto booted by the agent um I think then you you will see the romping popped up and then you can try out the features we provide.
23:39Yeah.
23:40Um, do we have a run just to get started?
23:45Yeah, I think it's just starting.
23:47Okay, still running saying um yeah, Claud is working.
24:05Uh question from the audience while you guys are doing that.
24:08Um one, is it possible to turn up the brightness on the dashboard or switch to light mode?
24:17Start again.
24:18I'm I'm right here.
24:20Can you turn the light mode on the dashboard?
24:22It's a little hard to see.
24:23I see.
24:28Light mode.
24:30Forgot.
24:31Where is it?
24:33Is there uh second question is can auto research apply to RL?
24:46What is the scalar metric when there are multiple things to optimize?
24:51What is the
24:53can auto research apply to RL?
24:56What is the scalar metric when there are multiple things to optimize?
25:00I see.
25:01Yeah.
25:01Um it can definitely be applied to IO like apply to IO training scripts.
25:08Um I think there is a um derived a project called sobound uh speedr runner that's actually exactly doing it like for post training aquin model uh for playing a
25:28game and uh yeah in that project you can apply auto research to the uh IR IR training scripts And um the second question you have a multiple objectives to optimize for.
25:45Um the idea is you eventually just combine them to a single metric like you either apply different weights on those different metrics or you treat some of the matrix
25:59as constraint and optimize for one of them.
26:03For example, if you're doing things like fraud detection, you might hold the uh accuracy and trying to optimize the precision.
26:13Um and uh for parameter golf exactly you hold the model size as a constraint and you optimize for the accuracy of the system.
26:25Yeah, we can see the um agent already generated uh one node.
26:35So the zero zero node is basically the root node the baseline solution and 3.93 means the validation BPB attached to that solution and uh you can see the wos auto research already generated
26:54one node here um um here we also have a summary of the uh ideas it's tried And uh seems like already getting improvement.
27:09Oh yeah, you can also
27:10also open the code and see what's actually been changed.
27:18Yeah.
27:21Do we have a stability?
27:24Okay.
27:24Uh for stability you can basically tell the agent for example say like uh explore the quantization method.
27:33Is it connected?
27:36Yeah.
27:36Uhization method do not change the not just change the model config because we don't like just want to do the hyperparame tuning.
27:47We also want to try like more aggressive method.
27:58I don't think it's connected.
28:07I think clock didn't follow the instruction.
28:17Yeah, basically we also support you bring your claw code agent to our dashboard.
28:24We call it a bridged uh session.
28:27Um so basically yeah you can you can see the dashboard the tree structure the metric and the uh clock code at the same time and then you can tear clock code okay this is the
28:43uh the direction I want to steer the whole uh search process to
28:49checked in with some of you I think internet might be a little better uh some of you guys are using hotspot I know there's some modal payment issues.
28:58We're going to move on to the final part which is the competition aspect.
29:03Uh you don't need modal for this and originally this was designed to be a 30inut competition but we will give you guys until the end of the day um
29:13a little bit more on what this is.
29:19You guys have all played Flappy Bird.
29:21Um we're going to do a Flappy Wo Cup.
29:24So essentially, everyone hopefully has played Flappy Bird.
29:29You tap to flap.
29:31You dodge the pipes and you fly as far as you can.
29:35And your final score is dependent on how far you fly.
29:40We have turned this into an auto research problem.
29:43So instead of playing it, there's a controller called should flap that decides every frame whether the bird should flap.
29:52Your job is to optimize that controller.
29:56The metric is the average score and you want that as high as possible with the highest score winning.
30:04With all games, there's a twist.
30:06Uh we're not just running this on Flappy Bird.
30:08Uh we're running this across four different worlds.
30:12So the first world is normal, which is the classic game.
30:17The second is moving gap where the pipe gap drifts up and down.
30:23The third world has wind pushing the bird around.
30:27And the fourth world is variable which mixes all the conditions.
30:32Your score is the average across all four of these worlds.
30:36So you can't just focus on one.
30:39Um a controller that does well in a normal game but dies in the wind won't win.
30:44And the leaderboard scores you on a hidden validation set that you never see.
30:49So you can't obviously overfit on the public game.
30:55A few rules uh mostly so it's fair and it doesn't break.
30:59Uh you only and we will share the GitHub repo for this shortly.
31:03Uh you only edit the controller which is a controller.py.
31:08You keep you should keep should flap signature exactly as it is.
31:11So you can't touch the game or the evaluator.
31:15No imports, external libraries, file IO.
31:18So we're keeping this self-contained.
31:21And in the spirit of keeping it fair, uh, no parallel compute allowed.
31:26So you would be using WICO for auto research.
31:29Um, everyone is defaulted to the same model, Opus 4.8. So it's a level playing field.
31:35And remember, the local score that you see on your laptop isn't going to match the leaderboard because of our hidden test set.
31:46Some tips.
31:47Um, you'll get about 50 steps on WCO.
31:51Uh, actually this is with the 30 minute time time frame.
31:55So, uh, I guess you can run unlimited steps until the end of the day.
32:02Um, can start from the baseline.
32:04uh the baseline already works.
32:06So improve the code rather than rewriting the controller.
32:09Uh think about making small focused edits.
32:13Um things like tuning the safety margin, the velocity keyboard, the bottom buffer, adjusting the behavior per world and um maybe implement some of the features we talked about today.
32:25Whether it's inspecting nodes that scored well, building on them, bringing in ideas from outside, um, and trying several directions.
32:37This is the instructions for the game.
32:41Um, by the way, you guys are playing for a Nintendo Switch.
32:45Uh the way that we're going to determine who wins is for your team name, put in your email and by the end of the day, the highest score uh we will reach out to the person that that is.
32:59In the meantime, we're going to continue answering questions from our Slido or uh please raise your hand if you have any questions.
33:07Okay, we're just about to wrap up here.
33:10Um scores look like they're coming in.
33:13I don't know who Fern Nog is, but a little suspicious because there's only one node and so we will be inspecting your runs to ensure that there's no cheating going on.
33:26Um,
33:29[laughter]
33:30nice try.
33:31Uh, we will be opening this competition uh till midnight tonight.
33:36um we will find a way to get to you can put in your email or we're also we also have your emails uh from the runs that you guys are doing or you can just contact us on our website.
33:48But um feel free feel free to keep going at it.
33:52And um thank you all for coming.
34:10[music]