WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=7RFJRDYroic NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/7RFJRDYroic.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.880 --> 00:00:14.000 Uh, welcome. s2 00:00:14.000 --> 00:00:16.080 Please grab a seat if you haven't already. s3 00:00:16.080 --> 00:00:17.119 My name is Vam. s4 00:00:17.119 --> 00:00:19.760 We have Jung Yao, Droo, and Dex. s5 00:00:19.760 --> 00:00:26.560 Before anything else, uh, a huge thank you to Sean and the AI engineers worldfare for having us. s6 00:00:26.560 --> 00:00:32.399 This is generally um a great room of AI and we're really really happy to be here. s7 00:00:32.399 --> 00:00:35.120 Um if you haven't been able to get set up yet, no worries. s8 00:00:35.120 --> 00:00:38.559 Uh we're going to have ample time to get it to to it later. s9 00:00:38.559 --> 00:00:44.719 Um Junga will give us a brief background on the basics behind auto research. s10 00:00:52.239 --> 00:00:53.199 Okay. s11 00:00:53.199 --> 00:01:01.280 Um in case any of you still don't know uh auto research is this is just a quick brief. s12 00:01:01.280 --> 00:01:06.799 So auto research is basically a loop of a try and errors. s13 00:01:06.799 --> 00:01:13.200 You give a language model your code and a replayable evaluation script. s14 00:01:13.200 --> 00:01:24.640 It will keep trying ideas and test them against the eval and it keeps the ideas that improve the score and discard everything else. s15 00:01:24.640 --> 00:01:31.920 So some example metrics includes accuracy, speed uh and cost. s16 00:01:31.920 --> 00:01:48.880 Similar loop has been exist uh in academic for a while like two years but it really got viral in this year when Kapasi uh posted about it in his model training s17 00:01:48.880 --> 00:01:54.640 scripts and turned out works really well for him. s18 00:01:55.360 --> 00:02:09.200 In just a few weeks, it just the use case of auto research spread from training to everything like a backend engineering, prompt and agent engineering, speed up model inference s19 00:02:09.200 --> 00:02:13.760 and it also quickly show up in production. s20 00:02:13.760 --> 00:02:26.000 For example, it reduce the latency of Shopify's CI/CD pipeline and improve the browser use the company's browser use agent. s21 00:02:27.920 --> 00:02:44.800 In April, our experimental research agent Aiden even won OpenAI's hiring challenge uh called the prime golf uh which we will tell you a bit more soon and uh I also have a talk on Wednesday specifically s22 00:02:44.800 --> 00:02:46.640 about it. s23 00:02:46.640 --> 00:03:06.000 Um so okay auto research can be really powerful but it's also hard to master with a normal coding agent you basically steer um agent every single step but for auto research it's like the agent is running in a loop s24 00:03:06.000 --> 00:03:20.400 on its own so it's much bigger multiplier on your prompt and your setup pointed at the right eval you basically let it boil the ocean for you, but otherwise you're just s25 00:03:20.400 --> 00:03:23.599 burning through all your RM tokens. s26 00:03:23.599 --> 00:03:34.720 Um so hopefully this workshop will give you a good starting point to learn about auto research and um apply that in your own work. s27 00:03:34.720 --> 00:03:38.400 Okay, we'll come back to Bay. s28 00:03:40.799 --> 00:03:41.760 Thanks Janelle. s29 00:03:41.760 --> 00:03:44.560 Um so here's how we'll spend the next two hours. s30 00:03:44.560 --> 00:03:50.959 Um, Dex will take you inside parameter golf, uh, the challenge that we entered our agent into. s31 00:03:50.959 --> 00:03:57.280 Um, what that challenge was actually all about and how our agent ended up as the most influential contributor. s32 00:03:57.280 --> 00:03:58.720 Um, then it's your turn. s33 00:03:58.720 --> 00:04:03.280 Uh, we'll get you set up and kick off an auto research run on your laptop. s34 00:04:03.280 --> 00:04:11.599 While your auto research is running in the background, we'll break down the basics of an auto research loop and the pieces that actually shape what an agent does. s35 00:04:11.599 --> 00:04:19.680 Um, and we'll go deeper and do a guided demo where you're able to run, steer, and inspect what your agent is doing live. s36 00:04:19.680 --> 00:04:23.840 And we'll finish with a fun part, which is a live competition. s37 00:04:23.840 --> 00:04:35.199 Um, you'll take everything that you learned today, point auto research at a game that we've all played before, and go head-to-head to see whose agent can get the highest score s38 00:04:35.199 --> 00:04:40.080 uh, with the winner walking away with a really cool prize. s39 00:04:42.000 --> 00:04:44.080 a little bit more on who we are. s40 00:04:44.080 --> 00:04:47.919 Um, so we've been working on auto research for quite some time now. s41 00:04:47.919 --> 00:04:55.919 Before WICO, uh, we released aid in April 2024, which was an autonomous ML engineering agent. s42 00:04:55.919 --> 00:05:04.080 The basic idea was give it a machine learning problem, let it write code, run experiments, read the results, and keep improving. s43 00:05:04.080 --> 00:05:17.120 When OpenAI built MLE Bench, aid was one of the first agents they evaluated independently and on that benchmark it scored about four times the medal rate of the next best agent. s44 00:05:17.120 --> 00:05:25.680 More recently, as we discussed, we pointed our agent to parameter golf and across more than a thousand research it researchers, it became the number one contributor. s45 00:05:25.680 --> 00:05:30.240 I'm going to hand it off to Dex who's going to tell us more about the competition. s46 00:05:39.680 --> 00:05:42.400 Hello everyone, my name is Dex. s47 00:05:42.400 --> 00:05:45.759 Uh so I'm going to talk about the paramet golf. s48 00:05:45.759 --> 00:05:50.080 So paramet golf is open as uh model craft challenge. s49 00:05:50.080 --> 00:05:56.000 The goal is to train the best language model you can under two hard constraints. s50 00:05:56.000 --> 00:06:06.400 uh once the four artifacts including model weights plus training code has to fit in 16 megabytes and the training has to finish within 10 minutes. s51 00:06:06.400 --> 00:06:14.080 Submissions are scored by bits per bpb on the fine wide validation set and the low is better. s52 00:06:14.080 --> 00:06:25.520 Openi provided a baseline data set and evaluation scripts so participants can for the ripple improve the model and submit their result through GitHub. s53 00:06:25.520 --> 00:06:27.600 So everything is open sourced. s54 00:06:27.600 --> 00:06:33.680 They also put up a million dollar in computer credits and reviewed every submission. s55 00:06:33.680 --> 00:06:46.880 During the 44 day competition, there were,6 researchers entered and filed 248 PR roughly about uh 250k of GPU compute burn. s56 00:06:46.880 --> 00:06:50.080 The final lead board is short. s57 00:06:50.080 --> 00:06:55.360 Only 47 PRs merge as records from 31 contributors. s58 00:06:55.840 --> 00:07:07.440 And for this competition, we build Aiden, a self-improving multi-aging system based on the cloud and GPT model families that run indefinitely to help clamp on a single metric. s59 00:07:07.440 --> 00:07:12.080 In permit golf, Aiden automated the four research loop. s60 00:07:12.080 --> 00:07:19.360 It pulls public page papers from archive, strong implementations from GitHub and its own part logs. s61 00:07:19.360 --> 00:07:30.800 Then propose new ideas, tests them, audit them for legality, keep the ones that work, write the implementation and run the parents. s62 00:07:30.800 --> 00:07:38.240 Once the results, clear the quality gates, it will create and submit a PR for review. s63 00:07:38.800 --> 00:07:40.720 And the results are remarkable. s64 00:07:40.720 --> 00:07:43.919 Aiden landed seven mer leaderboard records. s65 00:07:43.919 --> 00:07:45.919 That's twice the next best human. s66 00:07:45.919 --> 00:07:53.680 It also had the highest OPR H index in the whole competition which means Aiden didn't just produce accept records. s67 00:07:53.680 --> 00:08:10.160 It published reusable work that other participants kept building on 435 downstream PR stated and built on Aiden's work and it got there on a fraction of the budget from just 4% of the community's GPU compute. s68 00:08:10.160 --> 00:08:13.520 It got 15% of total merged records. s69 00:08:13.520 --> 00:08:18.319 That's 3.7 times more cost effective than the field average. s70 00:08:18.319 --> 00:08:24.960 And this wraps up a how Aiden becomes the number one contributor to the open herring challenge. s71 00:08:24.960 --> 00:08:27.199 Thank you. s72 00:08:30.720 --> 00:08:31.120 Cool. s73 00:08:31.120 --> 00:08:39.440 So, we're actually going to run auto research on a version of the same parameter golf challenge that Dex just walked you through. s74 00:08:39.440 --> 00:08:45.440 Uh, for future reference, we will just call this mini PG, which stands for mini parameter golf. s75 00:08:45.440 --> 00:08:50.480 Um, as Dex discussed, the real parameter golf competition was heavy. s76 00:08:50.480 --> 00:09:03.680 Every single experiment was training a language model on a massive data set, which is fine when the competition is over four weeks, but in a workshop like today, you'd never actually be able to see your auto research run. s77 00:09:03.680 --> 00:09:09.120 So, think of mini PG as a scale model of the challenge. s78 00:09:09.120 --> 00:09:21.600 uh you're still going to train a real language model, score it on the same fixed evaluator, and drive down the same single number valp, which is essentially measuring how well the model predicts text it hasn't seen. s79 00:09:21.600 --> 00:09:23.920 But what we did is we shrank this in scale. s80 00:09:23.920 --> 00:09:29.760 We're using a tiny data set instead of fine web and training an eval that only takes a few minutes. s81 00:09:29.760 --> 00:09:36.640 So you'll actually be able to get a loop going and see several steel several real steps live. s82 00:09:37.920 --> 00:09:42.800 So before I get to the instructions on the next slide, just a little bit of orientation. s83 00:09:42.800 --> 00:09:47.120 Um you're going to run auto research yourself on mini parameter golf. s84 00:09:47.120 --> 00:09:50.640 You're going to sign up for two platforms if you haven't already. s85 00:09:50.640 --> 00:09:58.880 One is we our auto research agent that runs a search and the other is modal uh the compute your experiments train and evaluate on. s86 00:09:58.880 --> 00:09:59.680 Free credits. s87 00:09:59.680 --> 00:10:01.120 Thank you modal. s88 00:10:01.120 --> 00:10:06.000 Um then you will clone our parameter golf repo uh and hand it to your coding agent. s89 00:10:06.000 --> 00:10:07.600 tell it to kick off a run. s90 00:10:07.600 --> 00:10:10.240 It handles the setup and it'll start the loop. s91 00:10:10.240 --> 00:10:17.600 Once it's going and it's running in the background, uh you can kick it off and we will check back in in around 15 or 20 minutes to take you s92 00:10:17.600 --> 00:10:20.160 inside the details. s93 00:10:22.640 --> 00:10:23.920 Instructions are here. s94 00:10:23.920 --> 00:10:27.360 Um sign up for WO on dashboard.ai. s95 00:10:27.360 --> 00:10:30.560 Um modal for moto modto.com-signup. s96 00:10:30.560 --> 00:10:33.279 Uh there's a code to claim your modal credits. s97 00:10:33.279 --> 00:10:39.680 Um, if you already have a modal account, you can go directly to modal.com/crediteds and enter your code there. s98 00:10:39.680 --> 00:10:50.800 Um, and once you're done with those two steps, uh, clone the repo shown here, um, and tell your agent to set up mini PG from the readme and kick off a run. s99 00:10:50.800 --> 00:11:01.120 If you run into any issues or have any questions, feel free to scan the QR code and enter it there and we'll be monitoring that or uh, raise your hand and we'll come to you. s100 00:11:01.120 --> 00:11:04.000 check back in in around 105 minutes. s101 00:11:04.000 --> 00:11:04.640 Thanks, guys. s102 00:11:04.640 --> 00:11:05.839 Uh, all right. s103 00:11:05.839 --> 00:11:07.680 So, sorry about the internet problem. s104 00:11:07.680 --> 00:11:16.160 I think it's being worked on, but until then, I think it would be good to maybe go through uh the anatomy of an auto research run. s105 00:11:16.160 --> 00:11:24.000 And we're going to be doing this in the context of uh WCO as the auto research system and mini PG as the task. s106 00:11:24.000 --> 00:11:28.079 But really, this this kind of contract that we have here is fairly general. s107 00:11:28.079 --> 00:11:35.360 Most auto research systems are going to have you ask very similar questions and uh think about the same things. s108 00:11:36.399 --> 00:11:36.720 Okay. s109 00:11:36.720 --> 00:11:39.279 So the first thing is the source file. s110 00:11:39.279 --> 00:11:43.120 Uh this is what the agent actually has to modify. s111 00:11:43.120 --> 00:11:46.399 This defines the search space and where it can make changes. s112 00:11:46.399 --> 00:11:50.079 So this can be a single file, it can be it can be multiple files. s113 00:11:50.079 --> 00:12:05.279 In the case of minip it's uh this one file called solution.py and solution.py PI essentially defines the model config and the uh uh the training dynamics that's going to go on while the model is training. s114 00:12:06.720 --> 00:12:14.320 And then the second thing and in my opinion this is probably the most crucial part of an auto research system is the evaluation. s115 00:12:14.320 --> 00:12:23.200 And so you want to have a very simple way of allowing your auto research system to evaluate every solution that it generates. s116 00:12:23.200 --> 00:12:28.800 Um, and you want to have it output a single scalar metric. s117 00:12:28.800 --> 00:12:34.160 And you want the eval to be fast, you want it to be cheap, you want it to have low noise. s118 00:12:34.160 --> 00:12:42.480 Uh, because this essentially leads to your ability to run more steps if you have a fixed budget or a fixed time constraint. s119 00:12:42.480 --> 00:12:52.160 Um, and if there's low noise, then both you as well as the agent are more confident as you're kind of hill climbing on, uh, whatever task you're, uh, autoresarching s120 00:12:52.160 --> 00:12:53.040 on. s121 00:12:53.040 --> 00:12:58.160 And so, uh, one thing to be aware of though is agents love to reward hack. s122 00:12:58.160 --> 00:13:03.279 So, they love to find holes in your evaluation protocol, holes in your benchmark. s123 00:13:03.279 --> 00:13:08.639 And so, you really want to figure out how to make your eval as robust as possible. s124 00:13:10.160 --> 00:13:13.279 The next two things are the metric and the goal. s125 00:13:13.279 --> 00:13:15.519 So you really want to ask yourself two questions. s126 00:13:15.519 --> 00:13:20.160 Um what do I want to improve and is higher or lower better? s127 00:13:20.160 --> 00:13:26.480 Um and in our case for mini PG, you want to minimize the validation BP. s128 00:13:26.480 --> 00:13:33.839 So you essentially want to find a model that is better at predicting on unseen text. s129 00:13:35.839 --> 00:13:41.839 The next thing is the number of steps you actually run the auto research system for. s130 00:13:41.839 --> 00:13:48.800 This is a bit tricky because it's difficult to know a priority how many steps you should give your your system. s131 00:13:48.800 --> 00:14:00.160 We've chosen 20 in this case just because we want to give your agents enough time to kind of explore different ideas and you can you can see what the agent is actually exploring, what works, what doesn't. s132 00:14:00.160 --> 00:14:07.120 Um but really more ideas that gets tried uh generally leads to more breakthroughs. s133 00:14:07.120 --> 00:14:17.680 And so uh this goes back to what I was saying earlier where if you have a fast eval um or if you have a cheap eval then when you're under some kind of time constraint s134 00:14:17.680 --> 00:14:20.959 or cost constraint, you can run more steps. s135 00:14:20.959 --> 00:14:28.480 Um and so uh again the eval is is always super important to uh consider. s136 00:14:30.240 --> 00:14:30.800 All right. s137 00:14:30.800 --> 00:14:39.839 So you also want to bound the uh time for which your eval runs for uh because a lot of things can go wrong. s138 00:14:39.839 --> 00:14:48.079 So loops can stall, hardware can fail, uh sometimes the agent will produce really bad code that causes the entire system to hang. s139 00:14:48.079 --> 00:14:52.079 And so you want to bound the the actual evaluation running. s140 00:14:52.079 --> 00:14:55.760 Um in our case, we set it to be 210 seconds. s141 00:14:55.760 --> 00:15:08.399 And the reason for that is we give 60 seconds for uh training the model, 60 seconds for testing the model on unseen text and then 90 seconds which is a buffer for s142 00:15:08.399 --> 00:15:20.079 you know modal to spin up the GPUs for training, spin it up for um uh for the testing and any other kinds of uh overheads that exist. s143 00:15:21.680 --> 00:15:28.240 And the last thing is the guidance that you give the agent at the start of your optimization. s144 00:15:28.240 --> 00:15:31.440 Uh so this is how you steer the agent from the start. s145 00:15:31.440 --> 00:15:37.120 And you really want to think about it as you know you've probably run experiments before. s146 00:15:37.120 --> 00:15:41.040 You've compounded a bunch of domain expertise and human knowledge. s147 00:15:41.040 --> 00:15:48.079 And you might have some ideas about things that work, things that might be interesting to try and maybe things that don't work. s148 00:15:48.079 --> 00:15:51.040 And so this is where you can provide that kind of guidance. s149 00:15:51.040 --> 00:15:58.880 Um in the case of WICO, you can simply point it to a markdown file, have all of your information and context in there. s150 00:15:58.880 --> 00:16:05.440 Um and the agent is able to use this and uh it gives the agent the best chance at uh at finding success. s151 00:16:05.440 --> 00:16:16.959 Um, in the case of mini PG, this can be uh some hard constraints or different ideas that seem to work quite well like uh focusing on compression uh different quantization s152 00:16:16.959 --> 00:16:22.639 methods uh maybe even weight sharing across different blocks in the model. s153 00:16:31.440 --> 00:16:39.759 Um we are just going to demo how to get started on auto research for you considering the internet issues. s154 00:16:39.759 --> 00:16:50.160 Um before we get to that a little bit more about what we think um makes auto production grade auto research actually production grade. s155 00:16:50.160 --> 00:16:52.720 Um the first is structure. s156 00:16:52.720 --> 00:17:04.640 So uh we have a treeb brace search uh where the agent shows you every attempt it's run so that ideas as duv had mentioned can sort of compound on each other instead of starting from scratch. s157 00:17:04.640 --> 00:17:06.959 The second is observability. s158 00:17:06.959 --> 00:17:16.400 Um you can sort of open any attempt read the exact code change behind its score so you can see why the agent is taking the approach it is. s159 00:17:16.400 --> 00:17:18.559 And the third thing is steerability. s160 00:17:18.559 --> 00:17:20.720 So you have a sense of control. s161 00:17:20.720 --> 00:17:28.959 You can inject your own ideas and knowledge while the loop is running and have it try new directions in case it's stuck. s162 00:17:29.440 --> 00:17:31.919 A little bit more about the treebased search. s163 00:17:31.919 --> 00:17:43.520 Um we will shortly demo this on our auto research dashboard, but um treebased auto research essentially treats the problem as a structured optimization process over code. s164 00:17:43.520 --> 00:17:45.120 Uh it starts with the root of the tree. s165 00:17:45.120 --> 00:17:51.840 As you can see the score was 0.5 um which is just the baseline program and a measured score. s166 00:17:51.840 --> 00:17:58.400 Every node after that route is a modified version of the code run against your eval with a score attached. s167 00:17:58.400 --> 00:18:01.679 You can think of this tree as an experiment graph. s168 00:18:01.679 --> 00:18:05.039 As you can see the search builds on what works. s169 00:18:05.039 --> 00:18:07.600 We're finding good candidates and pruning bad ones. s170 00:18:07.600 --> 00:18:12.960 So when a branch improves a score it builds on those ideas and goes deeper next. s171 00:18:12.960 --> 00:18:20.799 which means that you can inspect the exact changes your auto research system is making and build off of those. s172 00:18:22.799 --> 00:18:26.960 So the next idea we talked about was observability. s173 00:18:26.960 --> 00:18:35.600 Um because you can see every one of the nodes in this treebased search, you can open any one of them and see exactly what changed. s174 00:18:35.600 --> 00:18:41.600 You can see your metric move in real time correlating to a movement correlating the movement to a reason. s175 00:18:41.600 --> 00:18:48.240 You can hover over over a node, get a summary of what changed, and you can open it and read the exact diff. s176 00:18:48.240 --> 00:18:52.320 So when the metric jumps, you can read the change that caused it. s177 00:18:52.320 --> 00:18:56.320 And typically, these runs go for hours across hundreds of experiments. s178 00:18:56.320 --> 00:19:03.360 So once you can see what it's doing, the next natural step is to be able to change what it does. s179 00:19:03.760 --> 00:19:07.600 This is essentially when you stop watching and start to take over. s180 00:19:07.600 --> 00:19:17.440 Um, if I have an idea I want to try in plain English, um, I take the idea, it opens a new branch and you can see it try this new idea and have s181 00:19:17.440 --> 00:19:20.480 new nodes develop and a new curve. s182 00:19:20.480 --> 00:19:24.720 I can also put a budget to each node so it spends compute where I want. s183 00:19:24.720 --> 00:19:31.520 And this is the part that made it usable on real problems because you might have context and intuition that an agent doesn't. s184 00:19:31.520 --> 00:19:39.760 At least for now, we believe it's important to find ways to fold human judgment into the system whenever it's necessary. s185 00:19:40.320 --> 00:19:44.080 I want to ground that point uh steerability in something real. s186 00:19:44.080 --> 00:19:49.520 Um because parameter because in parameter golf steerability was a big part of why we did well. s187 00:19:49.520 --> 00:19:52.960 Um this is an actual breakthrough from the competition. s188 00:19:52.960 --> 00:19:56.640 It started with our agent uh which is the line in pink. s189 00:19:56.640 --> 00:20:00.320 um for five days from April 15th to April 19th. s190 00:20:00.320 --> 00:20:08.320 Uh it kept trying new things, um new attention, compression, a richer stack, but the score essentially plateaued. s191 00:20:08.320 --> 00:20:14.400 There was a human in the community uh that shipped a clever new tokenizer, which is the blue line. s192 00:20:14.400 --> 00:20:25.440 Um, and within eight hours of emerging that change, our system picked it up, fused it with the pieces it had already been working on, and shipped the biggest jump in weeks. s193 00:20:25.440 --> 00:20:30.640 Running auto research on something this hard is less of a set it and forget it. s194 00:20:30.640 --> 00:20:33.919 Uh, it's a collaboration where we steer the agent. s195 00:20:33.919 --> 00:20:36.159 We pull in ideas from the people around us. s196 00:20:36.159 --> 00:20:39.760 And steerability is essentially what makes that loop possible. s197 00:20:39.760 --> 00:20:43.280 It's how real breakthroughs actually happen. s198 00:20:44.320 --> 00:20:51.679 Um, I'm going to hand it over to Dex and Jungao to talk a little bit more about how to get auto research set up. s199 00:20:51.679 --> 00:20:56.880 Um, we'll essentially mimic the process that you guys were trying to do on your laptops. s200 00:20:56.880 --> 00:20:57.679 Okay. s201 00:20:57.679 --> 00:21:08.559 Um, so what we are going to do here is we just walk you guys through the high level idea like how how does it happen? s202 00:21:08.559 --> 00:21:11.520 We just use our mobile hotspot. s203 00:21:11.520 --> 00:21:15.600 Um so we will be freestyling a little bit here. s204 00:21:15.600 --> 00:21:18.080 Uh hope you don't mind. s205 00:21:18.080 --> 00:21:30.400 Um so yeah can you just uh yeah okay maybe we can also walk through the whole doc being fed to uh claude. s206 00:21:30.400 --> 00:21:30.720 So s207 00:21:30.720 --> 00:21:31.440 yeah. s208 00:21:31.440 --> 00:21:48.400 So basically uh so you can go into the rebel and some like simple instruction like start the wo raw based on the readme and it should be like if the internet is good I think it should start the s209 00:21:48.400 --> 00:21:58.480 uh default wle run and uh we can also go over via the like the how is the wle command working. s210 00:21:58.480 --> 00:22:01.679 Yeah, this case you have a source file. s211 00:22:01.679 --> 00:22:06.720 The solution is basically the training script of mini PG. s212 00:22:06.720 --> 00:22:21.600 I assume like a eval command is I think Z also mentioned this is additional guidance or prompt that will guide the whole uh search process and the metric is a validation BPB. s213 00:22:21.600 --> 00:22:29.200 Uh this is like a hold out um uh loss like a loss only hold out set. s214 00:22:29.200 --> 00:22:36.880 Uh go is minimize and others are just configs we just mentioned. s215 00:22:36.880 --> 00:22:37.600 Oh, sorry. s216 00:22:37.600 --> 00:22:56.960 Eval commander is actually the the bash command that uh you're going to run the uh uh run the eval and eventually it will output a uh score like it print a score to the terminal and it's going to be passed by the agent s217 00:22:56.960 --> 00:23:04.880 and that's the key uh signal that will drive the whole tri and error loops. s218 00:23:05.760 --> 00:23:06.159 Yeah. s219 00:23:06.159 --> 00:23:10.320 Um in meanwhile I can also show the dashboard. s220 00:23:10.320 --> 00:23:23.280 So if you go into the W dashboard um and I think there I did some wrong and you you basically see a lot of things and uh after the s221 00:23:23.280 --> 00:23:37.760 the command like auto booted by the agent um I think then you you will see the romping popped up and then you can try out the features we provide. s222 00:23:39.840 --> 00:23:40.559 Yeah. s223 00:23:40.559 --> 00:23:45.280 Um, do we have a run just to get started? s224 00:23:45.280 --> 00:23:47.679 Yeah, I think it's just starting. s225 00:23:47.679 --> 00:23:54.400 Okay, still running saying um yeah, Claud is working. s226 00:24:05.520 --> 00:24:08.880 Uh question from the audience while you guys are doing that. s227 00:24:08.880 --> 00:24:16.559 Um one, is it possible to turn up the brightness on the dashboard or switch to light mode? s228 00:24:17.760 --> 00:24:18.400 Start again. s229 00:24:18.400 --> 00:24:20.320 I'm I'm right here. s230 00:24:20.320 --> 00:24:22.320 Can you turn the light mode on the dashboard? s231 00:24:22.320 --> 00:24:23.600 It's a little hard to see. s232 00:24:23.600 --> 00:24:25.840 I see. s233 00:24:28.720 --> 00:24:30.640 Light mode. s234 00:24:30.640 --> 00:24:31.120 Forgot. s235 00:24:31.120 --> 00:24:33.039 Where is it? s236 00:24:33.039 --> 00:24:46.960 Is there uh second question is can auto research apply to RL? s237 00:24:46.960 --> 00:24:51.440 What is the scalar metric when there are multiple things to optimize? s238 00:24:51.919 --> 00:24:53.520 What is the s239 00:24:53.520 --> 00:24:56.000 can auto research apply to RL? s240 00:24:56.000 --> 00:25:00.480 What is the scalar metric when there are multiple things to optimize? s241 00:25:00.480 --> 00:25:01.360 I see. s242 00:25:01.360 --> 00:25:01.840 Yeah. s243 00:25:01.840 --> 00:25:08.720 Um it can definitely be applied to IO like apply to IO training scripts. s244 00:25:08.720 --> 00:25:28.640 Um I think there is a um derived a project called sobound uh speedr runner that's actually exactly doing it like for post training aquin model uh for playing a s245 00:25:28.640 --> 00:25:45.600 game and uh yeah in that project you can apply auto research to the uh IR IR training scripts And um the second question you have a multiple objectives to optimize for. s246 00:25:45.600 --> 00:25:59.360 Um the idea is you eventually just combine them to a single metric like you either apply different weights on those different metrics or you treat some of the matrix s247 00:25:59.360 --> 00:26:03.360 as constraint and optimize for one of them. s248 00:26:03.360 --> 00:26:13.279 For example, if you're doing things like fraud detection, you might hold the uh accuracy and trying to optimize the precision. s249 00:26:13.279 --> 00:26:25.679 Um and uh for parameter golf exactly you hold the model size as a constraint and you optimize for the accuracy of the system. s250 00:26:25.919 --> 00:26:35.279 Yeah, we can see the um agent already generated uh one node. s251 00:26:35.279 --> 00:26:54.400 So the zero zero node is basically the root node the baseline solution and 3.93 means the validation BPB attached to that solution and uh you can see the wos auto research already generated s252 00:26:54.400 --> 00:27:09.919 one node here um um here we also have a summary of the uh ideas it's tried And uh seems like already getting improvement. s253 00:27:09.919 --> 00:27:10.960 Oh yeah, you can also s254 00:27:10.960 --> 00:27:15.520 also open the code and see what's actually been changed. s255 00:27:18.720 --> 00:27:20.720 Yeah. s256 00:27:21.919 --> 00:27:24.080 Do we have a stability? s257 00:27:24.080 --> 00:27:24.480 Okay. s258 00:27:24.480 --> 00:27:33.840 Uh for stability you can basically tell the agent for example say like uh explore the quantization method. s259 00:27:33.840 --> 00:27:36.400 Is it connected? s260 00:27:36.400 --> 00:27:36.960 Yeah. s261 00:27:36.960 --> 00:27:47.200 Uhization method do not change the not just change the model config because we don't like just want to do the hyperparame tuning. s262 00:27:47.200 --> 00:27:51.679 We also want to try like more aggressive method. s263 00:27:58.480 --> 00:28:01.360 I don't think it's connected. s264 00:28:07.360 --> 00:28:11.760 I think clock didn't follow the instruction. s265 00:28:17.039 --> 00:28:24.080 Yeah, basically we also support you bring your claw code agent to our dashboard. s266 00:28:24.080 --> 00:28:27.679 We call it a bridged uh session. s267 00:28:27.679 --> 00:28:43.520 Um so basically yeah you can you can see the dashboard the tree structure the metric and the uh clock code at the same time and then you can tear clock code okay this is the s268 00:28:43.520 --> 00:28:49.200 uh the direction I want to steer the whole uh search process to s269 00:28:49.919 --> 00:28:58.640 checked in with some of you I think internet might be a little better uh some of you guys are using hotspot I know there's some modal payment issues. s270 00:28:58.640 --> 00:29:03.360 We're going to move on to the final part which is the competition aspect. s271 00:29:03.360 --> 00:29:13.919 Uh you don't need modal for this and originally this was designed to be a 30inut competition but we will give you guys until the end of the day um s272 00:29:13.919 --> 00:29:17.840 a little bit more on what this is. s273 00:29:19.520 --> 00:29:21.360 You guys have all played Flappy Bird. s274 00:29:21.360 --> 00:29:24.399 Um we're going to do a Flappy Wo Cup. s275 00:29:24.399 --> 00:29:29.840 So essentially, everyone hopefully has played Flappy Bird. s276 00:29:29.840 --> 00:29:31.279 You tap to flap. s277 00:29:31.279 --> 00:29:35.440 You dodge the pipes and you fly as far as you can. s278 00:29:35.440 --> 00:29:40.000 And your final score is dependent on how far you fly. s279 00:29:40.399 --> 00:29:43.760 We have turned this into an auto research problem. s280 00:29:43.760 --> 00:29:52.880 So instead of playing it, there's a controller called should flap that decides every frame whether the bird should flap. s281 00:29:52.880 --> 00:29:56.480 Your job is to optimize that controller. s282 00:29:56.480 --> 00:30:03.919 The metric is the average score and you want that as high as possible with the highest score winning. s283 00:30:04.480 --> 00:30:06.320 With all games, there's a twist. s284 00:30:06.320 --> 00:30:08.960 Uh we're not just running this on Flappy Bird. s285 00:30:08.960 --> 00:30:12.240 Uh we're running this across four different worlds. s286 00:30:12.240 --> 00:30:17.120 So the first world is normal, which is the classic game. s287 00:30:17.120 --> 00:30:23.840 The second is moving gap where the pipe gap drifts up and down. s288 00:30:23.840 --> 00:30:27.840 The third world has wind pushing the bird around. s289 00:30:27.919 --> 00:30:32.480 And the fourth world is variable which mixes all the conditions. s290 00:30:32.480 --> 00:30:36.640 Your score is the average across all four of these worlds. s291 00:30:36.640 --> 00:30:39.120 So you can't just focus on one. s292 00:30:39.120 --> 00:30:44.559 Um a controller that does well in a normal game but dies in the wind won't win. s293 00:30:44.559 --> 00:30:49.200 And the leaderboard scores you on a hidden validation set that you never see. s294 00:30:49.200 --> 00:30:53.760 So you can't obviously overfit on the public game. s295 00:30:55.200 --> 00:30:59.120 A few rules uh mostly so it's fair and it doesn't break. s296 00:30:59.120 --> 00:31:03.279 Uh you only and we will share the GitHub repo for this shortly. s297 00:31:03.279 --> 00:31:07.919 Uh you only edit the controller which is a controller.py. s298 00:31:08.080 --> 00:31:11.679 You keep you should keep should flap signature exactly as it is. s299 00:31:11.679 --> 00:31:15.360 So you can't touch the game or the evaluator. s300 00:31:15.360 --> 00:31:18.559 No imports, external libraries, file IO. s301 00:31:18.559 --> 00:31:21.200 So we're keeping this self-contained. s302 00:31:21.200 --> 00:31:26.080 And in the spirit of keeping it fair, uh, no parallel compute allowed. s303 00:31:26.080 --> 00:31:29.120 So you would be using WICO for auto research. s304 00:31:29.120 --> 00:31:35.840 Um, everyone is defaulted to the same model, Opus 4.8. So it's a level playing field. s305 00:31:35.840 --> 00:31:45.120 And remember, the local score that you see on your laptop isn't going to match the leaderboard because of our hidden test set. s306 00:31:46.000 --> 00:31:47.200 Some tips. s307 00:31:47.200 --> 00:31:51.200 Um, you'll get about 50 steps on WCO. s308 00:31:51.200 --> 00:31:55.440 Uh, actually this is with the 30 minute time time frame. s309 00:31:55.440 --> 00:32:02.080 So, uh, I guess you can run unlimited steps until the end of the day. s310 00:32:02.080 --> 00:32:04.559 Um, can start from the baseline. s311 00:32:04.559 --> 00:32:06.080 uh the baseline already works. s312 00:32:06.080 --> 00:32:09.919 So improve the code rather than rewriting the controller. s313 00:32:09.919 --> 00:32:13.200 Uh think about making small focused edits. s314 00:32:13.200 --> 00:32:25.120 Um things like tuning the safety margin, the velocity keyboard, the bottom buffer, adjusting the behavior per world and um maybe implement some of the features we talked about today. s315 00:32:25.120 --> 00:32:34.960 Whether it's inspecting nodes that scored well, building on them, bringing in ideas from outside, um, and trying several directions. s316 00:32:37.760 --> 00:32:41.120 This is the instructions for the game. s317 00:32:41.120 --> 00:32:45.200 Um, by the way, you guys are playing for a Nintendo Switch. s318 00:32:45.200 --> 00:32:58.399 Uh the way that we're going to determine who wins is for your team name, put in your email and by the end of the day, the highest score uh we will reach out to the person that that is. s319 00:32:59.360 --> 00:33:07.360 In the meantime, we're going to continue answering questions from our Slido or uh please raise your hand if you have any questions. s320 00:33:07.360 --> 00:33:10.159 Okay, we're just about to wrap up here. s321 00:33:10.159 --> 00:33:13.279 Um scores look like they're coming in. s322 00:33:13.279 --> 00:33:26.960 I don't know who Fern Nog is, but a little suspicious because there's only one node and so we will be inspecting your runs to ensure that there's no cheating going on. s323 00:33:26.960 --> 00:33:28.960 Um, s324 00:33:29.563 --> 00:33:30.000 [laughter] s325 00:33:30.000 --> 00:33:31.360 nice try. s326 00:33:31.360 --> 00:33:36.399 Uh, we will be opening this competition uh till midnight tonight. s327 00:33:36.399 --> 00:33:48.720 um we will find a way to get to you can put in your email or we're also we also have your emails uh from the runs that you guys are doing or you can just contact us on our website. s328 00:33:48.720 --> 00:33:52.399 But um feel free feel free to keep going at it. s329 00:33:52.399 --> 00:33:55.679 And um thank you all for coming. s330 00:34:10.828 --> 00:34:12.828 [music]