We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect

AI Engineer · 19 min · 142 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Hey um hi everyone.
00:14Uh thanks for being here.
00:16Uh yeah, I'm super happy today to talk about uh automated eye research and uh especially uh all those like frontier model uh perform at uh automated eye research task.
00:28Um so I'm Elie.
00:30I worked at Prime Intellect as a research engineer and uh yeah I will go through our work on on this subject.
00:36So first I want to basically explain a bit why we are doing that and why we think it's super important to do that in the open.
00:45Um so first uh I think we we all agree that uh we've heard about like big labs saying that this bad thing called recursive self-improvement is coming very soon.
00:59Uh so recursive self-improvement is like model training models uh without uh human intervention basically.
01:06Um but uh we don't have any benchmark to basically quantify if this is true or not right.
01:13Uh and even less we don't have like a third party benchmark by non- big labs to to to see if it's something coming soon or not.
01:23And the other part is that we think that uh it's super important to understand all those model uh do research because we think that a lot of the scientific research that will come into the coming years
01:36uh will be based also on AI tools.
01:39So it's super important to understand how those model do research not just only AI research.
01:45So we try to build kind of this environment to test the capabilities of the model to do so.
01:53So it all started with uh Andre Karpati uh that's basically had fun by doing this video where he trained uh GPT2 from scratch in like 90 minutes like GPT2 training takes like weeks
02:09and uh no in two years ago I think it only took like 90 minutes.
02:13So what does it mean to reprod reproduce uh GPT2 in 90 minutes?
02:18It means that in 90 minutes you achieve this target loss.
02:22Um and yeah and that's at this point when you have the same loss than GPT2 you consider that your model is somewhat of equal performance.
02:34Um then what happened is that the community took this repo uh this GitHub repo and create another one called moed nano GPT and this effort was leaded by someone called Keller Jordan.
02:47And what happened is that they basically took this 90 minutes then 45 minutes and then now we can train like GPT2 validation loss model in less than two minutes which is honestly
03:00crazy and it took like two years to to achieve this.
03:04So it's a very strong benchmark where uh a lot of very talented researcher iterated on um yeah so we decided to take this environment of speedun so it's kind of a game so the goal of the game is to achieve this loss in the fewest in the shortest amount of time so this is the nano GPT1
03:27and you can uh you don't have almost any constraints the only constraint that you shots that you need to use the same validation and training data, right?
03:38Um there is a new speedrun called the optimizer speedrun that was released uh a few months ago and here it's slightly different because uh you can only change the optimizer
03:49related parameters.
03:51So for instance nano GPT you can change the architecture uh do uh attention whatever uh optimizer sp you can only change like Adam to m shampoo or whatever optimizer
04:06is your favorite um yeah and so this is a bit more researchy because uh it's less about optimizing the program to be as fast as possible but more like finding the best method possible.
04:20no matter the the the time you put into the computer, right?
04:26So, um yeah, why take speedrun as an environment for automated AI research?
04:33First, uh we think that it's a good evaluation.
04:36We'll see later why.
04:37Uh and this is kind of the main focus of this talk.
04:40But we also think it's probably a good training environment because uh it's a way to give the model a reward.
04:47So the reward is positive if the model bit the speed run and beat the last record sorry and the reward is zero or negative if it didn't manage to to do it.
04:59So it's a good uh environment to train model.
05:01It's also quite fast like as you see uh previous record were around 2 minutes for the optimizer one.
05:08uh each run take about like 15 to 20 minutes and uh yeah and there is like clear rules basically and we also think it's like a good environment to make discovery
05:20so like kind of breakthrough in our research because uh there is those clear rule that you can verify or not.
05:28Um yeah.
05:29So yeah.
05:32Um so what we did uh so the release was like about two months ago and uh there was this optimizer speedrun and we decided to basically compete with the community
05:44by launching two AI agents.
05:46So Codex and Cloud Code.
05:48Codex was like GPT 5.5 with XI and uh cloud code was Opus 4.8 with XI.
05:56Um and yeah, we decided to basically let the agent free on our cluster uh and uh and just iterate on it.
06:04So we have like V1, V2, V3 is just basically us stopping the agent and then restarting.
06:10V3 uh was like one or two day before the release because we saw that our agents no longer have the best record.
06:18So we were like okay take all the the human uh record in the last few week and just try to to improve upon it and and and it worked.
06:28Yeah.
06:28And we also have this novelty track where the goal is to uh beat the record with only novel ideas.
06:36Um and we'll see that this this was more complex for the the models.
06:43So our RS is very simple.
06:46Honestly, we could have just replaced it with slashgo, but they there was no SLG goal at the time.
06:51So, we made our own goalm.
06:53It's actually quite fun that we choose the same name and we had the goal.
06:57MD and kind of agents that MD that define the rules and we let the agent propose ids and then it can submit a job with sbatch on our slum cluster
07:11and uh basically the way it works is that it can submit on nodes that are available but only under a certain permission which means that if someone want to use this node
07:22uh the model just like cancel the job it's called preemptable permission.
07:26So yeah, then it measure the it read basically the training logs then decide if it's a record or not.
07:32To validate a record you need to basically pass a statistical threshold to make sure that it's just not see the optimization and is just not random.
07:40Right?
07:42So yeah a few results from this experiment.
07:45The first one that was honestly very painful to work with is that code clothes code keep stopping every nine or 10 hours and basically said yeah I cannot improve the record it's too hard for me
07:58there is no way to to go beyond it and then I was just like okay continue explore new direction hey just go again for 10 hours and then say yeah I cannot
08:11beat the recall and so on.
08:12So basically onethird of the time the cloud code agent was idle because I had no way to basically monitor it and codeex totally the opposite just worked for all the
08:24all the time and uh yeah almost never idle never asked for question and and and very impressive in that way.
08:31Um we also give the option for the model to basically write uh a bunch of stuff into what we call a scratch pad which is basically the active memory of the model.
08:44Uh we observe that basically codeex writes a lot on the scratch patch.
08:50So each plot that I will show are kind of normalized by the number of active order.
08:55So this is not only about codex working more it's it's really different behavior.
09:02So yeah, you see that uh writes a lot more to to this scratch pad to this memory and uh the shape of the like the the I don't know the tone of the the each file
09:14was also super different like CL was super excited about getting new record with a bunch of emoji and so on and CEX was just like here is what I do here is the decision I take what I will do next like super
09:27robotic kind of um Yeah, we also have this plot where basically we saw that codex was spawning much more sub aents than cloud.
09:39Uh we saw that codex burn much more token than code.
09:42So I think in total it was like kind billion of token but it's like there is obviously this input tok uh input caching that make it it's not like one billion output token.
09:55Uh so yeah we also see that codex did a lot of compaction because it only had like 250k context window and cloud only do it like one per hour and codex is more like
10:08no it's even less than one power for I mean one for the full run for code and codex was like one uh was 20 every one hour.
10:18So yeah um yeah here is the main results.
10:23So what this plot shows is that basically we so in in white you see that the human record progression right and in red you see cloud I mean it's supposed to be orange but whatever and in blue uh you see codeex right
10:41and you see that at almost every time uh cloud and codex are better than the human record and code is super good at the beginning very very fast to achieve very good
10:53Um yeah and one thing that is super important is that the model have the ability to basically fetch the human records at any time and that's what codex did that's what cloud did sorry
11:05because when I restarted it it basically fetch the new record from human and improve upon it um yeah so the result is that uh I think at the time the best uh record was like
11:18uh 2,990 step and we beat it by like uh uh 50 or 60 step for code and codeex was like 20 step above.
11:30So I think it's both impressive and and yeah um so we so this is like not released yet.
11:38This is something that we are working on currently and basically the idea is that this is a cool experiment to do but it lack of structured right.
11:47uh if you want to do a real benchmark, you want to do multiple seed, you want to do uh yeah proper uh thing where you you you you basically put all the model and earnestness in the same condition, right?
12:00So this is what we are working on right now and basically um the idea is to do three different track uh one without any access to really like measure the capability of the models to do AI research based on
12:15only the model weight knowledge one with only archive paper and one with like full access.
12:22So it also have access to the the like the latest record by human.
12:27And for this we plan to do both uh the nano GPT track one which is the original one and the optimizer speedun where we we only launch uh we only constrain the the optimizer
12:39to be to be novel basically.
12:42Um yeah so I will present some result on the optimizer speedrun.
12:47Uh this is basically what we got.
12:49So we let the agent iterate for six day almost five days let's say and we see that uh codeex k and clothes uh are super effective so for GLM
13:03this is not finished run right so the model is actually still iterating on the cluster right now but we see that cloud is once again very good at it and we see that surprisingly Kim is also very competitive
13:17and kind of have this breakthrough on day four where he kind of beat Codex with a new record, right?
13:24It's also interesting to see that uh Claude is much more like progressive in the way it improved the record and Kim has really this step function where I kind of do a breakthrough and so on.
13:37Uh so this is an interesting plot because I mean six day is quite a lot for anal uh uh but you you can change this uh axis by also the number of output token
13:48and then kind of tell a different story because in max mode consumes so much more token than codeex and kimmy and you also see that Kimmy is actually super efficient
14:00uh for the number of token that uh it uses.
14:04So it's scheming K2.7 code.
14:06Um so yeah uh we also see that they have a different approach to uh using the literature and papers.
14:16Um so for instance like code is doing a lot of search on papers and actually include found a paper that no other model found and it actually lead to
14:26the best record.
14:27So it's kind of funny and uh yeah um one of the main issue of all of this is that uh when I when I launched this this agent and I think that's something important that I want you to to kind of uh remember for this co this talk is that when I launched this these different agents I
14:49was expecting them to come up with some crazy ideas on optimizer that's like no one of discover but honestly it wasn't the case.
14:58Uh they did some clever trick where basically they combine different papers.
15:03uh they kind of do plus one improvement over a bunch of method but there was really like no novel optimizer or mechanism that was uh coming from those model and I think that's
15:15kind of telling that even on something that is not simple but I'd say that it's kind of accessible for people right for like human researcher uh spending like days and weeks for the the model like
15:31cannot like find new uh optimizer and mechanism.
15:35So we believe that there is a way to basically make it more um make it better for discovery instead of evaluation and this is coming from uh this is very inspired from alpha evolve by Google and also a bunch of papers that have been released since then.
15:53It's kind of this multi- aent system that interact together uh bunch of generator.
16:00You have closed model but you also have open source model here that are super effective for the cost right.
16:06Uh they can suggest ideas then you run the speedrun so you get the reward then you have a judge that basically give a quality feedback can also be like
16:16the judge also have this taste.
16:18you can kind of have like the judge have a taste about the the method if it's good or not uh if it's outside the loop and then you can uh basically decide
16:30which method you want to scale to a larger number of parameters and number of token.
16:37Um so this is kind of the scale part of the speedrun because some a lot of method in the the speedrun community uh people are often saying that they doesn't work at large scale.
16:48So I think it's very important to also put scale elements in this loop.
16:53Uh and I think also that uh human are super useful here to basically judge the ID of agents kind of steer them in the right direction and so on.
17:05Um yeah so we didn't try it yet I mean we are kind of trying it right now and uh we hope that this will lead to to to new discovery in AI research at least and also a way
17:18is that you can define multiple speedrun so this is the next slide if you like it's from safe bank slides but if you if you don't have the reference good for you means that that you're not too online
17:32uh but the idea is that uh by changing the object objective and the constraints of the speedrun you can basically create a lot of diversity and constrain the model to go into a certain direction
17:44and uh yeah and make those discovery.
17:47So uh at HIM in Ind.
17:52Uh there is a bunch of stuff here that we I mean most of it we didn't release yet but we are working on GPU sandboxing to allow model to iterate into
18:02sandbox because you need GPU sandbox for this kind of stuff.
18:05We are working on our own agents that are very efficient for like RM framework.
18:12So it means like you have a file system and you can write information read from it.
18:17Uh and you also do like this programmatic tool coding thing.
18:21We also training a model to be good at it on top of like open source model.
18:26And uh the thing that we already released is that we have those set of liberation and product called verifier primaril training where you can basically train evaluate any environments on
18:38any RS and the model that you can train can be like GNM 5.2 too which is very big and and yeah we have like we work a lot on making those li very efficient
18:47to to ship the best quality for for our clients.
18:51Yeah.
18:52Uh I mean yeah super excited about this domain.
18:56Once again I think it's super important to have uh a part of like this recursive self-improvement to happen in the open because there is actually a lot of people
19:06working that are not on big labs.
19:08So you need to basically uh yeah make it easy for people to understand all those model work to do research and so on.
19:16So that's kind of our goal and uh yeah thanks a lot.