How Autoresearch is changing ML research — Zhengyao Jiang, Weco
https://www.youtube.com/watch?v=iCj_ATyThvc
AI Engineer · 16 min · 127 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] This April, OBI ran a hiring challenge, a competition called Parameter Golf.
[00:19] The top contributor was one candidate that they couldn't hire.
[00:25] It wasn't a person, it's an agent we build called Aiden.
[00:31] In parameter golf, the goal is to train the best language model you can under size and computation constraints.
[00:41] About 1,000 machine learning engineers, researchers participate.
[00:47] They fired 2,000 submissions.
[00:51] Only 47 passed open review and made into the leaderboard.
[00:57] Seven of those are actually agents.
[01:01] More than twice what any human contributed.
[01:06] You've seen a lot of auto research today.
[01:09] Agents are here climbing benchmarks.
[01:12] Those are really impressive results.
[01:15] The question I want to ask is a bit different here.
[01:18] Can the auto research agent produce work that a human community actually recognize beyond a good score agent is optimizing for something that other engineers can merge fork and build on.
[01:36] So instead of having an agent just here climbing locally, we build one that publishes its own work and that's Aiden.
[01:47] Quick context on us.
[01:49] Wiko is a auto research company that founded about two and a half years ago.
[01:54] Uh I'm co-founder and a CEO Jungao.
[01:57] Um got my PhD at UCR on reinforcement learning.
[02:01] About two years ago, we buil aid the top auto research agent independently evaluated by OpenAI in their MLE bench paper.
[02:14] Even though back then there's a no such name called auto research, people call it machine learning engineering agent.
[02:22] Aiden is the next step and a experimental prototype.
[02:29] It's a multi-agent self-improving system that can read public information like research papers and other PRs, run its own experiments and submit a PR once the findings pass a quality gate.
[02:46] We send Aiden to parameter golf competition and it ran for about 22 days.
[02:53] By the end, Aid has set seven leaderboard records.
[02:57] Each one is a new best for the competition stampled by OpenAI and the best human only made three.
[03:07] Passing the host review is a one signal for the quality.
[03:11] A second maybe more important one is whether other participants would build on your work.
[03:20] And it turns out Aiden's work had the highest impact within the whole community.
[03:26] Here we are using a inference measure that used widely in academia.
[03:34] It's called a H index.
[03:36] Roughly if you have X papers get cited X times then your Ach index is X. Computed over PRs.
[03:46] Aiden was 10 and the next human was seven.
[03:50] The whole community was building on a AI systems work including many of other leaderboard entries.
[04:01] To break it down a little bit, why can a autonomous AI system be so powerful?
[04:08] One obvious reason is that it's an AI.
[04:12] It can run tirelessly.
[04:14] Over 22 days, it ran about 1,300 experiments on a single H100 node.
[04:25] But the throughput isn't the whole picture.
[04:27] A well tuned AI system can also keep its output quality high.
[04:34] On the compute side, it uses at most 4% of competition's total compute.
[04:44] and it made about 15% of the records.
[04:49] Also, 28% of its submissions made the leaderboard.
[04:55] Roughly six times higher heat rate than the community average.
[05:00] So, Aiden actually lifted the signal noise ratio within the whole community's public communication channel, which is a PR.
[05:10] It didn't win through massive paralization even though auto research have a tons of a potential of paralyzation.
[05:21] By those numbers it might feel like auto research already dominates human experts on ML engineering and research but that's not the full story I want to tell.
[05:35] Humans and AI are actually contribute in very different ways.
[05:41] When we trace the ideas, Aiden Aiden's record PR almost all of them come from human research papers other participants in parameter golf or in similar communities like nano GBT.
[05:59] Those ideas are not necessarily a merged PR.
[06:03] Sometimes it's a note um a human researcher said oh I give up this idea because of some implementation implementation difficulty and the agent is good at finding them and actually implement them.
[06:19] There are also a very small fraction of original ideas Aiden came up by itself which emerged from its efforts to navigate the file size constraints.
[06:32] Here's a concrete example that traces the patterns I just talked about.
[06:39] So Aiden picked up an idea from Quen paper called gated attention and it worked but it introduced more parameters and it broke the 16 megapy file size limit.
[06:56] So it figure out a qualization mechanism to bring the file size down.
[07:02] But with those two primitives combined, the score barely moved.
[07:09] Then another contributor posted a tokenizer improvement.
[07:14] Aiden recognized the idea, combine it with architectural work.
[07:19] It just work for five days or so.
[07:23] And after this combination, the three takea the three ideas turns out to have a huge synergy that lead to a big jump in performance and they become one of the Aiden's leaderboard
[07:38] records.
[07:41] So to sum up how I interpret Aiden and in general auto research systems effectiveness, it's very strong at finding and implementing ideas.
[07:52] In the case we just saw, it brought an idea from a recent paper into a actual implementation in the competition and it's good at dug promising ingredients out of the
[08:05] primary golf community even though the public channel is actually very noisy information wise.
[08:14] It can also came up logically straightforward ideas.
[08:18] For example, in this case, once you add the parameters and it breaks the file size limit, one obvious next move is just a quantization.
[08:30] And it's really fast and really efficient at finding right combinations across a huge search space.
[08:41] Okay, maybe none of those sounds very sexy.
[08:44] Most of them are just a good execution.
[08:47] But in reality, execution is a mostly the bottleneck.
[08:53] What moves the frontier is usually exactly some belief on existing ideas and tons of good executions.
[09:06] Okay.
[09:06] To step back, the state of a human AI collaboration is a human collectively provide a lot of creative ideas and agent do the execution to solve a concrete challenge.
[09:22] What we are looking at is a a large group of a human and one AI system.
[09:27] Does this mean a single human engineer's contribution marginally get smaller?
[09:35] I didn't say even for that not really.
[09:38] In primate golf competition, it's easy to only focus on engineers that's actually doing hill climbing.
[09:46] But the design behind the competition itself is tremendously important.
[09:52] A bad design can make the whole community effort useless and their evil design work will have a few huge leverage in the auto research era.
[10:04] I really like one tweet from Andre Kapasi about 10 years ago where he said greeting descent can write code better than you.
[10:15] I'm sorry for the context about 10 years ago deep learning was starting to eat up a lot of software engineering like conventional coding work and his tweet was arguing against those people who thought they can
[10:32] handw write better code than a trained model.
[10:37] Okay, now obviously no one is seriously trying to handr write code to beat a model.
[10:43] However, software engineering I mean as a job still exist and so many people's job are just training those models and those are one of the most well paid job
[10:56] today.
[10:58] I think how gradient descent change coding is a great metaphor for how auto research will change research and ML engineering.
[11:08] It commonize certain execution skills.
[11:12] At the same time, it makes some higher level skills far more valuable.
[11:19] So actually doing auto research is a lot like training a model.
[11:23] Your codebased abstraction is essentially the architecture.
[11:28] It sets the constraint and the priorities um for what the agent can explore.
[11:35] Your eval is the loss function and the data.
[11:38] It sets what the agent optimizes for.
[11:43] Take the eval first.
[11:44] The eval is the signal you use to train a model.
[11:49] In this case, it's training your code.
[11:53] It plays the same role that like data and the loss function uh in model training or in a reinforcement learning setting.
[12:02] It's like a environment that the agent is training.
[12:07] Nowadays, no one would argue data or environments um don't matter.
[12:14] And uh this is where a vertical mode can also be built.
[12:19] You might have a proprietary data for evaluation or a unique understanding of a in a particular field what matters and how to measure it.
[12:30] and a good evaluation would be amplified more and more as auto research are getting stronger.
[12:40] The other one I think is really underrated is codebased abstraction.
[12:46] The abstraction provides the framework that auto research can iterate on and uh that's also that starting point hugely bias the whole search direction.
[13:01] This is a lot like a architecture design in neural networks.
[13:05] Different architecture in theory can represent the same function, but the architecture systematically makes some of the functions easier to be learned.
[13:19] And a good architecture biases the optimization towards solutions that generalize better, perform better, even when the training loss might looks the same.
[13:32] That's exactly the same for auto research.
[13:36] Here's an example.
[13:37] We run auto research for a um fraud detection pipeline um and we trying to optimize the data prep-processing and first we give it a loose API where the same function
[13:54] process both the training and testing data and the score looks great but the solution was polluted because there's a certain certain test set information got leaked to the training information.
[14:14] We then tighten the obstruction to a more strict API where the test data couldn't reach the training and the data leakage rate just dropped to zero.
[14:26] In this case, a good abstraction leads to better solutions.
[14:31] Even though if the agent really want they can steer reward hack.
[14:38] So my point is using auto research is a new craft.
[14:43] It's about the designing a here for an agent to climb and we are still very early on it.
[14:50] I think that makes this extremely exciting time to be an AI engineer.
[14:57] Other research will change what skills matter most.
[15:01] Creativity, the judgment to design a good eval or an abstraction.
[15:07] Those will soon get exponentially more important.
[15:12] Driving those system itself is where will be a new skill and that one is like barely existed one or two years ago.
[15:22] So the search is automated.
[15:25] the human would just move up the stack not out of it.
[15:32] Again, um we call is a auto research um product research lab.
[15:38] We we keep sharing what we are learning as we build uh on our blog and I will also post some of my thinking to on ax.
[15:48] If you think some of this uh useful to you, feel free to follow me on X. Thank you.