WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=kiqubc5b5Yo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/kiqubc5b5Yo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.680 --> 00:00:14.840
Well, thank you everyone for coming.

s3
00:00:14.840 --> 00:00:24.680
Um today I'll be presenting what we've been building at Morgan Stanley, uh sort of auto research agent to try and automate quant research.

s4
00:00:24.680 --> 00:00:37.080
Um and I wanted to just take the the first couple minutes to sort of explain, you know, give some context on our team and also um you know, why we're even pursuing trying to build this this auto research agent.

s5
00:00:37.080 --> 00:00:39.240
And so, our group are relatively small.

s6
00:00:39.240 --> 00:00:52.400
We're about 30 uh PhD AI researchers, and we kind of operate both like kind of half academic, so we're encouraged to um you know, publish papers, open-source code, share our research.

s7
00:00:52.400 --> 00:00:56.240
And then the other half is sort of more applied internal work.

s8
00:00:56.240 --> 00:01:02.680
And you know, I think a lot of our problems or or yeah, at least some of them are are can be fairly well-posed.

s9
00:01:02.680 --> 00:01:13.640
They kind of have a similar shape to Kaggle where we have you know, an input time series data set, and our task is to just, you know, predict future values and maybe with some other constraints

s10
00:01:13.640 --> 00:01:16.880
of wanting the model to be well-calibrated.

s11
00:01:16.880 --> 00:01:27.600
Um and so, I think, you know, being on the sell side, we kind of have maybe less sort of adversarial selection, and so it it kind of lends itself naturally to this auto research framework.

s12
00:01:27.600 --> 00:01:39.920
Um and you know, also I think there's we have a lot of sort of algorithms in production where we have a feeling if we could just put in more cycles, there might be some improvements to kind of squeeze out, whether it's

s13
00:01:39.920 --> 00:01:47.000
just better hyperparameter tuning or exploring a lot of different kind of ensembling these different methods together.

s14
00:01:47.000 --> 00:02:01.160
Also, you know, we have We work with a lot of different desks, and there's a feeling that like something we have built for say credit bonds could really transfer well to like muni bonds, and that kind of translation process should be something that can be

s15
00:02:01.160 --> 00:02:03.600
automatable with with agents.

s16
00:02:03.600 --> 00:02:14.160
And even like going to new trading desks and trying to build them new algorithms, there's sort of a lot of low-hanging fruit where uh they're not really using a lot of machine learning or or AI automation.

s17
00:02:14.160 --> 00:02:19.240
And so, you know, even like a reasonably trained model should be able to have a big impact.

s18
00:02:19.240 --> 00:02:30.000
And so, sort of in the last year and a half where these these agents became, you know, able to do long-horizon tasks, we've been interested in trying to do this automation.

s19
00:02:30.000 --> 00:02:41.440
Um but it wasn't really until December of 2025, and I think this is a pretty common sentiment now that um it really felt possible for the first time with like with Opus 4.5

s20
00:02:41.440 --> 00:02:51.200
and with, you know, these harnesses like Claude Code and Codex, where it really felt like the models were at a point where they could do these long-horizon tasks and also kind of the idea of

s21
00:02:51.200 --> 00:02:55.120
putting them in these harnesses that that allow them to do so.

s22
00:02:55.120 --> 00:03:01.800
And so, you know, really starting this year, we kind of made it a big effort to build this this auto research agent.

s23
00:03:01.800 --> 00:03:10.560
And of course, sort of the the top-line goal is just maximizing P&L and maximizing the the number of new algorithms we can put in production.

s24
00:03:10.560 --> 00:03:13.480
Um but, you know, we also had these design concerns.

s25
00:03:13.480 --> 00:03:23.000
Like, of course, we wanted to play nicely and integrate with all of our data and sort of the pre-built scaffolding we have around backtesting and evals.

s26
00:03:23.000 --> 00:03:25.760
We also wanted it to be able to run the full spectrum.

s27
00:03:25.760 --> 00:03:37.239
So, like, you know, on one side, maybe it's a new data set, and we just want to say like, here's the path to the data, a natural language description of what we hope to predict,

s28
00:03:37.239 --> 00:03:43.120
and the thing should go off and do research and build its own eval and start doing experimentation.

s29
00:03:43.120 --> 00:03:57.360
And then sort of on the other side of the spectrum, maybe it's like, no, we have our data scripts, we have our eval, we actually have have good models we've already produced, and we just wanted to kind of churn and do more cycles and see if we can find an improvement.

s30
00:03:57.360 --> 00:04:07.760
Um also, we wanted to build this to be really model agnostic, so we could use any of the frontier providers or increasingly, uh you know, any open-source model.

s31
00:04:07.760 --> 00:04:15.840
And also, we wanted to build it in a way that you know, as these models get better and better, it's not sort of consuming what we built.

s32
00:04:15.840 --> 00:04:18.920
Like, we kind of rise with the tide of the models.

s33
00:04:18.920 --> 00:04:30.080
And lastly, you know, really carefully think about um how do we encode sort of like our enterprise knowledge as Morgan Stanley and then our human expertise as as quant researchers.

s34
00:04:30.560 --> 00:04:39.280
And so, for for the rest of the talk, I want to start with sort of what I'm calling Alpha Lab 1.0, which is our first version we released.

s35
00:04:39.280 --> 00:04:46.720
I think it was early April, and we put out a full like 40-page tech report going through all the details and results.

s36
00:04:46.720 --> 00:04:49.680
We also open-sourced all the code on GitHub.

s37
00:04:49.680 --> 00:04:58.480
So, I I kind of want to cover that more at a high level because all the details are so public, but I'm happy to talk afterwards in depth about any part.

s38
00:04:58.480 --> 00:05:03.520
Um but then I really want to cover, you know, sort of what's happened since then.

s39
00:05:03.520 --> 00:05:05.960
So, what were the initial results?

s40
00:05:05.960 --> 00:05:18.160
What were the kind of failure cases since then because we we have encountered a lot of failures, and then talk about how we're really addressing those by building uh kind of our own rich set of evals and environments,

s41
00:05:18.160 --> 00:05:18.340
and how

s42
00:05:18.340 --> 00:05:18.440
[snorts]

s43
00:05:18.440 --> 00:05:32.960
that's sort of allowing us to improve the harness and climb towards this sort of self-recursive improvement and sort of our grand vision now for Alpha Lab 2.0. And so, to start, you know, Alpha Lab

s44
00:05:32.960 --> 00:05:43.640
is an agentic harness, and kind of going towards that that first side of the spectrum, the the goal is you have some data set, you know, let's say it's like a

s45
00:05:43.640 --> 00:05:53.240
uh exchange rate data set or something, and you can just provide the path to the file or maybe it lives on an API and you can just say here's the API access

s46
00:05:53.240 --> 00:05:54.920
and the API spec.

s47
00:05:54.920 --> 00:05:57.920
And then just in natural language say what you want to predict.

s48
00:05:57.920 --> 00:06:01.880
So maybe it's as simple as, you know, the simple exchange rate.

s49
00:06:01.880 --> 00:06:06.280
I would just I'm curious in predicting one day out what the the rate will be.

s50
00:06:06.280 --> 00:06:08.840
And the harness then works in these these three phases.

s51
00:06:08.840 --> 00:06:12.920
So the first phase is research and I'll cover these all more in depth.

s52
00:06:12.920 --> 00:06:24.160
The second phase is then actually building its own evaluation or backtesting and the third phase is sort of the mass experimentation, which is really the heart of the the harness.

s53
00:06:24.160 --> 00:06:32.640
And then as output you get a suite of trained machine learning models that are trying to, you know, do the the prediction you care about.

s54
00:06:34.120 --> 00:06:40.240
And one design choice we made, so the harness is actually all our own code.

s55
00:06:40.240 --> 00:06:44.600
So we decided not to use any off-the-shelf agent framework.

s56
00:06:44.600 --> 00:06:56.600
Um we wrote it all and really Claude wrote it all and I think like in the era of of Claude code, I I like this approach of kind of building your own from scratch because you get sort of

s57
00:06:56.600 --> 00:07:03.280
max freedom and max like, you know, you're you're free to kind of tweak anything you care about.

s58
00:07:03.280 --> 00:07:12.200
Um and so, you know, all the tool calls are done with these like functional tool calling and that also allows us to nicely really be provider agnostic.

s59
00:07:12.200 --> 00:07:21.280
So OpenAI Anthropic or open source providers, it's very easy to to adapt the the harness to any any of those.

s60
00:07:21.280 --> 00:07:31.520
As far as the actual tools, there's there's several but the sort of three main ones are one full shell access, so it can write any bash command.

s61
00:07:31.520 --> 00:07:37.360
So this is how it's writing code, setting up its Python environment, editing code, running code.

s62
00:07:37.360 --> 00:07:47.720
Um another important one is web search and this allows it to go read, you know, archive or technical blogs or anything like that and get sort of up to speed at

s63
00:07:47.720 --> 00:07:52.480
um you know, at least in the public domain like sort of what's the state of the art methods.

s64
00:07:52.480 --> 00:08:04.880
And then the third one is um we use Slurm to manage our GPU cluster, but the the higher level idea is just a a nice abstraction where the model can say like I'm training a

s65
00:08:04.880 --> 00:08:06.440
a fairly big model.

s66
00:08:06.440 --> 00:08:16.000
I need 4 H100s and this many CPUs and just kind of write the config config and submit the job and not have to worry about like doing hardware orchestration.

s67
00:08:17.040 --> 00:08:20.320
Uh yeah, and so going to the the actual phases.

s68
00:08:20.320 --> 00:08:23.600
So again, the first phase is this this research phase.

s69
00:08:23.600 --> 00:08:34.000
And the idea here is kind of like almost like a super cloud MD file or a super init where we just want the system to go off and kind of build enough context

s70
00:08:34.000 --> 00:08:39.400
such that it can start meaningfully uh you know, forming hypotheses and testing them.

s71
00:08:39.400 --> 00:08:46.880
Um and we built, you know, Alpha Lab is kind of this server-side running thing, but we built this lightweight UI on top.

s72
00:08:46.880 --> 00:08:52.080
And so sort of how we've done this is build this this scaffolding of the to-do list.

s73
00:08:52.080 --> 00:08:59.560
So it's first, you know, prompted to to build a to-do list such that if it completed every item, it'd be able to start experimenting.

s74
00:08:59.560 --> 00:09:04.240
And this allows us to kind of keep re-prompting should it try to exit early.

s75
00:09:04.240 --> 00:09:14.080
And so you can see, you know, it's talking about setting up its Python environment, um doing different data loading, doing different kinds of statistical testing.

s76
00:09:14.080 --> 00:09:19.960
And each item of the list it's instructed to build a like take notes in a markdown file.

s77
00:09:19.960 --> 00:09:26.960
And this also allows feature agents to smartly query that and kind of manage their context dynamically.

s78
00:09:26.960 --> 00:09:36.520
And and this is really where web search is used most because it will go off and read archive and and get kind of good context from the public domain as well.

s79
00:09:36.520 --> 00:09:43.320
And so this can vary a lot, but um it takes, you know, roughly around 3 to 4 hours.

s80
00:09:43.320 --> 00:09:53.560
The second phase and what um is most different in 2.0 is the eval building, but just to say sort of our first implementation, you know, of course the evaluation

s81
00:09:53.560 --> 00:10:05.080
is the most important piece and LLMs aren't malicious, but they can make, you know, very silly mistakes and if you're optimizing against a bad a bad eval, the whole thing kind of falls apart.

s82
00:10:05.080 --> 00:10:10.600
So our attempt to be more robust is have this sort of multi-agent framework.

s83
00:10:10.600 --> 00:10:18.840
So one is tasked with first building the eval, like actually writing all of the code, and then that goes off to two critic agents.

s84
00:10:18.840 --> 00:10:28.600
One that's told to be more high-level, like are there conceptual errors in our evaluation or any kind of forward leakage of information, and then one that's more programmatic.

s85
00:10:28.600 --> 00:10:31.720
So it's like writing unit tests and integration tests.

s86
00:10:31.720 --> 00:10:43.120
Um and they write up any issues they find of our issues, it goes back to the builder to fix, and this loop doesn't end until sort of all of them are happy that the eval is good.

s87
00:10:43.120 --> 00:10:48.600
And then the third phase and really the heart of everything is this mass experimentation.

s88
00:10:48.600 --> 00:10:57.080
And so we've we've chosen, you know, both in the code and in our UI, this is kind of formulated as a Jira board or Kanban board.

s89
00:10:57.080 --> 00:11:07.280
And so there's this sort of strategist agent that gets to look and query all the context from the previous steps, and it's just supposed to keep coming up with experiments

s90
00:11:07.280 --> 00:11:11.920
it wants to try, and it submits them to this like to implement column.

s91
00:11:11.920 --> 00:11:25.480
And then as new cards come in, those get pawned off to worker agents that are tasked with actually writing the code to implement the strategy, writing the Slurm config, you know, what hardware does it need, submitting it to our cluster,

s92
00:11:25.480 --> 00:11:35.400
waiting for the job to finish, and then look at, you know, the the machine learn learning training curves, like did it underfit, overfit, and then also looking at the eval results.

s93
00:11:35.400 --> 00:11:41.520
And then it sort of writes this postmortem analysis, which goes back to the strategist as each job finishes.

s94
00:11:41.520 --> 00:11:45.440
So, the strategist hopefully can do this kind of self-evolution.

s95
00:11:45.440 --> 00:11:54.640
So, it can see like I suggested, you know, three variants of transformers that actually didn't work too well, but uh XGBoost is is working really well.

s96
00:11:54.640 --> 00:11:58.560
So, I want to explore more tree methods or something like that.

s97
00:11:58.560 --> 00:12:02.880
Um And also in this step is where we as users can steer.

s98
00:12:02.880 --> 00:12:11.520
So, you can, you know, another reason we pick sort of this Jira formation is uh you can cancel cards, you can add your own cards.

s99
00:12:11.520 --> 00:12:21.520
There's also a chat feature that's sort of cut off here, but you can chat with the strategist and kind of steer it towards more creative or kind of give it intuition of different

s100
00:12:21.520 --> 00:12:23.360
methods that should try.

s101
00:12:23.360 --> 00:12:33.120
Um and then we keep this, you know, ever-growing leaderboard where you can see like given your eval and given a held-out private validation set, like what are the best-performing

s102
00:12:33.120 --> 00:12:34.000
models?

s103
00:12:34.000 --> 00:12:41.560
And you can click in and see like, you know, from inception of the idea to the code, and you can pull it out and play with it.

s104
00:12:41.560 --> 00:12:54.080
And then this is just sort of our UI 2.0, which is really just optimized to look cool, but you can see like, you know, it's suggesting experiments, and this is really sped up, but how each worker kind of pulls it through,

s105
00:12:54.080 --> 00:12:59.000
um you know, building, deploying, and then analyzing the results.

s106
00:12:59.680 --> 00:13:05.200
And um you know, in our paper, we did more sort of academic data sets.

s107
00:13:05.200 --> 00:13:16.520
So, we looked at CUDA kernels, we looked at uh an academic traffic time series data set, and we did the kind of classic like Karpathy-style LM speed running.

s108
00:13:16.520 --> 00:13:26.640
Um and it's sort of hard to benchmark AlphaLab, but we compared to more of a Karpathy-style like single agent in a loop going, and it did find a better

s109
00:13:26.640 --> 00:13:30.440
uh training config for training an LLM.

s110
00:13:30.440 --> 00:13:40.720
We also put this on a Kaggle competition, which was to It was, uh, hosted by Nvidia to fine-tune their NeMo Tron model to be a reasoning model.

s111
00:13:40.720 --> 00:13:44.400
Uh, and it got in the top 12% of submissions.

s112
00:13:44.400 --> 00:13:54.400
Which I think is decent, and also it only had sort of 10 iterations to work with because we joined late, and I think, you know, AlphaLab works best the more iterations it can explore.

s113
00:13:54.400 --> 00:13:59.960
So, presumably or hopefully it would have done better had it, uh, had more time.

s114
00:13:59.960 --> 00:14:13.320
And then I can say at a high level internally, there's been a handful of of models, all sort of of the flavor where we had a decent model already, but we just turn it over to AlphaLab to keep kind of churning on it.

s115
00:14:13.320 --> 00:14:20.040
Uh, where it's found meaningful improvements that are now working their way through risk and going into production.

s116
00:14:20.160 --> 00:14:29.600
And so with the my last couple minutes, I I want to talk about, um, you know, we we ran into some real issues and hard questions.

s117
00:14:29.600 --> 00:14:36.680
Uh, I think the first was, you know, okay, so we had some some good results, but we've also had cases where it's really failed.

s118
00:14:36.680 --> 00:14:45.160
And so it's kind of always in our head, like, how real is is any of this, you know, if it if it's failing on these hardest problems, how do we measure,

s119
00:14:45.160 --> 00:14:47.240
how, you know, how real this is.

s120
00:14:47.240 --> 00:14:58.240
And the second was, and I'm sure a lot of you might be thinking, like, okay, your research phase sounds reasonable, having a strategist and workers sounds reasonable, but isn't it sort of arbitrary,

s121
00:14:58.240 --> 00:15:01.640
you know, like, how do you motivate these design choices?

s122
00:15:01.640 --> 00:15:08.400
And how we feel now is like, and I think a big theme of this conference is, you know, should we be making these decisions at all?

s123
00:15:08.400 --> 00:15:11.120
Like, this itself is a verifiable loop.

s124
00:15:11.120 --> 00:15:15.400
Like, an LLM should be doing this this meta-optimization itself.

s125
00:15:15.400 --> 00:15:25.560
And then lastly, you know, in this this 1.0 formulation, it's sort of like we give the data, we give the goal, and the LLM just goes off and does it.

s126
00:15:25.560 --> 00:15:33.320
So, like how does that exactly encode our, you know, Morgan Stanley's enterprise knowledge, but just our expertise as as quants?

s127
00:15:33.320 --> 00:15:35.960
Like it's sort of missing from the picture.

s128
00:15:35.960 --> 00:15:42.320
And so, the answer to all three we feel is really in building our own evals and environments.

s129
00:15:42.320 --> 00:15:52.920
And so, to just sort of go one by one through the questions, you know, to the first point, and this is a lesson I learned over and over, like monthly, working with these models,

s130
00:15:52.920 --> 00:15:54.560
you have to start with good eval.

s131
00:15:54.560 --> 00:16:04.840
Like it's such an obvious thing I think we were kind of overeager and wanted to treat it more like a a human researcher, but you of course need like a very clear way to measure.

s132
00:16:04.840 --> 00:16:11.200
And so, the the biggest kind of change is now we're we're very opinionated about the eval.

s133
00:16:11.200 --> 00:16:13.600
It's very much like Kaggle.

s134
00:16:13.600 --> 00:16:22.120
And so, you know, we treat it as data and description in, the harness lives in the middle, and its only job is to submit containerized models.

s135
00:16:22.120 --> 00:16:31.800
And it gets sort of the the feedback of like a public leaderboard score, but you as a user get to see a private leaderboard, like held-out validation, how's the model performing.

s136
00:16:31.800 --> 00:16:39.320
And so, you know, in this case, you can measure like just for any given task, how well does your model do?

s137
00:16:39.320 --> 00:16:46.720
But of course, once you have this strict kind of format, you can think, okay, you know, to me evals and environments are the same thing.

s138
00:16:46.720 --> 00:16:49.120
It's just you train in environments.

s139
00:16:49.120 --> 00:16:57.480
So, now what we've done is built on the order of like 10 to 20 really careful environments, and that becomes a reinforcement learning signal.

s140
00:16:57.480 --> 00:17:00.839
And so, we're you know, quote-unquote AlphaLab meaning AlphaLab.

s141
00:17:00.839 --> 00:17:06.120
So, once you have that way to measure, you know, you can do human tuning of the the harness.

s142
00:17:06.120 --> 00:17:14.160
So, like maybe there should be two strategists, and maybe they should debate, or maybe there, you know, whatever kind of ideas you have, you at least have a way to

s143
00:17:14.160 --> 00:17:16.600
measure and kind of manually hill climb.

s144
00:17:16.600 --> 00:17:27.360
But what we're doing now is really this meta harness optimization, where the LLM is looking at the trace, is looking at the about, you know, the results, and improving the harness itself.

s145
00:17:27.360 --> 00:17:40.640
And also sort of a a tangential axis is we're now collecting good traces from open source model and really touching weights and doing like GRPL or or other, you know, on policy distillation methods.

s146
00:17:40.640 --> 00:17:50.760
And so, we see like the the best performing thing might be an orchestration of open source and close source models, but we're kind of optimizing the whole thing together.

s147
00:17:50.760 --> 00:17:58.400
And to the last point, you know, how do we as as experts encode our expertise, we again think it's through environments.

s148
00:17:58.400 --> 00:18:03.320
Like building environments, at least good environments, is really really hard work.

s149
00:18:03.320 --> 00:18:14.440
You know, it's you have to be there's of course like the data that goes in that's proprietary, but you know, designing the the verifiable metrics is maybe the easy part, but then we're also doing this qualitative

s150
00:18:14.440 --> 00:18:27.520
rubrics, where we kind of look at the traces and say, you know, what makes a good researcher, what's the thought process, and we can grade each roll out on, you know, how well it's following our research process.

s151
00:18:27.520 --> 00:18:37.280
And so, having these good rubrics that become the signal for the model to learn, we feel is really how we're we're building our own expertise into the system.

s152
00:18:37.280 --> 00:18:47.640
And so, just to kind of conclude the the 2.0 version is really having this strict environment and eval setup, and you know, whatever lives in the middle, we almost

s153
00:18:47.640 --> 00:18:49.280
in the limit kind of don't care about.

s154
00:18:49.280 --> 00:18:55.560
We can initialize it to Alpha Lab 1.0, but this it should really be this self-improving system.

s155
00:18:55.560 --> 00:19:07.200
Um and just to leave kind of the the bigger picture headline result, and my feeling is you know, this ability to do general auto research, I think will kind of become a commodity.

s156
00:19:07.200 --> 00:19:17.840
Like I think we're already seeing it with GLM 5.2, and so I really think all of your value as like an enterprise or human expert comes from building environments.

s157
00:19:17.840 --> 00:19:30.880
And so like, you know, temporarily for us that we're still doing like manual tuning against the environment, but I think in the limit the the auto research can research itself and just be the self-improving

s158
00:19:30.880 --> 00:19:32.760
um process.

s159
00:19:32.760 --> 00:19:43.720
And so, um my site's on there and then also the the project page again the 1.0 version we released everything and our plan is to keep just releasing because again,

s160
00:19:43.720 --> 00:19:47.080
we think the environments encode all of the the value.

s161
00:19:47.080 --> 00:19:49.640
Uh so, thank you.

s162
00:20:04.558 --> 00:20:06.558
[music]