WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=V-EDrhIhHzQ

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/V-EDrhIhHzQ.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.960 --> 00:00:13.960
Hey guys, how's it going?

s2
00:00:13.960 --> 00:00:14.600
Thanks for showing up.

s3
00:00:14.600 --> 00:00:17.240
This was a little bit of a last-minute assembly.

s4
00:00:17.240 --> 00:00:21.040
I know a few days ago I was like talking to Swix.

s5
00:00:21.040 --> 00:00:22.240
I was like, "Hey, can I still do a workshop?"

s6
00:00:22.240 --> 00:00:23.800
And he was like, "We have one slot left.

s7
00:00:23.800 --> 00:00:24.600
It's Monday at 4:30."

s8
00:00:24.600 --> 00:00:26.320
And I was like, "I'll take it."

s9
00:00:26.320 --> 00:00:34.760
Um and uh then yeah, um I wanted to kind of just do a bit of an update on uh some of the stuff we've been building at Primed and Loaded.

s10
00:00:34.760 --> 00:00:36.560
So, if you don't uh know me, hi.

s11
00:00:36.560 --> 00:00:37.280
I'm Will Brown.

s12
00:00:37.280 --> 00:00:39.160
I lead applied research at Primed and Loaded.

s13
00:00:39.160 --> 00:00:44.720
Uh we do a lot of stuff around uh every part of the kind of AI research infrastructure stack.

s14
00:00:44.720 --> 00:00:48.960
Uh today is going to be about post-training, which is where I spend a lot of my time thinking and building.

s15
00:00:48.960 --> 00:00:58.680
Um and especially want to be talking about uh the post-training tools that we build uh that are fully open source, uh the verifiers and Primed RL libraries, uh which kind of go hand in hand um

s16
00:00:58.680 --> 00:01:01.040
both on the environment side and the training infra side.

s17
00:01:01.040 --> 00:01:09.760
Um and show [clears throat] off some things we've been cooking over the past few months that I think is uh kind of the way that things have evolved as

s18
00:01:09.760 --> 00:01:21.880
the agent use cases have gotten more complex, but also kind of clearer in terms of what people want out of agents and the sorts of things that are needed to like do the sort of post-training that is needed to uh power like the real-world applications people are building nowadays.

s19
00:01:21.880 --> 00:01:39.000
And so, broadly at Primed and Loaded, we are our goal is to make doing large-scale open-source AI research easier and to enable companies to train their own models and deploy them and have them improve based on the scenarios that they actually see in production

s20
00:01:39.000 --> 00:01:44.480
in terms of use cases for applications and products and internal tasks and workflows.

s21
00:01:44.480 --> 00:01:52.840
Um and to give people an option to not just use the open-source models that are getting quite good, but to take them and make them even better on their own use cases.

s22
00:01:52.840 --> 00:01:57.480
Um and so, we use the phrase the open superintelligence stack to describe what we mean by this.

s23
00:01:57.480 --> 00:02:06.680
And I think when we said this phrase like a year ago, it felt a little more like marketing and now it feels a little bit more like, oh, yeah, that's that's kind of what it is.

s24
00:02:06.680 --> 00:02:08.640
Like the models are getting very, very good.

s25
00:02:08.640 --> 00:02:11.600
They are superhuman in many ways at lots of things.

s26
00:02:11.600 --> 00:02:19.516
Um and what we want to do is give people an open toolkit that they can use to do real training with them.

s27
00:02:19.516 --> 00:02:19.840
[clears throat]

s28
00:02:19.840 --> 00:02:26.160
And to have the control that they need to deploy it where they need to deploy it and customize it as much as they need to to kind of get the job done.

s29
00:02:26.160 --> 00:02:27.760
And so this is the stack that we built.

s30
00:02:27.760 --> 00:02:29.880
And it all kind of sits on top of compute.

s31
00:02:29.880 --> 00:02:34.120
So we operate a global marketplace of data centers around the world.

s32
00:02:34.120 --> 00:02:36.520
A lot of these are like quite large data centers.

s33
00:02:36.520 --> 00:02:40.040
We currently operate over 10,000 GPUs.

s34
00:02:40.040 --> 00:02:43.720
Many in like hundreds or thousands within a cluster.

s35
00:02:43.720 --> 00:02:45.640
We have our primary training framework.

s36
00:02:45.640 --> 00:02:50.120
We have environments built with the Verifiers library and our environments hub platform.

s37
00:02:50.120 --> 00:03:00.760
We have our platform for research workflows that we're now calling lab, which is an assembly of many pieces including the environments hub, hosted training evaluations, as well as inference and sandboxes.

s38
00:03:00.760 --> 00:03:05.080
And all of this is in service of empowering and unlocking frontier model training.

s39
00:03:05.080 --> 00:03:07.000
And so we do this ourselves.

s40
00:03:07.000 --> 00:03:11.400
We have our intellect model series with some exciting things there coming soon.

s41
00:03:11.400 --> 00:03:19.400
And we also train models with our customers where we have lots of people we work with who their goal is to do large scale model training on their own workflows.

s42
00:03:19.400 --> 00:03:31.640
And so to do all of this, we need to give people the tools they can assemble into the the the pipelines, the workflows, the research that allows them to actually get the results that they need at scale with everything they need to do it.

s43
00:03:31.640 --> 00:03:41.280
And so this talk is going to be about going deep into Verifiers and Primary L and showing off some of these new things, but all under the umbrella of what does modern post training look like?

s44
00:03:41.280 --> 00:03:45.280
What does it mean to kind of take a model and train it to be better at your task?

s45
00:03:45.280 --> 00:03:46.440
What are all the parts?

s46
00:03:46.440 --> 00:03:48.680
What are all the kind of gotchas?

s47
00:03:48.680 --> 00:04:01.520
And how do you orchestrate this into a system that is actually easy for people to use without needing to go build a massive research team and to be able to kind of have it be accessible in the sorts of things that anyone who's an AI engineer at

s48
00:04:01.520 --> 00:04:05.600
any like startup or enterprise that wants to invest in post-training can actually do.

s49
00:04:05.600 --> 00:04:09.640
And so there's a cookbook repo that is that's kind of like an alpha release right now.

s50
00:04:09.640 --> 00:04:14.120
It's still changing a bit, but it's a preview of kind of all the stuff we've been building over the past several months.

s51
00:04:14.120 --> 00:04:19.160
And so today we'll be kind of following along that framing a good bit.

s52
00:04:19.160 --> 00:04:23.200
And so I think the first thing we'll talk about is just kind of what is an environment.

s53
00:04:23.200 --> 00:04:28.600
People talk about environment in the context of RL and they think of like RL environments, but environments are more than just for RL.

s54
00:04:28.600 --> 00:04:32.280
They're for all sorts of things in post-training and evaluation.

s55
00:04:32.280 --> 00:04:37.320
We're going to talk about what we're going to call the the V1 version of the Verifiers library, which is a full overhaul.

s56
00:04:37.320 --> 00:04:41.200
Everything else still from before still works, but we're kind of we kind of wanted to redo it all.

s57
00:04:41.200 --> 00:04:47.800
And so we have a new way of doing everything that we think is going to make a lot more sense, be a lot more powerful for what people are looking to do going forward.

s58
00:04:47.800 --> 00:04:51.040
As well as kind of talk about how Primer RL has evolved as a library.

s59
00:04:51.040 --> 00:04:57.640
And so Primer RL is our like full-stack open-source training framework to support asynchronous reinforcement learning.

s60
00:04:57.640 --> 00:05:03.280
And we've got a lot of fun new bells and whistles to show off in terms of both scale and features.

s61
00:05:03.280 --> 00:05:05.280
A lot of this is in service of custom algorithms.

s62
00:05:05.280 --> 00:05:10.520
So making it much easier to do the kinds of things that people are interested in for modern post-training.

s63
00:05:10.520 --> 00:05:16.880
If you have been following the news on on policy distillation or self-distillation or all these other fun new algorithms that people are coming out with.

s64
00:05:16.880 --> 00:05:19.880
It is the the age of research indeed.

s65
00:05:19.880 --> 00:05:22.120
And we don't want to just like train small models.

s66
00:05:22.120 --> 00:05:23.640
We want to train big models.

s67
00:05:23.640 --> 00:05:29.320
We want to train them really efficiently because as models get bigger, the compute starts adding up.

s68
00:05:29.320 --> 00:05:34.960
And if you want to make this accessible to people, especially if you want to be able to iterate on it, it has to be fast.

s69
00:05:34.960 --> 00:05:36.320
It has to be cheap.

s70
00:05:36.320 --> 00:05:39.760
It has to be affordable and reliable.

s71
00:05:39.760 --> 00:05:42.160
And all of these kind of funnel into our lab platform.

s72
00:05:42.160 --> 00:05:46.880
We'll talk about both some of the things that we've already released there as well as some things that are coming soon.

s73
00:05:46.880 --> 00:05:57.480
And so the post-training loop in my mind kind of revolves around environments in the sense of environments are a language for specifying what you want your model to do.

s74
00:05:57.480 --> 00:06:11.240
Um, they are an encapsulation of the data you might have, the scenario you might want your agent to be in, uh, the way it'll interact with that environment, uh, as well as how to to score what good looks like, to determine what was good and bad.

s75
00:06:11.240 --> 00:06:17.560
Um, and often this is the first thing you want to do with an environment is just evals.

s76
00:06:17.560 --> 00:06:21.440
And so I think a lot of people are maybe nervous about getting into post-training.

s77
00:06:21.440 --> 00:06:23.120
They're like, "Oh, it seems like a lot of work.

s78
00:06:23.120 --> 00:06:24.720
There's a whole new tool chain.

s79
00:06:24.720 --> 00:06:34.680
Um, what if I'm already using like the frontier models and I want good results out of them, uh, or I'm getting good results out of them, or I want to like see what I can do at the harness level first or prompt optimization."

s80
00:06:34.680 --> 00:06:35.360
And that's all good.

s81
00:06:35.360 --> 00:06:37.960
Like we're not necessarily asking people to just like throw everything away.

s82
00:06:37.960 --> 00:06:45.280
I think in many cases what people will find and what we see with our customers is that um, the systems that work best for them involve using both.

s83
00:06:45.280 --> 00:06:54.680
And you kind of want to be able to make these decisions about what is the right where is the right place to train, uh, where's the right place to use a frontier model that is available via some API.

s84
00:06:54.680 --> 00:06:57.120
Um, and so evaluations are kind of very key to this.

s85
00:06:57.120 --> 00:07:00.960
And so like evals are the thing that opens the door to post-training.

s86
00:07:00.960 --> 00:07:04.200
And so environments and evals are essentially the same thing.

s87
00:07:04.200 --> 00:07:11.880
Um, but once you have evals, now this is the same kind of unit of, uh, like logic that you actually need to do post-training anyways.

s88
00:07:11.880 --> 00:07:17.440
And so, uh, building evals is like just good for your product hygiene no matter what you're doing.

s89
00:07:17.440 --> 00:07:24.240
If you want to kind of decide whether to use GPT or Claude, or decide do you need Opus or uh, Sonnet or Mythos for a task.

s90
00:07:24.240 --> 00:07:29.520
Like if you want to min-max on like intelligence versus dollars, um, evals are a very good way to do this.

s91
00:07:29.520 --> 00:07:32.240
But evals also then unlock this this flywheel.

s92
00:07:32.240 --> 00:07:42.840
And in terms of modern post-training, I think historically people have done SFT then RL as like the main uh, frontier model recipe, although on policy distillation has certainly found its way into a lot of workflows.

s93
00:07:42.840 --> 00:07:46.800
Um, and I think some people are also very eager about algorithms like self-distillation.

s94
00:07:46.800 --> 00:08:00.560
We can talk a bit about that and when it makes sense and when it doesn't, but uh, in particular one area where it does make sense to do like the whole on-policy distillation thing is when you're training experts where you have multiple different things you want your model to be good at, and people have found that if

s95
00:08:00.560 --> 00:08:13.000
you have a bunch of different environments that are all different things, and you want to have one model be really good at them, a nice way to do this is train individual RL experts on top of the same base model and then do distillation from those teachers into the same

s96
00:08:13.000 --> 00:08:14.040
uh, checkpoint.

s97
00:08:14.040 --> 00:08:16.919
Uh, that just generally ends up being more reliable.

s98
00:08:16.919 --> 00:08:24.280
Um, and then once you have this, you want to deploy the trained model, which could be a full uh, base model with a full weight training, or it could be a Lora adapter,

s99
00:08:24.280 --> 00:08:30.960
and you want to serve this at scale, and ultimately what is useful about this whole process is it's not just a thing you do once.

s100
00:08:30.960 --> 00:08:36.520
Like I think some people also say like, "Oh, why should I do post-training if the the frontier models are going to get better?"

s101
00:08:36.520 --> 00:08:38.240
Well, your model should get better, too.

s102
00:08:38.240 --> 00:08:40.479
Like it's not it like everything's going to get better.

s103
00:08:40.479 --> 00:08:43.120
Uh, the point of this is to have flywheels that make everything get better.

s104
00:08:43.120 --> 00:08:56.960
And so what you really want is to be able to not just post-train like today, but to be able to uh, have this iterative process of model refinement, and the sort of thing where you can kind of have the training compute end up be a pretty small fraction of your overall inference budget

s105
00:08:56.960 --> 00:09:03.360
that you amortize out such that your model is always getting better and better um, as you are getting more signal from the real world.

s106
00:09:03.360 --> 00:09:05.640
And getting this signal from the real world isn't trivial.

s107
00:09:05.640 --> 00:09:14.600
Like that's kind of largely an open question as to like how you go about um, getting information from real world feedback into your environments.

s108
00:09:14.600 --> 00:09:20.080
It's an engineering problem, it's a research problem, but it's the sort of thing we're all here at this conference to kind of think about and learn about.

s109
00:09:20.080 --> 00:09:24.600
And so I'll I'll touch on that a little bit uh, in the talk as to kind of how we we think about this.

s110
00:09:24.600 --> 00:09:28.400
But but really the goal is going to be thinking about what do these tools look like?

s111
00:09:28.400 --> 00:09:29.440
How do you actually do this?

s112
00:09:29.440 --> 00:09:30.480
What are the parts?

s113
00:09:30.480 --> 00:09:31.960
Um, and how do we build it?

s114
00:09:31.960 --> 00:09:36.400
Um, and so environments as evals, uh, Uh, what is an environment?

s115
00:09:36.400 --> 00:09:42.520
Um, I think it's useful to decompose environments into tasks and a harness.

s116
00:09:42.520 --> 00:09:47.920
Um, and this is foreshadowing some of the the refactoring we've done in verifiers over the past months.

s117
00:09:47.920 --> 00:10:03.400
Uh, if any of you have used the verifiers library before, you may be familiar with the the multi-turn environment pattern or tool environment pattern where there's kind of one loop that is owned by the environment that you can plug in various tools into and that was really great for a very long time for getting started for people,

s118
00:10:03.400 --> 00:10:09.280
especially back in the day when people were mostly just trying to graduate from single turn into multi-turn tool tool use.

s119
00:10:09.280 --> 00:10:21.240
Uh, but what we found and as we kind of iterated on different patterns and extended it in various different ways, we found ourselves repeating a lot of work of like adding patterns for a CLI agent or adding patterns for MCP.

s120
00:10:21.240 --> 00:10:26.840
Um, and we wanted to be able to like step back and rethink like how should an environment work.

s121
00:10:26.840 --> 00:10:29.400
Um, and what it really is there's a notion of a harness.

s122
00:10:29.400 --> 00:10:31.840
And so and you there's also a notion of a task.

s123
00:10:31.840 --> 00:10:40.720
And I think one of the reasons that this is kind of subtle and tricky and was a a design problem that we went over many iterations over the past 6 months really is um,

s124
00:10:40.720 --> 00:10:42.720
certain things it's not clear where they live.

s125
00:10:42.720 --> 00:10:45.800
Like there's certain things that might belong to the harness and might belong to the task.

s126
00:10:45.800 --> 00:10:52.240
There's certain tools that uh, in some cases it's I want this uh, task I'm going to do to use a certain tool.

s127
00:10:52.240 --> 00:10:54.360
In some case the harness has certain tools.

s128
00:10:54.360 --> 00:11:02.680
Same with skills or system prompts or many other pieces of the puzzle in assembling like the full world that your agent is going to be operating in or your model is going to be operating in.

s129
00:11:02.680 --> 00:11:06.400
Um, but ultimately the we're going to call all of these parts of the environment.

s130
00:11:06.400 --> 00:11:14.240
Um, and the goal of this is to have some notion of verification where you plug in a model uh, into the environment which includes a harness.

s131
00:11:14.240 --> 00:11:15.400
You give it a task.

s132
00:11:15.400 --> 00:11:18.960
It does a rollout and then you verify what it did.

s133
00:11:18.960 --> 00:11:30.640
Um, and this same process works both for evaluation offline just understanding which model is better as well as for doing reinforcement learning RL uh, as well as for generating data for SFT.

s134
00:11:30.640 --> 00:11:38.000
Uh, I think in many cases people think of SFT is this thing where they want to like upload a data set, but really often what they're doing there is they're essentially

s135
00:11:38.000 --> 00:11:49.280
cobbling together something that's essentially an environment and they're doing rollouts in it and saving it offline and putting it in one format and uploading it and then changing it to another format and then plugging it into a trainer.

s136
00:11:49.280 --> 00:11:57.760
And the way we've kind of approached this is saying, "Well, you can just kind of cut all that out and just like treat it like a a a problem where you're doing rollouts in an environment.

s137
00:11:57.760 --> 00:11:59.480
Just in this case there's a teacher."

s138
00:11:59.480 --> 00:12:03.760
And the teacher could be another it could be replaying from another data set.

s139
00:12:03.760 --> 00:12:07.360
But ultimately it's about collecting rollouts and training on those rollouts.

s140
00:12:07.360 --> 00:12:20.360
And then on policy distillation again takes the same form where you are doing rollouts in an environment just as you would for RL, but the scoring is from a teacher and the the log probs of the teacher the likelihood of the teacher rather than

s141
00:12:20.360 --> 00:12:22.600
the the reward signal itself.

s142
00:12:22.600 --> 00:12:26.240
And so these all in our framework are Python packages.

s143
00:12:26.240 --> 00:12:28.480
So you can have any task as you want.

s144
00:12:28.480 --> 00:12:31.560
You can pull data from anywhere you want.

s145
00:12:31.560 --> 00:12:37.080
There's a lot of flexibility that we've unlocked in terms of what these tasks can look like, what these harnesses can look like.

s146
00:12:37.080 --> 00:12:45.560
And our goal is to just make this a really flexible toolkit for all the kinds of evaluation things people want to do both for API models as well as for post training.

s147
00:12:45.560 --> 00:12:58.160
And so Verifiers V1 is what we're calling it, which is it's not actually released as V1 yet, but we took inspiration from VLUM doing this and decided that we were going to kind of have this be the

s148
00:12:58.160 --> 00:13:02.640
new pattern that we want to have everything kind of be centered around.

s149
00:13:02.640 --> 00:13:08.640
And the key pieces we broke things down into were a task set, a harness, and a runtime.

s150
00:13:08.640 --> 00:13:10.839
And so these are all composable.

s151
00:13:10.839 --> 00:13:16.760
You can mix and match them and they're all individually loadable in different ways.

s152
00:13:16.760 --> 00:13:25.600
But the way to think about it is that task sets are the data and the rules of what should be done that are agent agnostic.

s153
00:13:25.600 --> 00:13:28.560
So they're the sort of thing you could plug an agent into.

s154
00:13:28.560 --> 00:13:33.480
Um and we wanted to take a very general approach in supporting a lot of the great work being done throughout the ecosystem.

s155
00:13:33.480 --> 00:13:40.040
So, we integrate natively with Hugging Face datasets, with Harbor, uh with NeMo Gym and Open Ended.

s156
00:13:40.040 --> 00:13:48.280
And most other tools that you see out in the wild that are kind of under the umbrella of uh an RL environment, we would call these a task set.

s157
00:13:48.280 --> 00:14:00.440
Um we generally have found that it's useful to have these be harness agnostic where they represent the the back end of the server or some state that you're interacting with, but they don't own everything about

s158
00:14:00.440 --> 00:14:01.520
what the model is doing.

s159
00:14:01.520 --> 00:14:05.840
And so, it doesn't In some cases, it doesn't make sense to plug a model into a task set.

s160
00:14:05.840 --> 00:14:14.280
Especially because we're kind of gravitating towards an agent world where everything is running in a terminal or it has skills or it uh is using CLI tools.

s161
00:14:14.280 --> 00:14:19.520
Um and these things like often look more complex than just basic loops.

s162
00:14:19.520 --> 00:14:21.880
Um but we also want to support basic loops.

s163
00:14:21.880 --> 00:14:25.560
So, we want to kind of allow both the old way of doing things and the new way of doing things.

s164
00:14:25.560 --> 00:14:31.640
And so, uh everything that was the old way is now the default harness where it's system prompt and tools in a loop.

s165
00:14:31.640 --> 00:14:48.200
Um but the harness pattern also supports much more flexible execution of things like recursive language models or CLI agents like Codex, Cloud Code, Open Code, or classics from the research literature like Mini Sweep Agent or building your own with arbitrary Python libraries like LangChain or DSPy.

s166
00:14:48.200 --> 00:14:54.720
Um and so, we've been able to decouple these into a pattern where you get to write your harness independently of your task set.

s167
00:14:54.720 --> 00:15:01.600
Uh there are kind of some basic sanity checks about properties that like harnesses either do or don't support and task sets do or don't require.

s168
00:15:01.600 --> 00:15:03.080
Uh and these kind of click together.

s169
00:15:03.080 --> 00:15:05.760
And the runtime is where this executes.

s170
00:15:05.760 --> 00:15:16.760
And so, we've uh still been embracing a lot of the async IO patterns from before, but we've leaned a little more into having things be subprocesses uh where you can still run everything locally.

s171
00:15:16.760 --> 00:15:23.080
You don't have to use sandboxes, but you can use local Docker, or you can use our own Prime sandboxes layer.

s172
00:15:23.080 --> 00:15:27.120
You can use any other sandbox layer you'd like or kind of build from scratch.

s173
00:15:27.120 --> 00:15:32.120
And so the harness the runtime back end just is a place where the harness can run its code.

s174
00:15:32.120 --> 00:15:37.040
And so the harness just needs to be able to run code somewhere as a script essentially.

s175
00:15:37.040 --> 00:15:44.720
We've used a lot of the UV tooling where UV script is a very powerful pattern to be able to kind of mix and match and kind of contain dependencies.

s176
00:15:44.720 --> 00:15:51.600
But what happens is once you plug these together, you run a rollout on a task from a task set and you get a trace.

s177
00:15:51.600 --> 00:15:58.040
This is live on the verifiers main branch for prim and elect AI / verifiers on GitHub as well as it's released as a dev release.

s178
00:15:58.040 --> 00:16:05.160
The stable main release will be kind of coming to PyPI any minute now, but you can install the dev and play around with it if you want.

s179
00:16:05.160 --> 00:16:06.120
And so what do these look like?

s180
00:16:06.120 --> 00:16:09.360
So tasks are just like a row of a data set.

s181
00:16:09.360 --> 00:16:16.160
And the very basic version of it is you just start loading a data set from hugging face or anywhere else.

s182
00:16:16.160 --> 00:16:21.680
And so the new pattern here is from verifiers V1 I just to keep the old stuff separate.

s183
00:16:21.680 --> 00:16:28.360
The old stuff still works just fine, but this is how we have been able to kind of decouple and iterate on the new version.

s184
00:16:28.360 --> 00:16:30.760
As well as we've really embraced this decorator pattern.

s185
00:16:30.760 --> 00:16:31.800
We found it to be very useful.

s186
00:16:31.800 --> 00:16:35.160
We also if you were a rubric fan, we killed rubric.

s187
00:16:35.160 --> 00:16:39.320
Didn't make sense anymore if you were using old verifiers rubric patterns.

s188
00:16:39.320 --> 00:16:43.160
But still it's you have functions and loaders.

s189
00:16:43.160 --> 00:16:46.440
We are very heavy on PyDantic so everything is super typed.

s190
00:16:46.440 --> 00:16:50.800
We have lots of powerful config features where you can have everything in a toml file.

s191
00:16:50.800 --> 00:17:00.920
You can override it in the CLI and everything is kind of clean and guaranteed to kind of type check at like validation time rather than waiting for something to fail later down the road.

s192
00:17:00.920 --> 00:17:04.199
And so examples of this are things like sweet wrapper where you can do a genetic code search.

s193
00:17:04.199 --> 00:17:06.319
You can do the classic games like Wordle.

s194
00:17:06.319 --> 00:17:09.520
You can do search over documents with judges.

s195
00:17:09.520 --> 00:17:16.120
You can do complex things like harbor that support a lot of popular benchmarks now that need agents running in a terminal.

s196
00:17:16.120 --> 00:17:24.040
And all of these are going to be combinations of the the the task that pattern with uh pick your own runtime and pick your own harness.

s197
00:17:24.040 --> 00:17:32.880
Um and so rewards and metrics I think are also kind of uh just functions that take in the kind of records uh of what's happening in a rollout and return numbers.

s198
00:17:32.880 --> 00:17:36.320
Uh so rewards are the main thing that'll drive progress in RL.

s199
00:17:36.320 --> 00:17:39.800
Um metrics are just kind of like logging uh what has happened.

s200
00:17:39.800 --> 00:17:42.160
So counting tool use and counting errors.

s201
00:17:42.160 --> 00:17:45.640
Um these sorts of things are very useful to be able to expose in your dashboards.

s202
00:17:45.640 --> 00:17:46.640
Um and then group rewards.

s203
00:17:46.640 --> 00:17:57.880
I think this is something that we have fought hard to kind of make sure still is first class because we see it as very important to um a lot of the research pattern people want to do, but I think it's also

s204
00:17:57.880 --> 00:18:00.320
ignored in a lot of like uh tooling out there.

s205
00:18:00.320 --> 00:18:08.240
Where in in many RL frameworks it's actually quite hard to do group rewards because things are very decoupled and things kind of assume that all rollouts are going to live independently

s206
00:18:08.240 --> 00:18:09.480
and that they don't need to talk to each other.

s207
00:18:09.480 --> 00:18:18.920
But there's a lot of things where you really want to do pairwise judging or you want to do ranking or you want to give a bonus to the uh the shortest correct answer

s208
00:18:18.920 --> 00:18:20.560
uh in terms of tokens used.

s209
00:18:20.560 --> 00:18:35.480
Um and so these sorts of things are really flexible uh in terms of the we really design for flexibility in supporting the the things that we see as like the most exciting papers we've read or all the algorithms that we think people may want to innovate on um while still allowing people to have like the core primitives

s210
00:18:35.480 --> 00:18:38.480
that they kind of expect out of an RL framework.

s211
00:18:38.480 --> 00:18:43.040
Um and so like in group rewards, I think this concise this pattern is one that I find very useful a lot.

s212
00:18:43.040 --> 00:18:51.960
I think um like a big pattern that comes up a lot are doing post-training is uh models will love to like think and think and think if you let them.

s213
00:18:51.960 --> 00:19:01.920
Um and if you don't give them some kind of pressure to like be more efficient, uh I think a lot of people will notice that like open models often have really really long chains of thought.

s214
00:19:01.920 --> 00:19:10.920
Because on one hand it's like this is a useful strategy for a model, but it's also the sort of thing that will grow like out of control if you don't counteract it.

s215
00:19:10.920 --> 00:19:19.360
Um and so in reward design like one of the big things people will want to do is uh something like a length penalty um or a conciseness bonus.

s216
00:19:19.360 --> 00:19:23.040
Um, and so one of the reasons this is tricky is because you don't know the optimal length for a problem.

s217
00:19:23.040 --> 00:19:28.360
Like if I give you a math problem, I could say, "Oh, solve it in less than N tokens."

s218
00:19:28.360 --> 00:19:30.480
But also like who knows what the right N is.

s219
00:19:30.480 --> 00:19:32.880
It's also going to change as the model gets smarter over time.

s220
00:19:32.880 --> 00:19:34.480
It's going to be different for every problem.

s221
00:19:34.480 --> 00:19:36.880
And so you kind of can't know this up front.

s222
00:19:36.880 --> 00:19:39.960
And the only way you can do it is take advantage of variance.

s223
00:19:39.960 --> 00:19:44.680
So one of the nice things about RL is you have multiple samples typically.

s224
00:19:44.680 --> 00:19:49.800
Um, and this allows you to use the fact that you have multiple samples to shape the reward.

s225
00:19:49.800 --> 00:19:58.960
Um, and so if you have multiple rewards in a group, what you could do is uh look at all the ones that were the correct answer or just all the ones in general

s226
00:19:58.960 --> 00:20:00.920
and give a bonus to the ones that are the most concise.

s227
00:20:00.920 --> 00:20:11.200
Where if you also have a uh a correctness reward like these are going to uh ensure that um you're both incentivizing correctness as well as incentivizing efficiency.

s228
00:20:11.200 --> 00:20:17.520
And so juggling multiple objectives simultaneously is kind of one of the hard challenges in RL in reward design.

s229
00:20:17.520 --> 00:20:24.040
Um, but doing things like group level comparisons and kind of these sorts of bonuses are are quite useful in many cases.

s230
00:20:24.040 --> 00:20:36.840
Um, I also want to talk about like tools and user simulators, which I think have been uh becoming more important uh in a lot of complex applications where you have models that are In many cases there's like a core agent harness, but there's also

s231
00:20:36.840 --> 00:20:42.680
In many cases you are putting a model in a setting where it's going to be in some task where a user is giving it additional tools.

s232
00:20:42.680 --> 00:20:45.600
Whether these are In some cases you might want to model these as skills.

s233
00:20:45.600 --> 00:20:47.720
In some cases you might want to model them as MCP servers.

s234
00:20:47.720 --> 00:20:55.000
We use MCP as a kind of uh a back-end framework um that can interact with the runtime uh both for tools and for user simulators.

s235
00:20:55.000 --> 00:21:05.680
So user simulators, especially if you want to do training where there's a a user in the loop, um you don't want just your agent to go do some task, but you want to be able to do some task that involves understanding how a user will interact.

s236
00:21:05.680 --> 00:21:12.120
You can essentially have this user be an MCP as well where we make it so the model sees it as a user, not as a tool.

s237
00:21:12.120 --> 00:21:21.480
Um but behind the scenes, it is a server that has some script or something and it has some LLM that is going to get some context and it's going to be able to

s238
00:21:21.480 --> 00:21:24.160
be like a user in the context of a rollout.

s239
00:21:24.160 --> 00:21:35.080
Uh in many cases, benchmarks have found that this is very useful for simulating the realism of like having a multi-turn setting where there are users in the loop, especially given that people are building products now where there are these users in the loop.

s240
00:21:35.080 --> 00:21:41.280
And so you want kind of a first-class way to incorporate this into um your uh your RL environments.

s241
00:21:41.280 --> 00:21:46.360
And so we've found that it's useful to have all of these things be kind of modular and pluggable.

s242
00:21:46.360 --> 00:21:52.160
And so the harness can connect to each of these which run as a as a UV script.

s243
00:21:52.160 --> 00:21:56.920
Um we also have UV script support for grading in addition to the basic reward function patterns.

s244
00:21:56.920 --> 00:22:00.560
Um and we also are using this idea called an interception server.

s245
00:22:00.560 --> 00:22:08.800
So the harness, we want people to be able to use real harnesses and not have to like break the harness and like retrofit it into like an RL harness.

s246
00:22:08.800 --> 00:22:12.280
And so ideally, we don't know anything about the harness code.

s247
00:22:12.280 --> 00:22:23.080
And so the pattern that we use with the interception server is that um we are responsible for giving each harness rollout a fake base URL, which could be OpenAI compatible or Anthropic compatible.

s248
00:22:23.080 --> 00:22:25.040
And the harness just thinks it's talking to some endpoint.

s249
00:22:25.040 --> 00:22:28.080
So any harness that can just talk to some endpoint, we're good to go.

s250
00:22:28.080 --> 00:22:30.280
And then we intercept each request.

s251
00:22:30.280 --> 00:22:36.640
Uh we can do some some back-end maneuvering to make sure that we're getting the log probs and setting the right temperature.

s252
00:22:36.640 --> 00:22:41.160
Um and then we send this to our uh inference server for with the RL trainer.

s253
00:22:41.160 --> 00:22:44.320
Uh and then as it completes, we send back the request.

s254
00:22:44.320 --> 00:22:46.680
And so the the harness doesn't know that it's doing RL.

s255
00:22:46.680 --> 00:22:50.720
The harness just is a harness running as if it would be running in a real-world environment.

s256
00:22:50.720 --> 00:22:57.320
And so you can kind of very easily move between the RL setting and the deployment setting where your harness is just code.

s257
00:22:57.320 --> 00:23:00.880
Doesn't need to be anything specialized to verifiers or RL.

s258
00:23:00.880 --> 00:23:08.080
We also have found it really easy useful to be able to kind of go from this like local to global and like have this hot swap pattern.

s259
00:23:08.080 --> 00:23:14.280
Um so we have this new like uh eval CLI where um you can just like choose the harness you want.

s260
00:23:14.280 --> 00:23:18.280
You can have a task set where you're going to say I'm going to run an eval on this set of tasks.

s261
00:23:18.280 --> 00:23:20.440
Okay, I want to use recursive language models and RLMs.

s262
00:23:20.440 --> 00:23:21.640
I want to use Codex.

s263
00:23:21.640 --> 00:23:22.560
I want to run it locally.

s264
00:23:22.560 --> 00:23:24.040
I want to run it in sandboxes.

s265
00:23:24.040 --> 00:23:25.480
Uh I want to run it in Docker.

s266
00:23:25.480 --> 00:23:27.120
These are all just like interchangeable.

s267
00:23:27.120 --> 00:23:39.080
Um and so we found that this is super useful in the iteration loop as you go from testing something out at a small scale towards scaling it up towards being able to understand questions like what's the best harness for this model?

s268
00:23:39.080 --> 00:23:42.360
Um does this harness generalize across tasks?

s269
00:23:42.360 --> 00:23:54.400
Um as well as being able to both have the convenience of like local prototyping where you can kind of run things fast on your MacBook um without having to like wait for a cloud job to finish, but also you can like go right to the cloud when you need to.

s270
00:23:54.400 --> 00:23:59.760
And so there's been a lot of fun patterns we've had to kind of innovate on um as we've done this overhaul.

s271
00:23:59.760 --> 00:24:01.600
And we we're quite happy with how it's turned out.

s272
00:24:01.600 --> 00:24:12.040
It's made our lives a lot easier for both client projects and research and just uh being able to have have a lot more flexibility and power and control over the the kinds of agents we want to be training.

s273
00:24:12.040 --> 00:24:16.280
Um and so one of the fun things behind the scenes uh is what we call the trace graph.

s274
00:24:16.280 --> 00:24:25.200
And so we had kind of been having this grow out of control in terms of the old way of doing things and we decided this was another opportunity to like really overhaul

s275
00:24:25.200 --> 00:24:37.600
our system to like have really good support for sub agents and parallel branching trees while also still preserving the kind of linear sequential dependencies that you need for RL with uh careful token control.

s276
00:24:37.600 --> 00:24:44.040
Um and so here uh there's a a notion of a of branches that are kind of like at the message level.

s277
00:24:44.040 --> 00:24:52.000
So conceptually um the things that matter logically in environment space and in harness space are messages which are just text.

s278
00:24:52.000 --> 00:25:02.640
Uh the harnesses don't think about tokens, uh but if you've done any RL experimentation you may have uh encountered issues where uh re-tokenization or like some messages if a model will say something

s279
00:25:02.640 --> 00:25:05.960
and you turn it into text and you put it back through tokenizer, it can change a little bit.

s280
00:25:05.960 --> 00:25:09.320
That because tokenization isn't is many to one.

s281
00:25:09.320 --> 00:25:15.520
And so this causes lots of very subtle numerical problems, especially late in large scale training runs.

s282
00:25:15.520 --> 00:25:21.000
And so you want a really nice back and forth between uh messages and tokens.

s283
00:25:21.000 --> 00:25:32.760
And so the trace data structure that we created here partly is to enable this where we can store things both at trace level and then map them back into token level in the right sequences as needed.

s284
00:25:32.760 --> 00:25:45.200
And we also released a library called renderers recently, which is a standalone toolkit that anyone can use that we have found the sort of thing that we're working with some of the inference tooling to

s285
00:25:45.200 --> 00:25:45.920
support.

s286
00:25:45.920 --> 00:25:57.640
So renderers are really all about like essentially rethinking tokenizers and chat templates where behind the scenes it's just making calls to the tokenizer, but chat templates if people have

s287
00:25:57.640 --> 00:26:00.080
spent time debugging with them, it sucks.

s288
00:26:00.080 --> 00:26:01.800
Ginger is awful.

s289
00:26:01.800 --> 00:26:10.120
It's very very painful and there's so many subtle things that we kept running into where like a model would sometimes have an extra new line and the chat template would strip it out

s290
00:26:10.120 --> 00:26:17.880
and this would like cause a mismatch in your trainer and inference that would either force you to go off policy because you now have a trainer and inference mismatch

s291
00:26:17.880 --> 00:26:28.160
or it would cause a logical branch where a thing that is a branch in like uh it becomes a branch in token space even though it shouldn't be in logic space

s292
00:26:28.160 --> 00:26:30.720
because of tokenizer subtleties.

s293
00:26:30.720 --> 00:26:41.120
And so renderers as as an abstraction it was kind of pioneered by OpenAI's harmony with the GPT- OSS release and used prominently in thinking machines cookbooks as well for tinker,

s294
00:26:41.120 --> 00:26:43.840
but we found it was useful to just kind of make it a standalone thing.

s295
00:26:43.840 --> 00:26:47.240
And so this is just a Python library that doesn't depend on any other prime stuff.

s296
00:26:47.240 --> 00:26:59.080
You could use it with any inference engine you want just as a standalone thing that is really designed for being able to manage this token in token out concatenation

s297
00:26:59.080 --> 00:27:10.720
without thinking about it too much yourself because we kind of turn each of these chat templates for these the popular models into programmable artifacts where you can do things like

s298
00:27:10.720 --> 00:27:18.280
look up a history of secret you can use the kind of history of a trace to be able to understand like what is the right tokenization?

s299
00:27:18.280 --> 00:27:25.040
Like do I essentially have like a logical prefix hit in message space even if I don't in tokenization space after re-tokenizing?

s300
00:27:25.040 --> 00:27:32.400
And so this is the sort of thing where I think people have gone back and forth on like whether they want LM APIs to be stateful in general.

s301
00:27:32.400 --> 00:27:37.480
I think a lot of people were hoping that like we could just have every model API be stateless.

s302
00:27:37.480 --> 00:27:45.440
I think maybe people are less concerned about this now because we're moving towards this agent world where agents themselves are going to be stateful APIs.

s303
00:27:45.440 --> 00:27:53.120
But I think this has revealed to us like going through all of the the things here like why opening eye responses decided to be stateful.

s304
00:27:53.120 --> 00:28:03.240
There are some kind of like unavoidable issues that kind of come up when you're doing large-scale agentic rollouts where you you do need to kind of manage this very carefully

s305
00:28:03.240 --> 00:28:07.000
and it's kind of unavoidable just because of how tokenizers work.

s306
00:28:07.000 --> 00:28:15.040
And so you want to be able to maintain these dual streams of the logical text and the the tokens and you kind of want these to be cleanly interoperable

s307
00:28:15.040 --> 00:28:22.920
where users don't have to think about the tokens very much but the trainer gets to see everything nicely in token space as well as the inference engine.

s308
00:28:22.920 --> 00:28:29.160
Um, and so from the harness interception server we have clients that can be used both for training and inference.

s309
00:28:29.160 --> 00:28:40.880
And so you can kind of like swap between these modes without thinking about it because certain models like don't need to in in a training setting you need to be able to get log probs and set the temperature.

s310
00:28:40.880 --> 00:28:42.680
Some model APIs won't let you do this.

s311
00:28:42.680 --> 00:28:48.960
They won't return log probs because like open eye models with reasoning like won't show you the reasoning trace so there's no way they give you the log probs for everything.

s312
00:28:48.960 --> 00:29:00.600
And so like that's fine, it's just eval only and so we have this this client layer where you can go between eval and train to be able to support all these models, but we still use the interception server pattern either way because it allows us to have like

s313
00:29:00.600 --> 00:29:12.120
this notion of a dialect where like you can choose OpenAI chat completions or responses or Anthropic and all of these are kind of easily supportable as just like translation layers between a raw request

s314
00:29:12.120 --> 00:29:18.840
into something that'll get passed through a renderer potentially if you're on the train client side and formatted into a a message via tokens.

s315
00:29:18.840 --> 00:29:21.560
Um And so this brings us to Primer RL.

s316
00:29:21.560 --> 00:29:24.800
So Primer RL is our training framework that is consumes the environment.

s317
00:29:24.800 --> 00:29:34.240
So once you have an environment with your task set and your harness and your interception server and your runtime and your renderer and all those things, this plugs into what we call the orchestrator.

s318
00:29:34.240 --> 00:29:37.440
And so Primer RL has been async from the ground up.

s319
00:29:37.440 --> 00:29:45.120
Uh so I think async RL is one of those things that I think people were kind of one foot in and one foot out and a lot of training frameworks you see them

s320
00:29:45.120 --> 00:29:47.560
uh will still kind of support synchronous training.

s321
00:29:47.560 --> 00:29:50.880
Um some people I think have their reasons for wanting to do synchronous training.

s322
00:29:50.880 --> 00:29:52.440
I don't agree with them.

s323
00:29:52.440 --> 00:30:05.360
Um I think kind of want to bite the bullet of the off-policyness anyways for reasons that come up with agents um in terms of you want to be able to overlap long rollouts and not always be waiting on your slowest rollout.

s324
00:30:05.360 --> 00:30:11.200
And this kind of means you can't be fully on policy unless you want to kind of accept always waiting on your slowest rollout.

s325
00:30:11.200 --> 00:30:14.280
Um and so this is really why we went all in on async.

s326
00:30:14.280 --> 00:30:20.360
And so the orchestrator's job is to allow the inference and trainer to just be separate processes, separate servers.

s327
00:30:20.360 --> 00:30:21.880
Uh they don't share GPUs.

s328
00:30:21.880 --> 00:30:24.200
They don't really know about each other all that much.

s329
00:30:24.200 --> 00:30:26.080
They just consume from each other.

s330
00:30:26.080 --> 00:30:28.840
Um the but the orchestrator job is to really like manage the run.

s331
00:30:28.840 --> 00:30:35.040
And so the orchestrator uh will make sure that the environment is running with the endpoint mapping to the inference server.

s332
00:30:35.040 --> 00:30:36.840
It'll do rollouts.

s333
00:30:36.840 --> 00:30:39.080
Uh it'll package these up into a batch.

s334
00:30:39.080 --> 00:30:46.320
It'll send this batch to the trainer and it'll be up to the trainer to figure out what to do with the batch, uh which will be kind of printing some

s335
00:30:46.320 --> 00:30:51.000
uh sequence to feed into a loss function um based on the the specification.

s336
00:30:51.000 --> 00:30:57.600
Um and so the the server pattern we use for environments is just an engine that can like send requests to inference and like send batches back to your trainers.

s337
00:30:57.600 --> 00:30:59.040
It's very client-server.

s338
00:30:59.040 --> 00:31:04.280
Um and we found that this is just a really useful way to uh allow scaling concerns to be decoupled as well.

s339
00:31:04.280 --> 00:31:08.920
And so like for example, you can have a lot of environments running or you can have one environment running.

s340
00:31:08.920 --> 00:31:12.960
You can have um a bunch of infra replicas, you can have one infra replica.

s341
00:31:12.960 --> 00:31:16.280
Um you can have sandboxes or no sandboxes.

s342
00:31:16.280 --> 00:31:19.080
Uh and the trainer doesn't care about this, the inference doesn't care about this.

s343
00:31:19.080 --> 00:31:29.440
It's just separation of concerns at a system level uh allows you to kind of not really worry about these things as uh like combined units versus like in some cases

s344
00:31:29.440 --> 00:31:37.880
uh people will want to like have training and inference on the same stack where it's like especially the the closer that you kind of fold in your logical in one then it's like sometimes you can't even run your environments

s345
00:31:37.880 --> 00:31:40.320
if you're not doing RL, but that means then you can't really experiment.

s346
00:31:40.320 --> 00:31:41.760
You also can't use them as evals.

s347
00:31:41.760 --> 00:31:49.280
There's a lot of reasons why just like pulling everything apart into these pieces and just like having nice APIs for them to talk to each other uh makes your life way easier.

s348
00:31:49.280 --> 00:31:52.480
Um and we've also just been like really scaling it.

s349
00:31:52.480 --> 00:32:02.240
Um and so we've been doing a lot of work on uh like GLM-5 series and Kimi K2.5 2.6 series just to make sure that we can do like really good

s350
00:32:02.240 --> 00:32:03.440
large-scale RL efficiently.

s351
00:32:03.440 --> 00:32:05.760
And so some results we found we have recently.

s352
00:32:05.760 --> 00:32:10.240
This was uh the run was on GLM-5 before the 521 came out.

s353
00:32:10.240 --> 00:32:24.880
It supports 52 as well, but um on the latest primary RL version, we can do um a GLM-5 step on 28 nodes in less than 5 minutes for long horizon coding tasks with 131K context,

s354
00:32:24.880 --> 00:32:28.920
um which means you can do a 1,000-step run in 3 days.

s355
00:32:28.920 --> 00:32:32.920
And that costs for rental prices about 50K.

s356
00:32:32.920 --> 00:32:44.920
And so 50K is not cheap, but it's like if you're doing a full run on a frontier size model on like a proper real world agent environment, like it's a lot cheaper than what

s357
00:32:44.920 --> 00:32:46.240
OpenAI's raising for it.

s358
00:32:46.240 --> 00:32:54.640
Um it's a lot cheaper than like some of the clusters people are are uh selling, and it's the sort of thing that like just starts making sense for a lot more enterprises if you can like actually do this.

s359
00:32:54.640 --> 00:33:03.440
It becomes pretty justifiable if you can kind of get to the point where uh you have the the tooling chain to be able to like build ready valves and like uh do all this stuff.

s360
00:33:03.440 --> 00:33:11.280
Um and so this is like the sort of thing where it's like let's say you do you wanted to like you have a bunch of tasks that are like representative of like your coding workflows, and you want to like do a big RL run.

s361
00:33:11.280 --> 00:33:18.120
Like this is a pretty big RL run, but it's also the sort of thing that like you could people spend this much on tokens in a month sometimes.

s362
00:33:18.120 --> 00:33:27.480
Um and so you can uh start finding a lot of savings if you kind of think about doing large scale post training, and that's kind of what we're we're here to help people do.

s363
00:33:27.480 --> 00:33:37.240
Um I guess more on the async side uh one of the reasons why you really want to do async is that um there's a long tail of how long your coding agents take.

s364
00:33:37.240 --> 00:33:42.960
Like if you fire up a coding agent task, just like think your Claude coder Codex tasks, like how many minutes is it going to take you?

s365
00:33:42.960 --> 00:33:44.400
Sometimes it'll be 30 seconds.

s366
00:33:44.400 --> 00:33:45.640
Sometimes it'll be like 2 minutes.

s367
00:33:45.640 --> 00:33:49.000
Sometimes it'll be a goal that goes for like 3 hours.

s368
00:33:49.000 --> 00:33:51.240
Um and these can all be rollouts.

s369
00:33:51.240 --> 00:33:59.640
And so one of the goals of async RL is to have your like forward progress speed not be tied to the speed of your individual rollout.

s370
00:33:59.640 --> 00:34:10.120
And so this means you can kind of allow your rollouts to finish long after they start and just go into the first batch that they can accept them even if that is from a much different copy of the model.

s371
00:34:10.120 --> 00:34:15.120
Um and so then the the inference server is just always taking the latest version of the model.

s372
00:34:15.120 --> 00:34:21.560
And so um what people have generally found and what we've done with our experimentation and found as well is that you can go reasonably far off policy.

s373
00:34:21.560 --> 00:34:25.280
Like I think 16 is where we typically are often operating is like an average.

s374
00:34:25.280 --> 00:34:32.240
Um but this means that you just also can have a lot more room to not worry about certain things about speed.

s375
00:34:32.240 --> 00:34:38.720
Like you don't need to worry about the boot up time of your sandboxes as much or the weight sync time or like the time of any environment or your or your grading.

s376
00:34:38.720 --> 00:34:46.720
Like, it's it's fine if you have these things that take time cuz you can't If you're doing like grading with a judge, you can't like force your judge to be super fast all the time.

s377
00:34:46.720 --> 00:34:58.560
And so, you want to have a system where it's okay if there are pockets of your life cycle that don't use GPU time but do use time um and you can kind of overlap these without kind of wasting GPU cycles.

s378
00:34:58.560 --> 00:35:02.480
And so, that's really the the key benefit of of async RL in our eyes.

s379
00:35:02.480 --> 00:35:04.840
Um and we've done a lot of work on the loss function side.

s380
00:35:04.840 --> 00:35:12.040
The DPPO paper is one that I think has gotten popular that we've been using a lot as well of like how do you make sure that this like stuff stays stable.

s381
00:35:12.040 --> 00:35:15.840
Um we found that it's very stable up to thousands of uh steps.

s382
00:35:15.840 --> 00:35:22.440
Um we're pushing towards 10,000 in kind of current experiments um in terms of the the scale that we're trying to kind of get the stuff to reliably.

s383
00:35:22.440 --> 00:35:30.080
Um as well as doing this at big batch sizes where you kind of need to pull out all the bells and whistles on parallelism.

s384
00:35:30.080 --> 00:35:38.560
And so, what we found is that I guess I can go back to uh like doing like this uh expert parallel on the trainer as well as kind of parallel.

s385
00:35:38.560 --> 00:35:48.240
Um and then on inference doing uh wide expert parallel for the big MOEs uh meaning multi-node uh experts across multiple nodes as well as disintegrated prefill.

s386
00:35:48.240 --> 00:35:55.120
Um and you can kind of throw all the inference bells and whistles that people would do for normal serving into your RL stack and get same wins there as well.

s387
00:35:55.120 --> 00:36:00.760
Um and so, here to kind of fully kind of recap all the the advancements we've been pushing into the the stack.

s388
00:36:00.760 --> 00:36:11.880
We've been leaning in towards FP8 um wide expert YDP um disintegrated prefill lots of stuff at the routing and KV offloading and management of just kind of where things live.

s389
00:36:11.880 --> 00:36:13.080
Especially things like router re-place.

s390
00:36:13.080 --> 00:36:21.680
The router re-place is actually a really nasty systems problem because it requires tracking a lot of metadata per rollout because you have this for every single layer.

s391
00:36:21.680 --> 00:36:30.720
Um so, it's like it's a big multiple over just the like tokens and log probs uh that you actually have to store um because especially if you're multi routing to multiple experts for layer,

s392
00:36:30.720 --> 00:36:39.720
um and so the storage concerns for these, as well as like for multi-modal, um if you have like images that you need to store, like there's a lot of stuff where you want to kind of offload

s393
00:36:39.720 --> 00:36:46.320
the heavier artifacts onto some like object storage system or other file system, um and not just have it floating around in memory.

s394
00:36:46.320 --> 00:36:57.720
Um and so we've rebuilt a lot of our systems to support this, as well as just really pushing uh there's a lot of kind of like special cases that you need to worry about where it's like certain models will want certain types of context parallelism,

s395
00:36:57.720 --> 00:37:03.160
which have different considerations about what you can do uh in terms of like other aspects of the system.

s396
00:37:03.160 --> 00:37:10.400
Um and so we've just been trying to really refine the recipes for making sure that this works really well, especially for models like JLM.

s397
00:37:10.400 --> 00:37:12.480
Um and we've done this all on top of a Torch Titan base.

s398
00:37:12.480 --> 00:37:15.880
I think a lot of people are like, "Why do you use Torch Titan and not Megatron?"

s399
00:37:15.880 --> 00:37:24.400
Um and it's because Torch Titan is just like really easy to like hack, and Megatron is kind of this monolith um that I think some people will hack it, but it's I think uh

s400
00:37:24.400 --> 00:37:31.760
we just started with Torch Titan a long time ago and kind of find the pieces we want to bring in, and especially when a new model comes out it's like, "Well, you want to train on it."

s401
00:37:31.760 --> 00:37:37.600
Or you want to let's say you want to there's some new paper that you read that like has some new idea, you want to be able to like make everything really hackable and modular.

s402
00:37:37.600 --> 00:37:47.520
And so like our our team is not that big, like our whole research team that maintains Prim Rels like less than 10 people, um and the company is less than 40 people.

s403
00:37:47.520 --> 00:37:52.920
Um and so there's a lot of work that we want to make sure people can parallelize, but also move quickly.

s404
00:37:52.920 --> 00:38:01.280
Um and so uh this is all stuff that we've kind of been able to to figure out over the past few months as we've really been pushing it for scale.

s405
00:38:01.280 --> 00:38:04.240
Um another thing that's I think fun is algorithms.

s406
00:38:04.240 --> 00:38:10.000
So I think a lot of what researchers care about I think researchers some people will love thinking about the system problems of scale.

s407
00:38:10.000 --> 00:38:16.160
Um if you do talk to us, um but if you don't, I think what a lot of people want to spend their time thinking about is algorithms in terms of

s408
00:38:16.160 --> 00:38:24.880
the on-policy distillation stuff or like OP SD or uh if you saw the Echo paper, I think that got a lot of people excited of like thinking about how do you do world modeling with RL

s409
00:38:24.880 --> 00:38:27.960
um and folding these in as well as just basic stuff like SFT.

s410
00:38:27.960 --> 00:38:31.120
Um there's also this Max RL paper from a while back that was super cool.

s411
00:38:31.120 --> 00:38:34.080
And so we were just we were seeing all these papers and we were like we just want to do all of these.

s412
00:38:34.080 --> 00:38:41.400
We want it to be much easier to kind of like mix and match these and to not have to like add another if statement buried super deep in the code

s413
00:38:41.400 --> 00:38:43.560
um and like pipe a bunch of stuff all the way through.

s414
00:38:43.560 --> 00:38:52.440
And so we kind of decompose things into the loss which is like the thing that is taking the gradient as well as the algorithm which we say is the thing that's kind of like preparing the data.

s415
00:38:52.440 --> 00:38:58.600
Um and so we have different losses we can kind of like pipe things to um in terms of like the signal and the masking.

s416
00:38:58.600 --> 00:39:02.160
Uh and you can use these to kind of assemble uh different algorithms.

s417
00:39:02.160 --> 00:39:08.880
Um and so now algorithm is like a class where you have you can have a class that does different things in terms of scoring or uh the groups.

s418
00:39:08.880 --> 00:39:11.920
Um and and you'd kind of pick which loss you want to target with it.

s419
00:39:11.920 --> 00:39:19.920
Um and so we support all of these all the popular ones and you can kind of add your own that look like adding in a function to like assign advantages

s420
00:39:19.920 --> 00:39:21.120
uh to a rollout.

s421
00:39:21.120 --> 00:39:26.840
Um Uh yeah.

s422
00:39:26.840 --> 00:39:33.200
And so then then from your algorithm then from your configs you can just kind of say hey I want like this algorithm and it's just going to pick one from the registry.

s423
00:39:33.200 --> 00:39:34.800
You can add your own to the registry if you want.

s424
00:39:34.800 --> 00:39:39.080
Um and then this will be the algorithm used for your training run.

s425
00:39:39.080 --> 00:39:41.880
Um and you can do this on a per environment basis if you want.

s426
00:39:41.880 --> 00:39:51.120
Um but all of these algorithms that people are looking at kind of fall into this like table where there's questions about like what are your where are your rollouts coming from?

s427
00:39:51.120 --> 00:39:55.360
Are they coming from your current policy model or are they coming from some other source like a teacher?

s428
00:39:55.360 --> 00:40:01.960
Um and so any algorithm people call like on policy or like slightly off policy in terms of the async RL stuff.

s429
00:40:01.960 --> 00:40:06.440
This is one where like your your actor in RL sense is going to be the model you're training your policy.

s430
00:40:06.440 --> 00:40:07.880
Your policy is your actor.

s431
00:40:07.880 --> 00:40:11.920
Um in other cases you're doing stuff where your your actor is some other model.

s432
00:40:11.920 --> 00:40:25.480
So if you're doing context distillation or you're doing SFT, like these are ones where you are uh going to have some other like model or prompt potentially be the teacher that is generating the data that you're going to be training on.

s433
00:40:25.480 --> 00:40:27.040
And so all of these kind of fit within this family.

s434
00:40:27.040 --> 00:40:28.440
Then the other thing is the advantage.

s435
00:40:28.440 --> 00:40:37.720
And so the advantage is just like if you generalize it to like a score, like you can now call like just cross entropy loss or negative log likelihood is like everything has like advantage one.

s436
00:40:37.720 --> 00:40:45.480
Um there's a lot of all of the like OPD algorithms can kind of you can kind of think of it as the log probability is being the advantage.

s437
00:40:45.480 --> 00:40:52.960
And with RL like your your reward minus some baseline usually group mean or something like it is your advantage.

s438
00:40:52.960 --> 00:40:58.600
And so we just kind of like took a step back and looked at all these things and we're like, "Oh, we can just kind of like factor this all out pretty nicely."

s439
00:40:58.600 --> 00:41:07.720
And have almost everything else be shared but these things get kind of swapped and so your infra doesn't need to change just because your loss function needs to change.

s440
00:41:09.160 --> 00:41:20.880
And so for on policy distillation like it's just plugging in with a different loss target and it's like talking to the teacher and being like, "Okay, the thing I need to get is reference log probs from some teacher and I already have my like sequences.

s441
00:41:20.880 --> 00:41:27.640
So I just need to send these to a teacher as prefill in terms of like a like you can get prefilled by just like asking for like a one token response.

s442
00:41:27.640 --> 00:41:31.560
Now I I get my sequences and I can just like stick these in as my reference log probs."

s443
00:41:31.560 --> 00:41:38.600
For self distillation you can have like a hint that you're kind of putting in before your like this teacher.

s444
00:41:38.600 --> 00:41:49.040
So you're not saying the same prompt you're saying a different prompt using renders to kind of pack these together and then you get back the same like sequence that you can slice out into the original form

s445
00:41:49.040 --> 00:41:51.920
so that you can kind of add these as your reference log probs there.

s446
00:41:51.920 --> 00:41:55.200
You could do echo where you have like two different algorithm components.

s447
00:41:55.200 --> 00:42:04.200
You could have one that is targeting the uh the doing cross entropy on your environment tokens while doing like an RL objective on your your action tokens.

s448
00:42:04.200 --> 00:42:05.240
And you can mix and match these.

s449
00:42:05.240 --> 00:42:14.320
So you can have like different like you can decide which teacher is going to be done your you're going to be sampling from your student or your teacher.

s450
00:42:14.320 --> 00:42:18.800
Um you can like decide which algorithm you're going to be using on a per environment basis.

s451
00:42:18.800 --> 00:42:25.400
Um and you can have like this one we have like both um a normal like OPD as well as GRPO.

s452
00:42:25.400 --> 00:42:28.800
Um and then Ah yes.

s453
00:42:28.800 --> 00:42:32.480
So like this whole family we now support within Primerl natively.

s454
00:42:32.480 --> 00:42:41.840
Um and one thing you may also have poked around at or seen or tried is uh we have our hosted training platform uh which means you don't have to worry about GPUs at all.

s455
00:42:41.840 --> 00:42:44.280
Um And so this is just hosted Primerl.

s456
00:42:44.280 --> 00:42:52.080
The version we have that is kind of the the broad uh self-serve version today is multi-tenant Laura um which is focused mostly on RL.

s457
00:42:52.080 --> 00:43:03.640
Um what we have coming quite soon that we'll be rolling out is full fine-tuning um which supports changing as much as you want in Primerl in terms of the model and everything else where we still give you all the same

s458
00:43:03.640 --> 00:43:15.520
abstractions for uh not needing to think about the GPUs and kind of auto-scaling and uh magic restarts and uh having a dashboard where you can log everything and have unified billing for sandboxes and judges and all these things.

s459
00:43:15.520 --> 00:43:24.840
Um but also you get to develop your environments on CPU on your laptop, push them to the platform as environment packages, and specify them in your configs.

s460
00:43:24.840 --> 00:43:33.280
Um and then this allows you to um have a lot of flexibility in like deciding when you want to go to different levels of the stack.

s461
00:43:33.280 --> 00:43:37.160
So in many cases you don't actually want to change anything in the trainer.

s462
00:43:37.160 --> 00:43:38.920
You just want to change your reward function.

s463
00:43:38.920 --> 00:43:40.760
And you can do this in environment space.

s464
00:43:40.760 --> 00:43:50.280
Maybe you want to configure the trainer but not change it in which case you want to kind of poke into like slightly more complex knobs for uh things like different loss functions or learning rates.

s465
00:43:50.280 --> 00:44:00.640
Um additionally you may want to like actually go deeper into the trainer and like uh do new algorithms at the environment level at the algorithm class level or at the loss function level or maybe something else entirely that needs going even deeper.

s466
00:44:00.640 --> 00:44:04.600
And so we we support all of this with the the um the full fine-tuning as well.

s467
00:44:04.600 --> 00:44:12.040
And so uh multi-tenant Laura, if you're unfamiliar, is a a very useful pattern for allowing multiple people to do training runs on the same architecture.

s468
00:44:12.040 --> 00:44:13.800
Um on the same uh model weight copy.

s469
00:44:13.800 --> 00:44:22.560
It's so you have a base model that's kind of This is how all inference for like token-based pricing usually works as you're doing multi-tenant inference where you have one big copy of Claude

s470
00:44:22.560 --> 00:44:36.880
uh that is just getting everyone's requests and hitting a shared uh like KV pool uh that is managed um with This is really nice with Laura because you can just have multiple Lauras available that you can hot swap and each person can have their own Laura without needing to replace the base model.

s471
00:44:36.880 --> 00:44:41.160
Uh and you get one kind of inference pool that serves everybody at once even if they're using different models.

s472
00:44:41.160 --> 00:44:45.080
This allows you to do things like token-based pricing uh and not need to reserve GPUs.

s473
00:44:45.080 --> 00:44:48.840
Um For full fine-tuning, it kind of does have to be GPU-based.

s474
00:44:48.840 --> 00:44:55.160
Um but we've been getting a lot more GPUs and so we have the ability to let people run stuff on their GPUs.

s475
00:44:55.160 --> 00:45:07.200
Um and uh this allows going like pretty like going to full parameter training where especially for algorithms like large-scale SFT or mid-training, um you will want to kind of have more than just a Laura adapter

s476
00:45:07.200 --> 00:45:14.000
um as well as the ability to kind of really customize at at every layer you might want um like how your algorithm is going to work.

s477
00:45:14.000 --> 00:45:18.320
Um but so we The multi-tenant Laura one is already live and you can go use it today.

s478
00:45:18.320 --> 00:45:21.000
Um full fine-tuning is coming out in the next coming next couple weeks.

s479
00:45:21.000 --> 00:45:30.560
Um the V1 stuff from before is already uh out uh as a kind of a alpha feature that is a stable release coming the next I don't know, quite soon hopefully.

s480
00:45:30.560 --> 00:45:33.840
Um and then the cookbook I mentioned earlier is also out.

s481
00:45:33.840 --> 00:45:36.520
Um And that's mostly what I want to talk about.

s482
00:45:36.520 --> 00:45:42.640
Um bit informal and I can People have questions, I can uh dig into any individual parts people think are curious.

s483
00:45:42.640 --> 00:45:44.360
We're also uh we're hiring.

s484
00:45:44.360 --> 00:45:50.880
Uh we are small team based in mostly San Francisco um becoming a much larger team quickly.

s485
00:45:50.880 --> 00:45:58.800
Um and uh we'll have some other exciting news about that coming later this week uh to I don't know.

s486
00:45:58.800 --> 00:46:01.400
Uh demonstrate our commitment to scaling the team.

s487
00:46:01.400 --> 00:46:17.360
Um but yeah, we are we're growing and we're I think in a unique position of like we do very open research work like all this code is on GitHub if you want to go play with it but we also like our a real company that trains big models and makes money.

s488
00:46:17.360 --> 00:46:26.600
And so yeah, we think we figured out a good business model to do real open research and it's been an incredible journey thus far and would love to have

s489
00:46:26.600 --> 00:46:31.120
excited passionate talented people join the team.

s490
00:46:32.952 --> 00:46:34.952
[applause]

s491
00:46:49.982 --> 00:46:51.982
[music]