WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=k35LeKZEhiE

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/k35LeKZEhiE.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.400 --> 00:00:13.920
Yeah, thank you, Jack.

s2
00:00:13.920 --> 00:00:16.760
Um, really grateful for the opportunity to speak here.

s3
00:00:16.760 --> 00:00:27.000
Today, I'm going to be sharing some of our frontier work on post training and how we envision a future where agents can learn new skills on the job.

s4
00:00:27.440 --> 00:00:33.920
So, over the last year or so, we've seen agents develop really strong reasoning skills.

s5
00:00:33.920 --> 00:00:46.360
And they've learned to use, uh, agentic harnesses to solve longer and longer horizon tasks, which involve many turns and tool calls on complicated environment states.

s6
00:00:46.480 --> 00:01:02.760
We're seeing an increasing, uh, demand for agents that can just be deployed in a plug-and-play way into how enterprises use the agents, um, so, for instance, if they already have some method of

s7
00:01:02.760 --> 00:01:11.320
calling the agent to do a task, to do a task, they would want to be able to train a custom model to do that task instead.

s8
00:01:11.320 --> 00:01:25.000
And that requires new ways of looking at post training that allow you to, um, adapt to any harness, including ones that you don't necessarily have access to the source code of.

s9
00:01:25.760 --> 00:01:34.320
So, I wanted to talk about a few different levels of post training, where each one builds on top of the last.

s10
00:01:34.400 --> 00:01:45.680
One way that we kind of think of this, uh, is a framework comparing it to how humans do learning, where you learn simple tasks first, and you can sort of compound your understanding

s11
00:01:45.680 --> 00:01:48.040
to more and more complicated tasks.

s12
00:01:48.040 --> 00:02:02.720
So, over the last year, we've sort of, I would say, mastered or gotten a lot of reps with these simple single-turn Q&A tasks and um some longer horizon synthetic environment tasks.

s13
00:02:02.720 --> 00:02:12.560
But, what we're increasingly seeing is we want to be able to adapt to custom harnesses and be able to train directly on those instead.

s14
00:02:12.560 --> 00:02:26.000
And we kind of think of those kind of like internships where you want to the model to do a specific task, but you don't necessarily know how exactly the task will play out because you don't own the harness.

s15
00:02:26.000 --> 00:02:38.560
And then finally, I would want to share some visions we have for the future of the custom model training space, where we think that there will be these kind of agentic citizens, which you can just deploy once,

s16
00:02:38.560 --> 00:02:46.600
and they'll be able to adapt to many different types of out of distribution tasks and learn from their interactions.

s17
00:02:47.720 --> 00:02:54.160
So first, I just want to talk about the training setup for these simple Q&A tasks.

s18
00:02:54.880 --> 00:03:02.360
We have something that looks like this, where you have an orchestrator, and the orchestrator is in charge of driving the rollouts.

s19
00:03:02.360 --> 00:03:07.560
The orchestrator holds a task back, which you can think of for now is just a simple prompt and answer.

s20
00:03:07.560 --> 00:03:12.959
So, something like a math question and a corresponding numerical answer.

s21
00:03:13.040 --> 00:03:17.840
Um the orchestrator will send this prompt to a model and then get an answer back.

s22
00:03:17.840 --> 00:03:20.720
Then it will send the answer to a grader and have it be graded.

s23
00:03:20.720 --> 00:03:28.400
So, once all of this is done, we want to improve our model based on that interaction or maybe like a batch of interactions.

s24
00:03:28.400 --> 00:03:35.720
And the way we do that is through a training engine, which takes in the graded chats and produces a weight update.

s25
00:03:35.720 --> 00:03:39.880
That weight update is then synced to some inference engines.

s26
00:03:39.880 --> 00:03:52.000
And once those inference engines are updated, then we can start this entire process over again, where the orchestrator will have like new problems to um send to the model completion endpoint,

s27
00:03:52.000 --> 00:03:56.720
and then you'll be able to get more chats, grade them, and train again.

s28
00:03:56.720 --> 00:04:10.720
The key thing to note here is that the only thing you need for improving your model is the graded chats in some format, and once you have those, the training engine can compute weight updates to improve your model.

s29
00:04:11.160 --> 00:04:18.920
Uh what's important here is that the chats are in a very specific format because we're sort of constraining everything to be inside of our training stack.

s30
00:04:18.920 --> 00:04:24.760
So, in this simple setup for Q&A, you don't have anything living outside of the training stack.

s31
00:04:24.760 --> 00:04:34.080
You You basically have the code of how to run the rollout and how everything is formatted, so it's like in a very controlled environment.

s32
00:04:34.800 --> 00:04:40.560
However, this is kind of limited because we can only kind of do single-turn tasks in this way.

s33
00:04:40.560 --> 00:04:51.120
If we want to do longer and long-horizon tasks, and we want to build like higher-order skills into our models, we need to also increase the complexity of our environment.

s34
00:04:51.120 --> 00:05:00.120
So, with synthetic environments, we have a very similar setup, but we offload a lot of the environment state um outside of the training stack.

s35
00:05:00.120 --> 00:05:14.680
So, you still have the same orchestrator um from before, but um the task back is maybe a little bit more complicated, and the environment state is uh living outside of the training stack.

s36
00:05:14.680 --> 00:05:23.120
So, the task back might now include things like tool cost specs or like maybe an initial state for your environment, like a file system.

s37
00:05:23.120 --> 00:05:36.360
And the orchestrator is now in charge of running many turns in a in series, where maybe first they ask the model for how it wants to respond, and then if the model wants to call some tools, it'll then call the sandbox

s38
00:05:36.360 --> 00:05:43.320
to actually like modify the environment state or read the environment state, and then return those results back to the model.

s39
00:05:43.320 --> 00:05:51.720
After all that is said and done, you get a full task trace out of this, um, and that task trace is then sent to a grader for grading.

s40
00:05:51.720 --> 00:06:01.200
And very similar to what we had before, you'll be able to take the graded chats, you'll be able to then use them to do a weight update.

s41
00:06:01.200 --> 00:06:13.240
The main thing to highlight here is that this orchestrator and sandbox setup is replayable, which is basically just saying that for any specific prompt, you can always like roll back to the initial state and like rerun it.

s42
00:06:13.240 --> 00:06:16.600
Um, you can do that in parallel or you can do that in series.

s43
00:06:16.600 --> 00:06:32.919
But, the reason that's important is because the main sort of method that we use for reinforcement learning today is GRPO, and that involves comparing many rollouts for the same prompt, and then comparing like relatively

s44
00:06:32.919 --> 00:06:34.880
which one is better than the other.

s45
00:06:34.880 --> 00:06:46.680
And the training engine will then up like make an edit to the model to upweight the trajectories that were more successful, and then downweight the ones that were less successful.

s46
00:06:48.440 --> 00:07:03.080
So, some challenges that we face in this setup is that the environment is something that you want to basically, uh, use to replicate reality, so that after you're done training, like the improvements that you've seen actually translate to

s47
00:07:03.080 --> 00:07:05.280
when you deploy these models into production.

s48
00:07:05.280 --> 00:07:15.800
And the main sort of problem is like has kind of two names, which are both the same problem, environment fidelity and reward hacking.

s49
00:07:15.800 --> 00:07:27.520
Essentially, the agent is exposed to an environment and sort of any, uh, any quirks of your environment will end up being something that your agent may like learn a model of.

s50
00:07:27.520 --> 00:07:40.080
So, we have some examples that we've seen where in a trading run in the past, we had some like networking issues causing our environment to have tool calls that failed maybe around 10% of the time.

s51
00:07:40.080 --> 00:07:47.240
If that is the case, then we actually saw that the model would then start outputting shorter and shorter responses.

s52
00:07:47.240 --> 00:07:52.800
Now, this was really surprising to us because in our reward function, we actually didn't have any length penalty.

s53
00:07:52.800 --> 00:08:03.840
So, like we we couldn't really tell why this was happening, but really what's going on here is if you think about maybe the model is like a human like walking along a sidewalk and like

s54
00:08:03.840 --> 00:08:16.680
the tool call failures are like potholes in the sidewalk, like it makes a lot of sense that because there's so many potholes, the model doesn't want to run for that long uh because it might fall in a pothole and then get a zero reward for the rollout.

s55
00:08:16.680 --> 00:08:26.080
Um and then conversely, um it's also possible that your model just learns to like output more and more gibberish over time depending on like what your environment looks like.

s56
00:08:26.080 --> 00:08:39.400
So, in a different case, we had a training run where we um have sandbox timeouts uh just so that they don't run forever, and we usually like filter out the rollouts that timed out from being trained on.

s57
00:08:39.400 --> 00:08:53.440
One thing we saw was that if your tool calls take a long time, then if the model feels like the problem is really hard, it will actually just be incentivized to like abuse the tool calls and just like call a lot of them in quick succession

s58
00:08:53.440 --> 00:08:56.480
and try to time out the sandbox so it avoids getting a reward of zero.

s59
00:08:56.480 --> 00:09:00.080
It just gets the rollout dropped.

s60
00:09:00.160 --> 00:09:23.000
So, as we scale to like more and more complicated tasks, the the task of like replicating these environments becomes increasingly difficult because it's very, very difficult to like perfectly simulate reality and sort of any mistake that you make, even if it's not intentional, will end up inducing these like subtle undesirable behaviors in your model.

s61
00:09:23.000 --> 00:09:33.920
So, that brings us to our next topic of bring your own harness, where we're basically asking like if the agent learns the exact environment distribution, why don't we just

s62
00:09:33.920 --> 00:09:35.640
use that for our training?

s63
00:09:35.640 --> 00:09:40.720
Like just directly the real environment, you will no longer need to uh replicate anything.

s64
00:09:40.720 --> 00:09:45.000
You can just like use exactly how it's going to be used in production.

s65
00:09:45.000 --> 00:09:49.080
This solves a lot of problems and sounds really good.

s66
00:09:49.080 --> 00:09:54.600
The architecture looks something like this where we now have almost everything outside of our training stack.

s67
00:09:54.600 --> 00:10:03.120
The only thing we have left is the model completion endpoint and some way to uh record the requests and responses that go in and out of the model.

s68
00:10:03.120 --> 00:10:14.760
Everything else kind of lives outside of the training stack and can be run in whatever fashion uh that like an existing enterprise or like customer might be using.

s69
00:10:14.760 --> 00:10:24.760
So, these would be like existing enterprise harnesses and essentially the reason this is nice is because we can meet customers where they're at.

s70
00:10:24.760 --> 00:10:35.080
Like, if they're already using the model in a certain way, we can just take our like training methodology and just like plug it right in and then we can help them improve the model for like

s71
00:10:35.080 --> 00:10:37.240
exactly the way that they're using it.

s72
00:10:37.240 --> 00:10:43.520
So, all the orchestration loops and logic will now live outside of the training stack.

s73
00:10:43.520 --> 00:11:03.160
Now, this sounds really good, but um This sounds really good, but the challenge here is in The challenge here is in the like data where I as you're deploying this into

s74
00:11:03.160 --> 00:11:18.040
uh production and you have less and less control over how the rollouts uh play out, you also have like less signal to learn from because the data is not in like a familiar format.

s75
00:11:18.280 --> 00:11:22.200
And this topic is touched on in a related work by Nvidia.

s76
00:11:22.200 --> 00:11:34.560
Uh this is like a paper from around a month ago where they introduce uh Polar, which is essentially a way to think about transitioning from a harness where you are kind of in charge of micro managing

s77
00:11:34.560 --> 00:11:39.000
every aspect of the rollouts, kind of like what we were previously talking about.

s78
00:11:39.000 --> 00:11:49.280
And transitioning to some method of just uh listening in on a black box harness, and you would no longer know exactly what the logic in here is.

s79
00:11:51.480 --> 00:12:00.040
So, some challenges is that some challenges we face in this setting are non-re-playability and offline or off-policy data.

s80
00:12:00.040 --> 00:12:10.640
I think both of these are describing the same issue, which is just that because we've moved so much of the logic outside of our training stock, we just don't have any way of

s81
00:12:10.640 --> 00:12:15.120
like enforcing invariants or like data structures that we like.

s82
00:12:15.120 --> 00:12:22.880
We have to be more flexible about the way we do training, and because of that, it just becomes harder to train your model and it may create an update.

s83
00:12:22.880 --> 00:12:32.960
So, an example would be for GRPO, which is like the traditional method, you would want to have many rollouts in parallel for your task, and that may not be possible anymore.

s84
00:12:32.960 --> 00:12:51.280
If you think about um suppose like a customer chat customer support chat, and you have a record of how one of your chats went, there's not really a way that you could then go back and think, "Oh, if I like said or if I responded in this other way, like would the user have been happier?"

s85
00:12:51.280 --> 00:12:56.320
Like there's no way to then get the user's response again.

s86
00:12:56.720 --> 00:13:06.600
But, we're optimistic because like we think that humans can do this kind of learning, and so it should be possible to like formulate some kind of method that would work for models as well.

s87
00:13:06.600 --> 00:13:20.120
Like, if a human was in a customer support chat, they could understand somehow that like based on the customer's reaction, like what they said was wrong or what they said was good, and then be able to like internalize improvements

s88
00:13:20.120 --> 00:13:22.480
uh for for the future.

s89
00:13:22.480 --> 00:13:28.920
So, I want to talk about some of the frontier research directions we have um towards like solving this problem.

s90
00:13:28.920 --> 00:13:35.480
There's kind of three main topics, which are self-distillation, automated data pipelines, and qualitative feedback ingestion.

s91
00:13:35.480 --> 00:13:44.280
Self-distillation is a pretty new technique, which is still um, I would say relatively like narrowly scoped.

s92
00:13:44.280 --> 00:13:55.760
So, we've seen successes in inducing like specific new behaviors with models, but it's it's still an open uh, research question of like how general can we push it?

s93
00:13:55.760 --> 00:14:10.200
Automated data pipelines is an idea which maybe if you take like a big batch of traces, um, you would be able to like automatically like flag undesirable behaviors or failure modes, and then be able to like put together a

s94
00:14:10.200 --> 00:14:17.560
like nice batch of training data, um, automatically, and then send that to the model and help it improve.

s95
00:14:17.560 --> 00:14:30.200
Currently, this is like pretty manual or like human in the loop, where like we go through traces ourselves, and we're like looking for looking for these failure modes manually, and then like describing

s96
00:14:30.200 --> 00:14:35.320
how we can improve the model, and then um, looking curating those data sets ourselves.

s97
00:14:35.320 --> 00:14:39.200
And then finally, I I think an interesting direction is qualitative feedback ingestion.

s98
00:14:39.200 --> 00:14:47.200
So, as you move to these like production settings, sometimes you don't have access to like a clear-cut binary grade or like a numerical grade.

s99
00:14:47.200 --> 00:14:54.640
Um, often times, what you receive back is like, "Hey, for this chat, the customer had this like piece of feedback."

s100
00:14:54.640 --> 00:15:04.280
Um, if we can find a way to update our models based on that information, uh, that would also be prove extremely helpful.

s101
00:15:04.280 --> 00:15:14.520
Um, and in fact, like self-distillation is one way in which we're exploring how we can do that, but it's like a pretty um, it's still a pretty open question.

s102
00:15:16.280 --> 00:15:28.920
And then finally, I wanted to share a little bit about a vision for what the future of post-editing might look like if we sort of extrapolate out and take this sort of progression to its end.

s103
00:15:29.320 --> 00:15:39.200
I think eventually we might reach a setting where instead of just limiting ourselves to thinking about a specific task that we can improve the model on, we can actually just

s104
00:15:39.200 --> 00:15:44.600
think of the model as this one deployment that can interact in many, many different settings.

s105
00:15:44.600 --> 00:15:51.280
And like the task that you think about might just be the task of improving yourself on everything.

s106
00:15:51.280 --> 00:15:58.560
Um and this model may be used for all sorts of different tasks, maybe across different users as well.

s107
00:15:58.560 --> 00:16:11.000
And be able to sort of do some kind of reflection or introspection on like okay, for this sort of type of interaction, here's how I like self-evaluate and think that I'm doing.

s108
00:16:11.000 --> 00:16:15.280
And then for this other type of interaction, um here's how I think I'm doing.

s109
00:16:15.280 --> 00:16:25.400
And then being able to automatically take these interactions and compute weight updates from them and improve.

s110
00:16:27.440 --> 00:16:39.400
And so, yeah, going back to the point that the agent learns every nook and cranny in your environment, um the exact environmental distribution, what if like the environment was just like every interaction that the agent ever has?

s111
00:16:39.400 --> 00:16:44.800
And then in addition, we had some way that the model could evaluate itself.

s112
00:16:44.800 --> 00:17:00.839
Um basically one one like question that we've Or sorry, one challenge that we've seen is like if you're only focusing on like what improving on one task at a time or like flagging one failure mode at a time,

s113
00:17:00.839 --> 00:17:11.079
you're kind of playing a game of Whac-A-Mole where as soon as a new thing pops up, you need to scramble and like create new data or new environments and improve the model that way.

s114
00:17:11.079 --> 00:17:26.800
With this kind of like self-improving system that understands interactions from like understands every interaction from the environment you would like kind of get around this problem and you wouldn't need to worry about it anymore.

s115
00:17:28.439 --> 00:17:43.160
So I want to leave people on this quote from a paper on a year ago which I find extremely relevant now which is that AI is at the cusp of a new period in which experience will become the dominant medium of improvement

s116
00:17:43.160 --> 00:17:48.640
and ultimately dwarf the scale of human data used in today's systems.

s117
00:17:51.720 --> 00:17:52.920
And yeah.

s118
00:17:52.920 --> 00:18:00.840
If you all have any questions I'm happy to take them after and uh Yeah, you can also email me at that address.

s119
00:18:00.840 --> 00:18:02.727
Thank you.

s120
00:18:02.727 --> 00:18:04.727
[applause]

s121
00:18:17.137 --> 00:18:19.137
[music]