WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=zL1kLftVTlo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/zL1kLftVTlo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.200 --> 00:00:13.840
All right.

s3
00:00:13.840 --> 00:00:14.719
Hey everyone.

s4
00:00:14.719 --> 00:00:16.720
Uh, thanks for coming by over here.

s5
00:00:16.720 --> 00:00:21.119
Hope you enjoyed the chat by part on uh, how to measure continual learning.

s6
00:00:21.119 --> 00:00:23.600
Now I'm here to talk about how we scale it up.

s7
00:00:23.600 --> 00:00:25.279
So a bit of background about me.

s8
00:00:25.279 --> 00:00:30.960
I uh went to this company called WinSurf where I was growing the research team over there.

s9
00:00:30.960 --> 00:00:39.360
Uh we trained this model called sui 1 that ended up leading to the two billion acquisition at deep mind and then I ended up giving up all the acquisition money to start trajectory

s10
00:00:39.360 --> 00:00:42.559
where we're building the platform for continual learning.

s11
00:00:42.559 --> 00:00:51.520
So uh right to dive in I think it's useful to talk a little bit about what AI progress uh has looked like for the past couple years.

s12
00:00:51.520 --> 00:01:05.280
So we've been in this mode where we've been rapidly scaling up benchmarks that have saturated within first years and then months and that has continued on for the past couple years and and will continue to grow as well.

s13
00:01:05.280 --> 00:01:09.920
The problem is it's becoming more time consuming and more expensive.

s14
00:01:09.920 --> 00:01:18.320
Uh we're seeing domains where it takes 4 hours, 6 hours, 24 hours or even several days in order to scale up benchmarks.

s15
00:01:18.320 --> 00:01:26.960
And obviously you guys have seen the massive amount of money that the labs are pouring into scaling up RL environments and data.

s16
00:01:26.960 --> 00:01:38.400
Now the thing is uh we are left with a bunch of benchmarks that are getting more time consuming uh more and more expensive and and perhaps more concerningly uh they're not

s17
00:01:38.400 --> 00:01:42.320
tied to real world use cases where people are using AI.

s18
00:01:42.320 --> 00:01:58.719
Now the uh trillion uh token problem here is that we are actually spending hundreds of trillions of tokens every single day on inference uh and we're generating great amounts of data on how models in the real world are are failing, how they're doing well.

s19
00:01:58.719 --> 00:02:02.479
And that should be signal that we should be capturing and training on.

s20
00:02:02.479 --> 00:02:05.040
Uh and this is actually how humans learn, right?

s21
00:02:05.040 --> 00:02:09.280
We're continuously updating in the real world and getting smarter every single day.

s22
00:02:09.280 --> 00:02:22.720
And uh we're seeing now the entire field talking about continual learning from AI experts like Ilia Carpathi Schol and Demis to also many industry leaders as well.

s23
00:02:22.720 --> 00:02:33.120
Satya talking about how it helps companies and there was actually a great video by Daresh a couple days ago talking a lot about the intricacies of scaling up continual learning

s24
00:02:33.120 --> 00:02:46.720
and obviously the goal is we started with more data training on internet skill pre-training scaling up to benchmarks but the real unlock moving forward will be continual learning so what's the problem why aren't we here yet

s25
00:02:46.720 --> 00:02:55.120
the main thing is uh there's several problems right now with our current algorithms so first is we have a task distribution mismatch.

s26
00:02:55.120 --> 00:03:00.800
We're scaling up these benchmarks which might not even be tied to reality of what people are training on.

s27
00:03:00.800 --> 00:03:09.519
And second, uh oftent times some of these methods aren't even sampling truly on policy with what what's happening online.

s28
00:03:09.519 --> 00:03:21.360
Third is uh we have a paradigm right now where we're sending out multiple rollouts and this requires huge amounts of infrastructure to make sure our environments are onetoone copies of the real world.

s29
00:03:21.360 --> 00:03:26.000
uh and it ends up being a bias that we're adding to our training paradigm.

s30
00:03:26.000 --> 00:03:33.599
And then finally, third is we are shoving every single reward into one scaler in order to train on when the real world is messy.

s31
00:03:33.599 --> 00:03:38.239
It's noisy and it has rich amounts of data uh that should be per token signal.

s32
00:03:38.239 --> 00:03:43.280
So we have all of these kind of broken criteria uh and we're training on it.

s33
00:03:43.280 --> 00:03:50.159
And I want to take a little bit of a step through memory lane to to see how we've been dealing with algorithms in the past.

s34
00:03:50.159 --> 00:04:00.799
Um so the first part you know we started with SFT these were the the days of instruction fine-tuning GPT 3.5 chat GBT and uh if you take a look at these four criteria

s35
00:04:00.799 --> 00:04:10.159
I mean we had parallelism solved back then it was just one roll or one uh use case or example that we needed to train on so that was a solved problem

s36
00:04:10.159 --> 00:04:21.759
and then the reward for SFT is per token which is great but we are not sampling on policy and the task distribution is just some sort of benchmark that we've cured or some sort of data set.

s37
00:04:21.759 --> 00:04:25.360
Then we moved on to the DPO, RLHF age.

s38
00:04:25.360 --> 00:04:33.440
Uh this is when Chad GBT really started taking off and we finally got online task distributions that we were able to train on.

s39
00:04:33.440 --> 00:04:40.080
Uh but sampling it was a little bit better because we were actually sampling from a model, but it was still off policy.

s40
00:04:40.080 --> 00:04:46.720
But then we ended up losing some of the uh the key infrastructure, easy infrastructure that we had with SFT.

s41
00:04:46.720 --> 00:04:49.360
Now we suddenly had pairs that we were training on.

s42
00:04:49.360 --> 00:04:53.440
Uh and then the reward went back to sequence level.

s43
00:04:53.440 --> 00:04:57.040
Then we moved on to GRPO which is kind of the mode that we're in right now.

s44
00:04:57.040 --> 00:05:15.680
Uh we basically took a Fouian bargain and wanted to max on on policy rollouts which is extremely powerful right we now have models that are capable of amazing things uh because they are on policy and able to grow and and not have all of the catastrophic forgetting problems that we've had with previous methods.

s45
00:05:15.680 --> 00:05:20.560
But on the other hand, we're working with off policy task distributions.

s46
00:05:20.560 --> 00:05:25.919
Uh our parallelism is now exploded, meaning that we need really robust environment infrastructure.

s47
00:05:25.919 --> 00:05:31.919
And then finally, for rewards, uh we're back to this paradigm of training on the entire sequence.

s48
00:05:31.919 --> 00:05:45.759
So can we get to a world where all of these are true where we have an online task distribution where sampling on policy we have one parallelism so we don't need any of this crazy infrastructure

s49
00:05:45.759 --> 00:06:01.680
and then finally our reward is actually token level well this is what I'm excited to share and and see how we can scale this up so first let's just go through what our best algorithm of post training is today hopefully you guys are all familiar with how gpo

s50
00:06:01.680 --> 00:06:02.560
works high level.

s51
00:06:02.560 --> 00:06:05.360
Uh you start off with some sort of task as an input.

s52
00:06:05.360 --> 00:06:07.039
You do a bunch of parallel rollouts.

s53
00:06:07.039 --> 00:06:09.919
So we'll call these 01 through 04.

s54
00:06:09.919 --> 00:06:12.639
And then you have some sort of endstate reward.

s55
00:06:12.639 --> 00:06:16.080
So you classify these, let's say, as a couple numbers.

s56
00:06:16.080 --> 00:06:18.319
And then GRPO works on advantage.

s57
00:06:18.319 --> 00:06:29.919
The idea that you have some sort of mean that you're calculating over and you're trying to shift the distribution of the model to be uh better on the ones that you're better on uh and then obviously move away from the ones that were worse than the mean.

s58
00:06:29.919 --> 00:06:31.840
So this is how GRPO works.

s59
00:06:31.840 --> 00:06:44.560
Now let's take a look at a new continual learning algorithm called self-distillation policy optimization or sorry first we'll we'll talk through why GRPO actually is uh still not enough.

s60
00:06:44.560 --> 00:06:51.600
So um the first part is uh if you remember the tasks so these benchmarks are obviously very different.

s61
00:06:51.600 --> 00:06:57.759
Uh it requires a bunch of parallelism and then it's almost like you're drinking through a straw in order to get the reward.

s62
00:06:57.759 --> 00:07:04.240
The way to think about it is imagine you were trying to write an essay and your teacher just gave you a score of 87 out of 100.

s63
00:07:04.240 --> 00:07:10.880
You would have to run through so many different examples to get to the idea of what a good essay is and it's very simple and efficient.

s64
00:07:10.880 --> 00:07:14.479
And so these are the fundamental problems with RL.

s65
00:07:14.560 --> 00:07:20.319
Now uh we're we're going to go through a new algorithm uh called on policy self-distillation.

s66
00:07:20.319 --> 00:07:24.639
So you guys are probably familiar with distillation in general.

s67
00:07:24.639 --> 00:07:29.759
Uh there's been obviously a lot of talk about mythos and and blocking that from happening for a lot of open source models.

s68
00:07:29.759 --> 00:07:34.560
Uh the high level here is that you start with some sort of data set.

s69
00:07:34.560 --> 00:07:40.479
Uh it's usually uh off policy and not actually tied to reality but some sort of fixed data set.

s70
00:07:40.479 --> 00:07:44.479
And what you have is a student model and then a smarter teacher model.

s71
00:07:44.479 --> 00:07:53.840
So usually a bigger model uh and you're trying to basically fit the log probs of the student to the teacher and both of these are passed in with the same data.

s72
00:07:53.840 --> 00:07:54.080
All right.

s73
00:07:54.080 --> 00:07:56.240
So this is normal distillation.

s74
00:07:56.240 --> 00:08:00.000
Now there's a new innovation called on policy distillation.

s75
00:08:00.000 --> 00:08:07.759
The only thing we're doing is actually swapping out the fixed data for instead a roll out of what the student would have rolled out.

s76
00:08:07.759 --> 00:08:08.080
All right.

s77
00:08:08.080 --> 00:08:16.800
So we take a a now roll out of the student that is our trajectory and then we're trying to fit the log props again of a smarter teacher model to the student.

s78
00:08:16.800 --> 00:08:17.599
Great.

s79
00:08:17.599 --> 00:08:23.680
Now the problem is when we're trying to push the frontier we don't magically have some smarter model, right?

s80
00:08:23.680 --> 00:08:25.280
uh like a teacher.

s81
00:08:25.280 --> 00:08:28.879
And so now what do we do if we're we're already at the smartest model?

s82
00:08:28.879 --> 00:08:30.879
Well, here's the final algorithm.

s83
00:08:30.879 --> 00:08:34.560
Uh and this is where the self distillation part comes in.

s84
00:08:34.560 --> 00:08:44.240
If you basically take the the student model and give it some sort of what we call privileged information, a hint about the world and put that into the prompt.

s85
00:08:44.240 --> 00:08:47.839
Well, suddenly that student is a little bit smarter.

s86
00:08:47.839 --> 00:08:51.040
And that's essentially the key idea of on policy self-distillation.

s87
00:08:51.040 --> 00:08:59.440
You take what's called this hint, put it into the beginning of the prompt, and now you match the log props of the student without that hint to the teacher with that hint.

s88
00:08:59.440 --> 00:09:02.080
And this is an extremely powerful algorithm.

s89
00:09:02.080 --> 00:09:06.160
To visualize it a little bit more, let's say you have this student prompt, right?

s90
00:09:06.160 --> 00:09:08.880
Uh find the derivative of this function.

s91
00:09:08.880 --> 00:09:16.000
What a hint would be is if you had some sort of environment information or or some sort of guidance on how you should actually solve the problem.

s92
00:09:16.000 --> 00:09:20.640
The simplest form of this is is literally just a example of a golden solution.

s93
00:09:20.640 --> 00:09:25.839
You put that into the teacher prompt and then say here is guidance on how you would solve the problem.

s94
00:09:25.839 --> 00:09:35.920
And you can imagine now that solving that problem from the teacher's perspective becomes a little bit easier and we're we're trying to shift to those log problems had it known the answer in the first place.

s95
00:09:35.920 --> 00:09:41.279
Um and so by doing this we've solved several key problems that existed with RL.

s96
00:09:41.279 --> 00:09:51.360
Uh so if you remember uh the task distribution now we can suddenly take something that is truly from online without needing to have sort of benchmarks that are created.

s97
00:09:51.360 --> 00:09:54.880
Uh the second part is we're still on policy sampling which is great.

s98
00:09:54.880 --> 00:09:57.200
Uh but now there's no parallel rollouts.

s99
00:09:57.200 --> 00:10:02.320
We don't need a group of eight in order to roll out but just from a single example we're able to get information.

s100
00:10:02.320 --> 00:10:06.160
So that takes away the environment bottleneck and all of these other infrastructures.

s101
00:10:06.160 --> 00:10:10.560
And then finally the most exciting part is we're matching every single log problem of every token.

s102
00:10:10.560 --> 00:10:14.880
So there's massively rich feedback about what this algorithm is doing.

s103
00:10:14.880 --> 00:10:17.120
Uh to see it a little more in action.

s104
00:10:17.120 --> 00:10:28.000
And this is a really exciting part of self-distillation is it's actually not just the top token that you sampled uh that you're actually making better, but instead the entire vocabulary.

s105
00:10:28.000 --> 00:10:34.079
So for every single token, there's a vocabulary of let's say 65k tokens that you're optimizing over.

s106
00:10:34.079 --> 00:10:39.920
And let's say the model generated some sort of uh for loop or range in Python.

s107
00:10:39.920 --> 00:10:52.560
And essentially what we're doing is saying, hey, the teacher for some of these areas where it's blue, it wasn't the top sample token of the student, but instead we're pushing that distribution to sample a brand new token.

s108
00:10:52.560 --> 00:11:02.480
And this is really exciting because we're not just taking now a distribution like RL and slightly sharpening it, but we're instead actually shifting entire distributions.

s109
00:11:02.480 --> 00:11:05.920
So this is a really exciting part about opsd.

s110
00:11:05.920 --> 00:11:11.519
Now we might be saying uh you know opsd is is awesome and and we've solved this problem.

s111
00:11:11.519 --> 00:11:14.160
So I mean for short horizon tasks it works incredibly well.

s112
00:11:14.160 --> 00:11:28.160
If you take a look at live codebench uh actually we found that gpo saturates around sonnet level performance and and doesn't really push the frontier but because we're actually shifting distributions we're able to get to brand new territory

s113
00:11:28.160 --> 00:11:30.800
of results with a lot of data sets.

s114
00:11:30.800 --> 00:11:43.279
And then the really cool part is um with RL a fundamental limitation and if you guys have ever trained RL models is the models like to think a lot right the more tokens that you expend it's it's just going to do better

s115
00:11:43.279 --> 00:11:53.920
but with opsd you don't have that problem and so the actual tokens to solve some of these really difficult challenges actually collapses which is really exciting for token efficiency

s116
00:11:54.240 --> 00:12:04.079
and uh for some of these short horizon tasks you can actually plugandplay opsd right now with several open source projects So uh open claw is a really good example of this open claw rl

s117
00:12:04.079 --> 00:12:11.920
uh and you can actually use it to do simple chat bots learn from your behaviors with unstructured data uh which is super super awesome.

s118
00:12:11.920 --> 00:12:17.040
All right so we might be thinking okay continual learning is solved we can all go home and and be super happy.

s119
00:12:17.040 --> 00:12:23.920
Uh the thing is this works really well for small models short horizon tasks like something like a chatbot.

s120
00:12:23.920 --> 00:12:32.399
Uh but this is where academic papers kind of end uh and where you really need to scale things up to start to to see the limitations.

s121
00:12:32.399 --> 00:12:40.720
Uh so at trajectory we've been scaling up this algorithm to 120bs to 500 bs to one trillion parameter models.

s122
00:12:40.720 --> 00:12:50.480
And as soon as you get to the 120B range with not just one or two tool calls but 50 or 100 well things start to break apart a little bit.

s123
00:12:50.480 --> 00:13:02.399
So uh first of all eval accuracy is all over the place right the range is going really high uh run-to-run variance is extremely high and then also we start to see a lot of tool call errors

s124
00:13:02.399 --> 00:13:15.279
uh the model is not behaving accordingly to the format that it was trained on in the first place with the instruction fine-tuning all right so what are the problems that we started to see with this algorithm as we were scaling it up

s125
00:13:15.279 --> 00:13:20.240
so the first part is actually really funny and what I call it the buttweight problem.

s126
00:13:20.240 --> 00:13:23.519
So on shorter tasks, you're fitting to a distribution.

s127
00:13:23.519 --> 00:13:24.959
Everything's great.

s128
00:13:24.959 --> 00:13:30.720
But when you move on to longer tasks, what you get is is basically this student model.

s129
00:13:30.720 --> 00:13:34.000
It's going off and doing whatever it thinks is on policy.

s130
00:13:34.000 --> 00:13:42.000
And then at some point, because you're so divergent in a long task, the teacher is going to try to course correct every single time it gets a chance.

s131
00:13:42.000 --> 00:13:52.240
And so what you end up with is the teacher model just uh continuously trying to improve this token of wait or uh maybe or some of these kind of hedging words.

s132
00:13:52.240 --> 00:14:05.600
And there's this really interesting wordle in the the uh bottom left that you can see or wordcloud uh that as steps go on with opsd and you really scale it up uh you start to see some of these words like wait and then but

s133
00:14:05.600 --> 00:14:06.959
uh start to appear.

s134
00:14:06.959 --> 00:14:14.160
And then you actually end up in this really interesting local suboptimal position where everything just turns into maybe.

s135
00:14:14.160 --> 00:14:17.440
And on the right here you can kind of see some visualization of this.

s136
00:14:17.440 --> 00:14:21.360
As the task goes on you get two different distributions that are really divergent.

s137
00:14:21.360 --> 00:14:25.199
And what you end up with is the model trying to be in the middle of both of those.

s138
00:14:25.199 --> 00:14:27.760
Uh which is obviously really suboptimal.

s139
00:14:27.760 --> 00:14:28.079
All right.

s140
00:14:28.079 --> 00:14:30.320
So there's a couple ways to solve this.

s141
00:14:30.320 --> 00:14:35.920
Uh one potential solution is to define step level divergence.

s142
00:14:35.920 --> 00:14:40.560
So uh in a tool calling trajectory, let's say you have a 100 tool calls going on.

s143
00:14:40.560 --> 00:14:48.240
Well, what we can do is start to look at the KL divergence of the student model and the teacher model as time goes on.

s144
00:14:48.240 --> 00:14:51.920
And now we can use this as a waiting factor.

s145
00:14:51.920 --> 00:15:03.839
So not just like normal KL where we use that as a KL penalty, but instead we're actually multiplying the token weight uh of every single step based on this divergence property.

s146
00:15:03.839 --> 00:15:12.800
And now the cool part of this uh is that so on the left side we have a normal trajectory that is like pretty in distribution and everything's fine, right?

s147
00:15:12.800 --> 00:15:14.959
It's everything is a weight of one.

s148
00:15:14.959 --> 00:15:19.680
On the second part, we have a trajectory that diverges pretty heavily.

s149
00:15:19.680 --> 00:15:24.560
And so in here, we're only going to modify W1 as the first step.

s150
00:15:24.560 --> 00:15:28.800
Uh and and so that we can train that, get that right, and then move on, which is awesome.

s151
00:15:28.800 --> 00:15:37.680
And then finally, on the third part, and this is really cool, by having independent waiting for every step is we can actually have a scenario where uh the model might go off tracks.

s152
00:15:37.680 --> 00:15:43.839
We don't want to heavily weight that in, but then later on in the trajectory, it might get back on track again.

s153
00:15:43.839 --> 00:15:48.079
and and we're fine with that and we'll mildly shift the distribution.

s154
00:15:48.079 --> 00:15:53.040
So this is one way that we've been able to overcome this for a long horizon tool calling.

s155
00:15:53.040 --> 00:15:57.519
The second part and this is actually really nefarious with uh ops.

s156
00:15:57.519 --> 00:16:06.639
So with RL the number one problem that people face is reward hacking as you guys are are all aware of uh and that's a game that continuously RL researchers have had to play.

s157
00:16:06.639 --> 00:16:11.279
Well there is an equivalent for OPSD as well and that is hint leakage.

s158
00:16:11.279 --> 00:16:13.920
The way you can think about it is we're we're taking this hint, right?

s159
00:16:13.920 --> 00:16:16.240
Putting it in the the beginning of a prompt.

s160
00:16:16.240 --> 00:16:25.440
Well, if the student had no way of knowing what that hint would be, uh then you're going to end up with some weird scenarios and and kind of skipping some steps along the way.

s161
00:16:25.440 --> 00:16:33.759
So, the way this manifests u is let's take this example here where you're trying to find the last three digits of of this formula.

s162
00:16:33.759 --> 00:16:43.600
uh and a normal hint might be uh actually giving the correct steps and then maybe a final answer and saying, "Hey, actually hint, the last three digits are all zeros."

s163
00:16:43.600 --> 00:16:51.920
Well, you can see on the right here that when you actually roll out the model, uh you end up with something that's says, "Oh, actually, I know what the solution is.

s164
00:16:51.920 --> 00:16:57.440
It's 00 0. So, let me go back and put that into my reasoning trace and then figure out what's going on."

s165
00:16:57.440 --> 00:17:00.480
Well, you can imagine that this is not going to occur whatsoever in the real world.

s166
00:17:00.480 --> 00:17:03.120
And so we've ended up in this really strange position.

s167
00:17:03.120 --> 00:17:10.640
Uh and so there's a lot of care that you have to put into how you design these hints and making sure there's not leakage of information.

s168
00:17:10.640 --> 00:17:18.079
So uh there's one kind of trivial solution to this that you can imagine and this is just literally using an LLM to filter out these hints.

s169
00:17:18.079 --> 00:17:22.799
So an example of this is uh you know let's say a user can't log in.

s170
00:17:22.799 --> 00:17:32.000
you have some sort of problem like this uh and there might be some sort of information like exactly the solution right you'll find their SSO token that is expired

s171
00:17:32.000 --> 00:17:43.360
and you can have an LLM basically translate that into what is something reasonable that they should have known and that's the process of looking through the logs but not actually giving it the solution that it would shortcut some of its vital reasoning.

s172
00:17:43.360 --> 00:17:50.880
So this is one trivial uh solution and it works decently well but there's some more satisfying algorithmic approaches too.

s173
00:17:50.880 --> 00:17:53.679
One of these is called residual guidance.

s174
00:17:53.679 --> 00:18:01.760
The general idea is in a hint uh most of the time you need to actually get through the entire hint in order to get the full information.

s175
00:18:01.760 --> 00:18:14.480
So what if you let's say cut it in half then you have a partial hint and then you basically say okay this is the partial teacher we're able to get the log props of a slightly smart teacher

s176
00:18:14.480 --> 00:18:17.679
and then we have the full teacher as well and that's with the full hint.

s177
00:18:17.679 --> 00:18:27.600
Now what we can do is actually take the linear combination of both of these and this gives us a good idea of how strong the hint is and how out of distribution

s178
00:18:27.600 --> 00:18:30.000
it is for the original model.

s179
00:18:30.000 --> 00:18:40.880
And what you end up with here is so on the left you have normal OPD uh where you might be entirely shifting distributions and there's almost no overlap between your model and the original solution.

s180
00:18:40.880 --> 00:18:52.880
uh to this kind of cool world where uh let's say half the hint is actually quite close to your distribution but the full hint is very off and you take a linear combination and so then you're not shifting the model into unknown territory.

s181
00:18:52.880 --> 00:18:59.679
So these are just some of the solutions to uh a lot of the challenges that we've had with scaling up opsd.

s182
00:18:59.679 --> 00:19:11.919
Uh but it is a very very powerful algorithm and uh with a lot of these combinations together we're actually able to scale this up to uh a 12b model on mercur apex agents uh which often requires

s183
00:19:11.919 --> 00:19:20.240
100 and or plus tool calls in order to achieve and it's a really powerful algorithm that has even surpassed RL as well.

s184
00:19:20.559 --> 00:19:23.679
So now we've finally arrived at a algorithm.

s185
00:19:23.679 --> 00:19:33.280
It's not necessarily the algorithm to solve continual learning, but uh definitely one that is a huge step forward that keeps the on policy nature that makes RL so powerful.

s186
00:19:33.280 --> 00:19:37.760
But then it also has uh finally the task distribution that is online.

s187
00:19:37.760 --> 00:19:40.640
Uh it has parallelism uh that is singular.

s188
00:19:40.640 --> 00:19:42.400
So we don't need all of this infrastructure.

s189
00:19:42.400 --> 00:19:46.240
And then finally it is per token dense reward.

s190
00:19:46.400 --> 00:19:56.880
So the uh the really exciting part for us and and what we're really focused on now trajectory is uh this just gives you one taste of the entire continual learning loop

s191
00:19:56.880 --> 00:20:08.880
and there is a really exciting world that is about to come where software in general just gets smarter every single time it's used and that is the most exciting unlock that's going to happen in 2026

s192
00:20:08.880 --> 00:20:11.039
2027 and as we scale up.

s193
00:20:11.039 --> 00:20:13.120
So a little bit about trajectory.

s194
00:20:13.120 --> 00:20:23.440
We are uh building the platform that turns every interaction into model improvement, harness improvement, uh and the entire agentic loop just getting smarter over time.

s195
00:20:23.440 --> 00:20:28.960
Um and we're we're building this platform where we take in agent traces data from production.

s196
00:20:28.960 --> 00:20:36.000
We're able to optimize that uh as a self-s served loop and then deploy that uh as a continually learning system.

s197
00:20:36.000 --> 00:20:39.280
And uh we have a control plane that goes over all of that.

s198
00:20:39.280 --> 00:20:41.440
Um, so very quickly, our team is super awesome.

s199
00:20:41.440 --> 00:20:46.960
We're from DeepMind, Meta Super Intelligence, OpenAI, uh, and a lot of great product builders as well.

s200
00:20:46.960 --> 00:20:50.480
Um, and we we've given early access to a lot of companies.

s201
00:20:50.480 --> 00:20:54.880
So, Harvey, Dakagon, Rogo, um, and they're super excited about what we're doing.

s202
00:20:54.880 --> 00:21:01.760
Uh, if you're interested in any of the research that we're doing, uh, or any of the product that we're building, uh, definitely let me know.

s203
00:21:01.760 --> 00:21:05.840
Keep in touch and, uh, happy to answer any questions.

s204
00:21:06.784 --> 00:21:08.784
[applause]

s205
00:21:16.159 --> 00:21:24.799
Is it in the sense of like very little latency so that you can learn time or is this something that

s206
00:21:24.799 --> 00:21:26.159
Yeah, that's a great question.

s207
00:21:26.159 --> 00:21:34.640
Uh so what I would say is as a research community right now we're in this zone of what I call pseudo continual learning.

s208
00:21:34.640 --> 00:21:40.799
uh where there's some still level of like batch updates offline uh and then re-uploading the model.

s209
00:21:40.799 --> 00:21:42.880
I think it's partly an infrastructure question.

s210
00:21:42.880 --> 00:21:51.760
It's partly still an algorithmic question as well of how do you truly get when you have 10,000 rollouts going out in a product merging those together the infrastructure to pull all of those together.

s211
00:21:51.760 --> 00:21:54.559
So those are some of the problems that we are solving at trajectory.

s212
00:21:54.559 --> 00:21:59.679
Uh but I wouldn't say we're anywhere close to the the end solution.

s213
00:22:01.120 --> 00:22:04.640
Question about hardness.

s214
00:22:04.640 --> 00:22:11.280
studies in terms of effectiveness.

s215
00:22:11.520 --> 00:22:12.000
Totally.

s216
00:22:12.000 --> 00:22:18.960
I I think that's actually one of the most underexplored and most exciting questions is not only just harness improvement, right?

s217
00:22:18.960 --> 00:22:22.400
And I think there's some literature out there now starting to explore that.

s218
00:22:22.400 --> 00:22:27.760
But the really exciting part is how does the model and the harness interplay with each other as you're both updating them.

s219
00:22:27.760 --> 00:22:32.240
That's some of the stuff that we're now exploring with our current customers and and really doing those things online.

s220
00:22:32.240 --> 00:22:38.480
Uh, but it's completely unexplored territory and there's some really exciting innovations to be made there as well.

s221
00:22:41.280 --> 00:22:42.000
Cool.

s222
00:22:42.000 --> 00:22:42.320
All right.

s223
00:22:42.320 --> 00:22:44.799
Thanks so much, guys.