WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ZTA0GwpAUak

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ZTA0GwpAUak.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.400 --> 00:00:13.000
All right.

s3
00:00:13.000 --> 00:00:13.920
Can you hear me?

s4
00:00:13.920 --> 00:00:15.840
Yeah, take that as a yes.

s5
00:00:15.840 --> 00:00:16.600
Um cool.

s6
00:00:16.600 --> 00:00:18.560
So, we'll go ahead and get started here.

s7
00:00:18.560 --> 00:00:23.040
Uh today we're going to be talking about is bringing continual learning into enterprises.

s8
00:00:23.040 --> 00:00:25.120
Uh and how we're doing it at Applied Compute.

s9
00:00:25.120 --> 00:00:27.320
Uh bit of an intro, my name's Sam Denton.

s10
00:00:27.320 --> 00:00:31.120
Uh I lead the platform research team at Applied Compute.

s11
00:00:31.320 --> 00:00:33.360
So, here is our loose agenda for the day.

s12
00:00:33.360 --> 00:00:41.000
Uh we're going to start by talking about the distillation spectrum and how we think about different areas on the spectrum of distillation.

s13
00:00:41.000 --> 00:00:45.840
Then we're going to talk about where value accrues along this distillation spectrum.

s14
00:00:45.840 --> 00:00:50.160
We'll show a bunch of data on how distillation is working in enterprises.

s15
00:00:50.160 --> 00:00:58.120
Uh if I have time, I'll try and get to some tips and tricks we found at Applied Compute to making self-distillation work in continual learning work in the enterprise.

s16
00:00:58.120 --> 00:01:01.520
And then finally, we'll wrap up and talk about sort of what we've learned.

s17
00:01:01.520 --> 00:01:08.040
Um also, I'm going under the assumption that most people have some kind of context on distillation and self-distillation.

s18
00:01:08.040 --> 00:01:11.480
I figured by 3:00 p.m. on a continual learning day, you'd had enough of it.

s19
00:01:11.480 --> 00:01:15.240
So, I'm just going to dive right into it.

s20
00:01:15.440 --> 00:01:22.040
Um so, first I want to just talk about the distillation spectrum and how we think about the spectrum at Applied Compute.

s21
00:01:22.040 --> 00:01:28.440
So, uh I want to define this like offline and online distillation spectrum.

s22
00:01:28.440 --> 00:01:33.760
Um so, we'll start on sort of one end of the spectrum over here, which is offline distillation.

s23
00:01:33.760 --> 00:01:39.880
What this looks like is you get a single batch of traces from a production agent, and you're meant to just do something with it, right?

s24
00:01:39.880 --> 00:01:44.520
Like how do you learn from a bunch of production traces of some agent out in the wild?

s25
00:01:44.520 --> 00:01:49.160
Um you want to learn via hindsight, you want to learn via the mistakes the agent made.

s26
00:01:49.160 --> 00:01:58.280
Um whatever you can learn from this one-time batch of production traces, um and this is sort of the offline end of the spectrum.

s27
00:01:58.280 --> 00:02:02.360
In the middle, we have something that might look like a daily batch of production traces.

s28
00:02:02.360 --> 00:02:08.440
So, maybe you deploy a model and then at the end of every day, you collect a bunch of traces and you figure out what can I do with these traces?

s29
00:02:08.440 --> 00:02:10.440
How do I make use of these traces?

s30
00:02:10.440 --> 00:02:21.320
This is not as offline as a single sort of lump of production traces, but it's not fully online in the sense of all the way on the right here, we have this sort of unified engine

s31
00:02:21.320 --> 00:02:23.840
of putting inference and training together, right?

s32
00:02:23.840 --> 00:02:36.520
And this is sort of the holy grail of continual learning where I have a model that's serving production traffic, it does a rollout, it creates a trace, we figure out how to learn from that trace, we update the model, and then we serve the next production request.

s33
00:02:36.520 --> 00:02:45.680
So, there are a bunch of different points along this spectrum where we think that distillation and continual learning can be useful, but this is how we think about sort of the two ends of the spectrum

s34
00:02:45.680 --> 00:02:50.160
and how we try and do continual learning across the whole spectrum.

s35
00:02:50.560 --> 00:02:56.600
Um on the left side, on the offline side of traces, this is sort of where a lot of enterprises are today.

s36
00:02:56.600 --> 00:03:02.000
Basically, they say, "Okay, we have a bunch of production traffic, we have a bunch of traces, like figure out a way to make our agent better.

s37
00:03:02.000 --> 00:03:05.200
It's clearly doing something, but it clearly can be better."

s38
00:03:05.200 --> 00:03:11.760
And so, how do you make our agent better today given sort of this one-time batch of production traces?

s39
00:03:11.760 --> 00:03:15.959
And on the right side, this is sort of the full complete flywheel, right?

s40
00:03:15.959 --> 00:03:29.640
This is uh we have some enterprises who are willing to sort of start getting something into production, which looks like this fully online continual learning, where we essentially deploy a model, and then we're able to make updates as the model is serving production traffic.

s41
00:03:29.640 --> 00:03:33.840
Um our goal at Applied Compute is to meet enterprises where they are, right?

s42
00:03:33.840 --> 00:03:39.400
So, they're across this spectrum and we want to provide value across both ends of the spectrum.

s43
00:03:39.427 --> 00:03:39.880
[snorts]

s44
00:03:39.880 --> 00:03:47.920
So, this is sort of online and offline distillation, but there's a whole other axis there's a whole other axis here we think about, which is hinting, right?

s45
00:03:47.920 --> 00:03:56.720
So, the whole point of distillation is we have some kind of information that allows us to create a teacher model, which is smarter than the student model or the on-policy model.

s46
00:03:56.720 --> 00:04:04.160
Um in order to create a teacher that's smarter than this on-policy model, we need to create some kind of hint or have some kind of privileged information.

s47
00:04:04.160 --> 00:04:07.080
And so the question is where does this hint come from?

s48
00:04:07.080 --> 00:04:12.600
So in the offline hinting world, we're deriving hints from some static or offline data.

s49
00:04:12.600 --> 00:04:15.760
So this can be potentially known rubrics for a single task.

s50
00:04:15.760 --> 00:04:23.240
It can be general priors about behavior that needs to get better, such as like a customer support agent that is too willing to give refunds, for example, right?

s51
00:04:23.240 --> 00:04:26.320
This is some known behavior that you're trying to improve.

s52
00:04:26.320 --> 00:04:34.160
Um or it could just be general things that we've seen in production about sort of loss reports and saying, "Oh, the model tends to miss on questions like this."

s53
00:04:34.160 --> 00:04:36.800
But it's independent of the online model's rollout.

s54
00:04:36.800 --> 00:04:42.320
So there's a whole world of creating offline hints from static data.

s55
00:04:42.360 --> 00:04:44.640
There's also online hints, right?

s56
00:04:44.640 --> 00:04:50.840
And so online hinting is our hints that are dynamically constructed from the online rollout.

s57
00:04:50.840 --> 00:04:56.440
The idea here is that we can also inject other pieces of information like behavior priors and things like that.

s58
00:04:56.440 --> 00:05:05.560
But the goal is to create a hint that is completely dependent on the online rollout and the online policy that's doing the work.

s59
00:05:05.560 --> 00:05:08.120
So we have these two online and offline spectrums.

s60
00:05:08.120 --> 00:05:11.440
We have hinting and we have distillation.

s61
00:05:11.520 --> 00:05:14.720
Which leaves us with a very nice 2 by 2 grid, right?

s62
00:05:14.720 --> 00:05:18.800
And so here sort of like the four types of distillation that we see.

s63
00:05:18.800 --> 00:05:20.440
And again, these are all spectrums, right?

s64
00:05:20.440 --> 00:05:27.080
So I've drawn lines and put things in boxes where boxes sometimes don't make sense as boxes, but more as spectrums.

s65
00:05:27.080 --> 00:05:33.520
But in general, this is sort of like the the four quadrants of the continual learning distillation spectrum.

s66
00:05:33.520 --> 00:05:39.320
Um so in the first box, we have an offline hint paired with an offline production trace.

s67
00:05:39.320 --> 00:05:46.200
So what this looks like is we take a trace from production and we have some prior about how generally production traces aren't good enough.

s68
00:05:46.200 --> 00:05:51.960
Again, maybe customer support is not uh is too quick to give refunds, things like that.

s69
00:05:51.960 --> 00:05:57.080
And we give and we construct a hint that we apply to all of these offline production traces.

s70
00:05:57.080 --> 00:06:05.120
And then we do this distillation task, and eventually we sort of create some smarter model from offline hints paired with offline production traces.

s71
00:06:05.120 --> 00:06:10.680
In the second quadrant, we have offline hints paired with on policy and online production traces.

s72
00:06:10.680 --> 00:06:15.400
So again, what this looks like is we have a hint of a behavior we're trying to improve.

s73
00:06:15.400 --> 00:06:19.840
We let an online let an online agent do a rollout.

s74
00:06:19.840 --> 00:06:23.480
We inject some hint that we're injecting into all of our rollouts.

s75
00:06:23.480 --> 00:06:28.760
Try and update the policy model, and then continue serving production traffic.

s76
00:06:28.760 --> 00:06:37.720
In the third quadrant, we have off policy traces with some on policy step and hints that are constructed against that one on policy step.

s77
00:06:37.720 --> 00:06:47.360
Um I'll talk a little bit more about what an on policy step means later on, but basically the idea is that the trace that led to the point where I inject a hint was fully off policy.

s78
00:06:47.360 --> 00:06:51.600
It was some production some production trace that came from a few days ago.

s79
00:06:51.600 --> 00:06:58.800
And we are we are using our on policy model to just roll out one step without actually having to interact with the environment.

s80
00:06:58.800 --> 00:07:03.960
And we construct the hint based on what that on policy model did in that one step.

s81
00:07:03.960 --> 00:07:06.360
So again, we have an offline production trace.

s82
00:07:06.360 --> 00:07:15.720
We pick a moment in time to do some online step, and then we construct a hint based on what the on policy model did in that one online step.

s83
00:07:15.720 --> 00:07:21.800
And then finally, we have this fourth quadrant, which is sort of online hints with online production traces.

s84
00:07:21.800 --> 00:07:26.000
So what this means is we have an on policy model do a bunch of things in production.

s85
00:07:26.000 --> 00:07:26.880
It finishes.

s86
00:07:26.880 --> 00:07:27.520
We stop.

s87
00:07:27.520 --> 00:07:31.040
We look at what the on the online model did.

s88
00:07:31.040 --> 00:07:44.000
We then create a hint dynamically from the full rollout that the online model did, and then we construct a hint, and then we do some sort of distillation against this online production trace with an online hint.

s89
00:07:44.000 --> 00:07:55.360
So, in general at Applied Compute, we're really really Again, we do research across all four quadrants of this um sort of table, but we're really focused on quadrant one and quadrant four.

s90
00:07:55.360 --> 00:08:02.960
And so, quadrant one basically is how we meet enterprises who are who are ready to have their production agents improve today.

s91
00:08:02.960 --> 00:08:08.840
Um and so, what this looks like is we don't actually have to have replayability of a production environment, right?

s92
00:08:08.840 --> 00:08:19.040
We can take a bunch of production traces, and we just look at what happened, and then we can essentially construct We have these offline hints for behavior changes that we're trying to target and improve.

s93
00:08:19.040 --> 00:08:21.480
Again, not giving refunds quite as often.

s94
00:08:21.480 --> 00:08:27.120
Um I'll talk a little bit about formatting things or reasoning like the amount of reasoning we're trying to encourage.

s95
00:08:27.120 --> 00:08:31.360
But basically, these are off-policy traces where we inject some offline hint.

s96
00:08:31.360 --> 00:08:35.360
And this allows us to really target specific behaviors.

s97
00:08:35.880 --> 00:08:41.520
On quadrant four, um this is sort of our most scalable solution to climate overall emails.

s98
00:08:41.520 --> 00:08:46.800
Um so, this can be done with either a replayable environment or actually serving production traffic.

s99
00:08:46.800 --> 00:08:55.240
And because we're constructing these hints online in a dynamic way, that we can actually cater to a bunch of different behaviors via distillation, right?

s100
00:08:55.240 --> 00:09:07.520
Um and this is how we complete this flywheel where we have an online model serving production traffic, constructing hints online dynamically based on what it did, and then updating our model accordingly.

s101
00:09:07.520 --> 00:09:10.880
So again, um this is sort of our full training taxonomy.

s102
00:09:10.880 --> 00:09:18.760
Uh we've done work across all four, and today I'm really going to focus on quadrant one and quadrant four, which is where we spend most of our time.

s103
00:09:20.000 --> 00:09:26.720
Um in terms of how that grid maps to where value accrues, this is sort of how we think about it.

s104
00:09:26.720 --> 00:09:31.480
Um we can improve for free today, and we can raise all ceilings tomorrow.

s105
00:09:31.480 --> 00:09:35.400
We can improve for free today by using offline production traces.

s106
00:09:35.400 --> 00:09:37.200
Give us a dump of your production data.

s107
00:09:37.200 --> 00:09:39.480
We'll find a way to make it valuable.

s108
00:09:39.480 --> 00:09:49.480
And then as we deploy an online policy model that we can then raise the ceilings continuously by updating the model as it's serving production traffic.

s109
00:09:49.480 --> 00:09:58.320
Um and I think the most important thing I want to call out here is that when we think about how we do this, we want to do this without having access to some golden answer.

s110
00:09:58.320 --> 00:10:08.280
Um I think this is something that like generally frustrates me a lot in the distillation space is a lot of distillation work is done assuming you have some kind of golden answer that you can distill into the model.

s111
00:10:08.280 --> 00:10:09.840
And this is often not the case.

s112
00:10:09.840 --> 00:10:18.680
Um and so in general, we want to think about how we can do continual learning and distillation without having some beautifully golden rubric to accompany every task.

s113
00:10:18.680 --> 00:10:26.400
Um as the infra collapses between serving and training, we're automatically going to raise the ceiling continuously via online distillation.

s114
00:10:26.400 --> 00:10:38.200
So again, we have the spectrum, one-time batch of offline production data, and then online um traces, and our goal is to provide value across the full spectrum.

s115
00:10:40.040 --> 00:10:40.360
Cool.

s116
00:10:40.360 --> 00:10:45.960
So let's spend a little bit of time talking about the data and some of the results across these four quadrants.

s117
00:10:45.960 --> 00:10:50.720
Um so in the world of offline traces, offline hint, we have this setup, right?

s118
00:10:50.720 --> 00:10:59.600
Where our goal is to essentially take a Qwen 3.5 thinking model against SWE-bench and get it to submit its reasoning faster than it normally does.

s119
00:10:59.600 --> 00:11:05.480
So on SWE-bench, we found that this model was essentially taking like up to 80 turns to submit its answer.

s120
00:11:05.480 --> 00:11:11.320
What we wanted to do was encourage it to call a tool to submit its task before turn 40.

s121
00:11:11.320 --> 00:11:21.160
And the reason for that was basically to prove to ourselves that we could get it to sort of wrap up its reasoning quickly by turn 40 without letting it do its normal sort of full reasoning chain.

s122
00:11:21.160 --> 00:11:23.720
Um and so we'll we'll talk about three metrics here.

s123
00:11:23.720 --> 00:11:31.640
The first is the task complete rate, which is the percentage of the time that the agent calls this tool to {quote} finish a submission.

s124
00:11:31.640 --> 00:11:33.600
The second is test pass rate.

s125
00:11:33.600 --> 00:11:37.960
So, this is how we measure the regression in performance on sort of the base task.

s126
00:11:37.960 --> 00:11:49.600
And this is the percentage of the time that the environment passes all tests accompanying the SWE-bench task um irrespective of whether the agent, you know, submitted this submitted the task via this tool call.

s127
00:11:49.600 --> 00:11:55.240
And then finally, we have our SWE-bench task rate or pass rate, which is how we basically combine these two these two metrics.

s128
00:11:55.240 --> 00:11:57.600
It's the intersection of those two behaviors.

s129
00:11:57.600 --> 00:12:04.120
And so, the goal is we want to raise the SWE-bench pass rate performance while not degrading the test pass rate.

s130
00:12:04.120 --> 00:12:07.600
Uh and I've included a a hint here of what this looks like in practice.

s131
00:12:07.600 --> 00:12:10.160
It says something like you are near your 40-turn limit.

s132
00:12:10.160 --> 00:12:11.560
There's only about three turns left.

s133
00:12:11.560 --> 00:12:15.800
You have to keep You often keep exploring and forget to wrap up investigating.

s134
00:12:15.800 --> 00:12:22.280
So, finalize and verify your fix and then call this tool before uh you run out of time.

s135
00:12:22.280 --> 00:12:24.160
So, this is what the results look like.

s136
00:12:24.160 --> 00:12:35.080
Um so, we were able to target the specific behavioral change, which was to call this tool when it wanted to submit a task, and without any degradation to the overall performance.

s137
00:12:35.080 --> 00:12:38.600
So, you can see that sort of maroon color is the test pass rate.

s138
00:12:38.600 --> 00:12:39.680
It's relatively constant.

s139
00:12:39.680 --> 00:12:41.520
In fact, it goes up a little bit.

s140
00:12:41.520 --> 00:12:48.160
But, the task complete call rate increases dramatically from about 22% to 60%.

s141
00:12:48.160 --> 00:12:50.480
And so, we're able to add this behavior.

s142
00:12:50.480 --> 00:12:56.720
Um and I think the the really interesting surprising result here is again, this is in a fully sort of like offline world.

s143
00:12:56.720 --> 00:13:04.520
And so, we're we're taking a production trace or a trace that was created ahead of time that never basically never called this task complete tool.

s144
00:13:04.520 --> 00:13:18.160
And we're nudging with the student and uh teacher models uh the the the student model towards calling this task complete tool call without ever specifically changing the tokens for the tool call.

s145
00:13:18.160 --> 00:13:22.120
Because again, the rollout is conditioned on the {quote} "production trace," right?

s146
00:13:22.120 --> 00:13:25.880
And so, it never had the reasoning path to think to call the tool call.

s147
00:13:25.880 --> 00:13:28.320
And so, the teacher doesn't force the tool call.

s148
00:13:28.320 --> 00:13:35.640
It just starts to force the the reasoning path towards the tool call without ever actually changing the tool call.

s149
00:13:35.640 --> 00:13:38.759
Which I think was really cool and surprising to us.

s150
00:13:38.759 --> 00:13:39.200
[snorts]

s151
00:13:39.200 --> 00:13:43.480
Now, that being said, there actually is like a little bit of a cheat here that we can use.

s152
00:13:43.480 --> 00:13:51.520
Which is that you can and as I mentioned earlier, you can roll out just one step from the on policy model given an offline production trace.

s153
00:13:51.520 --> 00:14:00.240
And when we do this, we're obviously see that the student model sort of learns to wrap up its reasoning, and eventually the teacher starts encouraging it to actually call this tool token.

s154
00:14:00.240 --> 00:14:09.840
Um and so you can see by having something that's a little bit more on policy that we're able to increase sort of the sweet bench pass rate more than in in the fully offline world.

s155
00:14:09.840 --> 00:14:16.280
So again, this is sort of offline trace, offline hint with just one step on policy.

s156
00:14:17.480 --> 00:14:17.760
Cool.

s157
00:14:17.760 --> 00:14:25.800
So then what does it look like in sort of the fully online online trace, online hint world where we're serving production traffic?

s158
00:14:25.800 --> 00:14:36.920
So for a certain production use case we had, we needed to teach a coding agent to use very specific formatting for hyperlinks due to a certain hardens certain hardens nuance of one of our customers.

s159
00:14:36.920 --> 00:14:42.640
And obviously this coding agent needed to not regress on any of the base coding agent capabilities.

s160
00:14:42.640 --> 00:14:49.600
Um now the problem here is that these these hyperlink formats were very very out of distribution for previously post trained models.

s161
00:14:49.600 --> 00:14:59.320
And so when we tried things like giving in adding a reward for specific hyperlink formatting, or even doing SFT on traces where we knew the hyperlink was correctly formatted,

s162
00:14:59.320 --> 00:15:03.480
we saw that there was this sort of degradation in overall coding agent performance.

s163
00:15:03.480 --> 00:15:09.360
Uh and so what we did here is set this up as an online trace with an online hint.

s164
00:15:09.360 --> 00:15:17.720
Um so basically what this looked like is that we would do a roll out, then we would inject a hint specific to the roll out that occurred from the on policy model,

s165
00:15:17.720 --> 00:15:22.160
and then say, "In your prior roll out, you'd formatted hyperlinks like this.

s166
00:15:22.160 --> 00:15:27.600
Next time, make sure to make Next time, make sure to format hyperlinks in this way instead.

s167
00:15:27.600 --> 00:15:37.080
And so, what we were able to see is that the percentage of correct hyperlink formatting jumped drastically from about, I guess, 15% all the way up to around 80%.

s168
00:15:37.080 --> 00:15:42.600
And the other line The other graph here shows what happens if we try to do offline hinting.

s169
00:15:42.600 --> 00:15:50.120
So, this is basically for every single rollout, apply the same hint, which says, "Remember that when you do hyperlinks, you have to format it this way."

s170
00:15:50.120 --> 00:15:56.600
Um and you can see that we do climb the behavior a little bit, but far less than in this online hinting world.

s171
00:15:56.600 --> 00:15:59.000
So, here we've seen sort of like two different results.

s172
00:15:59.000 --> 00:16:04.400
One where we can use offline hinting and offline traces to climb from production traces.

s173
00:16:04.400 --> 00:16:13.520
And then another one where we're able to actually use sort of the on policyness of the model and online hints to improve the behavior when we're serving production traffic.

s174
00:16:13.720 --> 00:16:14.840
Um okay, cool.

s175
00:16:14.840 --> 00:16:18.680
I think I have enough time here to talk about a little bit of tips and tricks here.

s176
00:16:18.680 --> 00:16:25.600
Um so, the first is that we found that per step hinting is drama- is very, very important to making distillation work.

s177
00:16:25.600 --> 00:16:33.600
Um so, rather than injecting a hint to the beginning of a rollout, we use a judge to essentially decide where in the rollout we should be injecting hints.

s178
00:16:33.600 --> 00:16:42.480
And then uh actually have found that it's best to just do distillation on that next step that occurs or maybe a few steps forward rather than the entire rollout.

s179
00:16:42.480 --> 00:16:47.760
Because that's really the turn in the moment in time that you want to have the teacher teach something to the student.

s180
00:16:47.760 --> 00:16:57.720
Um I You can also see in this graph here that this KL learning signal basically goes down as you get further and further away from the hint, which which makes sense.

s181
00:16:57.920 --> 00:17:03.080
Uh Another trick that we've used is something called relevance mask self-distillation.

s182
00:17:03.080 --> 00:17:06.120
Um and there's a blog post on our website about how we've done this.

s183
00:17:06.120 --> 00:17:15.319
But essentially, the idea is that we use an LLM judge to sample and choose which tokens we actually learn from from our teacher from our teacher model.

s184
00:17:15.319 --> 00:17:24.760
Um because often, we'll see that the teacher model has preferences of certain connector words that are not really relevant to actually what we're trying to teach the student.

s185
00:17:24.760 --> 00:17:37.720
And we can see in sort of the graphs at the bottom that we're able to increase our ability to learn a very, very out of distribution behavior, while also being better about avoiding catastrophic degradation.

s186
00:17:40.040 --> 00:17:40.520
Cool.

s187
00:17:40.520 --> 00:17:42.640
So, overall, where does that leave us?

s188
00:17:42.640 --> 00:17:52.000
So, obviously, I assume everyone here is sort of on the distillation train, but it's a very, very valuable tool towards continual learning.

s189
00:17:52.000 --> 00:18:01.320
We introduced a spectrum of offline and online rollouts, as well as offline and online hinting, and how we use them towards distillation.

s190
00:18:01.320 --> 00:18:08.440
So, we use offline hinting with offline production traces to provide value on day one to enterprise clients.

s191
00:18:08.440 --> 00:18:11.760
Give us production traces and we can teach you a certain behavior.

s192
00:18:11.760 --> 00:18:22.320
We then use online hinting and online production traces to do this highest ceiling sort of continuous learning improvement across multiple improvement areas, because that judge is able to adapt

s193
00:18:22.320 --> 00:18:26.160
to whatever the online model does in production.

s194
00:18:27.000 --> 00:18:30.280
And finally, I just want to say thank you to the team that worked on this.

s195
00:18:30.280 --> 00:18:32.400
A lot of a lot of the work was done by others.

s196
00:18:32.400 --> 00:18:34.240
I just kind of got to present it.

s197
00:18:34.240 --> 00:18:38.480
Um and we're hiring, having a lot of fun working on research problems around continual learning.

s198
00:18:38.480 --> 00:18:44.400
So, if you're interested, reach out to hiring@appliedcompute.com or uh yeah, just email me as well.

s199
00:18:44.400 --> 00:18:46.960
So, thank you everyone.

s200
00:18:58.403 --> 00:19:00.403
[music]