WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=WiqDvX6isc4

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/WiqDvX6isc4.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.000 --> 00:00:13.840
All right.

s3
00:00:13.840 --> 00:00:15.160
Hi, everybody.

s4
00:00:15.160 --> 00:00:16.400
Uh my name's Jack.

s5
00:00:16.400 --> 00:00:23.800
I'm here to talk about scaling compute on context and also our startup N gram, which launched last week.

s6
00:00:23.800 --> 00:00:33.120
Um more This isn't going to be like a super detail-oriented talk where I go through a lot of experiments we've been running or talk too much about what our models do.

s7
00:00:33.120 --> 00:00:39.600
I just want to frame like the high-level problem of what we call scaling compute on context.

s8
00:00:39.600 --> 00:00:41.400
People have many names for this.

s9
00:00:41.400 --> 00:00:50.160
It's maybe like a a sub problem of continual learning or maybe even just the the answer that we see to the problem.

s10
00:00:50.160 --> 00:00:53.040
Um I guess a little bit about myself at first.

s11
00:00:53.040 --> 00:00:54.320
I'm Jack.

s12
00:00:54.320 --> 00:00:55.400
I'm a researcher.

s13
00:00:55.400 --> 00:00:57.520
I'm uh part of the startup N gram.

s14
00:00:57.520 --> 00:00:59.760
You can see me on the left in this picture.

s15
00:00:59.760 --> 00:01:03.160
We just launched last week with this picture on Twitter.

s16
00:01:03.160 --> 00:01:07.360
So, if you want to look us up and see more about what we do afterwards, please feel free.

s17
00:01:07.360 --> 00:01:13.160
I'm also happy to talk after the uh talk concludes.

s18
00:01:13.160 --> 00:01:17.600
Um so, I'm going to start with this question about breadth versus depth.

s19
00:01:17.600 --> 00:01:18.840
This is Terence Tao.

s20
00:01:18.840 --> 00:01:22.400
He's likely the world's most famous mathematician.

s21
00:01:22.400 --> 00:01:27.800
And he's also a really heavy user of AI, even like an advocate for AI in math.

s22
00:01:27.800 --> 00:01:42.960
And one of the things that you'll hear him talk about is how AI knows like every single public mathematical topic and it can uh make connections between things that you wouldn't expect and help sort of like

s23
00:01:42.960 --> 00:01:49.320
uh bridge gaps in the literature that in in a way that no human even can know because it's read so much.

s24
00:01:49.320 --> 00:01:59.840
But it maybe lacks the depth that you would look for from like, for example, a graduate student who spent 5 years practicing in in one area that gets this like almost

s25
00:01:59.840 --> 00:02:04.920
like subconscious intuition for for the problem space.

s26
00:02:04.920 --> 00:02:15.640
And so I think I think about what we're doing which is like training your data into models at at a high level as just sort of like learning facts and skills like traditional AI,

s27
00:02:15.640 --> 00:02:27.120
but at a low level it's it's about depth and like the thing that we're after like this idea of scaling compute on context is the pursuit of depth in in AI.

s28
00:02:27.120 --> 00:02:31.720
Um I can give you some examples of why I think our approach is important.

s29
00:02:31.720 --> 00:02:36.240
It's related to like continual learning in general like as other people propose it.

s30
00:02:36.240 --> 00:02:40.640
So like one thing that models don't have is knowledge of what happens after they're trained.

s31
00:02:40.640 --> 00:02:52.720
So even you know Fable whatever five that's coming out today was probably pre-trained with a cutoff I would guess at least a month ago and has no idea if Mexico won their game last night.

s32
00:02:52.720 --> 00:03:01.120
Actually I don't know I meant to check before the talk, but uh it's nothing to do with my pre-training cutoff because I'm a human I'm capable of acquiring new information.

s33
00:03:01.120 --> 00:03:11.120
Uh another thing that models are bad at is just sort of like hard difficult long-tail skill and knowledge acquisition that doesn't appear a lot in the training data.

s34
00:03:11.120 --> 00:03:14.239
So like models still are quite bad at writing AMD kernels.

s35
00:03:14.239 --> 00:03:18.440
There are not that many good kernels written on AMD GPUs that are public.

s36
00:03:18.440 --> 00:03:24.239
And they're intended to acquire this knowledge through pre-training, but they don't because it doesn't occur very often.

s37
00:03:24.239 --> 00:03:31.720
Um and then I think maybe the most pressing case for you is like why can't I have a model like ChatGPT that knows all of my emails or knows

s38
00:03:31.720 --> 00:03:38.880
the way that I like to write things or knows like you know where my family and I like to vacation every year and all these little details of your life.

s39
00:03:38.880 --> 00:03:44.600
And there's a pretty basic reason for this which is that ChatGPT and models like it are trained on public data.

s40
00:03:44.600 --> 00:03:50.920
So um they don't know things like about your partnership with another uh company like SunTrust Bank.

s41
00:03:50.920 --> 00:03:57.680
Uh they don't know really anything about you unless you happen to be famous enough to appear in the pre-training data.

s42
00:03:57.680 --> 00:04:02.760
So, I I feel this is not just like a intellectual or like academic problem.

s43
00:04:02.760 --> 00:04:11.080
It's like the core problem with the current paradigm in AI that models cannot acquire new knowledge after training in in a personalized way.

s44
00:04:11.080 --> 00:04:21.600
So, by definition, models have to be trained on data that's sort of open to the public and they can't learn the depth of like the things that you know.

s45
00:04:21.600 --> 00:04:24.800
So, just going back to Terrence Tao again, like how do we change this?

s46
00:04:24.800 --> 00:04:33.800
How do we teach new things to models in a way that lets them acquire uh this kind of expertise or like really deep skill set that we're looking for.

s47
00:04:33.800 --> 00:04:35.520
And I'll say there's a lot of names for this.

s48
00:04:35.520 --> 00:04:43.720
Like people call it um sleep time compute, continual learning, neural memory, write time compute, note taking, dreaming, studying, machine studying.

s49
00:04:43.720 --> 00:04:47.040
In classical AI, maybe it's called amortized inference.

s50
00:04:47.040 --> 00:04:49.560
And I think um I'm calling it scaling compute on context.

s51
00:04:49.560 --> 00:04:54.640
And it's almost all describing the same thing, which is like something people really want.

s52
00:04:54.640 --> 00:05:05.960
But I think the reason why it doesn't have like even a set agreed upon name is because the paradigm is like very early and hasn't been solidified the way, for example, pre-training or or post-training have.

s53
00:05:05.960 --> 00:05:10.360
Um so, maybe I'll take a second and and talk about scaling.

s54
00:05:10.360 --> 00:05:14.200
Um there's basically three axes that we use to scale AI models.

s55
00:05:14.200 --> 00:05:16.120
We can train them on more data.

s56
00:05:16.120 --> 00:05:18.560
We can train them for longer or add compute.

s57
00:05:18.560 --> 00:05:24.320
Or we can make the models themselves bigger, like give them more capacity to acquire new information.

s58
00:05:24.320 --> 00:05:33.400
And this is like the main driver of progress from the last really the entirety of the deep learning revolution comes from these three axes of scaling.

s59
00:05:33.400 --> 00:05:36.919
And the the results are extremely compelling.

s60
00:05:36.919 --> 00:05:40.960
Um but they're still limited to like these public data sources.

s61
00:05:40.960 --> 00:05:42.560
Models are really good at Wikipedia.

s62
00:05:42.560 --> 00:05:46.440
They know everything about Reddit, papers on archive, code on GitHub.

s63
00:05:46.440 --> 00:05:56.120
And now they have this new layer of post training data that's like experts that are hired through data acquisition companies like Scale AI, Surge AI, and Mercor.

s64
00:05:56.120 --> 00:06:02.600
But, they're still by definition creating publicly available data because it's something that the model could tell to a user.

s65
00:06:02.600 --> 00:06:07.640
So, scaling is basically only used on on public data, and yet it's so powerful.

s66
00:06:07.640 --> 00:06:15.920
So, I think I'm trying to go a bit faster, so I'm not going to dwell on this, but this is the plot from Meter about how um models get

s67
00:06:15.920 --> 00:06:20.560
better, you know, every month and can complete tasks that take a longer time.

s68
00:06:20.560 --> 00:06:23.160
And this is purely an artifact of scaling.

s69
00:06:23.160 --> 00:06:30.720
Um I think the core question that I want to talk to you about today is how do we apply scale to your data?

s70
00:06:30.720 --> 00:06:39.160
Um I think like scale is clearly the thing that drives progress, you know, it's not necessarily new algorithms or like great new ideas.

s71
00:06:39.160 --> 00:06:51.600
I think maybe there's an element of data that's important, but really the thing that makes like, you know, the new the new generation of models like Fable and GBT, whatever that's coming out next month, so good is that they they basically scale along all three axes.

s72
00:06:51.600 --> 00:06:55.600
I'm sure they have new data, they're certainly training for longer, and they make the models bigger.

s73
00:06:55.600 --> 00:06:58.960
And this is how models keep getting better and will continue to get better.

s74
00:06:58.960 --> 00:07:02.560
But, I think the missing element is that this is always on public data.

s75
00:07:02.560 --> 00:07:11.280
So, models are getting better at, you know, coding in the way that is public on GitHub, they're getting better at doing math in ways that are written in public textbooks,

s76
00:07:11.280 --> 00:07:15.440
but they're not getting more knowledge of you or your life or your work.

s77
00:07:15.440 --> 00:07:17.880
And that's what we're trying to change here.

s78
00:07:17.880 --> 00:07:24.880
So, if we approach the problem from first principles, um I think the core limitation is that you have a fixed data budget.

s79
00:07:24.880 --> 00:07:33.760
So, like say I want to scale in some fashion to train a model that knows the data from Ngram better, like our our company.

s80
00:07:33.760 --> 00:07:36.040
Um we can't create new data.

s81
00:07:36.040 --> 00:07:40.320
So, like the kind of data scaling axis is out the window.

s82
00:07:40.320 --> 00:07:45.960
Um I think we also probably agree that we can't train a model from scratch on our data.

s83
00:07:45.960 --> 00:07:49.880
So, we probably want to start from a pre-trained model.

s84
00:07:49.880 --> 00:08:01.480
Or maybe another way to look at it as is there's a ton of information about the outside world that is useful for understanding what happens within our company or like your own context of choice.

s85
00:08:01.480 --> 00:08:04.680
So, you very likely want to start from a pre-trained model.

s86
00:08:04.680 --> 00:08:08.480
Um, this leaves us with essentially one axis of scaling, which is compute.

s87
00:08:08.480 --> 00:08:13.040
And this brings us to the title of the talk today, which is scaling compute on context.

s88
00:08:13.040 --> 00:08:21.240
Um, just like a small tangent while I have you is that I think one thing that's been beneficial for us to realize is that the amount of data isn't really fixed.

s89
00:08:21.240 --> 00:08:24.800
There are a lot of ways that you can get more data afterwards.

s90
00:08:24.800 --> 00:08:32.039
Like maybe in the pure like math problem that that I'll propose, it's you know, you have this fixed data set and you want to train it into a model.

s91
00:08:32.039 --> 00:08:42.360
But really if you're like studying a textbook or trying to learn a new language, uh, it's not really like you're limited to the the words of the textbook itself.

s92
00:08:42.360 --> 00:08:43.800
There's like a lot of stuff you can do.

s93
00:08:43.800 --> 00:08:49.000
Like you can find other textbooks, you can go on the internet and search for related things.

s94
00:08:49.000 --> 00:08:54.800
You can even be proactive and talk to speakers of the language or people who know the thing that you're trying to learn.

s95
00:08:54.800 --> 00:08:59.040
So, in practice I think the data access is very interesting and not actually fixed.

s96
00:08:59.040 --> 00:09:09.640
But from like a core idealistic standpoint, the way we think about things is more or less how do you scale more compute given the same data?

s97
00:09:10.160 --> 00:09:23.200
Um, and for the for the math heads in the room, I'm I'm not going to write any equations, but I think you can think of this as like a a box that you're dropped into and all you have is this one pre-trained model.

s98
00:09:23.200 --> 00:09:33.240
Maybe it's I don't know, GLM 5.2. Maybe you somehow hacked into Anthropic and stole the weights of of Claude and now you're trying to do it that way, but you have the pre-trained model

s99
00:09:33.240 --> 00:09:38.400
and then you have this unstructured data set D. So, maybe this is like all the emails you've ever written.

s100
00:09:38.400 --> 00:09:42.280
It's all the transcripts from every meeting your company's ever had.

s101
00:09:42.280 --> 00:09:45.200
It's some very large unstructured corpus.

s102
00:09:45.200 --> 00:09:51.120
And the question is like, how do we create a better theta that knows D?

s103
00:09:51.120 --> 00:10:00.200
And I'm going to walk through like a few ideas that you could try or that people have tried and point to some links and you can also ask me questions at the end.

s104
00:10:00.200 --> 00:10:10.720
So, core question is something like how to produce a new model, like call it theta the star, that knows D. And I think the definition of know, like this is a very load-bearing

s105
00:10:10.720 --> 00:10:19.120
term in this in this question and maybe that's where people get the leeway to propose new ideas.

s106
00:10:19.120 --> 00:10:21.760
But, this is essentially what what you want to do.

s107
00:10:21.760 --> 00:10:25.160
This is what every continual learning like startup is trying to do.

s108
00:10:25.160 --> 00:10:27.680
This is more or less what we're doing at Engram.

s109
00:10:27.680 --> 00:10:32.040
So, I'll start with a very simple idea, which is, okay, maybe you can just train the model on the data.

s110
00:10:32.040 --> 00:10:36.960
Like you can use next token prediction and train it like an LLM.

s111
00:10:37.240 --> 00:10:47.960
And I think you'll find unless you have a D that's so wide it can sort of simulate the effect of pre-training, which no one has, then this doesn't work very well.

s112
00:10:47.960 --> 00:10:50.520
Um I'll walk through an example real quick.

s113
00:10:50.520 --> 00:10:55.360
Say we have this set of like 10K financial reports.

s114
00:10:55.360 --> 00:10:56.680
You want the model to like know these.

s115
00:10:56.680 --> 00:10:57.760
You want it to be in the weights.

s116
00:10:57.760 --> 00:11:00.280
You want the model to answer questions about them.

s117
00:11:00.280 --> 00:11:02.200
You want the model to be able to create new ones.

s118
00:11:02.200 --> 00:11:05.320
You want all these behaviors to be like encoded into theta.

s119
00:11:05.320 --> 00:11:09.920
Um and then you just train theta on the context that you have.

s120
00:11:09.920 --> 00:11:13.480
You can get to a loss of like 0.0001.

s121
00:11:13.839 --> 00:11:18.640
Um and you can end up with a model that knows the data perfectly well.

s122
00:11:18.640 --> 00:11:24.000
And then when you generate from it, it basically collapses.

s123
00:11:24.000 --> 00:11:30.280
So, this strategy, this kind of naive idea of, oh, take the the context that you have and train on it indefinitely.

s124
00:11:30.280 --> 00:11:37.600
I I one, it's clearly bounded because there's some information in that just gets perfectly transferred into the model and then you no longer learn.

s125
00:11:37.600 --> 00:11:41.960
So, this is like not an indefinite axis of scaling, but two, it just frankly doesn't work.

s126
00:11:41.960 --> 00:11:50.760
Like, just doing this kind of next token prediction on the data you have doesn't produce a model that has interesting generalization properties like normal models.

s127
00:11:50.760 --> 00:11:59.160
Like, it can't answer any question unless the question is perfectly encoded in the data with its answer, which is like never the case in in practice.

s128
00:11:59.160 --> 00:12:03.840
Um so, let's think about another idea since this is not quite as easy as we thought.

s129
00:12:03.840 --> 00:12:09.320
What if we try to kind of trick the model to think the data is in context?

s130
00:12:09.320 --> 00:12:12.160
Because we know models are really good when you paste stuff into context.

s131
00:12:12.160 --> 00:12:15.000
Like, in-context learning is is magical.

s132
00:12:15.000 --> 00:12:26.600
Um one idea is you can do like compaction, which is similar to the way that, you know, Claude code or Codex or open code, what have you, does compaction.

s133
00:12:26.600 --> 00:12:36.520
You take this really long context, which is D, and then you try to compress it into some set of KVs that can represent the data to the model in like a very

s134
00:12:36.520 --> 00:12:37.920
succinct way.

s135
00:12:37.920 --> 00:12:40.240
And there's like some interesting approaches to do this.

s136
00:12:40.240 --> 00:12:41.880
You can do it in like a learned way.

s137
00:12:41.880 --> 00:12:49.760
This is a very cute paper that has like a kind of greedy algorithm for approximating KV compaction.

s138
00:12:49.760 --> 00:13:00.000
So, basically, if your data is small enough to fit into context, there are some interesting ways to like compress it to something very small and like pretend like your model knows this.

s139
00:13:00.000 --> 00:13:01.920
I think there's like multiple problems with this.

s140
00:13:01.920 --> 00:13:10.800
The main one being it only applies to things that are in context, but it also misses, I think, some of the magic that you can get from from taking gradients.

s141
00:13:10.800 --> 00:13:17.000
So, there's like an alternate way of doing it, which is you can train the model to think the data is in context.

s142
00:13:17.000 --> 00:13:19.680
And there's some interesting approaches here.

s143
00:13:19.680 --> 00:13:23.160
Like, I think Ronak was talking about on-policy distillation.

s144
00:13:23.160 --> 00:13:32.960
This is a powerful tool for doing kind of knowledge transfer where you have text and you show it to the model and then you make the model think that the text is in context.

s145
00:13:32.960 --> 00:13:35.920
That's more or less the trick of on policy distillation.

s146
00:13:35.920 --> 00:13:39.640
The on policy part just means you kind of update the model throughout training.

s147
00:13:39.640 --> 00:13:41.880
Um it works.

s148
00:13:41.880 --> 00:13:44.000
It's it's a pretty good algorithm.

s149
00:13:44.000 --> 00:13:50.720
I think there are also some core problems with it, like maybe the main one being like what data do you actually do this with?

s150
00:13:50.720 --> 00:14:03.080
You can't really distill the raw documents, so techniques like self-study from the cartridges paper on on the left here try to generate like question and answer pairs conditioned on D

s151
00:14:03.080 --> 00:14:10.200
and then train the model to behave as if it is seeing D in context when it's answering questions.

s152
00:14:10.200 --> 00:14:19.080
I think this is like close to the behavior you want, but also has some properties that are not necessarily appealing that I'll get to you in a few slides.

s153
00:14:19.080 --> 00:14:24.160
Um I think there's one more idea that I think is interesting or maybe maybe two.

s154
00:14:24.160 --> 00:14:35.080
Um I think a lot of the magic in deep learning or in like LLMs, like the reason why you know, GPT-5 is so amazing is basically because of pre-training.

s155
00:14:35.080 --> 00:14:40.040
I think there was like a lot of caveats to this statement, but pre-training is amazing for knowledge acquisition.

s156
00:14:40.040 --> 00:14:48.320
Like I can ask Claude what uh I don't know, result I got in in a paper that I've written and it actually knows this, which is incredible.

s157
00:14:48.320 --> 00:14:55.000
And you could argue maybe they do some one of these synthetic data tricks, but it more or less is knowledge that's acquired through pre-training.

s158
00:14:55.000 --> 00:15:00.960
And so one way to teach data to a model is to simulate pre-training in some way.

s159
00:15:00.960 --> 00:15:13.080
And these are like three pretty interesting approaches to do that, to like craft synthetic data conditioned on D and then train data for longer on the synthetic data as if you're continuing pre-training.

s160
00:15:13.080 --> 00:15:17.160
I think there are caveats to this approach, like you sort of overwrite some of the pre-training.

s161
00:15:17.160 --> 00:15:21.040
I think it's difficult to scale, but I think this is like pretty promising.

s162
00:15:21.040 --> 00:15:25.080
Maybe like one blocker is you then have to post-train the model after doing this.

s163
00:15:25.080 --> 00:15:28.600
So, a lot of people don't actually start with good pre-trained base models.

s164
00:15:28.600 --> 00:15:31.320
They have post-trained models, which makes this hard.

s165
00:15:31.320 --> 00:15:37.960
But, I I do like this line of work, and these papers are interesting resources if you're interested in learning more.

s166
00:15:37.960 --> 00:15:39.120
I need to go faster.

s167
00:15:39.120 --> 00:15:43.080
There's one more Uh let's skip Andre.

s168
00:15:43.080 --> 00:15:49.400
There's one more interesting idea, which is you can craft unsupervised reinforcement learning environments and do RL.

s169
00:15:49.400 --> 00:15:58.680
It's It's pretty similar to the previous suggestion, except instead of doing some type of distillation, you're just using like RL loss like GRPO or whatever.

s170
00:15:58.680 --> 00:16:05.760
Um I think all of these are are promising, but also missing maybe some core component.

s171
00:16:05.760 --> 00:16:12.680
Uh the thing that we're really after is like to give the model more knowledge of D, or to get better depth of your domain.

s172
00:16:12.680 --> 00:16:19.240
We want to be able to add compute arbitrarily in a way that makes the model better.

s173
00:16:19.240 --> 00:16:23.760
So, I think none of the approaches I proposed do this.

s174
00:16:23.760 --> 00:16:32.080
Um basically for classical machine learning reasons, which is that whatever you do, you have to define the data set, and then you train on the data set, and eventually things saturate.

s175
00:16:32.080 --> 00:16:38.520
So, even if it's like really hard, unless your model is under parameterized, eventually it will learn all the data.

s176
00:16:38.520 --> 00:16:43.200
And this doesn't give the property the beautiful scaling properties that we see out of pre-training.

s177
00:16:43.200 --> 00:16:54.640
It's kind of like a data wall in in the synthetic sense, where when you create synthetic data from D and and train on it, you eventually hit this upper bound where like you've learned all of the synthetic data.

s178
00:16:54.640 --> 00:16:56.200
And then you have to do it again.

s179
00:16:56.200 --> 00:17:01.800
And so, I think a lot of the missing components here are are how do you do it again?

s180
00:17:01.800 --> 00:17:06.319
Like what's this second stage of of training look like?

s181
00:17:06.319 --> 00:17:08.640
Like so, you can do almost any of the techniques I just mentioned.

s182
00:17:08.640 --> 00:17:14.720
You could do, you know, the attention matching, or some type of self-study thing, or some continued pre-training.

s183
00:17:14.720 --> 00:17:26.839
But, eventually you will fit the data and you'll know some about D, but you won't know everything and you'll no longer have this property where you can add compute and give the model more depth.

s184
00:17:26.839 --> 00:17:31.920
So, I think a lot of the exciting work here comes from like this idea of like self-improvement.

s185
00:17:31.920 --> 00:17:42.280
Um it's a bit overworked as well, but I think this is actually the magic behind a lot of successful RL systems like AlphaGo is that AlphaGo makes its own training questions harder

s186
00:17:42.280 --> 00:17:44.000
by getting better through training.

s187
00:17:44.000 --> 00:17:54.920
And so, I think one thing that everyone is looking for is a technique that can make models better, which makes them train themselves better.

s188
00:17:54.920 --> 00:17:58.400
Or like this is maybe a long way of saying self-improvement.

s189
00:17:58.400 --> 00:18:04.680
Like you sort of generate data and then the model gets a bit better and then you generate better data um recursively.

s190
00:18:04.680 --> 00:18:20.320
And I think this is something that we're working on a lot at Ngram is like how do you make like when we started the company, we we generated curves that look just like this blue curve where no matter sort of how much data we generate or how much we train, we kind of do plateau because there's this

s191
00:18:20.320 --> 00:18:30.320
almost like natural upper bound to how much you can learn in one go from D. But, I think it turns out that there are more sophisticated things you can do that make the training gradually harder

s192
00:18:30.320 --> 00:18:32.960
that make the model better over time.

s193
00:18:32.960 --> 00:18:38.880
So, this is the kind of like platonic graph we're looking for when we talk about scaling compute on context.

s194
00:18:38.880 --> 00:18:50.880
It's like your emails, for example, the model is just generating a better and better internal model of who you are and what you do and what your goals are, what your value function looks like, and it can

s195
00:18:50.880 --> 00:18:53.600
help you more and more.

s196
00:18:54.120 --> 00:18:56.520
Um okay, so I think I'll wrap up here.

s197
00:18:56.520 --> 00:18:57.880
I have 1 minute left.

s198
00:18:57.880 --> 00:19:03.120
That's like a summary of maybe what we think about when we think about doing research at Ngram.

s199
00:19:03.120 --> 00:19:07.920
It's a little bit abstract, but hopefully that gives you an idea of like what the core research problems are.

s200
00:19:07.920 --> 00:19:11.160
If you're interested, you feel free to um talk to me after.

s201
00:19:11.160 --> 00:19:13.840
You can message me online, or even just apply.

s202
00:19:13.840 --> 00:19:20.760
Just go to ngr.am.com and we have job postings for various like engineering and research roles.

s203
00:19:20.760 --> 00:19:23.600
Um, yeah, thank you so much.

s204
00:19:23.948 --> 00:19:25.948
[applause]

s205
00:19:39.013 --> 00:19:41.013
[music]