WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=xbPriQWXtWM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/xbPriQWXtWM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.440 --> 00:00:13.200
Hi everyone.

s2
00:00:13.200 --> 00:00:14.000
My name is Varon.

s3
00:00:14.000 --> 00:00:16.800
I'm the pre-training lead at RCAI.

s4
00:00:16.800 --> 00:00:21.360
And the talk I'm going to be giving today is called the base model is dead.

s5
00:00:21.360 --> 00:00:22.800
Um but not really.

s6
00:00:22.800 --> 00:00:39.120
Um the idea of the base model that we have um kind of is like built on uh this idea of like training on super large-scale web text and the base model kind of being a reflection of like the whole

s7
00:00:39.120 --> 00:00:41.880
knowledge of like the uh human internet.

s8
00:00:41.880 --> 00:00:52.000
Um you can see in like uh these I've I've taken these uh from a bunch of uh different papers on like the entire LM training process.

s9
00:00:52.000 --> 00:01:00.640
Um our own model RC uh training large thinking, um the process looked kind of like the simplified diagram on the left.

s10
00:01:00.640 --> 00:01:18.480
I've taken the top one from uh GLM 4.5, the bottom one from GLM 5. Um all these have a pre-training phase and uh pre-training is like the stage where the model look accumulates world knowledge, builds useful representations

s11
00:01:18.480 --> 00:01:23.760
uh all through next token prediction on um web text.

s12
00:01:23.760 --> 00:01:30.280
Um I've got a simplified uh transformer diagram um decoder-only transformer.

s13
00:01:30.280 --> 00:01:49.920
And um a screenshot from the GPT-3 paper that talks about how um language models can uh learn how to do in-context learning through um unsupervised or self-supervised or um some some even just call it supervised

s14
00:01:49.920 --> 00:01:54.720
um learning on uh through next token prediction.

s15
00:01:56.240 --> 00:02:06.120
Um the way that uh older base models were trained was, like I said, mostly on uh things that reflected the entirety of human knowledge.

s16
00:02:06.120 --> 00:02:15.840
Uh so, Common Crawl, uh which is like a commonly available web scrape, uh made up most of the training data set for GPT-3.

s17
00:02:15.840 --> 00:02:18.959
Um WebText-2, another web scrape data set.

s18
00:02:18.959 --> 00:02:22.600
Um some sources from like uh books as well.

s19
00:02:22.600 --> 00:02:28.360
Um and Wikipedia as like a high-quality representation of human knowledge.

s20
00:02:28.360 --> 00:02:38.320
Um you can see that uh WebText alone here, including like Wikipedia, makes up like roughly 85% of the whole training mix.

s21
00:02:38.320 --> 00:02:54.800
Um looking at uh the bottom with Llama 3, um WebText still kind of makes up a majority of of the um model's training data with like 50% of the tokens corresponding to general knowledge.

s22
00:02:56.520 --> 00:03:11.040
Back then, post-training was mostly shaping the model to use um the parts it to like surface uh the knowledge that it accumulates and accumulated in pre-training uh in like a chat interface.

s23
00:03:11.040 --> 00:03:21.560
So, mostly allowing the model to adapt to a chat template, to the question-answer format, um and be be useful in an interaction that way.

s24
00:03:21.560 --> 00:03:36.120
Um RL was mostly just a cherry on top, um shaping the, you know, flavor of the interactions more than conferring extra um knowledge or quality onto the base model itself.

s25
00:03:36.120 --> 00:03:50.400
Now, in in like this realm of of how language models used to be, pre-training um and the base model kind of defined how good you were able to get a model.

s26
00:03:50.400 --> 00:04:00.680
Um it was like the bulk of the compute uh budget and it was um, the the like core of the training process.

s27
00:04:00.680 --> 00:04:11.400
Um, however, uh, this kind of changed a lot last year when um, OpenAI I guess 2024 actually.

s28
00:04:11.400 --> 00:04:28.080
OpenAI released 01, um, pioneering reasoning models and DeepSeek uh, also released R1 in January 2025, um, allowing the whole world to know how to build these types of language models.

s29
00:04:28.080 --> 00:04:44.880
And now we have this new uh, new um, use for reinforcement learning, which is no longer a cherry on top, but it can dramatically improve the performance of of the model on various different tasks.

s30
00:04:44.880 --> 00:04:53.280
Um, the the famous graphs from 01 there talking about uh, AIME performance, um, competitive math contest.

s31
00:04:53.880 --> 00:05:06.520
Um, and then even later in the year we we saw Cloud Code um, coming to being as a way for developers to easily um, kind of use uh, language models in a

s32
00:05:06.520 --> 00:05:12.960
in a terminal to build out applications as models got stronger and stronger on things like function calling.

s33
00:05:12.960 --> 00:05:17.680
Um, and then people realized you could uh, you know, RL this end-to-end.

s34
00:05:17.680 --> 00:05:28.240
Um, and now models could learn how to interact with software environments and build software and uh, perform really useful work.

s35
00:05:28.600 --> 00:05:45.880
And so, um, the question then becomes like, is your standard base model still uh, what the best uh, what will be the best um, prior for the for the this large-scale reinforcement learning phase

s36
00:05:45.880 --> 00:05:52.640
that uh, research knows and um agentic models now use.

s37
00:05:52.640 --> 00:05:59.680
And we can kind of see like in the in a few open research papers what the trend is where the trend is going.

s38
00:05:59.680 --> 00:06:07.560
Um and interestingly enough, the um it seems like not super clear yet.

s39
00:06:07.560 --> 00:06:18.080
Um I have my opinions on like synthetic data being the way forward, but I'm I've got like two contrasting uh perspectives here kind of in the slide.

s40
00:06:18.080 --> 00:06:29.360
The top image is from the MEI uh Thinking 1 paper where they make a make it really large point to not use any synthetic data or any uh data from any other language model.

s41
00:06:29.360 --> 00:06:33.200
Um and they really try to, you know, filter their web scripts for this as well.

s42
00:06:33.200 --> 00:06:46.680
Um in order uh to kind of adhere to like the previous um paradigm of like uh using human knowledge as a way to bootstrap uh model representations and like

s43
00:06:46.680 --> 00:06:48.520
uh capabilities.

s44
00:06:48.520 --> 00:07:01.640
Um But I would say that this is also, like even even though they stuck with no synthetic data, the data mix that they've chosen here is still um totally different from what you'd expect in like a

s45
00:07:01.640 --> 00:07:05.520
um in a in a classical uh language model.

s46
00:07:05.520 --> 00:07:14.200
And I mean, the the main reason for that is that web text, which used to make up like up to 85% of the train data in GPT uh 3,

s47
00:07:14.200 --> 00:07:17.160
is now all the way down at 15%.

s48
00:07:17.160 --> 00:07:31.600
And that I mean, that just shows that uh the value of like web text contributing to like the downstream um performance of like the models on RL and stuff is

s49
00:07:31.600 --> 00:07:43.680
kind of uh it it's still important, but taking a backseat to things like code and stem abilities as the models kind of gain more real-world use cases related to to those.

s50
00:07:43.680 --> 00:07:56.360
Um the other approach uh is to bring um post-training data and large-scale synthetic data back uh through pull it back through the process into the pre-training phase.

s51
00:07:56.360 --> 00:07:59.960
Um the the bottom chart I've taken from NeMo Tron 3 Ultra.

s52
00:07:59.960 --> 00:08:13.440
Um what they reveal there um the data recipe and not sure how readable it is, but these top three um on the left uh pie chart, the top three on the

s53
00:08:13.440 --> 00:08:20.600
kind of right right side of it, uh they're all labeled SFT with with SFT as a prefix.

s54
00:08:20.600 --> 00:08:31.640
And that's the type of question and answer kind of chat data set that you'd you'd expect to see only in post-training, but by pulling it back into the process, they're able to like

s55
00:08:31.640 --> 00:08:44.480
get the model to learn um the shape of these conversations and what kind of tasks they might be expected to do downstream um from the very beginning of the pre-training process.

s56
00:08:44.480 --> 00:08:55.800
Um and uh this follows like uh a similar um trend in like diminishing uh the amount of web text used in the model.

s57
00:08:55.800 --> 00:09:00.240
Um Yeah.

s58
00:09:00.560 --> 00:09:11.480
Um it's really interesting to see the the NeMo Tron series leans so heavily into synthetic data, but uh MAI Thinking 1 kind of lean in the opposite direction.

s59
00:09:11.480 --> 00:09:23.680
Um I've I've I've just got this slide here as like a con uh easy contrast that people can see on the like amount of web text and the amount of books and stuff

s60
00:09:23.680 --> 00:09:26.560
um being less of a percentage here.

s61
00:09:26.560 --> 00:09:40.560
And GPT-3 didn't even used to have any specific code data sets, but now code is like the dominating um data data subset that we have in uh pre-training recipes.

s62
00:09:41.400 --> 00:09:47.920
Um so I mentioned synthetic data, but what is actually uh like how is synthetic data used?

s63
00:09:47.920 --> 00:09:59.000
There's a lot of uh talk around synthetic data that, you know, blindly tossing it into a model can cause the model to collapse and uh and performance to tank, but

s64
00:09:59.000 --> 00:10:07.440
there's been a lot of work and uh even at like a large scale, you know, example of this uh turning out really well.

s65
00:10:07.440 --> 00:10:20.080
Um so in our own in our in our own uh model Trinity Lodge, we uh had a large amount of uh web uh web scale synthetic data um mostly through rephrasing,

s66
00:10:20.080 --> 00:10:28.720
where you take a seed data item and you sort of upsample it in the mix by uh generating synthetic rephrases of the same information.

s67
00:10:28.720 --> 00:10:32.360
So, the model sees the same information in like multiple ways.

s68
00:10:32.360 --> 00:10:46.240
Um the bottom two uh screenshots are from Kimikay2, um an even larger scale model that uh broadly used this um across the whole pre-training data set.

s69
00:10:46.240 --> 00:10:58.480
Um the top uh top right um screenshot is from a paper that uh resulted in the data sets Swallow Code and Swallow Math, which are early examples of this.

s70
00:10:58.480 --> 00:11:09.320
But, the trend seems to be that um synthetic data not only allows you to get more and more tokens, uh but also, you know, clean up tokens, get higher quality tokens,

s71
00:11:09.320 --> 00:11:24.160
and have um tokens that are shaped more like instruct or agentic tasks all the way back in pre-training and uh allowing the model to like learn those task representations from the very beginning.

s72
00:11:24.760 --> 00:11:37.600
Um another reason that it's uh beneficial to add post-training data early in pre-training is now with MOEs, um one of the biggest uh pain points in training an MOE is dealing with load balancing.

s73
00:11:37.600 --> 00:11:52.920
Um, where experts can specialize over the course of training and um they uh and load balancing objectives aim to uh achieve broadly equal utilization of the experts um in a given batch

s74
00:11:52.920 --> 00:11:55.560
or sequence depending on the objective.

s75
00:11:55.560 --> 00:12:11.000
Um without uh post-training data in uh early in pre-training uh with an MOE one really easy pitfall uh that we can fall into is this is kind of illustrated in the MAI thinking one report, which is that

s76
00:12:11.000 --> 00:12:19.480
the data distribution that the model sees uh in post-training is really really different uh compared to what it sees in pre-training.

s77
00:12:19.480 --> 00:12:30.880
And this can cause massive imbalances and um MAI overcame it by uh really cranking up the load balancing coefficient during the SFT stages.

s78
00:12:30.880 --> 00:12:43.400
Um but I mean, ideally you don't want to mess with the balance that far into training and the model should learn stable representations from really early on.

s79
00:12:44.000 --> 00:12:58.880
Another uh interesting thing that um is changing in base models now is that you is this whole advent of mid-training, uh which is exposing the model to the distribution that it would see during post-training in RL

s80
00:12:58.880 --> 00:13:07.600
and at a longer context, so for things like agentic traces to be allowed into the mix and uh to kind of help prepare the model that way.

s81
00:13:07.600 --> 00:13:19.720
Um a lot of models though are training with much longer context in pre-training and there's no reason that these data sets can't be pulled back into the mix to allow for more stable representations from the very beginning.

s82
00:13:20.040 --> 00:13:31.360
Um I think uh a better uh better way to understand the current uh phase of LM training uh isn't so much like pre-training, mid-training, post-training, RL.

s83
00:13:31.360 --> 00:13:42.640
It all gets a bit muddy that way, but there's two broad paradigms that are like that really help build a LM today, and that's supervised learning to next token prediction

s84
00:13:42.640 --> 00:13:44.440
and RL.

s85
00:13:44.440 --> 00:13:47.960
Um and RL is becoming more and more important.

s86
00:13:47.960 --> 00:14:04.480
Um The bottom thing is a screenshot from interview um with uh the head of Xiaomi's Mimo Labs, uh where she talks about how they allocate compute between research, pre-training, and post-training.

s87
00:14:04.480 --> 00:14:09.080
And pre-training and post-training in the final model have a roughly equal compute allocation.

s88
00:14:09.080 --> 00:14:21.320
Um Composer 2.5 takes us to the extreme where um Kosar really uh sank much, much more RL compute into the model than the model had ever seen in supervised learning.

s89
00:14:21.320 --> 00:14:34.280
But with RL dominating such a massive amount of the compute budget, uh it makes sense to view supervised learning as a way specifically to prepare the model for to build useful representations

s90
00:14:34.280 --> 00:14:43.880
for for RL instead of it being the bulk of like um what the model would be used for like previously.

s91
00:14:44.839 --> 00:14:50.839
Um there's been some some work on how supervised learning affects RL.

s92
00:14:50.839 --> 00:15:04.640
Um I really like this one paper where the main takeaways are basically that uh the base model needs to have some exposure to like uh like the atomic skills that it would need to compose during RL, and

s93
00:15:04.640 --> 00:15:13.280
um the model can learn to extrapolate from there during RL given like the environment has a sufficient level of difficulty.

s94
00:15:13.280 --> 00:15:20.600
Um I had to put in the classic AlphaGo graph there where RL eventually overtakes supervised learning.

s95
00:15:20.600 --> 00:15:30.920
Uh it's unclear if we'll see something like for language models because of course, you know, uh, human language is such an insane distribution to have to like learn through reinforcement learning alone.

s96
00:15:30.920 --> 00:15:45.600
Um, but it's definitely possible that we might see diminished supervised learning in more and more RL, uh, which makes this kind of thinking of a base model as, um, atomic skills for RL more and more valuable.

s97
00:15:46.120 --> 00:15:52.520
Another thing that some labs are doing, um, is kind of introducing novel data, uh, during supervised learning.

s98
00:15:52.520 --> 00:15:57.440
And by novel, I mean something that the model really wouldn't have seen the shape of before.

s99
00:15:57.440 --> 00:15:59.440
Uh, easy example is, you know, reasoning traces.

s100
00:15:59.440 --> 00:16:04.840
They don't really look like a like a ton of what, um, humans output.

s101
00:16:04.840 --> 00:16:19.480
And, um, another interesting thing is like training for test time compute, um, schemes, um, by kind of warming the model up to them during SFT, um, or even pre-training itself.

s102
00:16:19.480 --> 00:16:24.680
Um, this These screenshots were taken from, uh, Zephyr's Ziya 1 paper.

s103
00:16:24.680 --> 00:16:37.440
Um, and I think if they're, uh, very interesting ways of thinking about how data can affect, um, like, uh, the skills needed to explore well in RL.

s104
00:16:37.560 --> 00:16:46.920
Um, in conclusion, um, base models have kind of moved from general, uh, human knowledge and world priors to reasoning and agentic behavior priors.

s105
00:16:46.920 --> 00:16:57.920
Um, of course, that's kind of, uh, reductive in the in the in a way that reasoning reasoners and agents are like the main way we see, uh, bots, uh,

s106
00:16:57.920 --> 00:17:00.640
the main way we see these chatbots used now.

s107
00:17:00.640 --> 00:17:13.760
But, if a new paradigm were to take off, like a new way of interacting with the models, um, it makes sense to like think of a base model as building a prior for that instead of, um,

s108
00:17:13.760 --> 00:17:18.600
just building off like a a massive, uh, script of web text.

s109
00:17:18.600 --> 00:17:19.959
And yeah.

s110
00:17:19.959 --> 00:17:21.839
Thanks for Thanks for listening.

s111
00:17:21.839 --> 00:17:24.000
Thanks for your time.

s112
00:17:24.000 --> 00:17:25.391
And

s113
00:17:25.391 --> 00:17:27.391
[applause]