The Base Model Is Dead — Varun Singh, Arcee AI

AI Engineer · 17 min · 113 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Hi everyone.
00:13My name is Varon.
00:14I'm the pre-training lead at RCAI.
00:16And the talk I'm going to be giving today is called the base model is dead.
00:21Um but not really.
00:22Um the idea of the base model that we have um kind of is like built on uh this idea of like training on super large-scale web text and the base model kind of being a reflection of like the whole
00:39knowledge of like the uh human internet.
00:41Um you can see in like uh these I've I've taken these uh from a bunch of uh different papers on like the entire LM training process.
00:52Um our own model RC uh training large thinking, um the process looked kind of like the simplified diagram on the left.
01:00I've taken the top one from uh GLM 4.5, the bottom one from GLM 5. Um all these have a pre-training phase and uh pre-training is like the stage where the model look accumulates world knowledge, builds useful representations
01:18uh all through next token prediction on um web text.
01:23Um I've got a simplified uh transformer diagram um decoder-only transformer.
01:30And um a screenshot from the GPT-3 paper that talks about how um language models can uh learn how to do in-context learning through um unsupervised or self-supervised or um some some even just call it supervised
01:49um learning on uh through next token prediction.
01:56Um the way that uh older base models were trained was, like I said, mostly on uh things that reflected the entirety of human knowledge.
02:06Uh so, Common Crawl, uh which is like a commonly available web scrape, uh made up most of the training data set for GPT-3.
02:15Um WebText-2, another web scrape data set.
02:18Um some sources from like uh books as well.
02:22Um and Wikipedia as like a high-quality representation of human knowledge.
02:28Um you can see that uh WebText alone here, including like Wikipedia, makes up like roughly 85% of the whole training mix.
02:38Um looking at uh the bottom with Llama 3, um WebText still kind of makes up a majority of of the um model's training data with like 50% of the tokens corresponding to general knowledge.
02:56Back then, post-training was mostly shaping the model to use um the parts it to like surface uh the knowledge that it accumulates and accumulated in pre-training uh in like a chat interface.
03:11So, mostly allowing the model to adapt to a chat template, to the question-answer format, um and be be useful in an interaction that way.
03:21Um RL was mostly just a cherry on top, um shaping the, you know, flavor of the interactions more than conferring extra um knowledge or quality onto the base model itself.
03:36Now, in in like this realm of of how language models used to be, pre-training um and the base model kind of defined how good you were able to get a model.
03:50Um it was like the bulk of the compute uh budget and it was um, the the like core of the training process.
04:00Um, however, uh, this kind of changed a lot last year when um, OpenAI I guess 2024 actually.
04:11OpenAI released 01, um, pioneering reasoning models and DeepSeek uh, also released R1 in January 2025, um, allowing the whole world to know how to build these types of language models.
04:28And now we have this new uh, new um, use for reinforcement learning, which is no longer a cherry on top, but it can dramatically improve the performance of of the model on various different tasks.
04:44Um, the the famous graphs from 01 there talking about uh, AIME performance, um, competitive math contest.
04:53Um, and then even later in the year we we saw Cloud Code um, coming to being as a way for developers to easily um, kind of use uh, language models in a
05:06in a terminal to build out applications as models got stronger and stronger on things like function calling.
05:12Um, and then people realized you could uh, you know, RL this end-to-end.
05:17Um, and now models could learn how to interact with software environments and build software and uh, perform really useful work.
05:28And so, um, the question then becomes like, is your standard base model still uh, what the best uh, what will be the best um, prior for the for the this large-scale reinforcement learning phase
05:45that uh, research knows and um agentic models now use.
05:52And we can kind of see like in the in a few open research papers what the trend is where the trend is going.
05:59Um and interestingly enough, the um it seems like not super clear yet.
06:07Um I have my opinions on like synthetic data being the way forward, but I'm I've got like two contrasting uh perspectives here kind of in the slide.
06:18The top image is from the MEI uh Thinking 1 paper where they make a make it really large point to not use any synthetic data or any uh data from any other language model.
06:29Um and they really try to, you know, filter their web scripts for this as well.
06:33Um in order uh to kind of adhere to like the previous um paradigm of like uh using human knowledge as a way to bootstrap uh model representations and like
06:46uh capabilities.
06:48Um But I would say that this is also, like even even though they stuck with no synthetic data, the data mix that they've chosen here is still um totally different from what you'd expect in like a
07:01um in a in a classical uh language model.
07:05And I mean, the the main reason for that is that web text, which used to make up like up to 85% of the train data in GPT uh 3,
07:14is now all the way down at 15%.
07:17And that I mean, that just shows that uh the value of like web text contributing to like the downstream um performance of like the models on RL and stuff is
07:31kind of uh it it's still important, but taking a backseat to things like code and stem abilities as the models kind of gain more real-world use cases related to to those.
07:43Um the other approach uh is to bring um post-training data and large-scale synthetic data back uh through pull it back through the process into the pre-training phase.
07:56Um the the bottom chart I've taken from NeMo Tron 3 Ultra.
07:59Um what they reveal there um the data recipe and not sure how readable it is, but these top three um on the left uh pie chart, the top three on the
08:13kind of right right side of it, uh they're all labeled SFT with with SFT as a prefix.
08:20And that's the type of question and answer kind of chat data set that you'd you'd expect to see only in post-training, but by pulling it back into the process, they're able to like
08:31get the model to learn um the shape of these conversations and what kind of tasks they might be expected to do downstream um from the very beginning of the pre-training process.
08:44Um and uh this follows like uh a similar um trend in like diminishing uh the amount of web text used in the model.
08:55Um Yeah.
09:00Um it's really interesting to see the the NeMo Tron series leans so heavily into synthetic data, but uh MAI Thinking 1 kind of lean in the opposite direction.
09:11Um I've I've I've just got this slide here as like a con uh easy contrast that people can see on the like amount of web text and the amount of books and stuff
09:23um being less of a percentage here.
09:26And GPT-3 didn't even used to have any specific code data sets, but now code is like the dominating um data data subset that we have in uh pre-training recipes.
09:41Um so I mentioned synthetic data, but what is actually uh like how is synthetic data used?
09:47There's a lot of uh talk around synthetic data that, you know, blindly tossing it into a model can cause the model to collapse and uh and performance to tank, but
09:59there's been a lot of work and uh even at like a large scale, you know, example of this uh turning out really well.
10:07Um so in our own in our in our own uh model Trinity Lodge, we uh had a large amount of uh web uh web scale synthetic data um mostly through rephrasing,
10:20where you take a seed data item and you sort of upsample it in the mix by uh generating synthetic rephrases of the same information.
10:28So, the model sees the same information in like multiple ways.
10:32Um the bottom two uh screenshots are from Kimikay2, um an even larger scale model that uh broadly used this um across the whole pre-training data set.
10:46Um the top uh top right um screenshot is from a paper that uh resulted in the data sets Swallow Code and Swallow Math, which are early examples of this.
10:58But, the trend seems to be that um synthetic data not only allows you to get more and more tokens, uh but also, you know, clean up tokens, get higher quality tokens,
11:09and have um tokens that are shaped more like instruct or agentic tasks all the way back in pre-training and uh allowing the model to like learn those task representations from the very beginning.
11:24Um another reason that it's uh beneficial to add post-training data early in pre-training is now with MOEs, um one of the biggest uh pain points in training an MOE is dealing with load balancing.
11:37Um, where experts can specialize over the course of training and um they uh and load balancing objectives aim to uh achieve broadly equal utilization of the experts um in a given batch
11:52or sequence depending on the objective.
11:55Um without uh post-training data in uh early in pre-training uh with an MOE one really easy pitfall uh that we can fall into is this is kind of illustrated in the MAI thinking one report, which is that
12:11the data distribution that the model sees uh in post-training is really really different uh compared to what it sees in pre-training.
12:19And this can cause massive imbalances and um MAI overcame it by uh really cranking up the load balancing coefficient during the SFT stages.
12:30Um but I mean, ideally you don't want to mess with the balance that far into training and the model should learn stable representations from really early on.
12:44Another uh interesting thing that um is changing in base models now is that you is this whole advent of mid-training, uh which is exposing the model to the distribution that it would see during post-training in RL
12:58and at a longer context, so for things like agentic traces to be allowed into the mix and uh to kind of help prepare the model that way.
13:07Um a lot of models though are training with much longer context in pre-training and there's no reason that these data sets can't be pulled back into the mix to allow for more stable representations from the very beginning.
13:20Um I think uh a better uh better way to understand the current uh phase of LM training uh isn't so much like pre-training, mid-training, post-training, RL.
13:31It all gets a bit muddy that way, but there's two broad paradigms that are like that really help build a LM today, and that's supervised learning to next token prediction
13:42and RL.
13:44Um and RL is becoming more and more important.
13:47Um The bottom thing is a screenshot from interview um with uh the head of Xiaomi's Mimo Labs, uh where she talks about how they allocate compute between research, pre-training, and post-training.
14:04And pre-training and post-training in the final model have a roughly equal compute allocation.
14:09Um Composer 2.5 takes us to the extreme where um Kosar really uh sank much, much more RL compute into the model than the model had ever seen in supervised learning.
14:21But with RL dominating such a massive amount of the compute budget, uh it makes sense to view supervised learning as a way specifically to prepare the model for to build useful representations
14:34for for RL instead of it being the bulk of like um what the model would be used for like previously.
14:44Um there's been some some work on how supervised learning affects RL.
14:50Um I really like this one paper where the main takeaways are basically that uh the base model needs to have some exposure to like uh like the atomic skills that it would need to compose during RL, and
15:04um the model can learn to extrapolate from there during RL given like the environment has a sufficient level of difficulty.
15:13Um I had to put in the classic AlphaGo graph there where RL eventually overtakes supervised learning.
15:20Uh it's unclear if we'll see something like for language models because of course, you know, uh, human language is such an insane distribution to have to like learn through reinforcement learning alone.
15:30Um, but it's definitely possible that we might see diminished supervised learning in more and more RL, uh, which makes this kind of thinking of a base model as, um, atomic skills for RL more and more valuable.
15:46Another thing that some labs are doing, um, is kind of introducing novel data, uh, during supervised learning.
15:52And by novel, I mean something that the model really wouldn't have seen the shape of before.
15:57Uh, easy example is, you know, reasoning traces.
15:59They don't really look like a like a ton of what, um, humans output.
16:04And, um, another interesting thing is like training for test time compute, um, schemes, um, by kind of warming the model up to them during SFT, um, or even pre-training itself.
16:19Um, this These screenshots were taken from, uh, Zephyr's Ziya 1 paper.
16:24Um, and I think if they're, uh, very interesting ways of thinking about how data can affect, um, like, uh, the skills needed to explore well in RL.
16:37Um, in conclusion, um, base models have kind of moved from general, uh, human knowledge and world priors to reasoning and agentic behavior priors.
16:46Um, of course, that's kind of, uh, reductive in the in the in a way that reasoning reasoners and agents are like the main way we see, uh, bots, uh,
16:57the main way we see these chatbots used now.
17:00But, if a new paradigm were to take off, like a new way of interacting with the models, um, it makes sense to like think of a base model as building a prior for that instead of, um,
17:13just building off like a a massive, uh, script of web text.
17:18And yeah.
17:19Thanks for Thanks for listening.
17:21Thanks for your time.
17:24And
17:25[applause]