WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=KhYifX22yhE NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/KhYifX22yhE.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.640 --> 00:00:13.920 Hi everybody. s2 00:00:13.920 --> 00:00:16.000 Thanks for coming to our talk. s3 00:00:16.000 --> 00:00:18.320 Uh my name is Mara Abden. s4 00:00:18.320 --> 00:00:19.880 I'm I'm from Pulsar. s5 00:00:19.880 --> 00:00:21.600 I don't know if you're familiar with our models. s6 00:00:21.600 --> 00:00:31.960 Uh our data team and today my colleague Robert and I will be talking a little bit about um some of the challenges that we've seen as we scale our models s7 00:00:31.960 --> 00:00:33.240 over here. s8 00:00:33.240 --> 00:00:43.400 Uh particularly if you haven't heard, we've switched from recently from releasing our models towards enterprise to also releasing towards everybody. s9 00:00:43.400 --> 00:00:46.240 We've actually put out two open weight models. s10 00:00:46.240 --> 00:00:48.880 They're on Hugging Face a few weeks ago. s11 00:00:48.880 --> 00:00:50.840 We have Laguna M and Laguna XS. s12 00:00:50.840 --> 00:00:54.240 We also put out a tech report which has a ton of detail if you're interested. s13 00:00:54.240 --> 00:01:05.040 Um as you can see here by the Laguna M point one and XS point two, this uh this is this is actually is because we have switched out quite a few things s14 00:01:05.040 --> 00:01:12.600 between those two models and so big flavor of this talk is going to be about kind of how did we transition from one to two. s15 00:01:12.600 --> 00:01:21.160 And in fact, we've continued to do so and now we actually have a newer version and Robert will give a sneak peek about soon to be released model. s16 00:01:21.160 --> 00:01:21.760 Okay. s17 00:01:21.760 --> 00:01:25.840 So I will be particular talk about synthetic data part of things. s18 00:01:25.840 --> 00:01:33.320 So um there's three things that we did on the data side to kind of resolve some of the issues that we've seen with scale. s19 00:01:33.320 --> 00:01:43.200 One is that we implemented auto mixer that basically just gives us a chance to do a cheaper sweep on clusters of our sets before moving on to more expensive s20 00:01:43.200 --> 00:01:43.880 experiments. s21 00:01:43.880 --> 00:01:49.200 And then we improved we just rethought our sampling of web data for higher recall. s22 00:01:49.200 --> 00:01:56.720 Um and then the third one is that we relied a lot more on synthetic data in a few forms which I'm going to. s23 00:01:56.720 --> 00:02:08.800 Okay, so before we kind of going into what does that mean and what have we done, etc. One uh why would we kind of It's It's Sometimes it's fair at least to ask why synthetic data and the thing is that s24 00:02:08.800 --> 00:02:12.360 uh at least at Pulsar we don't see it as a way to replace organic data. s25 00:02:12.360 --> 00:02:17.959 I don't see it so in the current state of the world at least, but uh it is a way to kind of complement it. s26 00:02:17.959 --> 00:02:23.040 And the thing is that organic data has a lot in it that is basically kind of implicitly hidden. s27 00:02:23.040 --> 00:02:27.680 A lot of things that could teach the model or not very presented in the most optimal way sometimes. s28 00:02:27.680 --> 00:02:34.320 And so synthetic data gives us a track to extract some of these features and project them on some new planes. s29 00:02:34.320 --> 00:02:46.720 And this is how we get to expose um implicit rationale, implicit planning, implicit structure, and a way for us to fill gaps and regularize not only how we present the tokens, but also how we are teaching the model. s30 00:02:46.720 --> 00:02:50.680 Um for access point two in particular, we settled on 13% of the mix. s31 00:02:50.680 --> 00:02:53.760 This is only pre-training stages before post-training. s32 00:02:53.760 --> 00:02:57.160 Um and since then we've just been continuously generating more data in a bunch of directions. s33 00:02:57.160 --> 00:03:01.800 Now we have a six trillion token uh corpus that's continuously growing. s34 00:03:01.800 --> 00:03:03.160 Okay. s35 00:03:03.160 --> 00:03:16.080 Um so yeah, so uh what kind of what I I I I just said is that we saw some some I guess limitations switching from Luganem to uh point one to our point two models. s36 00:03:16.080 --> 00:03:23.480 And so one of those things is that we basically we started on data and this is really not a crazy kind of problem. s37 00:03:23.480 --> 00:03:39.680 We very intuitively started from a place on a smaller scale where we were um we were basically focusing on quality versus quantity um maybe a little too much because eventually when we started scaling our models, we had to scale our training budget and with that s38 00:03:39.680 --> 00:03:49.360 uh came some limitations because we started hitting repetition uh like non-optimal repetition on some of our high-quality data which saturated the model a little too early. s39 00:03:49.360 --> 00:04:02.320 So um we One of the ways we've particularly for this like a token uniqueness problem uh, we relied on which is a very common form of synthetic data rephrasing s40 00:04:02.320 --> 00:04:05.360 which you know you just may heard in like Beyond Web for example. s41 00:04:05.360 --> 00:04:07.600 It's become pretty trendy these days. s42 00:04:07.600 --> 00:04:22.120 Um, and you can see here that uh you know, this is a take this ablation this is an ablation result so take the numbers with a grain of salt but what persists pretty consistently is the diff between using the orange you would be just the seeds with repetition s43 00:04:22.120 --> 00:04:31.800 and then the green would be replacing some of those repeated tokens with higher like with the multi-mode reverse or at least yeah, all of them or at least reducing the repetition. s44 00:04:31.800 --> 00:04:42.120 Um, and so uh for rephrasing particular we did to the you know, what everyone's doing with the you know, generic kind of multi-mode very scalable pipeline but we also took it a step far s45 00:04:42.120 --> 00:04:50.720 and we did two other specialized pipelines one from to go from Morocco to code and text and one to go specifically for scan data. s46 00:04:50.720 --> 00:05:00.440 Just because this is a very cheap scalable pipeline so it kind of you kind of have to rely very heavily on the seed and we push a little further on that for the stem documents. s47 00:05:00.440 --> 00:05:01.200 Okay. s48 00:05:01.200 --> 00:05:12.919 So okay, so if you kind of think of everything is a kind of an modular way you can think of every synthetic data pipeline is composed of the same s49 00:05:12.919 --> 00:05:21.560 six components and so you have your seeds, your primary inputs, your metadata, your secondary inputs, your generator and function which can be an agent with tools or one you know, with some prompt s50 00:05:21.560 --> 00:05:25.960 with some prompt templates and then some supplementary functions like filters and validators and so on. s51 00:05:25.960 --> 00:05:31.440 And really you can compose just about all pipelines from like very simple to very to very expensive pipelines like this. s52 00:05:31.440 --> 00:05:42.720 And on on that note and kind of we've we have covered quite a bit of wide scope on the the axis complexity and you kind of can think of it if one end you have like the cheap scalable s53 00:05:42.720 --> 00:05:48.960 pipelines that have used smaller models and can get get with it because they're seed heavy examples of phrasing. s54 00:05:48.960 --> 00:05:54.680 And then on the other end you have more complex pipelines with a little more orchestration in the workflows. s55 00:05:54.680 --> 00:05:58.800 This is reserved when the we're building on something that's worth it. s56 00:05:58.800 --> 00:06:00.400 Educational data. s57 00:06:00.400 --> 00:06:06.040 But really it's this is how we're not blocked or limited by whatever teacher model can do. s58 00:06:06.040 --> 00:06:10.000 And this is how we can be ambitious in our synthetic data. s59 00:06:10.000 --> 00:06:15.440 Because the rule of thumb is if task is too hard for your model, then your model will start to fall on its face. s60 00:06:15.440 --> 00:06:17.240 Lose correctness, lose diversity. s61 00:06:17.240 --> 00:06:20.680 So break down the task, make it simpler. s62 00:06:20.680 --> 00:06:32.200 And yeah, I will give some examples of kind of shapes rather than just like something more concrete about how do we use this this modularity. s63 00:06:32.200 --> 00:06:35.160 And one shape is the formula writing is just rephrasing. s64 00:06:35.160 --> 00:06:36.400 We already talked about this. s65 00:06:36.400 --> 00:06:39.640 Multi-stage pipelines and multi-stage workflows. s66 00:06:39.640 --> 00:06:41.520 Basically this is what I also just said. s67 00:06:41.520 --> 00:06:44.480 You take a step and you break it down into multiple steps. s68 00:06:44.480 --> 00:06:48.800 You can aggregate the processing. s69 00:06:48.800 --> 00:06:51.400 Slowly build up the the generation. s70 00:06:51.400 --> 00:06:59.160 Example of this were if you wanted to generate a novel for example, it's you could generate one chapter at a time, but you could also, you know, take it a little slowly. s71 00:06:59.160 --> 00:07:07.360 One first like generate, you know, the setting, the character names, the character styles, the plot, you know, some twists. s72 00:07:07.360 --> 00:07:11.160 And then and then from there go into generate the chapters one by one. s73 00:07:11.160 --> 00:07:13.040 You will absolutely get a better novel. s74 00:07:13.040 --> 00:07:20.160 Um Okay, third is cross-domain porting which really is just like like moving from one mode to another. s75 00:07:20.160 --> 00:07:22.280 Example would be changing like translating code. s76 00:07:22.280 --> 00:07:27.560 Another example would be something we did which is take our math problems and convert them to code. s77 00:07:27.560 --> 00:07:29.760 The last one is multi-turn role. s78 00:07:29.760 --> 00:07:38.440 What I mean by that all I mean is that instead of having kind of a very one like very singular or or linear or even non-linear kind of view of things you have more of an iteration. s79 00:07:38.440 --> 00:07:40.840 This encapsulates pretty much everything. s80 00:07:40.840 --> 00:07:52.240 And and that like an example of that would be multi-turn chats when you have two agents talking to each other or a task evolution pipeline where, you know, you have a judge and an evolver going back and forth for some k amount of time. s81 00:07:52.240 --> 00:07:54.160 Um, and so on. s82 00:07:54.160 --> 00:07:54.720 Okay. s83 00:07:54.720 --> 00:08:04.880 So, lastly, uh, forewarning off to Robert, I do want to kind of just mention that we because of the modularity of the way we think about this, we can implement an infrastructure that's pretty configurable. s84 00:08:04.880 --> 00:08:06.800 Um, so this is how we present Hive. s85 00:08:06.800 --> 00:08:16.080 Hive is basically a way for us to easily, um, that uh, to easily construct uh, generations where now you have uh, a queue of agents that you define. s86 00:08:16.080 --> 00:08:20.000 Each one has, you know, its prompt, its parameters, its model, etc., inputs, outputs. s87 00:08:20.000 --> 00:08:25.040 But it also has when you can configure when enters the queue, when it exits, how many have frequency to come in. s88 00:08:25.040 --> 00:08:27.560 And then we have orchestration in the middle between agents. s89 00:08:27.560 --> 00:08:38.560 And with this orchestrators are really useful because they give you more flexibility and kind of presenting a hierarchy between LLMs that are generating is very uh, it's this is how you police them, basically. s90 00:08:38.560 --> 00:08:46.400 But also give them some some form of creativity and dynamically change instructions for the next agent or choose which agent goes next, which agent skipped, and so on. s91 00:08:46.400 --> 00:08:53.400 Um, and lastly, you have the supervisor, which basically someone who polices the orchestrator and has more of a global view. s92 00:08:53.400 --> 00:08:54.920 Um, cool. s93 00:08:54.920 --> 00:08:56.720 Okay, that's uh, that's it for me and Zendaya. s94 00:08:56.720 --> 00:08:58.520 I hope you learned something interesting. s95 00:08:58.520 --> 00:09:03.000 Handing over to Robert for uh, pre-training stuff. s96 00:09:03.320 --> 00:09:05.080 All right, thank you, Mara. s97 00:09:05.080 --> 00:09:12.560 Um, cuz I will talk a little bit more on the um, actual pre-training side rather than just data. s98 00:09:12.560 --> 00:09:19.520 I liked in the previous talk, um, the the speaker made a point that we should treat different data mixes holistically, different training stages. s99 00:09:19.520 --> 00:09:28.160 I want to make the same point that we should treat uh, data and implementation of your training code base, uh, correctness of it, and so on, also holistically. s100 00:09:28.160 --> 00:09:30.080 If you've got data that sucks, you can't train a good model. s101 00:09:30.080 --> 00:09:32.760 If you've got a training code base that sucks, you also can't. s102 00:09:32.760 --> 00:09:39.400 Um, so I specifically focus on architecture work on and distributed training and so on. s103 00:09:39.400 --> 00:09:43.480 Um and the way we we look at things in my team is uh we don't trust anything. s104 00:09:43.480 --> 00:09:52.000 There's so many things that can go wrong when you scale models to billions of parameters to hundreds of billions of parameters um training on thousands of GPUs and so on. s105 00:09:52.000 --> 00:10:00.200 And I want to show you some of the learnings that we got from training Laguna m.1. Um And yeah, some of the surprising things that happen at scale. s106 00:10:00.200 --> 00:10:04.320 Um so one thing we do is we've got these um model replica hashtags. s107 00:10:04.320 --> 00:10:07.720 So essentially when we train a model we've got multiple replicas of the same model, right? s108 00:10:07.720 --> 00:10:09.800 Um distributed data parallel. s109 00:10:09.800 --> 00:10:14.280 And uh we know there's an invariant, the weight should always be the same across all of these replicas. s110 00:10:14.280 --> 00:10:15.440 That's something you can verify, right? s111 00:10:15.440 --> 00:10:19.520 We you can calculate a hash over the weights and you know that should always be the same across all replicas. s112 00:10:19.520 --> 00:10:22.320 So we do that in training and periodically compare them. s113 00:10:22.320 --> 00:10:25.560 Um if all of these hashes are identical, then we know we can continue training. s114 00:10:25.560 --> 00:10:30.080 If they're not identical, we know something has gone seriously wrong uh because that should never happen. s115 00:10:30.080 --> 00:10:31.640 And we crash the training. s116 00:10:31.640 --> 00:10:36.120 And I'll give you some examples now of things that we've not shared before publicly like this. s117 00:10:36.120 --> 00:10:38.080 Um so I hope they're interesting. s118 00:10:38.080 --> 00:10:43.720 Um so the first example here of uh that happens at scale are broken GPUs. s119 00:10:43.720 --> 00:10:50.320 On the left-hand side we've got two loss curves uh and on the right-hand side the corresponding uh gradient norms um that we observed during training. s120 00:10:50.320 --> 00:10:52.680 And you can see that these loss curves look quite different, right? s121 00:10:52.680 --> 00:10:55.640 Like the purple one has got quite some bumps, looks a bit spiky. s122 00:10:55.640 --> 00:10:57.800 The gradient norms are huge for that run. s123 00:10:57.800 --> 00:11:05.080 And there's actually no difference uh in terms of model configuration, training data, training implementation um between these runs. s124 00:11:05.080 --> 00:11:06.520 They're exactly the same run. s125 00:11:06.520 --> 00:11:10.240 Just in one of them we were got unlucky and we had a broken GPU included. s126 00:11:10.240 --> 00:11:16.760 That broken GPU caused silent data corruption and um therefore made the training behave the way it did. s127 00:11:16.760 --> 00:11:24.240 Uh and that is one of those cases that you can catch with these hashtags because you know this computation should be the same across all replicas, but it wasn't. s128 00:11:24.240 --> 00:11:29.520 Um Which brings me to the next uh instance of that happens at scale. s129 00:11:29.520 --> 00:11:31.720 Uh in this case exploding gradients. s130 00:11:31.720 --> 00:11:35.840 Um again, we're looking at two different loss curves and the corresponding gradient norm curves. s131 00:11:35.840 --> 00:11:42.200 Um, the purple run is our initial training run for Laguna M 1. We're a bit further into training here around 50,000 steps or so. s132 00:11:42.200 --> 00:11:43.840 And you can see it stops converging, right? s133 00:11:43.840 --> 00:11:45.360 Like it just flattens out. s134 00:11:45.360 --> 00:11:53.280 And the reason here was that during training the uh activations grew and grew um right before the um LM head, the unembedding. s135 00:11:53.280 --> 00:12:00.520 And um we have to perform some sort of accumulation here uh because we use tensor parallel for the uh unembedding. s136 00:12:00.520 --> 00:12:04.760 And that accumulation um was performed in BF16 by default. s137 00:12:04.760 --> 00:12:13.600 And because of the growing scale that we observed in the activations, um there wasn't enough um numerical precision available anymore um to do this accurately. s138 00:12:13.600 --> 00:12:15.120 And hence the model just couldn't learn anymore. s139 00:12:15.120 --> 00:12:21.920 And this is also very uh dramatic point for this to happen because it from there on it really like back propagates into the full model trunk. s140 00:12:21.920 --> 00:12:25.160 Um, the orange curve is essentially just adding a fix on that. s141 00:12:25.160 --> 00:12:27.200 So, we took the checkpoint from the purple curve. s142 00:12:27.200 --> 00:12:32.560 We moved that accumulation into FP32 and from there on the model started converging again. s143 00:12:32.560 --> 00:12:35.600 Um, the gradient norm, as you can see, actually started decreasing. s144 00:12:35.600 --> 00:12:38.040 Um, before then we had an increasing trend. s145 00:12:38.040 --> 00:12:40.960 And um this is also something you can only observe at scale. s146 00:12:40.960 --> 00:12:44.560 And that will break your model if you if you're not careful about it. s147 00:12:44.560 --> 00:12:51.480 So, as Mara said, we took all of these insights on data, on um numerics, and so on, and we turned them from M 1 into XGen-2. s148 00:12:51.480 --> 00:12:53.839 That's why we say it's a new generation model. s149 00:12:53.839 --> 00:13:06.760 Um, this included uh increasing diversity, reducing data repetitions, um all of these numerical things I just mentioned, and just adding more observability and checks on that side, as well as generally optimizing the training in the architecture. s150 00:13:06.760 --> 00:13:09.160 And XGen-2, if you look at it, it's open weights, right? s151 00:13:09.160 --> 00:13:11.800 So, you can download uh you can download and use it for free. s152 00:13:11.800 --> 00:13:15.760 Um, it's one of the most competitive models for its size and for coding specifically. s153 00:13:15.760 --> 00:13:18.080 That's what we focus on Agentic coding. s154 00:13:18.080 --> 00:13:19.640 Um, so we're pretty happy with it. s155 00:13:19.640 --> 00:13:23.920 However, you can say that um the model with 33 billion parameters is pretty small. s156 00:13:23.920 --> 00:13:27.160 Like you wouldn't probably observe any issues anyways training it. s157 00:13:27.160 --> 00:13:30.040 So, what was important to us was to scale this, right? s158 00:13:30.040 --> 00:13:31.600 And this is where Laguna S comes in. s159 00:13:31.600 --> 00:13:34.240 This model is not public yet, so this is a preview. s160 00:13:34.240 --> 00:13:43.600 Um as I said, we treated XS as a test bed in a sense, and uh with Laguna S we scaled this to a model that's 118 billion uh total parameters s161 00:13:43.600 --> 00:13:45.280 and 8B active parameters. s162 00:13:45.280 --> 00:13:48.280 Again, we trained it on 30 trillion tokens on 4,000 GPUs. s163 00:13:48.280 --> 00:13:59.080 So, the scale was sufficient to not only test whether uh all the improvements we made on the data and architectural side had hold, but also if any of these numerical issues come up again. s164 00:13:59.080 --> 00:14:01.400 And um of course, something happened. s165 00:14:01.400 --> 00:14:06.040 Um in this case, it doesn't actually have anything to do with scale, so it was just unfortunate. s166 00:14:06.040 --> 00:14:13.440 Um in this case, we had a race condition because we added FP8 training uh based on Deep Chem FP8 kernels that are also like open source. s167 00:14:13.440 --> 00:14:21.720 Um we noticed these because we hit illegal memory accesses as well as NaNs in the gradients, uh which after a while of debugging we traced back to those kernels. s168 00:14:21.720 --> 00:14:28.000 Um there's also an unobservable unobservable effect um that you wouldn't know about if you don't know that there's an issue. s169 00:14:28.000 --> 00:14:33.480 Uh in our case, we noticed about 0.5% of the gradient gets silently corrupted, essentially replaced by random values. s170 00:14:33.480 --> 00:14:36.839 Um we do have a fix available that's in a PR right now. s171 00:14:36.839 --> 00:14:41.080 Uh it's not been merged to Deep Chem yet, but it's public on that QR code if anyone is interested. s172 00:14:41.080 --> 00:14:45.480 Um and it's also an interesting point because it's a blind check and a blind spot in the hash checks. s173 00:14:45.480 --> 00:14:53.920 In real training runs, you don't have any redundancy where you have the same model weights and the same data, so you can never check if forward and backward actually behave the same across different model replicas. s174 00:14:53.920 --> 00:14:56.600 So, you can also never check if there's a race condition in that. s175 00:14:56.600 --> 00:15:03.800 Uh that's something that we're working on right now uh to essentially have a hash checker um that can also do that as a at least as a dry run. s176 00:15:03.800 --> 00:15:14.800 Um and I want to end on um some early results from uh this new model um and demonstrating how it performs against some open weight models and also against our previous models. s177 00:15:14.800 --> 00:15:17.640 So, first I want to caveat this with these are base model evals, right? s178 00:15:17.640 --> 00:15:23.520 They are partly indicative of how the final model will look, but also not perfectly, right? s179 00:15:23.520 --> 00:15:25.360 Um there's still post training happening. s180 00:15:25.360 --> 00:15:28.720 Uh not all of these will translate one-to-one to the final model. s181 00:15:28.720 --> 00:15:39.240 Um but if we look at them uh specifically on the coding part of of the E valves, so for instance multiple E love code bench, big code bench, uh Laguna S is not only stronger than s182 00:15:39.240 --> 00:15:44.920 access to two, which is our previous uh smaller model that performed very well, but also than the much larger M dot one. s183 00:15:44.920 --> 00:15:55.160 Um and it's also much better than GLM 4.5 air, which is admittedly a bit older, and then we turn 360, which is quite recent, and then deep seek V4 flash max, which is quite recent and a fair bit larger. s184 00:15:55.160 --> 00:15:59.200 Um we can see it's competitive on big bench hard for instance. s185 00:15:59.200 --> 00:16:03.760 It doesn't achieve the the top E valve results compared to these models, but it's quite close. s186 00:16:03.760 --> 00:16:15.760 Um we also see it's quite close on E valve plus, and uh quite importantly for us, it does very well on sweep bench agent less multilingual, um which we use to sort of proxy agentic performance during pre-training. s187 00:16:15.760 --> 00:16:21.200 Um and in that case it performs much better than all the other all other models we tested here. s188 00:16:21.200 --> 00:16:25.440 Um I also want to point out that of course there are like it's not the strongest model in the world, right? s189 00:16:25.440 --> 00:16:34.360 Like for instance MMLU pro knowledge benchmark is something we don't care about that much compared to coding because we want to build the strongest agentic coding models. s190 00:16:34.360 --> 00:16:41.240 Um so here like compared to Nema tron and deep seek uh we have to say that they perform much better, and this mainly comes down due to data, right? s191 00:16:41.240 --> 00:16:44.320 It's a data gap um that we could plug if we wanted to. s192 00:16:44.320 --> 00:16:49.920 Um but I think the the point is all of the things we we found before were included in the recipe. s193 00:16:49.920 --> 00:16:53.720 The recipe held, it scaled, and we will continue scaling it from here. s194 00:16:53.720 --> 00:16:57.400 So this model will also be available sometime in the future relatively soon. s195 00:16:57.400 --> 00:17:01.760 Um again open weights, so all of you can download it and use it for free. s196 00:17:01.760 --> 00:17:06.040 And um with that I want to thank everyone for attending our talk. s197 00:17:06.040 --> 00:17:11.400 Um I added also a link to our careers page and our Twitter page if you want to uh check it out. s198 00:17:11.400 --> 00:17:14.640 And yeah, thank you very much. s199 00:17:28.667 --> 00:17:30.640 [music] s200 00:17:30.640 --> 00:17:32.640 Hey.