WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=maRzp4kImJ4 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/maRzp4kImJ4.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.400 --> 00:00:13.440 All right, cool. s3 00:00:13.440 --> 00:00:14.719 Hi everyone. s4 00:00:14.719 --> 00:00:17.600 Uh, hope you all have a good time at the conference. s5 00:00:17.600 --> 00:00:19.119 Uh, I'm N from Moto. s6 00:00:19.119 --> 00:00:28.240 Uh, at Moto, we spend a lot of time thinking about GPU capacity, like where it exists, how do we make it elastic, and what kind of workload can we actually use it. s7 00:00:28.240 --> 00:00:34.160 Today I want to talk about one place where everything became like gets really interesting the IO post training. s8 00:00:34.160 --> 00:00:43.600 A lot of IO discussion right now is about algorithm and the environments sandbox PO GRPO like to call maybe low precision training maybe deterministic kernels. s9 00:00:43.600 --> 00:00:49.200 Um but when you run those experiments at a scale the problem became more physical. s10 00:00:49.200 --> 00:00:50.399 Where are the GPUs? s11 00:00:50.399 --> 00:00:51.760 Are they in the same region? s12 00:00:51.760 --> 00:00:53.600 Uh do they have fast fabric? s13 00:00:53.600 --> 00:00:55.760 Uh can we get them right now? s14 00:00:55.760 --> 00:00:58.879 Maybe the default shape of IO comput is too restrictive. s15 00:00:58.879 --> 00:01:06.320 Maybe some of the work we usually force them one cluster uh like can actually run on scattered autoscaled capacity. s16 00:01:06.320 --> 00:01:08.479 So can we do our cross globe? s17 00:01:08.479 --> 00:01:10.640 So this is my talk about many about. s18 00:01:10.640 --> 00:01:15.360 So to make this more concrete let's start with the IO loop itself. s19 00:01:15.360 --> 00:01:24.080 So in the standard uh our post training loop we can see there's a one trainer and the trainer updates the policy rollout worker or maybe people call it sampler s20 00:01:24.080 --> 00:01:26.960 use those policy to generate trajectories. s21 00:01:26.960 --> 00:01:30.320 The environment will be returning the reward and observations. s22 00:01:30.320 --> 00:01:34.320 Those trajectory will go back to trainer for the next updates. s23 00:01:34.320 --> 00:01:44.400 The important error here is the way sync in the default setup trainer and the rollout will be living in the same cluster and the the way sync will be super fast with RDMA. s24 00:01:44.400 --> 00:01:48.960 But that also couple the rollout fleet to the to the trainer cluster. s25 00:01:48.960 --> 00:01:59.920 If the rollout needs to more to have more capacity maybe no more nodes during runs um you are normally limited by the fixed size during your trainer maybe your trainer cluster. s26 00:01:59.920 --> 00:02:06.719 So the ne next question is what kind of compute shape did we actually force everything into? s27 00:02:06.719 --> 00:02:13.760 On the left side is the cathedral uh one region one faster interconnect uh many GPUs wild together. s28 00:02:13.760 --> 00:02:15.920 This is the right shape for the trainer. s29 00:02:15.920 --> 00:02:17.599 On the right it is bro. s30 00:02:17.599 --> 00:02:22.319 This is where a lot of like usable useful usable compute actually lives. s31 00:02:22.319 --> 00:02:27.040 Different providers different regions different price and different availability. s32 00:02:27.040 --> 00:02:32.080 There's still a lot of capacity out there, but it's not one perfect RDM in the island. s33 00:02:32.080 --> 00:02:33.519 This is the mismatch. s34 00:02:33.519 --> 00:02:40.080 Available computer is distributed, but the default IO loop ask one tightly coupled cluster. s35 00:02:40.080 --> 00:02:43.599 And that cluster is exactly the hot part hard to get. s36 00:02:43.599 --> 00:02:46.160 IO wants all four of these at the same time. s37 00:02:46.160 --> 00:02:50.959 Enough GPU, same region, fast fabric, and available now. s38 00:02:50.959 --> 00:02:56.000 Any of these like is manageable, but all four of them that are pretty hard to get at the same time. s39 00:02:56.000 --> 00:03:00.800 ADMIC capacity is not elastic in the way the inference capacity is elastic. s40 00:03:00.800 --> 00:03:08.159 You cannot assume you can grow the trainer cluster uh halfway through a run just because roll out wants more nodes for a trainer. s41 00:03:08.159 --> 00:03:18.400 So if the whole out loop has no live like has to live inside the the one cluster rollout inherents the hottest part capacity constraint in here. s42 00:03:18.400 --> 00:03:25.120 So that leads to the key question does the whole out loop actually need that like this kind of shape. s43 00:03:25.120 --> 00:03:29.040 So let's dive into this training is one tightly coupled job. s44 00:03:29.040 --> 00:03:39.360 Every step has collectives all reduced and the model parallel communication that part actually wants one fast fabric a RDM connected rollout is a fleet of serving jobs. s45 00:03:39.360 --> 00:03:45.840 It generates trajectories call environments or maybe tools and they will be sending back data back to the trainer. s46 00:03:45.840 --> 00:03:49.840 So cross rollout jobs there's no global or reduce. s47 00:03:49.840 --> 00:03:52.879 So the thing I want to move here is not back propagation. s48 00:03:52.879 --> 00:03:55.040 Back propagation should stay in the cluster. s49 00:03:55.040 --> 00:03:59.360 The run the rollout fleet is the one that can leave. s50 00:03:59.360 --> 00:04:10.319 More precisely, the movable unit is the rollout serving island a coherent endpoint or maybe a local group of endpoint or that they can be serving one policy version inside island. s51 00:04:10.319 --> 00:04:13.200 A large model may still be having like local parallelism. s52 00:04:13.200 --> 00:04:14.879 They can do PD segregation. s53 00:04:14.879 --> 00:04:17.519 They can have like local serving constraints. s54 00:04:17.519 --> 00:04:20.639 So across islands the dependency is much lighter right now. s55 00:04:20.639 --> 00:04:24.639 policy version in and the trajectory and the metadata out. s56 00:04:24.639 --> 00:04:29.440 So once we define the unit that way the architecture is much much more natural. s57 00:04:29.440 --> 00:04:32.560 Once we define the movable unit the architecture is very straightforward. s58 00:04:32.560 --> 00:04:41.840 In this case we just have trainer standing audic cluster and that's where the back proper and the collective go the rollout side will be fing out across the SPR. s59 00:04:41.840 --> 00:04:48.720 Each rollout island will be can be single engine will be a local serving group depending on the model and the serving topology there. s60 00:04:48.720 --> 00:05:01.199 across island there is no global or reduce that's the most important thing there the global interface is very simple the trainer send policy weight version out and the rollout sending trajectory and the metadata back s61 00:05:01.199 --> 00:05:16.240 at this point the architecture depends on one remaining link the weight update so if you want to if you want to send the full parameter like full checkpoint from disk or maybe through the network then everything is like minimless and like it will be breaking immediately s62 00:05:16.240 --> 00:05:24.880 so uh after this aggregation The things we will be discussing about like the size of like go the size of the full parameters go through the disk. s63 00:05:24.880 --> 00:05:30.240 Naively that means shipping all full checkpoints every time rollout needs a new way version. s64 00:05:30.240 --> 00:05:32.479 At this scale the checkpoint is very huge. s65 00:05:32.479 --> 00:05:42.320 So a Kim scale NVIP for checkpoint you have like 500 gigabytes normally take a minute or maybe normally take multiple minutes to hours to just do the way. s66 00:05:42.320 --> 00:05:55.199 So moving that over commodity link is might not be the smartest choice because like uh when you're doing async maybe even even fully async training you still want way updates latency to be as low as possible like within seconds. s67 00:05:55.199 --> 00:06:02.639 So the problem here is not whether rollout can leave the cluster the problem is like the full checkpoint is a wrong unit of synchronization. s68 00:06:02.639 --> 00:06:09.360 So the next question is can we keep the exact same serve version there but send a much smaller object. s69 00:06:09.360 --> 00:06:10.880 So this is the bet. s70 00:06:10.880 --> 00:06:20.400 What if less than 1% of rollout visible weights got changed from one version to another one by rollout visible weights I mean the weights in the served rollout checkpoint s71 00:06:20.400 --> 00:06:32.400 uh not the FP32 optimizer states not the atom like moments like the weights are the rollouts engine which will actually use to serve maybe let's say the FBA or maybe MVP4 format s72 00:06:32.400 --> 00:06:42.319 if that's true we do not need to shift the entire full parameter over the network we just need to shift the change the server view the precision data got different. s73 00:06:42.319 --> 00:06:45.759 The important part here is like still a bit wise reconstruction. s74 00:06:45.759 --> 00:06:49.440 The rollout engine gets the same served served version. s75 00:06:49.440 --> 00:06:53.440 You would have gotten to as sync to the full checkpoint there. s76 00:06:53.440 --> 00:07:03.120 So if this works then the link shrinks from hundreds of gigabytes to maybe hundreds of megabytes and this is something smaller enough we can just send it across like the network. s77 00:07:03.120 --> 00:07:06.400 So right now we need to justify the less than 1% like claim. s78 00:07:06.400 --> 00:07:10.720 Why would this roll of visible like weights barely change? s79 00:07:10.720 --> 00:07:13.360 Uh now we get we will get into this mechanism. s80 00:07:13.360 --> 00:07:17.039 So it's kind of small atom like step meets finite precision. s81 00:07:17.039 --> 00:07:20.720 We need we need a two uh prerequisite. s82 00:07:20.720 --> 00:07:22.560 The first one two two ingredients. s83 00:07:22.560 --> 00:07:24.400 The ingredient one is the precision. s84 00:07:24.400 --> 00:07:31.680 The optimizer may keep very high precision master weights but the next four will pass really a BF6 visible view. s85 00:07:31.680 --> 00:07:36.240 That view has final resolution around a value of magnitude theta. s86 00:07:36.240 --> 00:07:40.400 uh BF16 spacing is roughly theta over 128. s87 00:07:40.400 --> 00:07:45.039 Uh that spacing is people call it oop basically the unit in the last piece. s88 00:07:45.039 --> 00:07:49.039 Basically it's the distance between the adjacent representable BF systeming value. s89 00:07:49.039 --> 00:07:53.840 But the update only needs to cross the near surrounding boundary to be viable to be visible. s90 00:07:53.840 --> 00:07:55.919 That boundary is about half of the oop. s91 00:07:55.919 --> 00:08:00.479 So roughly like the over 256 for weight around one. s92 00:08:00.479 --> 00:08:07.919 The BF systeming loop is around uh 0.78 and the near surrounding boundary is about 00039. s93 00:08:07.919 --> 00:08:20.080 If the optimizer notched the master weight by something like smaller than that uh the BFC visible value will run back so you will not see any change from the weight perspective rollout weight perspective. s94 00:08:20.080 --> 00:08:21.520 So that's the floor. s95 00:08:21.520 --> 00:08:24.240 The second primitive there is we call push. s96 00:08:24.240 --> 00:08:35.599 So for Adam maybe add here we just like we ignore the weight decay turn the per per sorry the the per parameter updates is the learning rate times a normalized direction s97 00:08:35.599 --> 00:08:45.839 the raw gradient can be dense and can have very different magnitudes across parameters and add like device by running uh gradients statistics. s98 00:08:45.839 --> 00:08:49.200 So the per wise push is usually on the order of learning rates. s99 00:08:49.200 --> 00:08:52.880 The paper passed I sight there they prove a bound. s100 00:08:52.880 --> 00:08:56.640 The addon step is at most B times the learning rates. s101 00:08:56.640 --> 00:09:00.080 So you do not need to actually remember the exact like bound there. s102 00:09:00.080 --> 00:09:05.120 Like the important notes is the adden makes the push small and very controlled. s103 00:09:05.120 --> 00:09:09.120 So at our post training learning rates the push is very very tiny. s104 00:09:09.120 --> 00:09:11.279 So that's the push. s105 00:09:11.279 --> 00:09:24.880 Combining these two primitives now we have to we have a whole like better picture assert the value changed only if the push clear the floor the push is the addent step roughly the learning rate the floor is the nearest the BF16 s106 00:09:24.880 --> 00:09:42.160 rounding boundary roughly theta over 256 take theta equal one the BF6 boundary is about 039 a typical added step here is around 3 millions so the update is more than a thousand smaller than the boundary so the BF visible value will not change. s107 00:09:42.160 --> 00:09:46.880 This is not saying the master weights uh in is frozen forever. s108 00:09:46.880 --> 00:09:51.760 It is saying the value that rollout engine would serve does not change on this part. s109 00:09:51.760 --> 00:09:56.160 So the whole magnet is pushing is is pushed versus four. s110 00:09:56.160 --> 00:09:58.399 Let's visualize this to have better understanding. s111 00:09:58.399 --> 00:10:03.360 The x-axis is the weight magnitude and the y-axis is the update magnitude. s112 00:10:03.360 --> 00:10:04.720 First we look at the red line. s113 00:10:04.720 --> 00:10:07.200 The red diagonal is the bxis invisible boundary. s114 00:10:07.200 --> 00:10:09.279 It's like theta over 256. s115 00:10:09.279 --> 00:10:11.200 And now we look at the green bound. s116 00:10:11.200 --> 00:10:12.480 This is the atom push. s117 00:10:12.480 --> 00:10:16.959 Is this roughly around the learning rate and with a conservative upper bound? s118 00:10:16.959 --> 00:10:20.800 So now we ask like where the most point fits for most of the weights. s119 00:10:20.800 --> 00:10:23.279 The red floor is above the green push. s120 00:10:23.279 --> 00:10:28.640 Those updates exists in the master weights but they are not visible in the served BI6 view. s121 00:10:28.640 --> 00:10:32.399 This step small weights on the left can move. s122 00:10:32.399 --> 00:10:33.519 Large weight on the right. s123 00:10:33.519 --> 00:10:34.560 They will just stay the same. s124 00:10:34.560 --> 00:10:36.160 They will be absorbed it. s125 00:10:36.160 --> 00:10:37.680 This is the addon absorption. s126 00:10:37.680 --> 00:10:41.839 This is why the serve update become very sparse. s127 00:10:41.839 --> 00:10:49.519 In this case, the object will be shipped is just a diff uh is just a diff not the entire FP32 automated state. s128 00:10:49.519 --> 00:10:51.519 We first look at the rollout view. s129 00:10:51.519 --> 00:10:56.800 The weight cast or projected to the dype that the rollout engine will be serving. s130 00:10:56.800 --> 00:11:08.560 Then we will be comparing the version t minus one and the version t. Uh in this view the patch is the patch is the change the position plus like uh replacement bits and also like some metadata. s131 00:11:08.560 --> 00:11:10.480 There are multiple lossless encoding. s132 00:11:10.480 --> 00:11:12.240 People can do selective overrides. s133 00:11:12.240 --> 00:11:14.880 They can people can also do xor. s134 00:11:14.880 --> 00:11:19.279 The important part is like there are bit equivalent bit level equivalent. s135 00:11:19.279 --> 00:11:21.360 So it's not a floating point addition. s136 00:11:21.360 --> 00:11:24.320 So there's no additive uh delta drift. s137 00:11:24.320 --> 00:11:30.880 If a roller engine applied the patch correctly, it reconstruct the sync server version bitwise. s138 00:11:30.880 --> 00:11:35.440 So everything so far we explained is about the full parameter which is hot power. s139 00:11:35.440 --> 00:11:46.480 So in full parameter reinforcement learning the automizer update the whole model but the roll of view patch is sparse as the thing we just explained lower is a small for a different purpose for for a different reason. s140 00:11:46.480 --> 00:11:50.880 The base model is frozen and the adapter is small enough by construction. s141 00:11:50.880 --> 00:11:56.480 So you do not need to pu So you do not need to have the push versus flow arguments here. s142 00:11:56.480 --> 00:12:03.279 So full parameter delta is small by absorption and the lower updates are small by construction. s143 00:12:03.440 --> 00:12:05.760 Let's dive into deeper about the paper itself. s144 00:12:05.760 --> 00:12:08.800 So the paper they mentioned more stats I will be showing here. s145 00:12:08.800 --> 00:12:10.880 The measurement is not great in sposity. s146 00:12:10.880 --> 00:12:12.880 It's not optimized state sposity. s147 00:12:12.880 --> 00:12:27.519 They cast weights to BF16 compare consecutive version stness and they compare the the version bitwise and they count what did not change over time across model family the result around 99% of the time is bit identical per step s148 00:12:27.519 --> 00:12:38.800 is also it also survives stillness even when the roller lax the changes set remain very small the important part is not only the number it is the patch is lossless s149 00:12:38.800 --> 00:12:44.480 change index plus imprint value reconstruct the exact same version. s150 00:12:44.800 --> 00:12:49.760 So a common misconception there is like the work it works because all our gradients are sparse. s151 00:12:49.760 --> 00:12:50.720 They are not. s152 00:12:50.720 --> 00:12:52.880 The paper reports the gradients are dense. s153 00:12:52.880 --> 00:12:56.079 About 99% of the parameter gets non-zero gradients. s154 00:12:56.079 --> 00:12:59.040 The FP32 master update is also dense. s155 00:12:59.040 --> 00:13:00.000 It's just small. s156 00:13:00.000 --> 00:13:06.000 The main thing is like the rollout weight change is just 1% from the perspective of rollout engine. s157 00:13:06.000 --> 00:13:18.639 So far we mostly talk about BF16 but the rollout of open serving even lower precision such as like MFP4 FA and NVF4 and we can see many many model providers doing this in the roll out. s158 00:13:18.639 --> 00:13:25.120 Uh this is not a training precision the training is just like in the normal BFC although people can do QA on that. s159 00:13:25.120 --> 00:13:31.920 So for fixed stale flow format the visibility for is roughly theta over two to the mantisa plus one. s160 00:13:31.920 --> 00:13:42.399 So as you can see the F4 will be higher and FA also will be between BF60 and F4 which means in even lower precision there will be less weight changed. s161 00:13:42.959 --> 00:13:45.360 So plane flows are easy to reason about. s162 00:13:45.360 --> 00:13:54.880 So each element has its own rounding sling one value cross the floor one value they just flip and changed group scale such in4 they are they're a bit different. s163 00:13:54.880 --> 00:13:59.040 This is the regime where many low precision serving system are moving towards right now. s164 00:13:59.040 --> 00:14:11.600 For in four each ways is quantized against a shared group scale and we can apply the same rationale and also we can observe similar thing for NBF4 is hierarchical scales and uh we can see there are different s165 00:14:11.600 --> 00:14:15.519 encoding and the displaying mechanism for MPV4. s166 00:14:16.240 --> 00:14:18.959 So this is the from one internal run. s167 00:14:18.959 --> 00:14:29.440 So here's the model we serve like GM 4.7 air in FP8 and we can see in the beginning there are only like 0.15% of weights got changed in the first step where the learning is high s168 00:14:29.440 --> 00:14:40.720 and after we have more training step like when the adden is going relatively stable and you can see the entire curve goes stable when you got only one 05% s169 00:14:40.720 --> 00:14:43.920 uh weight change during each step. s170 00:14:43.920 --> 00:14:46.480 So we can see this pattern showing more generally. s171 00:14:46.480 --> 00:14:58.079 We have a different research uh we saw different research sync have a similar conclusion and we saw a different model providers such as cursor composer 2 mi they all using add in the post training s172 00:14:58.079 --> 00:15:02.560 at at this point assume we can produce exact rollout weights version cheaply. s173 00:15:02.560 --> 00:15:07.120 The next question is how do we do this in practice? s174 00:15:07.120 --> 00:15:10.160 So from sparse delta to to our cross globe. s175 00:15:10.160 --> 00:15:22.720 How do we do this with elastic rollout engines and also explicit stness this is the whole shape the trainer saying a cluster after it got updated it publish immutable s176 00:15:22.720 --> 00:15:34.639 rollout based version to a shared bulletboard rollout engine leave outside of the training cluster which means they don't need to be RDM connected with the trainer they can be in different region or different providers. s177 00:15:34.639 --> 00:15:36.160 This is also request lane. s178 00:15:36.160 --> 00:15:39.600 You can see a request does not just say give me the completion. s179 00:15:39.600 --> 00:15:48.160 You will also say which version you will be sending request to and which version you will be accepting and the response will come back with the version and also s180 00:15:48.160 --> 00:15:52.959 like exact same information as if as if they are in the same cluster as the trainer. s181 00:15:52.959 --> 00:16:00.079 They will be returning tokens log prop like router replay information and many more metadata. s182 00:16:00.240 --> 00:16:10.639 The trainer writes immutable version to the broad after optimizer state uh after optimizer step engine pool version and the materialize it locally in the checkpoint layout. s183 00:16:10.639 --> 00:16:17.680 So they can just like serve directly the artifact defined version the engine choose how to how to load and shot it. s184 00:16:17.680 --> 00:16:31.279 It does not change it does it does not choose a different server version since it will be displaying in a hugging HF will be saved tensor format which is accepted widely by many rollout engines such as sjet and VLM. s185 00:16:31.279 --> 00:16:40.800 So we can support any compatible back end attention back end back end different parallelism compatible serving these type and any compatible GPUs there. s186 00:16:40.800 --> 00:16:43.040 We can talk more about the scar itself. s187 00:16:43.040 --> 00:16:46.560 The scikar is basically what makes a normal roller engine version aware. s188 00:16:46.560 --> 00:16:52.800 If the version is already at acceptable commit version, the psychar just proxy the request. s189 00:16:52.800 --> 00:16:58.240 If the engine is behind but can they can catch up the psychar just apply the missing transition. s190 00:16:58.240 --> 00:17:04.480 If they cannot get there the the psychar just simply return not ready. s191 00:17:04.480 --> 00:17:13.919 So this will be supporting elasticity rollout and any idle GPU can just be used with design with this design to support this aggregated roll out. s192 00:17:13.919 --> 00:17:20.480 So this is more like a system latency analyst incluster way syncing is fast because they have RDMA. s193 00:17:20.480 --> 00:17:34.080 Uh a full checkpoint across regions through network is pretty slow and but if we use the delta if we use exactly what we described previously we can decrease the number of like transfer s194 00:17:34.080 --> 00:17:37.520 transfer size from like 500 gigabytes to 5 500 megabytes. s195 00:17:37.520 --> 00:17:41.600 So it will be like extremely fast in seconds. s196 00:17:41.840 --> 00:17:44.320 So everything above was very general protocol. s197 00:17:44.320 --> 00:17:49.919 Stitch is one of very concrete implementation from model that we imple implementing everything above. s198 00:17:49.919 --> 00:17:59.840 So on the on the trainer side stitch publish uh what defines a rollout weights version and on the contract side you will be pulling out and record everything on the bulletboard s199 00:17:59.840 --> 00:18:08.000 and the only rollout side you will be pulling the latest weights and it start doing way sync across different region and different providers. s200 00:18:08.000 --> 00:18:13.200 So stitch itself is a very framework agnostic about trainer and engine and also transport. s201 00:18:13.200 --> 00:18:17.840 It's very async first and also agent first agentic first. s202 00:18:17.840 --> 00:18:22.240 By doing this we can have rollouts uh engines like autoscale globally. s203 00:18:22.240 --> 00:18:27.840 Each one self-s sync it weights serve accept version and return rollout metadata. s204 00:18:27.840 --> 00:18:33.679 That means scattered inference cap capacity became one elastic rollout fleet. s205 00:18:33.679 --> 00:18:38.480 Instead of being limited by the trainer cluster rollout can be the global pool. s206 00:18:38.480 --> 00:18:42.880 So inference capacity can now become our capacity. s207 00:18:43.919 --> 00:18:48.480 Last section we have some uh ongoing explorations. s208 00:18:48.640 --> 00:18:56.960 So we can see a lot of model providers such as moonshot and also deepc4 they have like they're adopting muan in their post training. s209 00:18:56.960 --> 00:19:02.559 Um does the spicy still hold for muan because a lot of thing we discussed previously only for Adam. s210 00:19:02.559 --> 00:19:15.280 Second question is async RL at the scale right now we can use the compute across the globe then how like how scalable is the fully async RL this is a very open-ended question there s211 00:19:15.280 --> 00:19:32.480 and uh third question is like does it generalize pass because like we have pre-training mid training and SFT like do we have can we apply same paradigm there last but not least we are working on some very hot problem and come work with us you can check the link there model s212 00:19:32.480 --> 00:19:34.720 Thank you.