Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli

AI Engineer · 17 min · 145 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Hello everyone.
00:14Uh Thank you for coming and uh I'll start the talk now.
00:21So This talk is uh around a paper that I did um which is very simple.
00:33The proposition is very clear.
00:36It's basically two lines of algebra that make uh the RMS norm layer in transformers cheaper, quicker, and kind of improve improve it as like a layer in the transformer
00:51architecture.
00:53Similar to how layer norm once used to be the standard and then it was substituted by RMS norm.
01:01This follows along um this way of thinking.
01:06And I got the chance to kind of meet some people from the open source world and um I co-authored this paper together with uh Niels Graf who was the
01:18kind of the creator of this.
01:20And the work follows from there.
01:24So this is uh presented on archive.
01:26You can have a look, read it, test it out.
01:29There is a repo as well.
01:32And the concept the let's say the idea and the way of thinking it's easiest to explain with maybe flash attention.
01:40So in a similar way of how um flash attention kind of weights until there's a multiplication and tries to limit this uh communications between memory so that the whole process is faster.
01:56This is kind of a similar thought along those lines and it does certain improvements that make the RMS norm process much quicker and in in effect improve the whole
02:14transformer.
02:16And one question is okay, why RMS norm since that layer does almost none of the math?
02:24And that's true.
02:26So the share of the kind of math portion if you look at it is quite small.
02:31However, the clock time or wall time as they say is quite big and for example in one decode step.
02:40So right when like inference is performed the RMS norm can be started like 33 times.
02:47Of course, it depends on the model and so on.
02:49In the paper, you have the specific models and how this was tested.
02:54Um and the question is how this can be improved and how this wait for the matrix multiplication can be kind of avoided.
03:07And the reason why this is slow is because the GPUs are not slow or bad at math, but they're bad at everything else around the actual math.
03:20So that means starting the work, the actual work.
03:24So for example, starting the process as it happens in some of the experiments 33 times that takes a long time and for example, fusing um each normalization into the matrix multiplication
03:42can help avoid this.
03:44Also doing weight folding can help in kind of moving data between memory and um that's a process that's also slow for GPUs.
03:55And also waiting.
03:56Uh so, for example, deferring the division that's done in the RMS norm layer is also a way to avoid this waiting step.
04:06So, basically, what this paper does is it improves all these three aspects by doing a few algebraic tricks in the way RMS norm is computed.
04:19That's it.
04:22And math-wise, these are the tricks.
04:25Uh it's mainly around the first two propositions.
04:29One is weightless normalization.
04:31Uh you can see that here.
04:33Um and deferred normalization, so um that's the second one.
04:37And now in more newer architectures, there is a situation where um RMS can be kind of can appear twice.
04:45Uh for example, in Gemma 4, this happens.
04:48So, canceling the pre-normalization also works.
04:53Um and all of this is algebraically proven in the paper.
04:58And the first proposition is this where kind of the the gain and the weight fold folded to one matrix W, uh you can see here with an asterisk.
05:09And that is computed offline, similar to how maybe in flash attention, you compute some stuff on the side so that there is no uh communication between memory all the time.
05:20So, this is one step that's kind of um done, this weight folding.
05:26And the other step is um deferring um the scalar the scalar divide of the matmul so that they can be done in parallel.
05:35So, in a normal case, you would have to compute once, then wait, and compute again.
05:42In this case, the idea is to kind of split this so that it can be parallelized.
05:48And the third one, which is kind of a version of this is that um there is kind of if there are two um because this is scale invariant, one of them can be dropped and this still works.
06:02And this is applicable to newer models um that can have this architecture and implementation.
06:10So, in order to make this happen in real life, especially this proposition number two, um so for this one, for example, it's easy.
06:20There is a repo called Transformer Tricks.
06:22You can just apply this to any model and it works.
06:26But in order to do this, there is some kernel work.
06:29So, it's not as straightforward to do.
06:33So, in order for me to do that, I was implementing this and I came out with this experiment once.
06:43So, it looks okay in general, where it's like, "Okay, the prompt is the Transformer architecture revolutionally revolutionized NLP because and then there is some kind of expected output."
06:54But in the output I got, I saw this repetition and one-step lag, as you can see here, the word because appears again.
07:01And there was something happening with the GPU streams and I was trying to figure out what was happening.
07:10And I was getting this one-step lag and kind of um outputs that were from the past in a way.
07:18Um and in debugging all of this, I realized that um in the process of building something like this, so as I explained the proposition two or deferring these two operations,
07:32um in CUDA, you can do two things.
07:35You can do like tensor cores that do one part of the matrix multiplication and you can do CUDA cores that kind of run stuff like element-wise operations, reductions, square roots, and so on.
07:47So, the idea was to do this in parallel and get the benefit of what I was explaining in the paper to actually test out this concept.
07:58So, this is how it was supposed to look like.
08:00So, there is if you do things sequentially, there is this idle waiting time when you when the vector unit computes the RMS and scaling, and then there is a matrix multiplication.
08:12So, the idea was okay, with flash norm, which is the technique in the paper, you're supposed to do those both in parallel.
08:18So, the matrix unit computes the matmul and the vector unit computes the RMS.
08:24So, in that way you save uh time.
08:26However, you cannot just do this in Python, you have to go a bit lower.
08:31And I did that with CUDA code like this.
08:36And this looked in general okay at my uh at that time.
08:42However, um I realized that I did something slightly wrong.
08:47And that thing was that the join in the end, where you're supposed to join the two streams, was implicit in my case.
08:58And when I tested this out, the unit tests worked, the quality seemed similar, like perplexity testing, and so on, because it's just like um similar generation, but over long generation, I was able to see this problem.
09:11So, I had no idea what this was.
09:15And the reason was that when I was doing this uh implicit um join, basically, one of the streams hadn't finished the work, so I got race conditions that kind of read the past
09:29from the unfinished matrix multiplication.
09:32So, the idea that I had to fix this was um around the fact that I had to be explicit about the join and wait until one of the operations is finished so that I'm certain
09:44that when I join I'm not reading from the past.
09:48So, that was the realization um in this exploration of CUDA streams.
09:54And this is how I had things done.
09:57So, the join was implicit.
10:00So, the post um scale read like an old uh buffer value.
10:06And how this is fixed is with this where basically you need to mark the end of the matrix multiplication, then mark the end of the RMS, and then post scale wait for
10:20the first stream and then wait for the second stream.
10:24And that fixed the bug and made kind of the paper work and the model speak forwards instead of backwards.
10:33And that was the cool maybe academic perspective, but I also wanted to try things, right?
10:40Deploy this, test it out, see how I can make it work um in maybe a more production setting.
10:47And you can also read the paper and see all the tests.
10:50Um some are done most are done around llama models, but like this works for other architectures as well.
10:57Um so, what you can do for this specific paper is um for example, the weight folding that I explained the pre-position one, you can just do it with um some code in the repo
11:10that's like flash you say flashify and it does that.
11:13However, with this second thing that I mentioned, you need to do a bit of kernel work if you want to do that uh like I explained in my example.
11:22And these are some results that are based on llama models and there are different kind of details that you can have a look at as well as well.
11:29Like what happens if you do only the third normalization, what happens if you do a full fused kernel.
11:35Um so there are a lot of experiments of going lower here to test all the propositions, and this have been our results um in different, let's say, levels of um scrutiny
11:46and detail.
11:48But even the simple one with like weight folding um shows some improvement.
11:54And this also works with like the day-to-day tools that you use in the models.
11:59It's not like you have to reinvent the wheel or, you know, do things from scratch.
12:05So it works with uh torch compile um because the it's kind of like a new checkpoint, and that's it.
12:12Flash attention does similar tricks at a different layer, and also it works with quantized models.
12:18So it's totally cool to actually apply this, and you can get a model that has this cool new normalization layer.
12:27And where you can get this um details and code to actually run this is this transformer tricks repo.
12:34So uh it has different algebraic tricks like I explained, as well as this paper that I mentioned.
12:42And also there is the GitHub uh not the GitHub, but the Hugging Face uh model repo where I've done this with some models, and you can have a Hugging Face link to that model and test it out.
12:54Um and what you also can do with this Hugging Face models is to deploy them in production.
13:00And so when I was thinking about doing this, um I realized that, okay, now that, let's say, the science is done and there is a link to a Hugging Face model,
13:11um Superlinked's uh inference engine was a cool way to actually deploy any um uh Hugging Face model, and we've done this at hackathons where people would bring like a custom Hugging Face
13:24model or checkpoint that they have with their fine-tuned stuff, and you can test out like even if you have some version of this algebraic tricks that you want to improve a model and
13:35test on test your own research ideas, you can actually try that out and have a deployed version on a cluster of this model and not have to worry about this glue code around
13:47deploying models.
13:49So, that's um pretty cool.
13:51And the the point is that if you have the full cluster open source and the model inference open source, you can actually test out this kind of maybe more novel research ideas where
14:05if you want to do kernel manipulation or flash norm and things like that, it's much more difficult to do that do this at a rented endpoint where you don't own the inference.
14:18It's you want something that's portable and flexible to actually allow you to do this stuff, but it's also production ready enough so that you can test things out at scale.
14:29And you can, for example, use site to combine this with other models like, as you can see in the top left, there is you can have this flashified models with different other models to do agentic tasks if you want
14:43and kind of do that end-to-end bigger use case.
14:48And the way site works is this production cluster helps you deploy the models, so you can have a look at site's repo as well for more details on this.
14:58Um and also there is a smarter queuing mechanism that helps you, especially if you work with smaller models cuz when doing the flash norm stuff, I worked with like smaller llama models and also with small agents from hugging face.
15:11So, having a way to deploy smaller models that can also work on like the same GPU so that you don't have to spend your money on GPU cost, but actually kind of
15:25switch models around, especially smaller models.
15:28It was quite useful.
15:30And you can also control the model configs through an API as well as the cluster, which is also pretty convenient without having like an infra guy supporting you in your open source research.
15:40So, that's cool as well.
15:43Um, and you own your cloud, which is useful if you want open weights, open models, open source.
15:51And there's also like a catalog that Sci has of different models, um, not just the ones I mentioned, but you can have a look.
15:59There's also re-ranking embedding models if you're building something along those lines.
16:04And with that I'm kind of finishing this story of my research journey where I co-authored this paper, um, around the technique that improves the transformer, but also found a way kind of to bring this to,
16:18let's say, production and test it out and find a way to play around with this open source models.
16:24And feel free to contact me on LinkedIn, maybe if you have any questions or contributions.
16:30A lot of this stuff that I've mentioned, like some of them are PRs on like vLLM or on Hugging Face.
16:36You might find them all around.
16:38You can also see the check out the paper.
16:40That's the archive link that you have there.
16:43Um, and you also have the Sci repo and my LinkedIn.
16:47Um, so thank you very much for attending.
16:53[applause]
16:53[cheering]
16:57And you can catch me for questions.
16:59We'll be here, close by.
17:16[music]