WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=c1hGBoWw20A

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/c1hGBoWw20A.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.960 --> 00:00:14.120
Hello everyone.

s2
00:00:14.120 --> 00:00:21.480
Uh Thank you for coming and uh I'll start the talk now.

s3
00:00:21.480 --> 00:00:33.280
So This talk is uh around a paper that I did um which is very simple.

s4
00:00:33.280 --> 00:00:36.400
The proposition is very clear.

s5
00:00:36.400 --> 00:00:51.640
It's basically two lines of algebra that make uh the RMS norm layer in transformers cheaper, quicker, and kind of improve improve it as like a layer in the transformer

s6
00:00:51.640 --> 00:00:53.520
architecture.

s7
00:00:53.520 --> 00:01:01.000
Similar to how layer norm once used to be the standard and then it was substituted by RMS norm.

s8
00:01:01.000 --> 00:01:06.239
This follows along um this way of thinking.

s9
00:01:06.440 --> 00:01:18.400
And I got the chance to kind of meet some people from the open source world and um I co-authored this paper together with uh Niels Graf who was the

s10
00:01:18.400 --> 00:01:20.520
kind of the creator of this.

s11
00:01:20.520 --> 00:01:24.080
And the work follows from there.

s12
00:01:24.080 --> 00:01:26.680
So this is uh presented on archive.

s13
00:01:26.680 --> 00:01:29.600
You can have a look, read it, test it out.

s14
00:01:29.600 --> 00:01:32.520
There is a repo as well.

s15
00:01:32.680 --> 00:01:40.960
And the concept the let's say the idea and the way of thinking it's easiest to explain with maybe flash attention.

s16
00:01:40.960 --> 00:01:56.080
So in a similar way of how um flash attention kind of weights until there's a multiplication and tries to limit this uh communications between memory so that the whole process is faster.

s17
00:01:56.080 --> 00:02:14.200
This is kind of a similar thought along those lines and it does certain improvements that make the RMS norm process much quicker and in in effect improve the whole

s18
00:02:14.200 --> 00:02:16.040
transformer.

s19
00:02:16.040 --> 00:02:24.840
And one question is okay, why RMS norm since that layer does almost none of the math?

s20
00:02:24.840 --> 00:02:26.440
And that's true.

s21
00:02:26.440 --> 00:02:31.520
So the share of the kind of math portion if you look at it is quite small.

s22
00:02:31.520 --> 00:02:40.720
However, the clock time or wall time as they say is quite big and for example in one decode step.

s23
00:02:40.720 --> 00:02:47.200
So right when like inference is performed the RMS norm can be started like 33 times.

s24
00:02:47.200 --> 00:02:49.680
Of course, it depends on the model and so on.

s25
00:02:49.680 --> 00:02:54.200
In the paper, you have the specific models and how this was tested.

s26
00:02:54.200 --> 00:03:06.720
Um and the question is how this can be improved and how this wait for the matrix multiplication can be kind of avoided.

s27
00:03:07.040 --> 00:03:20.120
And the reason why this is slow is because the GPUs are not slow or bad at math, but they're bad at everything else around the actual math.

s28
00:03:20.120 --> 00:03:24.120
So that means starting the work, the actual work.

s29
00:03:24.120 --> 00:03:42.560
So for example, starting the process as it happens in some of the experiments 33 times that takes a long time and for example, fusing um each normalization into the matrix multiplication

s30
00:03:42.560 --> 00:03:44.280
can help avoid this.

s31
00:03:44.280 --> 00:03:55.160
Also doing weight folding can help in kind of moving data between memory and um that's a process that's also slow for GPUs.

s32
00:03:55.160 --> 00:03:56.640
And also waiting.

s33
00:03:56.640 --> 00:04:06.640
Uh so, for example, deferring the division that's done in the RMS norm layer is also a way to avoid this waiting step.

s34
00:04:06.640 --> 00:04:19.720
So, basically, what this paper does is it improves all these three aspects by doing a few algebraic tricks in the way RMS norm is computed.

s35
00:04:19.720 --> 00:04:22.000
That's it.

s36
00:04:22.240 --> 00:04:25.760
And math-wise, these are the tricks.

s37
00:04:25.760 --> 00:04:29.360
Uh it's mainly around the first two propositions.

s38
00:04:29.360 --> 00:04:31.800
One is weightless normalization.

s39
00:04:31.800 --> 00:04:33.640
Uh you can see that here.

s40
00:04:33.640 --> 00:04:37.840
Um and deferred normalization, so um that's the second one.

s41
00:04:37.840 --> 00:04:45.760
And now in more newer architectures, there is a situation where um RMS can be kind of can appear twice.

s42
00:04:45.760 --> 00:04:48.880
Uh for example, in Gemma 4, this happens.

s43
00:04:48.880 --> 00:04:53.040
So, canceling the pre-normalization also works.

s44
00:04:53.040 --> 00:04:57.720
Um and all of this is algebraically proven in the paper.

s45
00:04:58.400 --> 00:05:09.640
And the first proposition is this where kind of the the gain and the weight fold folded to one matrix W, uh you can see here with an asterisk.

s46
00:05:09.640 --> 00:05:20.640
And that is computed offline, similar to how maybe in flash attention, you compute some stuff on the side so that there is no uh communication between memory all the time.

s47
00:05:20.640 --> 00:05:26.600
So, this is one step that's kind of um done, this weight folding.

s48
00:05:26.600 --> 00:05:35.520
And the other step is um deferring um the scalar the scalar divide of the matmul so that they can be done in parallel.

s49
00:05:35.520 --> 00:05:42.080
So, in a normal case, you would have to compute once, then wait, and compute again.

s50
00:05:42.080 --> 00:05:47.560
In this case, the idea is to kind of split this so that it can be parallelized.

s51
00:05:48.120 --> 00:06:02.400
And the third one, which is kind of a version of this is that um there is kind of if there are two um because this is scale invariant, one of them can be dropped and this still works.

s52
00:06:02.400 --> 00:06:09.400
And this is applicable to newer models um that can have this architecture and implementation.

s53
00:06:10.360 --> 00:06:20.480
So, in order to make this happen in real life, especially this proposition number two, um so for this one, for example, it's easy.

s54
00:06:20.480 --> 00:06:22.480
There is a repo called Transformer Tricks.

s55
00:06:22.480 --> 00:06:26.400
You can just apply this to any model and it works.

s56
00:06:26.400 --> 00:06:29.480
But in order to do this, there is some kernel work.

s57
00:06:29.480 --> 00:06:33.360
So, it's not as straightforward to do.

s58
00:06:33.480 --> 00:06:43.160
So, in order for me to do that, I was implementing this and I came out with this experiment once.

s59
00:06:43.160 --> 00:06:54.400
So, it looks okay in general, where it's like, "Okay, the prompt is the Transformer architecture revolutionally revolutionized NLP because and then there is some kind of expected output."

s60
00:06:54.400 --> 00:07:01.840
But in the output I got, I saw this repetition and one-step lag, as you can see here, the word because appears again.

s61
00:07:01.840 --> 00:07:10.880
And there was something happening with the GPU streams and I was trying to figure out what was happening.

s62
00:07:10.880 --> 00:07:18.680
And I was getting this one-step lag and kind of um outputs that were from the past in a way.

s63
00:07:18.680 --> 00:07:32.720
Um and in debugging all of this, I realized that um in the process of building something like this, so as I explained the proposition two or deferring these two operations,

s64
00:07:32.720 --> 00:07:35.640
um in CUDA, you can do two things.

s65
00:07:35.640 --> 00:07:47.040
You can do like tensor cores that do one part of the matrix multiplication and you can do CUDA cores that kind of run stuff like element-wise operations, reductions, square roots, and so on.

s66
00:07:47.040 --> 00:07:58.360
So, the idea was to do this in parallel and get the benefit of what I was explaining in the paper to actually test out this concept.

s67
00:07:58.360 --> 00:08:00.360
So, this is how it was supposed to look like.

s68
00:08:00.360 --> 00:08:12.280
So, there is if you do things sequentially, there is this idle waiting time when you when the vector unit computes the RMS and scaling, and then there is a matrix multiplication.

s69
00:08:12.280 --> 00:08:18.760
So, the idea was okay, with flash norm, which is the technique in the paper, you're supposed to do those both in parallel.

s70
00:08:18.760 --> 00:08:24.000
So, the matrix unit computes the matmul and the vector unit computes the RMS.

s71
00:08:24.000 --> 00:08:26.800
So, in that way you save uh time.

s72
00:08:26.800 --> 00:08:31.680
However, you cannot just do this in Python, you have to go a bit lower.

s73
00:08:31.680 --> 00:08:36.479
And I did that with CUDA code like this.

s74
00:08:36.479 --> 00:08:42.159
And this looked in general okay at my uh at that time.

s75
00:08:42.159 --> 00:08:47.800
However, um I realized that I did something slightly wrong.

s76
00:08:47.800 --> 00:08:58.200
And that thing was that the join in the end, where you're supposed to join the two streams, was implicit in my case.

s77
00:08:58.200 --> 00:09:11.040
And when I tested this out, the unit tests worked, the quality seemed similar, like perplexity testing, and so on, because it's just like um similar generation, but over long generation, I was able to see this problem.

s78
00:09:11.040 --> 00:09:15.160
So, I had no idea what this was.

s79
00:09:15.160 --> 00:09:29.600
And the reason was that when I was doing this uh implicit um join, basically, one of the streams hadn't finished the work, so I got race conditions that kind of read the past

s80
00:09:29.600 --> 00:09:32.240
from the unfinished matrix multiplication.

s81
00:09:32.240 --> 00:09:44.600
So, the idea that I had to fix this was um around the fact that I had to be explicit about the join and wait until one of the operations is finished so that I'm certain

s82
00:09:44.600 --> 00:09:48.640
that when I join I'm not reading from the past.

s83
00:09:48.640 --> 00:09:54.360
So, that was the realization um in this exploration of CUDA streams.

s84
00:09:54.360 --> 00:09:57.280
And this is how I had things done.

s85
00:09:57.280 --> 00:10:00.000
So, the join was implicit.

s86
00:10:00.000 --> 00:10:06.000
So, the post um scale read like an old uh buffer value.

s87
00:10:06.000 --> 00:10:20.240
And how this is fixed is with this where basically you need to mark the end of the matrix multiplication, then mark the end of the RMS, and then post scale wait for

s88
00:10:20.240 --> 00:10:24.560
the first stream and then wait for the second stream.

s89
00:10:24.560 --> 00:10:33.720
And that fixed the bug and made kind of the paper work and the model speak forwards instead of backwards.

s90
00:10:33.720 --> 00:10:40.560
And that was the cool maybe academic perspective, but I also wanted to try things, right?

s91
00:10:40.560 --> 00:10:47.360
Deploy this, test it out, see how I can make it work um in maybe a more production setting.

s92
00:10:47.360 --> 00:10:50.160
And you can also read the paper and see all the tests.

s93
00:10:50.160 --> 00:10:57.240
Um some are done most are done around llama models, but like this works for other architectures as well.

s94
00:10:57.240 --> 00:11:10.200
Um so, what you can do for this specific paper is um for example, the weight folding that I explained the pre-position one, you can just do it with um some code in the repo

s95
00:11:10.200 --> 00:11:13.680
that's like flash you say flashify and it does that.

s96
00:11:13.680 --> 00:11:22.160
However, with this second thing that I mentioned, you need to do a bit of kernel work if you want to do that uh like I explained in my example.

s97
00:11:22.160 --> 00:11:29.800
And these are some results that are based on llama models and there are different kind of details that you can have a look at as well as well.

s98
00:11:29.800 --> 00:11:35.960
Like what happens if you do only the third normalization, what happens if you do a full fused kernel.

s99
00:11:35.960 --> 00:11:46.680
Um so there are a lot of experiments of going lower here to test all the propositions, and this have been our results um in different, let's say, levels of um scrutiny

s100
00:11:46.680 --> 00:11:48.800
and detail.

s101
00:11:48.800 --> 00:11:54.960
But even the simple one with like weight folding um shows some improvement.

s102
00:11:54.960 --> 00:11:59.520
And this also works with like the day-to-day tools that you use in the models.

s103
00:11:59.520 --> 00:12:05.240
It's not like you have to reinvent the wheel or, you know, do things from scratch.

s104
00:12:05.240 --> 00:12:12.800
So it works with uh torch compile um because the it's kind of like a new checkpoint, and that's it.

s105
00:12:12.800 --> 00:12:18.480
Flash attention does similar tricks at a different layer, and also it works with quantized models.

s106
00:12:18.480 --> 00:12:27.280
So it's totally cool to actually apply this, and you can get a model that has this cool new normalization layer.

s107
00:12:27.280 --> 00:12:34.280
And where you can get this um details and code to actually run this is this transformer tricks repo.

s108
00:12:34.280 --> 00:12:42.320
So uh it has different algebraic tricks like I explained, as well as this paper that I mentioned.

s109
00:12:42.800 --> 00:12:54.640
And also there is the GitHub uh not the GitHub, but the Hugging Face uh model repo where I've done this with some models, and you can have a Hugging Face link to that model and test it out.

s110
00:12:54.640 --> 00:13:00.000
Um and what you also can do with this Hugging Face models is to deploy them in production.

s111
00:13:00.000 --> 00:13:11.520
And so when I was thinking about doing this, um I realized that, okay, now that, let's say, the science is done and there is a link to a Hugging Face model,

s112
00:13:11.520 --> 00:13:24.040
um Superlinked's uh inference engine was a cool way to actually deploy any um uh Hugging Face model, and we've done this at hackathons where people would bring like a custom Hugging Face

s113
00:13:24.040 --> 00:13:35.000
model or checkpoint that they have with their fine-tuned stuff, and you can test out like even if you have some version of this algebraic tricks that you want to improve a model and

s114
00:13:35.000 --> 00:13:47.600
test on test your own research ideas, you can actually try that out and have a deployed version on a cluster of this model and not have to worry about this glue code around

s115
00:13:47.600 --> 00:13:49.120
deploying models.

s116
00:13:49.120 --> 00:13:51.720
So, that's um pretty cool.

s117
00:13:51.720 --> 00:14:05.000
And the the point is that if you have the full cluster open source and the model inference open source, you can actually test out this kind of maybe more novel research ideas where

s118
00:14:05.000 --> 00:14:18.360
if you want to do kernel manipulation or flash norm and things like that, it's much more difficult to do that do this at a rented endpoint where you don't own the inference.

s119
00:14:18.360 --> 00:14:29.240
It's you want something that's portable and flexible to actually allow you to do this stuff, but it's also production ready enough so that you can test things out at scale.

s120
00:14:29.839 --> 00:14:43.560
And you can, for example, use site to combine this with other models like, as you can see in the top left, there is you can have this flashified models with different other models to do agentic tasks if you want

s121
00:14:43.560 --> 00:14:48.000
and kind of do that end-to-end bigger use case.

s122
00:14:48.000 --> 00:14:58.200
And the way site works is this production cluster helps you deploy the models, so you can have a look at site's repo as well for more details on this.

s123
00:14:58.200 --> 00:15:11.440
Um and also there is a smarter queuing mechanism that helps you, especially if you work with smaller models cuz when doing the flash norm stuff, I worked with like smaller llama models and also with small agents from hugging face.

s124
00:15:11.440 --> 00:15:25.920
So, having a way to deploy smaller models that can also work on like the same GPU so that you don't have to spend your money on GPU cost, but actually kind of

s125
00:15:25.920 --> 00:15:28.480
switch models around, especially smaller models.

s126
00:15:28.480 --> 00:15:30.760
It was quite useful.

s127
00:15:30.760 --> 00:15:40.720
And you can also control the model configs through an API as well as the cluster, which is also pretty convenient without having like an infra guy supporting you in your open source research.

s128
00:15:40.720 --> 00:15:43.080
So, that's cool as well.

s129
00:15:43.080 --> 00:15:51.360
Um, and you own your cloud, which is useful if you want open weights, open models, open source.

s130
00:15:51.520 --> 00:15:59.280
And there's also like a catalog that Sci has of different models, um, not just the ones I mentioned, but you can have a look.

s131
00:15:59.280 --> 00:16:04.080
There's also re-ranking embedding models if you're building something along those lines.

s132
00:16:04.080 --> 00:16:18.920
And with that I'm kind of finishing this story of my research journey where I co-authored this paper, um, around the technique that improves the transformer, but also found a way kind of to bring this to,

s133
00:16:18.920 --> 00:16:24.400
let's say, production and test it out and find a way to play around with this open source models.

s134
00:16:24.400 --> 00:16:30.760
And feel free to contact me on LinkedIn, maybe if you have any questions or contributions.

s135
00:16:30.760 --> 00:16:36.560
A lot of this stuff that I've mentioned, like some of them are PRs on like vLLM or on Hugging Face.

s136
00:16:36.560 --> 00:16:38.240
You might find them all around.

s137
00:16:38.240 --> 00:16:40.560
You can also see the check out the paper.

s138
00:16:40.560 --> 00:16:43.520
That's the archive link that you have there.

s139
00:16:43.520 --> 00:16:47.080
Um, and you also have the Sci repo and my LinkedIn.

s140
00:16:47.080 --> 00:16:53.120
Um, so thank you very much for attending.

s141
00:16:53.296 --> 00:16:53.951
[applause]

s142
00:16:53.951 --> 00:16:55.951
[cheering]

s143
00:16:57.839 --> 00:16:59.440
And you can catch me for questions.

s144
00:16:59.440 --> 00:17:02.640
We'll be here, close by.

s145
00:17:16.221 --> 00:17:18.221
[music]