WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=y2W4FNAuPEA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/y2W4FNAuPEA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:13.599 --> 00:00:16.000
Uh so good afternoon everyone.

s2
00:00:16.000 --> 00:00:20.320
Um my name is Hershel Jan and he is Tanisha.

s3
00:00:20.320 --> 00:00:26.640
Uh and we would like to welcome you all in this two hours workshop on the LLM inference.

s4
00:00:26.640 --> 00:00:41.200
Uh so the goal of this workshop is to understand this domain from the first principles uh dive deeper into it and like understand what's going on throughout the industry.

s5
00:00:41.440 --> 00:00:44.480
Uh a bit of background about us.

s6
00:00:44.480 --> 00:00:48.160
So I am a senior software engineer at Audible.

s7
00:00:48.160 --> 00:01:02.000
uh have been building MLA data platforms for the past five years and on the sides I have been writing this opensource handbook on LLM inference and Tanme he is the

s8
00:01:02.000 --> 00:01:21.920
senior quantitative modeler at XAN cup bank corporation he recently completed his PhD and he has been actively doing research in the agent verifiers and the world models Uh so a quick show of hands here.

s9
00:01:21.920 --> 00:01:26.640
Uh Vu here is like brand new to the LLM inference.

s10
00:01:27.200 --> 00:01:28.320
Okay, great.

s11
00:01:28.320 --> 00:01:33.119
And Vu here has like deployed these models in production.

s12
00:01:33.119 --> 00:01:34.720
They have been tuning it.

s13
00:01:34.720 --> 00:01:38.320
They have been serving the production traffic.

s14
00:01:38.720 --> 00:01:41.119
Okay, great.

s15
00:01:41.119 --> 00:01:47.200
So this workshop is targeted towards the beginner and the intermediate level.

s16
00:01:47.200 --> 00:01:50.960
U and all of the slides and exercises they are in the repo.

s17
00:01:50.960 --> 00:01:53.840
I will share that soon.

s18
00:01:55.119 --> 00:01:58.640
Here is the quick agenda for the workshop.

s19
00:01:58.640 --> 00:02:00.640
We will start with the problem statement.

s20
00:02:00.640 --> 00:02:06.399
We will try to understand few of the pain points around LLM inference.

s21
00:02:06.399 --> 00:02:12.160
uh then we understand what causes those pain points and build our foundations from there.

s22
00:02:12.160 --> 00:02:21.360
Then we will dive into like two kind of the optimizations that we do like the model optimizations and the serving optimizations.

s23
00:02:21.360 --> 00:02:38.480
uh and then we start learning about different serving engines that are available to deploy our LLM inference solutions in production and we will showcase some benchmarks and the decision chart on like which engine to use.

s24
00:02:40.239 --> 00:02:40.720
Cool.

s25
00:02:40.720 --> 00:02:47.519
So to understand the pain points first we need to know what is like LLM inference.

s26
00:02:47.519 --> 00:02:51.120
So, and probably a lot of us already know this.

s27
00:02:51.120 --> 00:03:04.080
Um, but yeah, anything that you ask your AI to do like whether it be generate a video, audio, analyze any text, uh, analyze your medical reports or like your tax bills,

s28
00:03:04.080 --> 00:03:08.080
all of that is like an LLM inference.

s29
00:03:08.080 --> 00:03:14.640
And this market is like approximately $23 billion today.

s30
00:03:15.040 --> 00:03:29.440
uh semi analysis recently shared that if you want to model like a Google search queries with LLMs, you need like a profit drain of like $36 billion and query cost

s31
00:03:29.440 --> 00:03:35.840
has to be less than 0.5 cents to keep your search business profitable.

s32
00:03:35.840 --> 00:03:48.560
On the other hand, the business insider mentioned like your AI has to be put on diet and everyone has to start auditing and budgeting their token usage and all of this is happening.

s33
00:03:48.560 --> 00:03:48.959
Why?

s34
00:03:48.959 --> 00:04:03.760
Because your hardware is limited, compute is expensive, your inference is expensive and with the growing need of like more and more AI usage, this inference cost is rising more and more.

s35
00:04:03.760 --> 00:04:11.599
So this stat, it's an old stat from the open AI, but it's still it's still true.

s36
00:04:11.599 --> 00:04:18.000
So if you look at the like training cost of the GPT3, it was like around $4.6 million.

s37
00:04:18.000 --> 00:04:20.000
It was a one-time cost.

s38
00:04:20.000 --> 00:04:37.600
But if you see the inference cost that has been like uh it's a recurring cost because it's a operating cost that scales with every user that comes in that every token that comes in every session that uh is being initiated on the like AI

s39
00:04:39.520 --> 00:04:43.360
and there are only two ways to basically counter this.

s40
00:04:43.360 --> 00:04:47.919
Uh one way is you reduce your token usage.

s41
00:04:47.919 --> 00:04:58.479
um alternative is you should try to optimize your inference solutions as a inference service provider for your customers and for yourself.

s42
00:04:58.479 --> 00:05:06.080
And so this um we have been seeing like lot and lot of like new solutions coming out every then and now.

s43
00:05:06.080 --> 00:05:17.360
Um and so the idea would be like okay we will try to build those foundations that will help us understand and evaluate like whatever ships next.

s44
00:05:17.360 --> 00:05:31.440
Um so yeah to get started like we will do a quick demo like it's a short demo of like what are the different pain points around inference u and so this is the repo

s45
00:05:31.440 --> 00:05:45.520
uh I mean you can pull it or you can also open it on the GitHub uh it's called LLM inference at scale a bit of background here like four months back when I didn't knew anything on the LLM inference

s46
00:05:45.520 --> 00:05:49.520
um I started learning it I saw like lot of resources were scattered.

s47
00:05:49.520 --> 00:05:56.720
So we started putting it uh like all together in one place uh so that it could benefit people.

s48
00:05:56.720 --> 00:05:58.400
Um yes.

s49
00:05:58.400 --> 00:06:12.800
So let me actually get out of this slideshow mode and probably go into I will go to this extended mode.

s50
00:06:13.120 --> 00:06:19.199
Um okay great.

s51
00:06:19.199 --> 00:06:27.840
Uh yeah so in this uh repository if you see a readme file there is like a link to the slides.

s52
00:06:27.840 --> 00:06:36.000
Uh so this it will be like this folder where you have like a pptx and there is like a benchmark report in there.

s53
00:06:36.000 --> 00:06:44.080
uh you can always like download it and then for the demo purposes uh we have couple of Jupyter notebooks.

s54
00:06:44.080 --> 00:06:54.960
Uh we have like collaborated with Moab who are the like Google collab alternative and what they basically provide you is like a free RTX 6000 GPU.

s55
00:06:54.960 --> 00:06:58.240
It's a 100 GB V RAM GPU.

s56
00:06:58.240 --> 00:07:10.800
So and we have like already set up these notebooks so that it becomes easy to like experiment with and like all of the assets and everything are preset for you.

s57
00:07:10.800 --> 00:07:31.039
Uh so uh we will start with like a simple demo a probably let me just see Okay.

s58
00:07:31.280 --> 00:07:33.199
Um yeah.

s59
00:07:33.199 --> 00:07:40.800
So when it comes to the inference, you need to do an inference on a certain model, right?

s60
00:07:40.800 --> 00:07:45.680
Uh for the workshop purposes, we are using a simple ML 7B model.

s61
00:07:45.680 --> 00:07:49.599
Uh it's a small model of around 15GB in size.

s62
00:07:49.599 --> 00:07:53.759
So we we are going to like load that into the GPU.

s63
00:07:53.759 --> 00:07:58.160
So, and we would look like some of the GPU stats as well.

s64
00:07:58.160 --> 00:08:01.840
So, we see like okay, we are working on the 6,000 Blackwell.

s65
00:08:01.840 --> 00:08:08.479
Uh, and you might be thinking I'm not running the cells because I don't trust the Wi-Fi at conferences.

s66
00:08:08.639 --> 00:08:09.759
So, yeah.

s67
00:08:09.759 --> 00:08:16.720
So, I would probably be just going over the results uh that we kind of ran previously.

s68
00:08:18.479 --> 00:08:19.360
Uh, yeah.

s69
00:08:19.360 --> 00:08:23.280
So, we have like a GPU which is like 102GB.

s70
00:08:23.280 --> 00:08:31.280
Uh now the first thing that comes to my mind is like what's my memory consumption looks like when I do the LLM inference.

s71
00:08:31.280 --> 00:08:37.200
So I load this model uh and I see like okay I have like a 15 GB here.

s72
00:08:37.200 --> 00:08:40.880
So I have roughly like 87.5GB.

s73
00:08:40.959 --> 00:08:53.200
And now when I do the like inference here uh what I notice is like the more the number of inputs I pass more is the memory that I need.

s74
00:08:53.200 --> 00:08:56.451
uh and it's increasing slowly but it's still increasing.

s75
00:08:56.451 --> 00:08:56.640
[snorts]

s76
00:08:56.640 --> 00:09:04.399
So imagine like if you have a context length of like around 4,000 or 16,000 or 32,000 uh tokens.

s77
00:09:04.399 --> 00:09:13.120
Uh so this memory could like really grow big and it you could actually get like all of those out of memory issues.

s78
00:09:13.120 --> 00:09:19.040
Uh so definitely this is like your problem one like your memory increasing with the increase in tokens.

s79
00:09:19.040 --> 00:09:24.000
So in form of like a simple visualization it looks like this.

s80
00:09:25.200 --> 00:09:32.399
The second problem that you would see is like the time to your first token it's very very slow.

s81
00:09:32.399 --> 00:09:35.920
Uh we measure it by a metric called TTFT.

s82
00:09:35.920 --> 00:09:38.800
It's a short short form of it.

s83
00:09:38.800 --> 00:09:53.200
uh and when you try to like measure the TTFT uh with the like input size you would see like longer the context you would see like this uh TTFT being slow.

s84
00:09:53.200 --> 00:09:54.640
So now there are two problems.

s85
00:09:54.640 --> 00:09:57.279
Your memory increases with the token size.

s86
00:09:57.279 --> 00:10:00.800
Your TTFT increases with the token size.

s87
00:10:00.800 --> 00:10:09.519
Uh sorry not the token size, the context Uh and then the third is the like throughput.

s88
00:10:09.519 --> 00:10:17.040
The throughput is like how many tokens can you serve per second and then how many users can you serve per second.

s89
00:10:17.040 --> 00:10:24.640
So if you take a very very vanilla implementation on your local system uh it would be like very sequential.

s90
00:10:24.640 --> 00:10:32.640
So if you send like five requests all those five requests would be catered like sequentially rather than parallelly.

s91
00:10:32.640 --> 00:10:41.040
Uh and so like your request basically takes more time to complete if you have like multiple users.

s92
00:10:41.040 --> 00:10:43.440
So these are the like three problems.

s93
00:10:43.440 --> 00:10:44.320
There is a fourth one.

s94
00:10:44.320 --> 00:10:45.920
I haven't described it here.

s95
00:10:45.920 --> 00:10:50.480
probably we will build that intuition as we move forward.

s96
00:10:50.480 --> 00:10:57.279
Uh but let's remember like these are the three problems the memory TTFT and the throughput.

s97
00:10:58.160 --> 00:10:58.640
Cool.

s98
00:10:58.640 --> 00:11:03.279
Uh I will go back to the slides.

s99
00:11:07.680 --> 00:11:10.160
Okay, perfect.

s100
00:11:15.680 --> 00:11:19.040
Uh so it should be this.

s101
00:11:20.880 --> 00:11:23.680
Uh is it visible?

s102
00:11:35.120 --> 00:11:39.279
Oh, you you need the Oh, okay.

s103
00:11:39.279 --> 00:11:40.399
Workshop.

s104
00:11:40.399 --> 00:11:42.399
Okay.

s105
00:11:47.600 --> 00:11:48.079
Yeah.

s106
00:11:48.079 --> 00:11:59.440
So, within that repository, if you see a workshop folder, you see that readme and then the readme has all the links, the slides and the demos.

s107
00:12:04.959 --> 00:12:07.760
Uh, does that work?

s108
00:12:09.680 --> 00:12:12.079
Okay, perfect.

s109
00:12:16.399 --> 00:12:17.040
Okay.

s110
00:12:17.040 --> 00:12:29.920
uh so let's start working through the foundations like let's start understanding uh what are the reasons behind those pain points and for that like we have to look at this inference pipeline

s111
00:12:29.920 --> 00:12:47.040
um so we get like an input text uh that text could have like any number of words you convert those into the tokens so for simplicity you can assume one word equal to one token

s112
00:12:47.040 --> 00:13:02.399
uh then you kind of convert them into like the embeddings and then you send it to the like transformers uh like there are 32 layers of transformers but that's specific to the Mistl 7B different models have different kind number of layers

s113
00:13:02.399 --> 00:13:13.040
uh and then you generate a new token and that token basically goes back to the input then you generate another token and that keeps on going now in this entire pipeline

s114
00:13:13.040 --> 00:13:18.800
you would see like 95% of your compute is like taken by these transformer layers.

s115
00:13:18.800 --> 00:13:24.399
So it's worth looking at like what goes within this transformer layer.

s116
00:13:25.120 --> 00:13:28.959
Within this transformer layer you would have like more layers.

s117
00:13:28.959 --> 00:13:35.920
You have like a normalization layer, you have an attention layer, you have a feed forward layer and all.

s118
00:13:35.920 --> 00:13:40.560
And attention layer is the one I think that has been very very famous.

s119
00:13:40.560 --> 00:13:42.480
Attention is all you need paper.

s120
00:13:42.480 --> 00:13:44.720
I think that's very well known.

s121
00:13:44.720 --> 00:13:54.560
So attention is the most compute intensive layer and we need to understand what goes within that attention layer.

s122
00:13:54.560 --> 00:13:57.040
So what does attention do?

s123
00:13:57.040 --> 00:14:06.720
Attention uh so if you have an input text it needs to find the attention scores of every token with respect to all of the previous tokens.

s124
00:14:06.720 --> 00:14:14.880
And to do that what it needs to do is like it needs to project every token into like a key query and the value space.

s125
00:14:14.880 --> 00:14:27.120
So in like in a simpler terms just u understand this like if you have 10 tokens then it needs like the 10 different query key and the value vectors.

s126
00:14:27.120 --> 00:14:31.440
If there are 100 tokens you would need 100 key and the value vectors.

s127
00:14:31.440 --> 00:14:35.360
If there are thousand tokens you would need thousand key value vectors.

s128
00:14:35.360 --> 00:14:42.160
And so like your number of the key and the value vectors they increase as you increase the input size.

s129
00:14:44.720 --> 00:14:54.399
Uh and if you calculate the like KV size per token uh for a mist 7B it comes out to be 131 KV.

s130
00:14:54.399 --> 00:15:04.800
Uh this is because like uh you have two vectors K and V. You have to multiply the size uh one vector is like 128 dimensions.

s131
00:15:04.800 --> 00:15:17.519
you have to multi multiply it by 32 transformer layers and then you have to multiply it by the KV heads for ML 7B it's gate KV heads it's not like 32

s132
00:15:17.519 --> 00:15:31.440
because it uses a different kind of an attention mechanism u which we will talk about for sure but yeah so the KV size per token is like your 131 KB

s133
00:15:31.440 --> 00:15:37.440
now imagine if you have 4K context uh So that size becomes like half a GB.

s134
00:15:37.440 --> 00:15:42.880
Uh if you do like 16k context that size becomes 2.1 GB.

s135
00:15:42.880 --> 00:15:58.800
Uh now multiplay by the users like assume you can serve multiple users together at the same time within that GPU you could have like 42GB with a 4K context and 80 users.

s136
00:15:58.800 --> 00:16:05.040
Um and if your GPU is only like let's say 24GB, you are already running out of the memory.

s137
00:16:05.040 --> 00:16:09.680
So you cannot serve that many users with that many context.

s138
00:16:12.320 --> 00:16:15.279
To visualize this, look at a GPU memory.

s139
00:16:15.279 --> 00:16:19.839
So the GPU memory has like a model weights which are pretty fixed.

s140
00:16:19.839 --> 00:16:21.759
These are pre-trained weights.

s141
00:16:21.759 --> 00:16:25.279
Uh there is like an overhead that is also fixed.

s142
00:16:25.279 --> 00:16:41.759
that also like that changes but it does not change that much u overall you can assume it's fixed and then there is like a leftover memory so this leftover memory is what being used by your KB me like key and the value vectors

s143
00:16:41.759 --> 00:16:57.040
so assume like you have a one user you can only serve that many key and the value vectors or that many tokens which can like fit in this entire 80GB

s144
00:16:57.040 --> 00:17:01.920
uh like memory that is left.

s145
00:17:02.959 --> 00:17:08.559
Uh so we can show this with a simple demo too.

s146
00:17:08.559 --> 00:17:21.280
Uh okay, let me Okay, great.

s147
00:17:25.600 --> 00:17:31.120
Uh, let me see if I can actually run this.

s148
00:17:31.520 --> 00:17:38.480
What probably?

s149
00:17:38.799 --> 00:17:42.160
Where the heck is this happening?

s150
00:17:49.840 --> 00:17:52.240
Okay, great.

s151
00:17:52.640 --> 00:17:53.120
Yeah.

s152
00:17:53.120 --> 00:17:57.280
So you would see like the GPU is attached.

s153
00:18:07.600 --> 00:18:14.559
So here we are just trying to confirm the like memory based on the maths and based on the intuition that we have built.

s154
00:18:14.559 --> 00:18:23.280
So the model memory is like let's say if you have a 7 billion parameters you are doing a 16 bit precision your total memory comes out to be 14.6

s155
00:18:23.280 --> 00:18:28.160
GB with you can basically verify that with the maths.

s156
00:18:28.160 --> 00:18:33.280
So if you do all that math that comes out to be the 14.6GB.

s157
00:18:33.919 --> 00:18:37.919
Uh now comes the KVK and the KV size.

s158
00:18:37.919 --> 00:18:42.559
So this KV size is like your 131 KB per token.

s159
00:18:42.559 --> 00:18:48.799
Uh and if you do that maths and you try to like visualize this.

s160
00:18:49.232 --> 00:18:51.232
[sighs]

s161
00:18:52.160 --> 00:18:54.559
Oh, sure.

s162
00:19:00.640 --> 00:19:02.640
Wait.

s163
00:19:03.840 --> 00:19:05.039
Okay.

s164
00:19:05.039 --> 00:19:09.520
And then let's just visualize this.

s165
00:19:09.919 --> 00:19:10.320
Okay.

s166
00:19:10.320 --> 00:19:11.840
Great.

s167
00:19:11.840 --> 00:19:15.440
Uh yeah so this is the like a memory chart.

s168
00:19:15.440 --> 00:19:21.679
So if you see like as your context increases your memory keeps increasing.

s169
00:19:21.679 --> 00:19:29.039
Then another thing to realize is like as your users increase uh then also your memory increases.

s170
00:19:29.039 --> 00:19:40.160
So if you want to serve like 160 users uh on a GPU you can support like uh you can only support like a lesser context length.

s171
00:19:40.160 --> 00:19:55.520
So there is always a tradeoff between what context length you can serve versus how much cost you can save by like putting your multiple uh users or the concurrent users into like a single GPU.

s172
00:19:55.520 --> 00:20:03.600
So you have to always take that tradeoff and and we will go through that like uh in couple of more slides.

s173
00:20:06.320 --> 00:20:09.280
Uh can you repeat please?

s174
00:20:14.400 --> 00:20:18.160
Uh I'm sorry I'm cannot hear you.

s175
00:20:24.880 --> 00:20:26.880
Yeah.

s176
00:20:33.200 --> 00:20:35.039
Cool.

s177
00:20:35.039 --> 00:20:36.400
Uh okay.

s178
00:20:36.400 --> 00:20:37.600
Great.

s179
00:20:37.600 --> 00:20:41.039
So let me pull back.

s180
00:20:41.120 --> 00:20:43.440
So that was like memory.

s181
00:20:43.440 --> 00:20:54.240
Uh we need to understand uh why we had like uh slower time to first token uh when we increase the context length.

s182
00:20:54.240 --> 00:21:01.600
So for that like we need to understand the two phases of inference and those phases are like the prefill and the decode phase.

s183
00:21:01.600 --> 00:21:06.960
I think you would all seen like a lot of articles but we just wanted to explain it.

s184
00:21:06.960 --> 00:21:21.440
Uh so when you send like a lot of uh like when you send these input tokens what you want to do is uh you want to build those key and the value vectors that I mentioned for all the tokens.

s185
00:21:21.440 --> 00:21:27.760
Then you want to compute the attention scores of every token with respect to the previous token.

s186
00:21:27.760 --> 00:21:38.799
All this operation that you do it's a very very metricsh heavy uh it's a very very compute heavy operation and we all know like the GPUs they are like

s187
00:21:38.799 --> 00:21:54.080
very well suited for a heavy compute workload so we call like a prefill to be like a compute bound uh and it does take some time to complete so whatever time that this phase takes to complete

s188
00:21:54.080 --> 00:22:03.679
that's your time to the first token So if if you have like a more input tokens, you have to generate more key value vectors.

s189
00:22:03.679 --> 00:22:12.559
You have to do lot more attention math and because of that your TTFT becomes more more slower.

s190
00:22:12.640 --> 00:22:21.120
Where is if once you generate one token you need to keep doing this to generate another tokens sequentially one after another.

s191
00:22:21.120 --> 00:22:34.559
But in that process every time you have to build the key and the value vectors of all the previous tokens which is same as prefill like you were building key value vectors there also here also

s192
00:22:34.559 --> 00:22:48.559
but in decode phase you are only computing the attention math for the new token and that is why it's a very less it's lesser compute oriented and it's also called as memory bound.

s193
00:22:48.559 --> 00:22:53.280
We will see it shortly why it's called as memory bound.

s194
00:22:54.960 --> 00:23:00.799
So in a classic timeline you would see prefill and decode phase like this.

s195
00:23:00.799 --> 00:23:13.760
So time taken by prefill that's your time to first token and then your time taken by every decode step that's your uh basically your inter token latency.

s196
00:23:13.760 --> 00:23:23.600
So that's like the fourth metric uh that you need to worry about like what's the time being taken by your decode step.

s197
00:23:25.760 --> 00:23:27.200
Okay, cool.

s198
00:23:27.200 --> 00:23:38.880
Uh now why why does the like decode step or why does decode takes time and why it's being called as like a memory bound operation?

s199
00:23:38.880 --> 00:23:40.960
Let's try to understand that.

s200
00:23:40.960 --> 00:23:48.640
uh to understand that we need to look at how the metrics map basically works on the GPU on a high level.

s201
00:23:48.640 --> 00:23:51.120
So GPU has two kind of memories.

s202
00:23:51.120 --> 00:23:53.360
You have a high bandwidth memory.

s203
00:23:53.360 --> 00:23:56.480
You have a shared memory.

s204
00:23:56.480 --> 00:24:02.240
So the high bandwidth memory is a larger size but a lower me like lower bandwidth.

s205
00:24:02.240 --> 00:24:09.520
By lower bandwidth I mean like you can transfer data out of it at a lower rate compared to the shared memory.

s206
00:24:09.520 --> 00:24:14.559
So the shared memory is smaller in size but it has a very very high bandwidth.

s207
00:24:14.559 --> 00:24:19.440
Uh that means you can transfer data in and out of it with a very first thing.

s208
00:24:19.440 --> 00:24:31.039
So when a when you have to do a metric math so you have to pick the data in chunks from the high bandwidth memory you have to put it into the shared memory.

s209
00:24:31.039 --> 00:24:36.720
Do that math write back the result into the high bandwidth memory.

s210
00:24:37.840 --> 00:24:50.559
uh for the prefill phase when you have to do this you have to do this matrix math only once but for the decode phase you have to do this metric

s211
00:24:50.559 --> 00:25:09.760
math uh like uh again and again because you're generating each and every token sequentially and so like you it doesn't matter like how fast is your decode because now you can transfer your data out of the

s212
00:25:09.760 --> 00:25:23.520
high bandwidth memory into the S shared memory at a certain speed because you are limited by the high bandwidth memory bandwidth speed and so that governs your like token sealing like

s213
00:25:23.520 --> 00:25:29.679
at what rate can you actually generate tokens out of the decode step.

s214
00:25:32.559 --> 00:25:43.279
If you look at this in the roof line plot uh so there is a left section which is called to be a memory bound.

s215
00:25:43.279 --> 00:25:47.760
Mathematically it's governed by the arithmetic intensity.

s216
00:25:47.760 --> 00:25:55.440
Arithmetic intensity is the number of flip-flop operations that you perform per bite of data being transferred.

s217
00:25:55.440 --> 00:26:14.240
So for the decode step uh decode step since you are transferring lot of data like the key and the value vectors of the all the previous tokens the model weights but you are doing the like less computation because you're computing attention math for only one token.

s218
00:26:14.240 --> 00:26:30.480
Uh so it's arithmetic intensity is very low but for a prefill phase you are transferring the data once but then like you are doing this heavy computation and so it's arithmetic intensity is very high.

s219
00:26:30.480 --> 00:26:42.080
So now you know like in terms of mathematics like why the computer like why the arithmetic intensity of prefill is very high compared to your decor.

s220
00:26:43.440 --> 00:26:43.460
Uh

s221
00:26:43.460 --> 00:26:44.159
[snorts]

s222
00:26:44.159 --> 00:26:50.320
okay so this is like another small small demo.

s223
00:26:50.320 --> 00:26:56.320
Uh every time I have to Okay.

s224
00:26:57.679 --> 00:27:00.000
Okay, great.

s225
00:27:00.000 --> 00:27:03.120
I hope this is already running.

s226
00:27:03.120 --> 00:27:06.880
So yeah, again we are loading the model.

s227
00:27:07.520 --> 00:27:10.000
Now this is the like the prefill cost.

s228
00:27:10.000 --> 00:27:21.279
So what we are basically doing is uh we are getting the like um the input uh text and then we are trying to generate this um the prefill step

s229
00:27:21.279 --> 00:27:22.799
the amount of time it takes.

s230
00:27:22.799 --> 00:27:29.440
We see like as we increase the like size of the input tokens this prefill is increasing.

s231
00:27:29.440 --> 00:27:37.120
So you and this is the reason why your TDF increases and then like your decode time.

s232
00:27:37.120 --> 00:27:41.919
So the decode time is like on average it stays about the same.

s233
00:27:41.919 --> 00:27:50.799
Uh and so if it is like assuming like you ignore the like cold start your decode time is like approximately around the average line.

s234
00:27:50.799 --> 00:27:57.279
it it it it is still impacted by like u the input size.

s235
00:27:57.279 --> 00:28:07.760
It's not like it's a constant uh time and it is because it still needs to pull the key and the value vectors from the memory for all the previous tokens.

s236
00:28:07.760 --> 00:28:15.919
So there is still like uh that uh basically small increase in time that you would see with the decode step.

s237
00:28:15.919 --> 00:28:21.200
And then this is the like classic uh roof line plot.

s238
00:28:21.919 --> 00:28:24.320
Uh okay.

s239
00:28:27.120 --> 00:28:29.120
Presentation.

s240
00:28:31.279 --> 00:28:32.720
Okay.

s241
00:28:32.720 --> 00:28:33.360
Five.

s242
00:28:33.360 --> 00:28:33.760
Okay.

s243
00:28:33.760 --> 00:28:35.600
Great.

s244
00:28:35.600 --> 00:28:36.799
Okay.

s245
00:28:36.799 --> 00:28:43.840
So now now let's try to understand like uh the throughput dimension.

s246
00:28:43.840 --> 00:28:57.520
You want to understand how many users you can actually serve and I think we saw like a diagram of the GPU memory where we saw okay there is some memory that is free for the key and the value vectors to grow.

s247
00:28:57.520 --> 00:29:12.480
So assume like you have just a single user uh what's the total KV size that you have you can basically support it's defined by your context limit.

s248
00:29:12.480 --> 00:29:26.320
uh the max users that you can support is like whatever is your GPU uh availability like whatever is the memory that is available in the GPU you divide it by the key and the value size per user

s249
00:29:26.320 --> 00:29:32.640
uh and when you do that like it comes out to be like your the concurrent users.

s250
00:29:33.120 --> 00:29:40.320
Now assume like your GPU is fixed, your model is fixed.

s251
00:29:40.320 --> 00:29:46.720
Uh so your KV size per token is fixed.

s252
00:29:46.720 --> 00:29:53.360
There are only two dimensions that are left here which is context and your concurrent users.

s253
00:29:53.360 --> 00:29:57.840
If you want to serve more concurrent users, you have to reduce the context length.

s254
00:29:57.840 --> 00:30:02.399
If you reduce the context length, you could impact your quality.

s255
00:30:02.480 --> 00:30:08.399
Uh so these are the two dimensions right now that we are trading off.

s256
00:30:09.279 --> 00:30:17.520
Then if we but can you actually serve the like max number of concurrent users?

s257
00:30:17.520 --> 00:30:28.159
Uh in an ideal world probably not because every business has like a latency SLO that we have to meet.

s258
00:30:28.159 --> 00:30:38.159
So if you remember like in the decode step I said the time for the decode still increases if you have more inputs.

s259
00:30:38.159 --> 00:30:41.440
It also increases if you have more users.

s260
00:30:41.440 --> 00:30:54.080
So ultimately uh your inter token latency also gets impacted if you have like a higher batch size and your TTF also gets impacted.

s261
00:30:54.080 --> 00:30:58.799
So now there is a third dimension you have to worry about which is like your latency.

s262
00:30:58.799 --> 00:31:04.480
So the three dimensions that you have is like a quality latency and the throughput.

s263
00:31:04.480 --> 00:31:11.520
So it comes out to be like this trade-off triangle where you have to choose between the two.

s264
00:31:11.520 --> 00:31:21.840
So for a premium chat application you would want to prioritize definitely the quality and you want to prioritize the like the latency.

s265
00:31:21.840 --> 00:31:31.200
You would not want your users to wait infinitely for the like or like not infinitely but probably for the larger latency.

s266
00:31:31.200 --> 00:31:45.440
You can always sacrifice the number of users you can support on the GPU and probably take that costed being more customers in form of like and and like if you consider like an agent

s267
00:31:45.440 --> 00:32:00.720
uh sorry the async agent workload you would want to like prioritize definitely quality and the throughput uh because these are the longunning tasks uh and you would want to like serve as many as concurrent tasks.

s268
00:32:00.720 --> 00:32:05.279
fast as possible but with a very very higher quality.

s269
00:32:07.840 --> 00:32:19.039
And often like we think like okay if the GPU is like a very expensive GPU uh that might not be a good fit for us.

s270
00:32:19.039 --> 00:32:26.720
Uh but it turns out that could actually serve you the lowest cost per million uh tokens.

s271
00:32:27.519 --> 00:32:41.519
Uh but you really have to trust your kind of calculations on the max users that you want and like uh you really have to make those estimations uh correctly.

s272
00:32:43.840 --> 00:33:02.159
Uh so we do have like uh let me just Where is this?

s273
00:33:02.320 --> 00:33:03.679
Okay, great.

s274
00:33:03.679 --> 00:33:12.559
So, for the capacity calculator, uh there is like a link to the collab because I was facing certain issues with molab.

s275
00:33:12.559 --> 00:33:17.279
I had to migrate out the wall widget library and I didn't have time.

s276
00:33:17.279 --> 00:33:21.279
So, being lazy, I just picked collab there.

s277
00:33:21.279 --> 00:33:24.480
Uh apologies to Moab.

s278
00:33:25.519 --> 00:33:31.760
Uh so my VR is connected.

s279
00:33:35.840 --> 00:33:37.840
Okay.

s280
00:33:46.080 --> 00:33:48.720
Wi-Fi probably.

s281
00:33:51.919 --> 00:33:52.320
Okay.

s282
00:33:52.320 --> 00:33:54.320
Great.

s283
00:34:02.159 --> 00:34:12.399
So what we have done over here is we have like shaded some like the GPUs with their V RAMs, bandwidths, the flip-flops and the cost per hours.

s284
00:34:12.399 --> 00:34:19.919
Um then we kind of like built this simple uh like uh capacity calculator.

s285
00:34:19.919 --> 00:34:37.280
This is just a KV visualizer uh where you kind of like when you increase the number of tokens uh you see like your KV size it increases and when you increase the number of users your size is like increasing at a much faster rate

s286
00:34:37.919 --> 00:34:49.200
and then in this capacity calculator uh let it run.

s287
00:34:49.599 --> 00:34:57.679
So we have like a model which we which is like a 7 billion parameter model that we selected.

s288
00:34:58.079 --> 00:35:01.359
We set the like precision to be FP16.

s289
00:35:01.359 --> 00:35:15.599
Uh now we decide the way we basically go by the GPU decision is you have to decide what's your like you have to fix one dimension first which you care about the most.

s290
00:35:15.599 --> 00:35:27.040
for premium chat I mentioned like latency is definitely the one uh and then like for the async workloads the batch the minimum batch size that you want to serve

s291
00:35:27.040 --> 00:35:39.680
for from like a single GPU that is the second dimension so you want to fix these first so I will go about like in a premium chat application uh

s292
00:35:39.680 --> 00:35:56.240
so I can go ahead with like 10 milliseconds latency a minimum batch size I don't care like I can so I'm okay with like probably two uh okay so probably with the seven

s293
00:35:56.240 --> 00:36:10.640
concurrent users on a single GPU and then like my context limit is very important to me because I want to focus on the quality as well uh and so like I do see like some of the GPUs

s294
00:36:10.640 --> 00:36:20.000
so the H18GB it's like a $8 per hour but like am I 300x is it?

s295
00:36:20.000 --> 00:36:20.480
Yeah.

s296
00:36:20.480 --> 00:36:33.920
So it's like around $10 per hour but if you do all that throughput math that we shared in the mathematics before you could find like your cost per million dollar tokens

s297
00:36:33.920 --> 00:36:37.599
that could be very very that could be like lesser.

s298
00:36:37.599 --> 00:36:49.119
So you need to do such calculations by fixing those dimensions and you need to decide your GPU to like reduce your kind of inference cost.

s299
00:36:49.119 --> 00:36:54.960
This is at least the first step that you can take towards optimizing the inference.

s300
00:36:56.720 --> 00:36:59.359
Okay, cool.

s301
00:37:01.520 --> 00:37:05.200
So the next slide.

s302
00:37:05.200 --> 00:37:10.560
So let me Okay, great.

s303
00:37:10.560 --> 00:37:14.800
And so like now the next thing is about the model optimization.

s304
00:37:14.800 --> 00:37:26.480
So we are now basically have built that foundation where we understood some of the pain points, reason behind those pain points, why those were happening um how we could like

s305
00:37:26.480 --> 00:37:29.359
address that GPU capacity thing.

s306
00:37:29.359 --> 00:37:33.839
We need to understand what can we do like what can we further do about it.

s307
00:37:33.839 --> 00:37:48.640
So it it is about the model optimization and I think I would like to invite Tan I he can talk more about these model optimizations provided he has worked uh on this like during his research times

s308
00:37:48.640 --> 00:38:07.680
okay I can control yeah here okay hi everyone uh mic check am I audible at last yeah okay so hi I'm Tesha I work as a senior quant modeler and also I am an AI researcher.

s309
00:38:07.680 --> 00:38:13.119
My work focuses on a agent verification and right now building world models.

s310
00:38:13.119 --> 00:38:25.119
So for this one model optimization before we start model optimization so I created a research template so that it will be easy for us to understand all these complex things.

s311
00:38:25.119 --> 00:38:27.920
I so our template is simple.

s312
00:38:27.920 --> 00:38:30.400
First we will identify the problem.

s313
00:38:30.400 --> 00:38:34.400
Second step we will solve the problem using two algorithms.

s314
00:38:34.400 --> 00:38:35.920
These are just fake algorithms.

s315
00:38:35.920 --> 00:38:39.280
So first algorithm is called ostrich algorithm.

s316
00:38:39.280 --> 00:38:46.480
Whenever we see uh just like ostrich whenever we see a problem ostrich put their head into the sand.

s317
00:38:46.480 --> 00:38:51.359
So same thing we will do whenever we face a problem we will just ignore it.

s318
00:38:51.359 --> 00:38:54.880
So this is an important algorithm we should follow.

s319
00:38:54.880 --> 00:38:59.359
Second one is created it is called world cup algorithm.

s320
00:38:59.359 --> 00:39:03.359
For example, we don't know who will win this FIFA World Cup.

s321
00:39:03.359 --> 00:39:12.880
So, what organizers did, they uh break the 48 teams into 12 groups, uh then round 32.

s322
00:39:12.880 --> 00:39:15.599
So, round 32 right now is currently going on.

s323
00:39:15.599 --> 00:39:21.359
Uh then round 16, then quarterfinals, uh then semi-finals and finals.

s324
00:39:21.359 --> 00:39:30.640
So what they are doing is that uh they are breaking it into a smaller problems and the useful results are moving forward.

s325
00:39:30.640 --> 00:39:38.960
So same analogy or same algorithm we will use uh to understand this model optimization all those things.

s326
00:39:38.960 --> 00:39:41.440
So yeah let's start.

s327
00:39:41.440 --> 00:39:46.400
So I have one H100 GPU.

s328
00:39:46.400 --> 00:39:54.640
I have to use this open-source model what is called GPTOSS 120 billion parameter model.

s329
00:39:54.640 --> 00:40:02.560
So right now I think it's so they have trained it on BF float 16 and weight is 240 GB.

s330
00:40:02.560 --> 00:40:04.880
What should I do?

s331
00:40:04.880 --> 00:40:06.960
This is the problem we have.

s332
00:40:06.960 --> 00:40:16.240
So first thing what we have to deal do is that 240 GB and 80 uh GB H100.

s333
00:40:16.240 --> 00:40:21.680
So and I have to fit only in one GPU or not in multiple GPU.

s334
00:40:21.680 --> 00:40:22.880
So what can we do?

s335
00:40:22.880 --> 00:40:27.680
I think simple step is that just compress it.

s336
00:40:27.680 --> 00:40:29.280
But how should we compress it?

s337
00:40:29.280 --> 00:40:30.720
Uh that's the another challenge.

s338
00:40:30.720 --> 00:40:42.160
So if we compress BF BF float 16 to FP8 uh then it will be around 120 GB but our GPU H100 is still 80 GB.

s339
00:40:42.160 --> 00:40:53.359
So what I think they did is that they compressed it into further MX uh MX FP4 and I think size is around 65 GB.

s340
00:40:53.359 --> 00:41:09.040
So this is something we can do uh compress but question so and we will use over this ostrich algorithm we are assuming that uh there is no loss in compressing a bigger model into a

s341
00:41:09.040 --> 00:41:10.880
a smaller size.

s342
00:41:10.880 --> 00:41:25.359
Second thing in uh in this one okay yeah so in this one in this slide we have used this mistral 7B so 7 billion parameters so it's a small model 7 billion parameters so

s343
00:41:25.359 --> 00:41:39.680
uh so if you multiply it by two bytes so it so weight of it's around is 14 14.5 GB which can easily fit into H100 or even a a40 so

s344
00:41:39.920 --> 00:41:52.079
so Next uh what we can do is that like mistral 7B instead of compressing it a floating point 16 we can apply different techniques like int 8 or int4

s345
00:41:52.079 --> 00:41:53.119
or nf4.

s346
00:41:53.119 --> 00:42:10.079
So basically we have to just use ostrich algorithm and just believe that uh there is no quality loss kind of things but somehow we also have to mathematically prove that by doing some kind of test testing on some external benchmark that whether it is working or not.

s347
00:42:10.079 --> 00:42:16.000
So the and this comes under post training quantization kind of thing.

s348
00:42:16.000 --> 00:42:22.560
One can also do uh this one uh during finetuning one can also do this kind of quantization.

s349
00:42:22.560 --> 00:42:25.760
This comes under a quant training kind of thing.

s350
00:42:25.760 --> 00:42:30.720
So uh let's move to our next problem.

s351
00:42:31.520 --> 00:42:50.960
So we have this huge matrices just just imagine imagine uh 1,000 by 1,000 uh dimension matrix A and another matrix matrix um 1,000 by 1,000.

s352
00:42:50.960 --> 00:43:04.319
So if we multiply uh if we multiply by this two matrices so number of operations will be 1,000 raised to the power q and this is kind of a problem in terms of

s353
00:43:04.319 --> 00:43:06.319
uh uh in terms of computing.

s354
00:43:06.319 --> 00:43:12.720
So we wondered our matrix multiplication should be fast and it should save memory.

s355
00:43:12.720 --> 00:43:15.200
So what should we do?

s356
00:43:15.200 --> 00:43:17.200
We have a giant matrix.

s357
00:43:17.200 --> 00:43:19.040
Okay, let's take this one.

s358
00:43:19.040 --> 00:43:22.880
Uh, Mr. 4096 by 4096.

s359
00:43:22.880 --> 00:43:33.440
What should we do uh to solve our problem of speeding up the things and saving the memory 4096 by 4096.

s360
00:43:33.599 --> 00:43:36.880
So first thing is that we will use just our world cup algorithm.

s361
00:43:36.880 --> 00:43:42.160
We can decide a random number just break the block vertically.

s362
00:43:42.160 --> 00:43:45.440
It does not matter what you are choosing it.

s363
00:43:45.440 --> 00:43:56.960
So you have so let's say uh we have 4096 uh columns we will break it uh we will break it into a group of 128 column each.

s364
00:43:56.960 --> 00:44:13.440
So 128 128 128 128 uh vertical vert uh vertically so we will get a 30 we will get this 32 blocks if we divide this 4096 then what will happen by doing this thing?

s365
00:44:13.440 --> 00:44:21.680
So if we just divide this one vertical vertically then we can use a multiple GPU to speed up the process.

s366
00:44:21.680 --> 00:44:27.119
So this kind of thing is called multi head attention.

s367
00:44:27.280 --> 00:44:29.599
So what else can we do?

s368
00:44:29.599 --> 00:44:36.720
We have a big matrix like uh as I have mentioned that ostrich algorithm.

s369
00:44:36.720 --> 00:44:40.160
So our main problem is sizing.

s370
00:44:40.160 --> 00:44:53.920
So what we what we can do is that instead of having all those 32 uh 32 vertical blocks we will throw away uh 31 blocks and we will assume that one block is sufficient enough

s371
00:44:53.920 --> 00:44:57.599
that all the queries uh can handle those blocks.

s372
00:44:57.599 --> 00:45:08.560
Our loss will be almost negligible and we come up with this algorithm uh which uh and this algorithm is called a multiquery attention.

s373
00:45:08.560 --> 00:45:12.400
So as we can see right now we are at two spectrum.

s374
00:45:12.400 --> 00:45:28.079
One is multi head attention where we split it into 32 blocks and use different uh different uh GPUs or do some parallel processing and at the same time we are just throwing 31 blocks and

s375
00:45:28.079 --> 00:45:32.400
uh we are calling this is as a multi-query attention.

s376
00:45:32.400 --> 00:45:46.240
So uh so at both extreme we should be come up with a middle ground like something we can say that instead of throwing all the 31 uh maybe we can group we can group

s377
00:45:46.240 --> 00:45:58.480
we can group some of the blocks together so that uh uh and we can assume that uh similar blocks will attend to a um similar kind of uh queries.

s378
00:45:58.480 --> 00:46:04.240
So this kind of technique comes under grouped query attention which is very popular right now.

s379
00:46:04.240 --> 00:46:11.359
Uh even in uh even in mistral or in other models this grouped query attention works.

s380
00:46:11.359 --> 00:46:23.280
So right now we have understand that we have a big matrix uh we can divide it the way we want and doing some mathematical calculation prove that loss is almost negligible kind of thing.

s381
00:46:23.280 --> 00:46:26.480
So what else we can do?

s382
00:46:26.880 --> 00:46:42.960
So after that uh after this grouped query attention uh see uh we have a big matrix uh one is one is key and one is value.

s383
00:46:42.960 --> 00:46:55.520
Let's compress that matrix into a latent vector and then come up with some algorithm to uh reconstruct from latent vector uh to our original matrix.

s384
00:46:55.520 --> 00:47:10.560
So this kind of a strategy comes under this one um multi head latent latent attention but again it has some problems with rope because rope is position dependent and uh and it is position independent kind of thing.

s385
00:47:10.560 --> 00:47:17.520
So yeah one needs to also include some uh index for keys also so that one can map it.

s386
00:47:17.520 --> 00:47:24.880
But again main problem is that why why we are why we are multiplying all those big matrices.

s387
00:47:24.880 --> 00:47:34.160
So because that's how this attention mechanism works that each token will pay attention to every token.

s388
00:47:34.160 --> 00:47:43.440
So how about let's don't pay attention to all the previous token only pay attention to the important tokens uh which is important for us.

s389
00:47:43.440 --> 00:47:47.680
So this is a kind of uh this kind of field is uh evolving.

s390
00:47:47.680 --> 00:47:51.599
So this comes under sparse uh deepseek sparse attention.

s391
00:47:51.599 --> 00:48:02.640
So uh yeah and yeah yeah yeah so okay next yeah so next one is flash attention.

s392
00:48:02.640 --> 00:48:20.800
So uh so in flash attention so main pro so main problem is that uh uh so so currently so so currently not currently so right now almost everyone uses flash attention but way in 2022 or 2023

s393
00:48:20.800 --> 00:48:35.040
uh so that's how it works that's how it works is that uh so uh this Q K query and A and key matrices they were in HBM.

s394
00:48:35.040 --> 00:48:47.200
Uh it loads uh it uh first uh it loads into uh this one uh tensor core and it do some uh it do some calculation and then it will uh write it back to

s395
00:48:47.200 --> 00:48:51.760
uh HBM and then uh this process goes on multiple times.

s396
00:48:51.760 --> 00:49:04.559
So in flash attention uh what they did is that uh is that instead of multiplying the whole matrices so they just divided it into like our world cup algorithm divided the

s397
00:49:04.559 --> 00:49:18.559
bigger matrices into a small tile and only put those small tiles uh into a SBM so that uh it can process multiplication fast and just uh keep uh keeping track of this some three variables

s398
00:49:18.559 --> 00:49:22.640
so that they can calculate this online softmax.

s399
00:49:23.599 --> 00:49:25.200
Yeah.

s400
00:49:25.200 --> 00:49:26.720
Next one.

s401
00:49:26.720 --> 00:49:27.440
So, yeah.

s402
00:49:27.440 --> 00:49:29.200
So, so this is just mathematics.

s403
00:49:29.200 --> 00:49:47.839
So, if we have a multi head attention if it is 524 uh KV uh then it depends upon how much how much grouping we want and so if instead of 32 KV head we only want to use uh 8 KV heads.

s404
00:49:47.839 --> 00:50:00.880
So uh so so we can get a compression of 4x times and this multi head latent attention this formula depends on the model to model how many layers your model have.

s405
00:50:00.880 --> 00:50:18.160
So in the original deepseek paper uh I think they have some 128 dimension 128 d 12 I don't remember the exact dimension but according to that uh they have used uh this one latent vector in which they have used 512

s406
00:50:18.160 --> 00:50:24.079
as a dimension and some 64 for for rope index.

s407
00:50:24.079 --> 00:50:34.000
So and then they show that it is a 50x 56x uh more compressed than multi head attention.

s408
00:50:37.280 --> 00:50:39.280
Okay.

s409
00:50:39.839 --> 00:50:40.240
Yeah.

s410
00:50:40.240 --> 00:50:46.000
So uh so yeah so this is uh so this is the uh this is the trade-off uh trade-off diagram.

s411
00:50:46.000 --> 00:50:50.960
So here I think we have not talked about this linear attention or mamba.

s412
00:50:50.960 --> 00:50:55.119
So main problem is just all this m Matrix multiplication.

s413
00:50:55.119 --> 00:50:57.359
Right now everyone is using attention.

s414
00:50:57.359 --> 00:51:11.040
Suppose in future uh if we don't want to use attention or rather than generating tokens sequentially just use maybe diffusion models where we can generate everything simultaneously.

s415
00:51:11.040 --> 00:51:14.480
So all these algorithms will change also.

s416
00:51:14.480 --> 00:51:16.319
But here I think they have two more.

s417
00:51:16.319 --> 00:51:19.440
One is linear attention and one is mamba.

s418
00:51:19.440 --> 00:51:33.839
So according to uh this slide so if we are not compressing anything so MHA is just we are parallelizing the process so there is no quality loss so it's a good and then this

s419
00:51:33.839 --> 00:51:48.559
uh grouped query attention which is I think almost uh every model is using uh just GQA and DSA kind of thing or yeah I think same thing we are providing in the attention mechanism

s420
00:51:48.559 --> 00:52:04.800
scorecard So uh so I think uh this one mha quality is good throughput is uh throughput is okay and for grouped query attention it depends upon your use case also though yeah though

s421
00:52:04.800 --> 00:52:20.079
quality is almost similar to uh multi head attention but use case also matters a lot yeah multi-query attention is just one extreme we are I don't know why but we are just assuming that we only need one block

s422
00:52:20.079 --> 00:52:24.640
and all the queries will attend to that smaller smaller block.

s423
00:52:24.640 --> 00:52:32.559
So, so quality is not that great for M for MQA and this multi head latent attention.

s424
00:52:32.559 --> 00:52:48.319
So yeah if you have tried some this deep seat models so I think uh they are doing great job yeah in in quality wise besides that sliding window so all these are sub techniques which

s425
00:52:48.319 --> 00:53:02.559
yeah yeah all these are some techniques like I just slide the windows all those things and instead of yeah instead of multiplying everything so linear attention is just saying that sum summarize everything first

s426
00:53:02.559 --> 00:53:08.880
uh and then look up into it and then mamba this is just a state space model.

s427
00:53:08.880 --> 00:53:13.839
Yeah, I can cover that.

s428
00:53:13.839 --> 00:53:15.520
Okay.

s429
00:53:15.520 --> 00:53:16.240
Uh cool.

s430
00:53:16.240 --> 00:53:27.839
Uh thank you T. So for the model like optimizations we also have like the two notebooks here.

s431
00:53:28.720 --> 00:53:36.160
So there will be I have to go to this.

s432
00:53:42.800 --> 00:53:43.359
Okay.

s433
00:53:43.359 --> 00:53:52.240
Uh so for the quantization uh like the demo uh this is is this already run?

s434
00:53:52.240 --> 00:53:57.200
No. Let me just run this.

s435
00:54:03.280 --> 00:54:03.680
Okay.

s436
00:54:03.680 --> 00:54:09.280
So we are loading the model which is like uh ML 7B.

s437
00:54:12.480 --> 00:54:21.920
Uh so this one is like with the FP16 baseline.

s438
00:54:29.760 --> 00:54:31.520
Wait.

s439
00:54:31.520 --> 00:54:34.559
Uh, did it run?

s440
00:54:35.839 --> 00:54:36.319
Okay.

s441
00:54:36.319 --> 00:54:40.160
So, it's uh two millisecond run.

s442
00:54:40.160 --> 00:54:41.280
Did this run?

s443
00:54:41.280 --> 00:54:42.720
Okay.

s444
00:54:42.720 --> 00:54:49.359
So, yeah, this time it's fetching that model with the FP16 precision.

s445
00:54:57.520 --> 00:54:59.839
the Wi-Fi.

s446
00:55:03.040 --> 00:55:06.000
It's going to take time.

s447
00:55:07.440 --> 00:55:09.440
Okay.

s448
00:55:11.200 --> 00:55:16.240
Yeah, it because it's downloading the weights from the hugging face.

s449
00:55:17.040 --> 00:55:19.040
Huh.

s450
00:55:20.480 --> 00:55:20.800
Yeah.

s451
00:55:20.800 --> 00:55:23.119
So, MOLAB is like running online.

s452
00:55:23.119 --> 00:55:25.119
Yes.

s453
00:55:26.240 --> 00:55:54.880
because it needs to make the network call through to the hugging phase and like it fetching I don't know like but it's taking time to download probably Okay.

s454
00:55:55.920 --> 00:55:56.960
So good.

s455
00:55:56.960 --> 00:55:57.520
Okay.

s456
00:55:57.520 --> 00:56:05.440
So here we see like the memory size is like 15 GB around approximately with the FP16 precision.

s457
00:56:05.440 --> 00:56:12.480
We are trying to do the 2x compression as Tmet talked about with the int8.

s458
00:56:13.839 --> 00:56:15.280
Let's download.

s459
00:56:15.280 --> 00:56:15.599
Okay.

s460
00:56:15.599 --> 00:56:21.200
So we do see like your memory size is now like 7.5 GB.

s461
00:56:21.200 --> 00:56:27.200
What that means is now you have a more s more memory for your KV to basically grow.

s462
00:56:27.200 --> 00:56:34.640
That means you can either serve higher context limit or you can serve the higher concurrent users there.

s463
00:56:36.319 --> 00:56:45.599
If you do the like in your basic you are doing the 4x compression so that with the 4x compression it would be more lower.

s464
00:56:45.599 --> 00:56:51.200
It would be I think around 3 to 4 GB.

s465
00:56:51.200 --> 00:56:51.520
Yeah.

s466
00:56:51.520 --> 00:56:57.680
4.5 GB and Yep.

s467
00:56:57.680 --> 00:57:01.280
So this is Wait.

s468
00:57:02.000 --> 00:57:09.839
So this is just a basic plot of like so these are the like theoretical numbers.

s469
00:57:09.839 --> 00:57:19.680
uh we are not doing the like any throughput test here but uh usually you would see like your memory increases so pro you would also have like a bit of higher

s470
00:57:19.680 --> 00:57:21.440
uh throughput.

s471
00:57:21.440 --> 00:57:30.000
Uh from some of the benchmarks that we studied we saw like the intate uh compression it does have like a lower throughput.

s472
00:57:32.960 --> 00:57:34.160
Okay.

s473
00:57:34.160 --> 00:57:40.799
And then there is like a demo on the like the attention mechanisms.

s474
00:57:42.480 --> 00:57:49.040
So for the attention okay I have to run this.

s475
00:57:58.480 --> 00:58:03.040
Uh okay so it has run.

s476
00:58:03.040 --> 00:58:04.480
Oh, wait.

s477
00:58:04.480 --> 00:58:08.000
Why does it say no GPU detected?

s478
00:58:09.440 --> 00:58:13.520
It should say the GPU should be detected.

s479
00:58:18.880 --> 00:58:21.440
Oh, okay.

s480
00:58:37.920 --> 00:58:52.799
Wait, but this is surprising.

s481
00:58:57.200 --> 00:59:02.720
Yeah, I guess it's not like able to detect the GPU for some reason.

s482
00:59:04.319 --> 00:59:08.799
Uh we do have like a GPU here.

s483
00:59:11.280 --> 00:59:14.319
Uh okay, never mind.

s484
00:59:14.319 --> 00:59:14.480
Yeah.

s485
00:59:14.480 --> 00:59:14.640
Yeah.

s486
00:59:14.640 --> 00:59:34.079
So, but the like basic idea here was more like as you try to move towards like compressing the computation like by using different attention mechanisms like moving from the multi head to the grouped query attention and then to the MLA

s487
00:59:34.079 --> 00:59:37.680
you would start seeing some optimizations.

s488
00:59:38.000 --> 00:59:42.640
Um I think yesterday night we were doing some benchmarking.

s489
00:59:42.640 --> 00:59:45.520
Uh I wanted to correct this part.

s490
00:59:45.520 --> 00:59:50.640
Uh so it wasn't like 50 56x it was 14x.

s491
00:59:50.640 --> 01:00:00.400
Uh basically the demo had a mistake of like a computation uh where it did not multiply the number of layers.

s492
01:00:01.040 --> 01:00:03.920
Uh yeah so apologies for that.

s493
01:00:03.920 --> 01:00:12.480
Uh so this MLA is like a 14x savings work in comparison to like your multi head attention.

s494
01:00:14.319 --> 01:00:30.319
Uh so now that we have understanding of the pain points, the foundations, the one side of the optimizations which is the model optimizations, we want to talk about what can you do on the

s495
01:00:30.319 --> 01:00:33.280
like the serving side.

s496
01:00:33.520 --> 01:00:48.720
So the first thing is we saw like when you perform like a simple decode step you are pulling it you are basically pulling the model weights and then you are recomputing the key and the value vectors

s497
01:00:48.720 --> 01:00:56.960
for all the previous tokens even though you already computed the those vectors for the tokens.

s498
01:00:56.960 --> 01:01:01.520
So there is definitely like a lot of compute wastage.

s499
01:01:01.520 --> 01:01:08.640
Uh and if you kind of analyze the time complexity of it, it would come out to be O of N².

s500
01:01:08.640 --> 01:01:13.119
Uh and the way to resolve that is like a classic trade-off against the memory.

s501
01:01:13.119 --> 01:01:20.000
You can maintain a memory of those vectors against the tokens and you can reference that memory.

s502
01:01:20.000 --> 01:01:24.240
So that memory was called as like KV cache.

s503
01:01:24.559 --> 01:01:36.559
uh and the like the flow looks something like this and then based on this KV cache there were like four optimizations that were really possible.

s504
01:01:36.559 --> 01:01:42.799
Um the first one is about the page detention.

s505
01:01:42.799 --> 01:01:45.520
So what's the different what's the problem today?

s506
01:01:45.520 --> 01:02:01.359
So when you send like multiple requests as the input to the GPU these requests are in a batch uh every request is allocated like a continuous memory storage let's say of

s507
01:02:01.359 --> 01:02:19.280
I'm just taking an example like let's set uh 2 KB however like your request needed only let's say uh 1 KB so there is like u 50% of that memory fragmentation.

s508
01:02:19.839 --> 01:02:24.480
Uh and this fragmentation basically leads to the memory wastage.

s509
01:02:24.480 --> 01:02:36.319
That means there was a space in the memory where you could have served more requests but you could not because you were looking for that contigious block of the memory.

s510
01:02:36.319 --> 01:02:47.760
So an inspiration to was being taken from like how the OS works like you maintain a logical memory and you basically have a physical memory.

s511
01:02:47.760 --> 01:03:04.799
So in the logical memory it would still feel like uh that the KV vector for the like every token is like a contiguous but it will be mapping to a different physical address.

s512
01:03:06.960 --> 01:03:11.599
So that really helped like saving a lot of memory.

s513
01:03:11.599 --> 01:03:27.440
Uh and it was only possible because you they considered like memory as a set of blocks and you would be dynamically allocating those blocks as the request need as the like new tokens comes in and they need that

s514
01:03:27.440 --> 01:03:29.920
kind of memory.

s515
01:03:30.960 --> 01:03:49.920
The another lever is like when you are sending multiple requests in the batch GPU is like taking those requests but it does not accepts the new batch unless all the requests in that batch gets completed.

s516
01:03:49.920 --> 01:03:59.760
So the diagram looks more like a page retention but here it is more about like when is GPU available to take the next batch.

s517
01:03:59.760 --> 01:04:15.200
So there is a time period where GPU is like sitting really idle and you want to like resolve for that and for that like the idea was like okay let's do that continuous batching.

s518
01:04:17.520 --> 01:04:24.960
So the continuous batching also really helped with like throughput because now you can ship more requests pretty quickly.

s519
01:04:24.960 --> 01:04:33.680
Keep making sure like GPU always uh get is always like occupied and it's not like uh sitting idle.

s520
01:04:33.680 --> 01:04:36.640
So you are saving on that compute.

s521
01:04:36.640 --> 01:04:39.520
The third is the like prefix caching.

s522
01:04:39.520 --> 01:04:47.599
So you remember like the KV cache helped you save the computation for a single request across the tokens.

s523
01:04:47.599 --> 01:04:53.440
But what if like you have the same tokens across multiple requests?

s524
01:04:53.440 --> 01:04:56.559
How do you basically save against that?

s525
01:04:56.559 --> 01:05:11.280
So the prefix caching uh which was introduced by VLM exactly counters that and then the third is like we talked about the fourth actually.

s526
01:05:11.280 --> 01:05:21.520
So we talked about quantizing the model but you could also you can also like quantize the KV weights.

s527
01:05:21.520 --> 01:05:28.240
So that means now you you need like a lesser space for your key and the value vectors.

s528
01:05:28.240 --> 01:05:32.480
That means you can serve more key and the value vectors in the memory.

s529
01:05:32.480 --> 01:05:34.799
And that means like you can serve more tokens.

s530
01:05:34.799 --> 01:05:38.240
That means you can serve more context context limit.

s531
01:05:38.240 --> 01:05:51.920
And that means like you can serve more model quality and all of this is like uh already present in the VLM.

s532
01:05:51.920 --> 01:06:05.520
You don't really need to reinvent that wheel uh and you can like deploy this VLM in production and you could see that basically growth.

s533
01:06:05.599 --> 01:06:09.440
So next we have like a benchmark that we did.

s534
01:06:09.440 --> 01:06:22.880
So this benchmark was let me see if I have that here the demos.

s535
01:06:23.359 --> 01:06:35.200
So doing this benchmark takes like around 1 hour because you have to continuously stop and like restart the VLM servers and you have to load the models and all.

s536
01:06:35.200 --> 01:06:42.720
So it does take a lot of time in doing the testing but I can like really tell you here what we are doing.

s537
01:06:42.720 --> 01:06:48.240
So we have kept the model as same like the Mistful 7B.

s538
01:06:48.240 --> 01:06:54.880
Uh and then we have like the set of input questions that we are sending.

s539
01:06:54.880 --> 01:06:57.280
Uh consider them as the prompts.

s540
01:06:57.280 --> 01:07:02.319
Then we have couple of helper functions here like checking the server is up or not.

s541
01:07:02.319 --> 01:07:05.760
The server is the VLM server.

s542
01:07:05.760 --> 01:07:10.880
Then there are helper functions to get the VLM metrics.

s543
01:07:10.880 --> 01:07:14.079
uh and I will talk about like what those metrics are.

s544
01:07:14.079 --> 01:07:23.280
Uh then there are like lot of the benchmarks and all and then you have to measure uh the KV usage and all.

s545
01:07:23.280 --> 01:07:25.200
So these are the like helper functions.

s546
01:07:25.200 --> 01:07:30.160
So the baseline is very simple like we have a hugging phase baseline.

s547
01:07:30.160 --> 01:07:36.880
Uh this is the raw like sending the text to the LLM getting back the response.

s548
01:07:36.880 --> 01:07:38.720
We see some results here.

s549
01:07:38.720 --> 01:07:44.240
We saw like hugging phase has a throughput of like around 51 tokens per second.

s550
01:07:44.240 --> 01:07:50.000
Time to first token was like 54 and then the inter token latency was 19.

s551
01:07:50.000 --> 01:07:54.480
Uh this bas uh this was all run on the h100.

s552
01:07:55.200 --> 01:08:00.720
Uh and then we start like a very default VLM server.

s553
01:08:00.720 --> 01:08:08.240
So by default VLM provides you the page detention, continuous batching and the KV caching.

s554
01:08:08.240 --> 01:08:25.040
So three things are present by default and when you try to compare those benchmarks you see your throughput is like almost 15x you are able to serve more tokens per second

s555
01:08:25.040 --> 01:08:45.759
then your time to the first token uh that also rises and then your v the inter token latency kind goes down and then your KV versus users and the versus context rate increases for sure.

s556
01:08:47.920 --> 01:09:02.640
Now when you apply the prefix caching to it so with the prefix caching you see like your throughput increases more your TDF decreases your inter token latency is approximately

s557
01:09:02.640 --> 01:09:16.880
same uh and then your KV cache usage versus the users it's kind of going down the vers context it's not going down it's approximately same I think this is also approximately

s558
01:09:16.880 --> 01:09:27.120
same it's like not that uh big of a deal when you apply the like KV quantization on top of it.

s559
01:09:27.600 --> 01:09:34.159
So it becomes like so so you see like your throughput is like almost similar.

s560
01:09:34.159 --> 01:09:36.640
Your time to first token is similar.

s561
01:09:36.640 --> 01:09:42.159
Your token latency is similar but then your KV usage actually goes down.

s562
01:09:42.159 --> 01:09:48.560
And this is because like you have quantized your key value space.

s563
01:09:48.560 --> 01:09:54.960
Uh and then there is a concept of speculative decoding that TME will talk about.

s564
01:09:54.960 --> 01:10:08.560
Uh so when you try to benchmark those so you also see like there is a uh like a bit of like the less KV usage there although like the results are approximately

s565
01:10:08.560 --> 01:10:10.560
same.

s566
01:10:15.040 --> 01:10:26.000
So yeah, I mean overall like these are the like the metrics across probably I should zoom out.

s567
01:10:26.000 --> 01:10:27.760
Okay, it's not zoom out.

s568
01:10:27.760 --> 01:10:30.159
It's not working.

s569
01:10:30.239 --> 01:10:31.440
Great.

s570
01:10:31.440 --> 01:10:35.280
So yeah, this is the like VLM benchmarks.

s571
01:10:35.280 --> 01:10:38.320
Um it's your production default by the way.

s572
01:10:38.320 --> 01:10:48.000
uh we will also share that decision tree uh when we try to talk about like the other engines.

s573
01:10:49.040 --> 01:11:04.000
So yeah, so we should talk about like what are some of the other inference optimizations we can do on top of it and what were some of the other solutions that came out.

s574
01:11:05.199 --> 01:11:08.480
Uh so I would like to again invite Tanme.

s575
01:11:08.480 --> 01:11:12.640
He's going to talk about like some of these optimizations.

s576
01:11:18.480 --> 01:11:19.199
Oh, sorry.

s577
01:11:19.199 --> 01:11:20.400
Uh I'm so sorry.

s578
01:11:20.400 --> 01:11:24.159
Uh I didn't enable the slides.

s579
01:11:25.280 --> 01:11:29.840
Uh what was the Okay, great.

s580
01:11:30.000 --> 01:11:30.800
Perfect.

s581
01:11:30.800 --> 01:11:31.600
Which one?

s582
01:11:31.600 --> 01:11:32.800
The speculative.

s583
01:11:32.800 --> 01:11:33.440
Yeah.

s584
01:11:33.440 --> 01:11:34.480
Thank you, Hersel.

s585
01:11:34.480 --> 01:11:36.080
Yeah.

s586
01:11:36.080 --> 01:11:47.199
So, so all these are like speculative decoding all these are the uh so so what we say uh different flavors of same kind of soda.

s587
01:11:47.199 --> 01:11:52.239
So this uh this technique comes under decoding accelerator.

s588
01:11:52.239 --> 01:12:07.360
So first one so we are only talking about this speculative decoding but there are other variants like self speculative eagle medusa I only like I think uh this one eagle algorithm

s589
01:12:07.440 --> 01:12:23.679
personally I don't think speculative decoding works because main problem is alignment okay so let's start with what is uh speculative decoding main problem is that in transformer architecture All these tokens are generated

s590
01:12:23.679 --> 01:12:27.040
sequentially one by one by one.

s591
01:12:27.040 --> 01:12:43.600
How about just use a smaller model and let a smaller model to generate maybe let's say four or five tokens and this teacher model or we can say according to our world cup algorithm we can say referee.

s592
01:12:43.600 --> 01:13:01.120
So referee will decide how many uh tokens it accept and this loop keeps on going on and our assumption is that there are certain domain where this kind of things will work like maybe in decode

s593
01:13:01.120 --> 01:13:09.040
maybe in coding or where almost there is no creativity uh each uh code or syntax is almost similar.

s594
01:13:09.040 --> 01:13:11.440
So maybe it can help it.

s595
01:13:11.440 --> 01:13:19.199
But uh based on personal testing, I didn't find this speculative decoding useful at all.

s596
01:13:19.199 --> 01:13:35.520
But other techniques like uh self speculative decoding where teacher model also have one head auxiliary head and it will do same similar kind of things what this base model or small model is doing it.

s597
01:13:35.520 --> 01:13:50.640
But then this eagle came Eagle 1 2 3 I don't know how many version versions are but it is just saying that instead of creating instead of generating tokens uh let's uh train a small model inside

s598
01:13:50.640 --> 01:14:03.679
train a small model and just take a features from one of its uh one of main models layer so that instead of generating token uh it will generate uh this features so

s599
01:14:03.679 --> 01:14:18.159
so uh so eagle is uh Eagle is better compared to this other kind of technologies and then another one is Medusa which is just saying that just generate all the tokens parallelly.

s600
01:14:18.159 --> 01:14:19.199
Uh okay.

s601
01:14:19.199 --> 01:14:23.040
So here so here in this slide

s602
01:14:23.520 --> 01:14:25.199
yeah the next slide.

s603
01:14:25.199 --> 01:14:27.199
Okay.

s604
01:14:29.440 --> 01:14:29.760
Okay.

s605
01:14:29.760 --> 01:14:31.040
Yeah.

s606
01:14:31.040 --> 01:14:31.679
Okay.

s607
01:14:31.679 --> 01:14:35.280
Now we come to uh now we will come to this one prefix caching.

s608
01:14:35.280 --> 01:14:48.480
So I don't know whether people are using this one static prefix caching or not but thing is that main problem with prefix caching is that sometimes we type and make a small kind of mistake

s609
01:14:48.480 --> 01:14:59.360
and this standard static prefix caching is basically it takes a prompt do some hashing and then next time when user asks similar kind of question it will try to match the hash.

s610
01:14:59.360 --> 01:15:10.400
So if hash is uh if hash is equal then then it will instead of recomputing all those K and B it will just uh take it from from the storage

s611
01:15:10.400 --> 01:15:20.640
but you know that sometimes we make a mistake or maybe we can just change a word or letter something like that then we have a very higher uh cache uh hit cache

s612
01:15:20.640 --> 01:15:31.199
miss hit rate so that's why uh this one uh radics tree so radics tree is becoming very popular and also also because of agent.

s613
01:15:31.199 --> 01:15:44.080
So I think almost everyone is doing agent and most of the computation is going during TT during test time inference kind of thing where we keep on asking same kind of questions and prompt for example

s614
01:15:44.080 --> 01:15:49.120
you are an expert software engineer multiply by 200 times.

s615
01:15:49.120 --> 01:16:03.679
This kind of loop keeps on going inside this uh agentic agentic kind of things where it is al necessary to keep uh or store similar kind of things in a radics tree.

s616
01:16:03.679 --> 01:16:18.159
So radics tree is just so so radic tree is just advanced version of this prefix tree where where we will just where we will just collapse a node if it does not have a does not have any branch

s617
01:16:18.159 --> 01:16:37.040
and for this kind of work where keep on repeating same thing this uh red x tree helps a lot and st lang uh use this kind of algorithm for prefix caching.

s618
01:16:37.600 --> 01:16:38.080
Okay.

s619
01:16:38.080 --> 01:16:38.320
Yeah.

s620
01:16:38.320 --> 01:16:39.520
Then there is another thing.

s621
01:16:39.520 --> 01:16:42.000
One is tensor RT LLM.

s622
01:16:42.000 --> 01:16:43.360
This is very confusing.

s623
01:16:43.360 --> 01:16:46.880
When I first started, I was I was just confused.

s624
01:16:46.880 --> 01:16:49.600
What is tensor RTLM?

s625
01:16:49.600 --> 01:16:50.080
So yeah.

s626
01:16:50.080 --> 01:16:56.000
So tensor RT is just a uh it's just a standard uh SDK kind of thing.

s627
01:16:56.000 --> 01:17:02.480
Tensor RTLM is just an inference engine just like VLM, SG lang.

s628
01:17:02.480 --> 01:17:06.400
But problem is that it is related to Nvidia.

s629
01:17:06.400 --> 01:17:13.920
They optimized each and every layer and every problem as I mentioned in our world cup algorithm.

s630
01:17:13.920 --> 01:17:19.679
They just break everything and optimized everything at hardware level also.

s631
01:17:19.679 --> 01:17:22.159
So uh yeah.

s632
01:17:22.159 --> 01:17:25.199
So okay next.

s633
01:17:27.040 --> 01:17:27.520
Yeah.

s634
01:17:27.520 --> 01:17:39.920
So for this workshop we also uh did some benchmarking like which is best uh so our setup was something similar was so so we did two kind of testing.

s635
01:17:39.920 --> 01:17:54.480
First one is without uh without agentic testing where we just so we use this shared GPT uh this one data set and uh just ask those questions uh using VLM

s636
01:17:54.480 --> 01:17:57.280
and SG lang.

s637
01:17:57.920 --> 01:17:59.920
Okay.

s638
01:18:06.000 --> 01:18:06.880
Yeah.

s639
01:18:06.880 --> 01:18:08.880
Okay.

s640
01:18:09.120 --> 01:18:13.120
And let me just zoom it up.

s641
01:18:13.120 --> 01:18:14.800
Okay, great.

s642
01:18:14.800 --> 01:18:15.040
Okay.

s643
01:18:15.040 --> 01:18:15.360
Yeah.

s644
01:18:15.360 --> 01:18:27.360
So, yeah, for this workshop, we used H100 and of our first testing was that uh we just uh we just asked uh we take questions from shared GPT and put it into

s645
01:18:27.360 --> 01:18:35.679
VLM, SG lang and we found that actually there's no statistical difference between which one is better.

s646
01:18:35.679 --> 01:18:37.840
So both have almost similar kind.

s647
01:18:37.840 --> 01:18:44.080
So both are fulfilling similar kind of request per second uh TTFT and latency.

s648
01:18:44.080 --> 01:18:50.960
So but only difference we have seen during agentic uh agentic branching.

s649
01:18:50.960 --> 01:19:05.360
So uh what we did was that we asked that similar kind of question that you are the best this one software engineer in the world just solve the problem of traffic congestion in this city kind of thing.

s650
01:19:05.360 --> 01:19:08.480
Then we put this into LLM.

s651
01:19:08.480 --> 01:19:10.960
LLM generates some output.

s652
01:19:10.960 --> 01:19:14.320
Then we did another uh round two also.

s653
01:19:14.320 --> 01:19:28.719
So once this LLM generates this output, then in round two we have specially mentioned that uh provide uh review the proposal and give ratings from 1 to 10.

s654
01:19:28.719 --> 01:19:35.360
So this uh two turns we did uh and this loop keeps on uh repeating it.

s655
01:19:35.360 --> 01:19:47.520
Uh what we found is that for this kind of uh workflow where everything is standard all those prompts and context engineering comes into the picture.

s656
01:19:47.520 --> 01:19:55.040
If we do proper this agentic branching then I think uh this HG lang is three to four times better.

s657
01:19:55.040 --> 01:20:03.280
But again this depends upon the different setup maybe uh if you do it uh you may get different results.

s658
01:20:03.280 --> 01:20:04.320
Okay.

s659
01:20:04.320 --> 01:20:04.800
Yeah.

s660
01:20:04.800 --> 01:20:09.600
So I think uh did we uploaded it on GitHub?

s661
01:20:09.600 --> 01:20:10.400
Okay.

s662
01:20:10.400 --> 01:20:10.880
Yeah.

s663
01:20:10.880 --> 01:20:15.600
So the PDF is like also in the drive.

s664
01:20:15.600 --> 01:20:19.280
Uh it's the same link as the slides.

s665
01:20:19.280 --> 01:20:23.280
So a quick summary here.

s666
01:20:23.280 --> 01:20:32.000
So on a standard API workload throughput you would see like a VLM and the SG lang would behave same.

s667
01:20:32.000 --> 01:20:36.960
So if you don't have if you have like a standard workload definitely go with VLM.

s668
01:20:36.960 --> 01:20:39.360
It's the production default anyways.

s669
01:20:39.360 --> 01:20:56.239
But what Tanme was also saying is when you try to like make it like agentic workloads that is where like your SG link really shines uh and uh it kind of like provides you all the benefits.

s670
01:20:57.679 --> 01:21:05.920
So yeah keep like VLM as a default but if you have agentic workloads probably try to move as the towards the SG lang.

s671
01:21:05.920 --> 01:21:08.800
if you're not happy with DB LLM.

s672
01:21:08.800 --> 01:21:13.440
Uh but uh okay.

s673
01:21:13.440 --> 01:21:23.040
Uh let me Okay.

s674
01:21:26.000 --> 01:21:36.719
And then like there is like the like a comparison that is done at the 120 billion like for the GPTO OSS 120 billion.

s675
01:21:36.719 --> 01:21:42.800
Um this is a benchmark that was prepared by clarify.

s676
01:21:42.800 --> 01:21:46.400
So there is like a blog link here.

s677
01:21:46.400 --> 01:21:48.640
Oh nice.

s678
01:21:48.960 --> 01:21:50.400
Okay.

s679
01:21:50.400 --> 01:21:50.800
Yeah.

s680
01:21:50.800 --> 01:21:57.840
So they did the similar benchmark and they included like a tensor RT LLM in it.

s681
01:21:58.400 --> 01:22:05.840
Definitely you can always go through these benchmarks and try to understand which basically suits your use case.

s682
01:22:05.840 --> 01:22:14.560
As we mentioned like Tensor RT they try to optimize the hardware side as well having the peak hardware performance.

s683
01:22:17.520 --> 01:22:24.000
Wait uh this is okay.

s684
01:22:26.159 --> 01:22:27.920
Yeah.

s685
01:22:27.920 --> 01:22:42.239
And then like in terms of when you want to dep pick like your engines once you figure out like between VLM, SG lang tenserati so that there are some new engines that are popping up

s686
01:22:42.239 --> 01:22:44.719
Nvidia Dynamo for sure.

s687
01:22:44.719 --> 01:22:49.679
Uh so they are also for the agentic uh session routing.

s688
01:22:49.760 --> 01:22:51.840
Uh hugging phase is always there.

s689
01:22:51.840 --> 01:22:54.159
It's a simple no server.

s690
01:22:54.159 --> 01:23:01.199
Then there is like an MSAR engine that was recently proposed by Stanford.

s691
01:23:01.199 --> 01:23:04.880
They are for like the multimodel.

s692
01:23:05.280 --> 01:23:19.199
Uh so definitely you could explore those and when you try to basically just to like give a quick summary uh you we start with like a baseline we try to find

s693
01:23:19.199 --> 01:23:22.880
what model could fit our use cases.

s694
01:23:22.880 --> 01:23:31.040
Um, so you could pick like uh Deep Seek, you could pick like don't pick like a Mistral 7B.

s695
01:23:31.040 --> 01:23:33.040
I mean, it's not good.

s696
01:23:33.040 --> 01:23:44.719
Uh, but yeah, so you pick your model and you want to like have a smaller memory and you want to try to fit that bigger model into smaller memory

s697
01:23:44.719 --> 01:23:48.080
so that you could save cost on the GPU cost.

s698
01:23:48.080 --> 01:23:58.880
So you can do like all those com quantization then you can apply all those serving optimizations by using the right serving engine under the hood.

s699
01:23:58.880 --> 01:24:05.679
So that can really provide you that throughut that you really want.

s700
01:24:07.920 --> 01:24:23.520
And now something that you can do uh after going back home pro because we cannot like actually go over all the material here uh is definitely reading about some of the source informations

s701
01:24:23.520 --> 01:24:37.679
like different attention mechanisms different like these engines like try to just read the different benchmarks which are present online as well and then there are a lot of like

s702
01:24:37.679 --> 01:24:45.600
in-depth guides or the next phases of it which is like learning about some KV eviction strategies.

s703
01:24:45.600 --> 01:24:50.639
So world is moving towards having a separate KV cache engineering domain.

s704
01:24:50.639 --> 01:24:53.040
So you want to understand what's going on in there.

s705
01:24:53.040 --> 01:24:58.080
So KV cache KV eviction cache compressions hybrid memories.

s706
01:24:58.080 --> 01:25:01.679
So there are like lot of solutions that are happening around there.

s707
01:25:01.679 --> 01:25:15.440
So always try to stick to those foundations or like the fundamentals or the first principles and try to see like which solution basically solves what problem and whether you actually need that

s708
01:25:15.440 --> 01:25:26.480
problem to be solved for your use case and then there is like distributed LLM inference which is like a different painoint altogether.

s709
01:25:26.480 --> 01:25:31.760
Uh you would probably need like a two-hour workshop there as well.

s710
01:25:31.760 --> 01:25:37.679
uh to like go over like all the internals do all the hands-on.

s711
01:25:41.280 --> 01:25:41.840
Yes.

s712
01:25:41.840 --> 01:25:51.679
And this is something we are trying to propose for the AI engineer New York session uh which is to like dive deeper into the advanced sections of the LLM inference.

s713
01:25:51.679 --> 01:25:56.400
So this workshop was more for the like beginner and the intermediate level.

s714
01:25:56.400 --> 01:26:02.800
Um so in this form we do have like a feedback as well plus also the interest.

s715
01:26:02.800 --> 01:26:14.960
Um if uh you think like we need certain improvements on certain sections definitely give that feedback as well and if you want to see this workshop in like New York

s716
01:26:14.960 --> 01:26:21.840
uh fair you I mean definitely feel free to enroll your interest.

s717
01:26:22.639 --> 01:26:24.639
Uhhuh.

s718
01:26:24.880 --> 01:26:27.360
How is it possible?

s719
01:26:28.000 --> 01:26:37.199
Well, let me just check.

s720
01:26:46.800 --> 01:26:48.480
Huh?

s721
01:26:48.480 --> 01:26:50.320
URL works, right?

s722
01:26:50.320 --> 01:26:51.520
Not the QR code.

s723
01:26:51.520 --> 01:26:52.320
Okay.

s724
01:26:52.320 --> 01:26:56.639
Probably I forgot to link those two together.

s725
01:27:00.239 --> 01:27:06.560
Z G A five.

s726
01:27:11.760 --> 01:27:13.280
Okay, cool.

s727
01:27:13.280 --> 01:27:13.520
Yes.

s728
01:27:13.520 --> 01:27:34.480
So if you can give that feedback let me just okay that will be fine um and yeah I think we would like to wrap this workshop then I'm sure like lot of you would be having a lot of questions so we can take all those like offline

s729
01:27:34.480 --> 01:27:39.440
uh we can meet uh and we can uh like talk about those questions.

s730
01:27:39.440 --> 01:27:41.120
Yeah sure.

s731
01:27:41.120 --> 01:27:42.400
Uh thank you everyone.

s732
01:27:42.400 --> 01:27:44.000
Thanks for joining.

s733
01:27:44.000 --> 01:27:49.840
Uh I think it was really meaningful and all of you like came here.

s734
01:27:49.840 --> 01:27:51.360
Uh thanks a lot.

s735
01:27:51.360 --> 01:27:51.600
Yeah.

s736
01:27:51.600 --> 01:27:53.600
Thanks.