Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

AI Engineer · 88 min · 736 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:13Uh so good afternoon everyone.
00:16Um my name is Hershel Jan and he is Tanisha.
00:20Uh and we would like to welcome you all in this two hours workshop on the LLM inference.
00:26Uh so the goal of this workshop is to understand this domain from the first principles uh dive deeper into it and like understand what's going on throughout the industry.
00:41Uh a bit of background about us.
00:44So I am a senior software engineer at Audible.
00:48uh have been building MLA data platforms for the past five years and on the sides I have been writing this opensource handbook on LLM inference and Tanme he is the
01:02senior quantitative modeler at XAN cup bank corporation he recently completed his PhD and he has been actively doing research in the agent verifiers and the world models Uh so a quick show of hands here.
01:21Uh Vu here is like brand new to the LLM inference.
01:27Okay, great.
01:28And Vu here has like deployed these models in production.
01:33They have been tuning it.
01:34They have been serving the production traffic.
01:38Okay, great.
01:41So this workshop is targeted towards the beginner and the intermediate level.
01:47U and all of the slides and exercises they are in the repo.
01:50I will share that soon.
01:55Here is the quick agenda for the workshop.
01:58We will start with the problem statement.
02:00We will try to understand few of the pain points around LLM inference.
02:06uh then we understand what causes those pain points and build our foundations from there.
02:12Then we will dive into like two kind of the optimizations that we do like the model optimizations and the serving optimizations.
02:21uh and then we start learning about different serving engines that are available to deploy our LLM inference solutions in production and we will showcase some benchmarks and the decision chart on like which engine to use.
02:40Cool.
02:40So to understand the pain points first we need to know what is like LLM inference.
02:47So, and probably a lot of us already know this.
02:51Um, but yeah, anything that you ask your AI to do like whether it be generate a video, audio, analyze any text, uh, analyze your medical reports or like your tax bills,
03:04all of that is like an LLM inference.
03:08And this market is like approximately $23 billion today.
03:15uh semi analysis recently shared that if you want to model like a Google search queries with LLMs, you need like a profit drain of like $36 billion and query cost
03:29has to be less than 0.5 cents to keep your search business profitable.
03:35On the other hand, the business insider mentioned like your AI has to be put on diet and everyone has to start auditing and budgeting their token usage and all of this is happening.
03:48Why?
03:48Because your hardware is limited, compute is expensive, your inference is expensive and with the growing need of like more and more AI usage, this inference cost is rising more and more.
04:03So this stat, it's an old stat from the open AI, but it's still it's still true.
04:11So if you look at the like training cost of the GPT3, it was like around $4.6 million.
04:18It was a one-time cost.
04:20But if you see the inference cost that has been like uh it's a recurring cost because it's a operating cost that scales with every user that comes in that every token that comes in every session that uh is being initiated on the like AI
04:39and there are only two ways to basically counter this.
04:43Uh one way is you reduce your token usage.
04:47um alternative is you should try to optimize your inference solutions as a inference service provider for your customers and for yourself.
04:58And so this um we have been seeing like lot and lot of like new solutions coming out every then and now.
05:06Um and so the idea would be like okay we will try to build those foundations that will help us understand and evaluate like whatever ships next.
05:17Um so yeah to get started like we will do a quick demo like it's a short demo of like what are the different pain points around inference u and so this is the repo
05:31uh I mean you can pull it or you can also open it on the GitHub uh it's called LLM inference at scale a bit of background here like four months back when I didn't knew anything on the LLM inference
05:45um I started learning it I saw like lot of resources were scattered.
05:49So we started putting it uh like all together in one place uh so that it could benefit people.
05:56Um yes.
05:58So let me actually get out of this slideshow mode and probably go into I will go to this extended mode.
06:13Um okay great.
06:19Uh yeah so in this uh repository if you see a readme file there is like a link to the slides.
06:27Uh so this it will be like this folder where you have like a pptx and there is like a benchmark report in there.
06:36uh you can always like download it and then for the demo purposes uh we have couple of Jupyter notebooks.
06:44Uh we have like collaborated with Moab who are the like Google collab alternative and what they basically provide you is like a free RTX 6000 GPU.
06:54It's a 100 GB V RAM GPU.
06:58So and we have like already set up these notebooks so that it becomes easy to like experiment with and like all of the assets and everything are preset for you.
07:10Uh so uh we will start with like a simple demo a probably let me just see Okay.
07:31Um yeah.
07:33So when it comes to the inference, you need to do an inference on a certain model, right?
07:40Uh for the workshop purposes, we are using a simple ML 7B model.
07:45Uh it's a small model of around 15GB in size.
07:49So we we are going to like load that into the GPU.
07:53So, and we would look like some of the GPU stats as well.
07:58So, we see like okay, we are working on the 6,000 Blackwell.
08:01Uh, and you might be thinking I'm not running the cells because I don't trust the Wi-Fi at conferences.
08:08So, yeah.
08:09So, I would probably be just going over the results uh that we kind of ran previously.
08:18Uh, yeah.
08:19So, we have like a GPU which is like 102GB.
08:23Uh now the first thing that comes to my mind is like what's my memory consumption looks like when I do the LLM inference.
08:31So I load this model uh and I see like okay I have like a 15 GB here.
08:37So I have roughly like 87.5GB.
08:40And now when I do the like inference here uh what I notice is like the more the number of inputs I pass more is the memory that I need.
08:53uh and it's increasing slowly but it's still increasing.
08:56[snorts]
08:56So imagine like if you have a context length of like around 4,000 or 16,000 or 32,000 uh tokens.
09:04Uh so this memory could like really grow big and it you could actually get like all of those out of memory issues.
09:13Uh so definitely this is like your problem one like your memory increasing with the increase in tokens.
09:19So in form of like a simple visualization it looks like this.
09:25The second problem that you would see is like the time to your first token it's very very slow.
09:32Uh we measure it by a metric called TTFT.
09:35It's a short short form of it.
09:38uh and when you try to like measure the TTFT uh with the like input size you would see like longer the context you would see like this uh TTFT being slow.
09:53So now there are two problems.
09:54Your memory increases with the token size.
09:57Your TTFT increases with the token size.
10:00Uh sorry not the token size, the context Uh and then the third is the like throughput.
10:09The throughput is like how many tokens can you serve per second and then how many users can you serve per second.
10:17So if you take a very very vanilla implementation on your local system uh it would be like very sequential.
10:24So if you send like five requests all those five requests would be catered like sequentially rather than parallelly.
10:32Uh and so like your request basically takes more time to complete if you have like multiple users.
10:41So these are the like three problems.
10:43There is a fourth one.
10:44I haven't described it here.
10:45probably we will build that intuition as we move forward.
10:50Uh but let's remember like these are the three problems the memory TTFT and the throughput.
10:58Cool.
10:58Uh I will go back to the slides.
11:07Okay, perfect.
11:15Uh so it should be this.
11:20Uh is it visible?
11:35Oh, you you need the Oh, okay.
11:39Workshop.
11:40Okay.
11:47Yeah.
11:48So, within that repository, if you see a workshop folder, you see that readme and then the readme has all the links, the slides and the demos.
12:04Uh, does that work?
12:09Okay, perfect.
12:16Okay.
12:17uh so let's start working through the foundations like let's start understanding uh what are the reasons behind those pain points and for that like we have to look at this inference pipeline
12:29um so we get like an input text uh that text could have like any number of words you convert those into the tokens so for simplicity you can assume one word equal to one token
12:47uh then you kind of convert them into like the embeddings and then you send it to the like transformers uh like there are 32 layers of transformers but that's specific to the Mistl 7B different models have different kind number of layers
13:02uh and then you generate a new token and that token basically goes back to the input then you generate another token and that keeps on going now in this entire pipeline
13:13you would see like 95% of your compute is like taken by these transformer layers.
13:18So it's worth looking at like what goes within this transformer layer.
13:25Within this transformer layer you would have like more layers.
13:28You have like a normalization layer, you have an attention layer, you have a feed forward layer and all.
13:35And attention layer is the one I think that has been very very famous.
13:40Attention is all you need paper.
13:42I think that's very well known.
13:44So attention is the most compute intensive layer and we need to understand what goes within that attention layer.
13:54So what does attention do?
13:57Attention uh so if you have an input text it needs to find the attention scores of every token with respect to all of the previous tokens.
14:06And to do that what it needs to do is like it needs to project every token into like a key query and the value space.
14:14So in like in a simpler terms just u understand this like if you have 10 tokens then it needs like the 10 different query key and the value vectors.
14:27If there are 100 tokens you would need 100 key and the value vectors.
14:31If there are thousand tokens you would need thousand key value vectors.
14:35And so like your number of the key and the value vectors they increase as you increase the input size.
14:44Uh and if you calculate the like KV size per token uh for a mist 7B it comes out to be 131 KV.
14:54Uh this is because like uh you have two vectors K and V. You have to multiply the size uh one vector is like 128 dimensions.
15:04you have to multi multiply it by 32 transformer layers and then you have to multiply it by the KV heads for ML 7B it's gate KV heads it's not like 32
15:17because it uses a different kind of an attention mechanism u which we will talk about for sure but yeah so the KV size per token is like your 131 KB
15:31now imagine if you have 4K context uh So that size becomes like half a GB.
15:37Uh if you do like 16k context that size becomes 2.1 GB.
15:42Uh now multiplay by the users like assume you can serve multiple users together at the same time within that GPU you could have like 42GB with a 4K context and 80 users.
15:58Um and if your GPU is only like let's say 24GB, you are already running out of the memory.
16:05So you cannot serve that many users with that many context.
16:12To visualize this, look at a GPU memory.
16:15So the GPU memory has like a model weights which are pretty fixed.
16:19These are pre-trained weights.
16:21Uh there is like an overhead that is also fixed.
16:25that also like that changes but it does not change that much u overall you can assume it's fixed and then there is like a leftover memory so this leftover memory is what being used by your KB me like key and the value vectors
16:41so assume like you have a one user you can only serve that many key and the value vectors or that many tokens which can like fit in this entire 80GB
16:57uh like memory that is left.
17:02Uh so we can show this with a simple demo too.
17:08Uh okay, let me Okay, great.
17:25Uh, let me see if I can actually run this.
17:31What probably?
17:38Where the heck is this happening?
17:49Okay, great.
17:52Yeah.
17:53So you would see like the GPU is attached.
18:07So here we are just trying to confirm the like memory based on the maths and based on the intuition that we have built.
18:14So the model memory is like let's say if you have a 7 billion parameters you are doing a 16 bit precision your total memory comes out to be 14.6
18:23GB with you can basically verify that with the maths.
18:28So if you do all that math that comes out to be the 14.6GB.
18:33Uh now comes the KVK and the KV size.
18:37So this KV size is like your 131 KB per token.
18:42Uh and if you do that maths and you try to like visualize this.
18:49[sighs]
18:52Oh, sure.
19:00Wait.
19:03Okay.
19:05And then let's just visualize this.
19:09Okay.
19:10Great.
19:11Uh yeah so this is the like a memory chart.
19:15So if you see like as your context increases your memory keeps increasing.
19:21Then another thing to realize is like as your users increase uh then also your memory increases.
19:29So if you want to serve like 160 users uh on a GPU you can support like uh you can only support like a lesser context length.
19:40So there is always a tradeoff between what context length you can serve versus how much cost you can save by like putting your multiple uh users or the concurrent users into like a single GPU.
19:55So you have to always take that tradeoff and and we will go through that like uh in couple of more slides.
20:06Uh can you repeat please?
20:14Uh I'm sorry I'm cannot hear you.
20:24Yeah.
20:33Cool.
20:35Uh okay.
20:36Great.
20:37So let me pull back.
20:41So that was like memory.
20:43Uh we need to understand uh why we had like uh slower time to first token uh when we increase the context length.
20:54So for that like we need to understand the two phases of inference and those phases are like the prefill and the decode phase.
21:01I think you would all seen like a lot of articles but we just wanted to explain it.
21:06Uh so when you send like a lot of uh like when you send these input tokens what you want to do is uh you want to build those key and the value vectors that I mentioned for all the tokens.
21:21Then you want to compute the attention scores of every token with respect to the previous token.
21:27All this operation that you do it's a very very metricsh heavy uh it's a very very compute heavy operation and we all know like the GPUs they are like
21:38very well suited for a heavy compute workload so we call like a prefill to be like a compute bound uh and it does take some time to complete so whatever time that this phase takes to complete
21:54that's your time to the first token So if if you have like a more input tokens, you have to generate more key value vectors.
22:03You have to do lot more attention math and because of that your TTFT becomes more more slower.
22:12Where is if once you generate one token you need to keep doing this to generate another tokens sequentially one after another.
22:21But in that process every time you have to build the key and the value vectors of all the previous tokens which is same as prefill like you were building key value vectors there also here also
22:34but in decode phase you are only computing the attention math for the new token and that is why it's a very less it's lesser compute oriented and it's also called as memory bound.
22:48We will see it shortly why it's called as memory bound.
22:54So in a classic timeline you would see prefill and decode phase like this.
23:00So time taken by prefill that's your time to first token and then your time taken by every decode step that's your uh basically your inter token latency.
23:13So that's like the fourth metric uh that you need to worry about like what's the time being taken by your decode step.
23:25Okay, cool.
23:27Uh now why why does the like decode step or why does decode takes time and why it's being called as like a memory bound operation?
23:38Let's try to understand that.
23:40uh to understand that we need to look at how the metrics map basically works on the GPU on a high level.
23:48So GPU has two kind of memories.
23:51You have a high bandwidth memory.
23:53You have a shared memory.
23:56So the high bandwidth memory is a larger size but a lower me like lower bandwidth.
24:02By lower bandwidth I mean like you can transfer data out of it at a lower rate compared to the shared memory.
24:09So the shared memory is smaller in size but it has a very very high bandwidth.
24:14Uh that means you can transfer data in and out of it with a very first thing.
24:19So when a when you have to do a metric math so you have to pick the data in chunks from the high bandwidth memory you have to put it into the shared memory.
24:31Do that math write back the result into the high bandwidth memory.
24:37uh for the prefill phase when you have to do this you have to do this matrix math only once but for the decode phase you have to do this metric
24:50math uh like uh again and again because you're generating each and every token sequentially and so like you it doesn't matter like how fast is your decode because now you can transfer your data out of the
25:09high bandwidth memory into the S shared memory at a certain speed because you are limited by the high bandwidth memory bandwidth speed and so that governs your like token sealing like
25:23at what rate can you actually generate tokens out of the decode step.
25:32If you look at this in the roof line plot uh so there is a left section which is called to be a memory bound.
25:43Mathematically it's governed by the arithmetic intensity.
25:47Arithmetic intensity is the number of flip-flop operations that you perform per bite of data being transferred.
25:55So for the decode step uh decode step since you are transferring lot of data like the key and the value vectors of the all the previous tokens the model weights but you are doing the like less computation because you're computing attention math for only one token.
26:14Uh so it's arithmetic intensity is very low but for a prefill phase you are transferring the data once but then like you are doing this heavy computation and so it's arithmetic intensity is very high.
26:30So now you know like in terms of mathematics like why the computer like why the arithmetic intensity of prefill is very high compared to your decor.
26:43Uh
26:43[snorts]
26:44okay so this is like another small small demo.
26:50Uh every time I have to Okay.
26:57Okay, great.
27:00I hope this is already running.
27:03So yeah, again we are loading the model.
27:07Now this is the like the prefill cost.
27:10So what we are basically doing is uh we are getting the like um the input uh text and then we are trying to generate this um the prefill step
27:21the amount of time it takes.
27:22We see like as we increase the like size of the input tokens this prefill is increasing.
27:29So you and this is the reason why your TDF increases and then like your decode time.
27:37So the decode time is like on average it stays about the same.
27:41Uh and so if it is like assuming like you ignore the like cold start your decode time is like approximately around the average line.
27:50it it it it is still impacted by like u the input size.
27:57It's not like it's a constant uh time and it is because it still needs to pull the key and the value vectors from the memory for all the previous tokens.
28:07So there is still like uh that uh basically small increase in time that you would see with the decode step.
28:15And then this is the like classic uh roof line plot.
28:21Uh okay.
28:27Presentation.
28:31Okay.
28:32Five.
28:33Okay.
28:33Great.
28:35Okay.
28:36So now now let's try to understand like uh the throughput dimension.
28:43You want to understand how many users you can actually serve and I think we saw like a diagram of the GPU memory where we saw okay there is some memory that is free for the key and the value vectors to grow.
28:57So assume like you have just a single user uh what's the total KV size that you have you can basically support it's defined by your context limit.
29:12uh the max users that you can support is like whatever is your GPU uh availability like whatever is the memory that is available in the GPU you divide it by the key and the value size per user
29:26uh and when you do that like it comes out to be like your the concurrent users.
29:33Now assume like your GPU is fixed, your model is fixed.
29:40Uh so your KV size per token is fixed.
29:46There are only two dimensions that are left here which is context and your concurrent users.
29:53If you want to serve more concurrent users, you have to reduce the context length.
29:57If you reduce the context length, you could impact your quality.
30:02Uh so these are the two dimensions right now that we are trading off.
30:09Then if we but can you actually serve the like max number of concurrent users?
30:17Uh in an ideal world probably not because every business has like a latency SLO that we have to meet.
30:28So if you remember like in the decode step I said the time for the decode still increases if you have more inputs.
30:38It also increases if you have more users.
30:41So ultimately uh your inter token latency also gets impacted if you have like a higher batch size and your TTF also gets impacted.
30:54So now there is a third dimension you have to worry about which is like your latency.
30:58So the three dimensions that you have is like a quality latency and the throughput.
31:04So it comes out to be like this trade-off triangle where you have to choose between the two.
31:11So for a premium chat application you would want to prioritize definitely the quality and you want to prioritize the like the latency.
31:21You would not want your users to wait infinitely for the like or like not infinitely but probably for the larger latency.
31:31You can always sacrifice the number of users you can support on the GPU and probably take that costed being more customers in form of like and and like if you consider like an agent
31:45uh sorry the async agent workload you would want to like prioritize definitely quality and the throughput uh because these are the longunning tasks uh and you would want to like serve as many as concurrent tasks.
32:00fast as possible but with a very very higher quality.
32:07And often like we think like okay if the GPU is like a very expensive GPU uh that might not be a good fit for us.
32:19Uh but it turns out that could actually serve you the lowest cost per million uh tokens.
32:27Uh but you really have to trust your kind of calculations on the max users that you want and like uh you really have to make those estimations uh correctly.
32:43Uh so we do have like uh let me just Where is this?
33:02Okay, great.
33:03So, for the capacity calculator, uh there is like a link to the collab because I was facing certain issues with molab.
33:12I had to migrate out the wall widget library and I didn't have time.
33:17So, being lazy, I just picked collab there.
33:21Uh apologies to Moab.
33:25Uh so my VR is connected.
33:35Okay.
33:46Wi-Fi probably.
33:51Okay.
33:52Great.
34:02So what we have done over here is we have like shaded some like the GPUs with their V RAMs, bandwidths, the flip-flops and the cost per hours.
34:12Um then we kind of like built this simple uh like uh capacity calculator.
34:19This is just a KV visualizer uh where you kind of like when you increase the number of tokens uh you see like your KV size it increases and when you increase the number of users your size is like increasing at a much faster rate
34:37and then in this capacity calculator uh let it run.
34:49So we have like a model which we which is like a 7 billion parameter model that we selected.
34:58We set the like precision to be FP16.
35:01Uh now we decide the way we basically go by the GPU decision is you have to decide what's your like you have to fix one dimension first which you care about the most.
35:15for premium chat I mentioned like latency is definitely the one uh and then like for the async workloads the batch the minimum batch size that you want to serve
35:27for from like a single GPU that is the second dimension so you want to fix these first so I will go about like in a premium chat application uh
35:39so I can go ahead with like 10 milliseconds latency a minimum batch size I don't care like I can so I'm okay with like probably two uh okay so probably with the seven
35:56concurrent users on a single GPU and then like my context limit is very important to me because I want to focus on the quality as well uh and so like I do see like some of the GPUs
36:10so the H18GB it's like a $8 per hour but like am I 300x is it?
36:20Yeah.
36:20So it's like around $10 per hour but if you do all that throughput math that we shared in the mathematics before you could find like your cost per million dollar tokens
36:33that could be very very that could be like lesser.
36:37So you need to do such calculations by fixing those dimensions and you need to decide your GPU to like reduce your kind of inference cost.
36:49This is at least the first step that you can take towards optimizing the inference.
36:56Okay, cool.
37:01So the next slide.
37:05So let me Okay, great.
37:10And so like now the next thing is about the model optimization.
37:14So we are now basically have built that foundation where we understood some of the pain points, reason behind those pain points, why those were happening um how we could like
37:26address that GPU capacity thing.
37:29We need to understand what can we do like what can we further do about it.
37:33So it it is about the model optimization and I think I would like to invite Tan I he can talk more about these model optimizations provided he has worked uh on this like during his research times
37:48okay I can control yeah here okay hi everyone uh mic check am I audible at last yeah okay so hi I'm Tesha I work as a senior quant modeler and also I am an AI researcher.
38:07My work focuses on a agent verification and right now building world models.
38:13So for this one model optimization before we start model optimization so I created a research template so that it will be easy for us to understand all these complex things.
38:25I so our template is simple.
38:27First we will identify the problem.
38:30Second step we will solve the problem using two algorithms.
38:34These are just fake algorithms.
38:35So first algorithm is called ostrich algorithm.
38:39Whenever we see uh just like ostrich whenever we see a problem ostrich put their head into the sand.
38:46So same thing we will do whenever we face a problem we will just ignore it.
38:51So this is an important algorithm we should follow.
38:54Second one is created it is called world cup algorithm.
38:59For example, we don't know who will win this FIFA World Cup.
39:03So, what organizers did, they uh break the 48 teams into 12 groups, uh then round 32.
39:12So, round 32 right now is currently going on.
39:15Uh then round 16, then quarterfinals, uh then semi-finals and finals.
39:21So what they are doing is that uh they are breaking it into a smaller problems and the useful results are moving forward.
39:30So same analogy or same algorithm we will use uh to understand this model optimization all those things.
39:38So yeah let's start.
39:41So I have one H100 GPU.
39:46I have to use this open-source model what is called GPTOSS 120 billion parameter model.
39:54So right now I think it's so they have trained it on BF float 16 and weight is 240 GB.
40:02What should I do?
40:04This is the problem we have.
40:06So first thing what we have to deal do is that 240 GB and 80 uh GB H100.
40:16So and I have to fit only in one GPU or not in multiple GPU.
40:21So what can we do?
40:22I think simple step is that just compress it.
40:27But how should we compress it?
40:29Uh that's the another challenge.
40:30So if we compress BF BF float 16 to FP8 uh then it will be around 120 GB but our GPU H100 is still 80 GB.
40:42So what I think they did is that they compressed it into further MX uh MX FP4 and I think size is around 65 GB.
40:53So this is something we can do uh compress but question so and we will use over this ostrich algorithm we are assuming that uh there is no loss in compressing a bigger model into a
41:09a smaller size.
41:10Second thing in uh in this one okay yeah so in this one in this slide we have used this mistral 7B so 7 billion parameters so it's a small model 7 billion parameters so
41:25uh so if you multiply it by two bytes so it so weight of it's around is 14 14.5 GB which can easily fit into H100 or even a a40 so
41:39so Next uh what we can do is that like mistral 7B instead of compressing it a floating point 16 we can apply different techniques like int 8 or int4
41:52or nf4.
41:53So basically we have to just use ostrich algorithm and just believe that uh there is no quality loss kind of things but somehow we also have to mathematically prove that by doing some kind of test testing on some external benchmark that whether it is working or not.
42:10So the and this comes under post training quantization kind of thing.
42:16One can also do uh this one uh during finetuning one can also do this kind of quantization.
42:22This comes under a quant training kind of thing.
42:25So uh let's move to our next problem.
42:31So we have this huge matrices just just imagine imagine uh 1,000 by 1,000 uh dimension matrix A and another matrix matrix um 1,000 by 1,000.
42:50So if we multiply uh if we multiply by this two matrices so number of operations will be 1,000 raised to the power q and this is kind of a problem in terms of
43:04uh uh in terms of computing.
43:06So we wondered our matrix multiplication should be fast and it should save memory.
43:12So what should we do?
43:15We have a giant matrix.
43:17Okay, let's take this one.
43:19Uh, Mr. 4096 by 4096.
43:22What should we do uh to solve our problem of speeding up the things and saving the memory 4096 by 4096.
43:33So first thing is that we will use just our world cup algorithm.
43:36We can decide a random number just break the block vertically.
43:42It does not matter what you are choosing it.
43:45So you have so let's say uh we have 4096 uh columns we will break it uh we will break it into a group of 128 column each.
43:56So 128 128 128 128 uh vertical vert uh vertically so we will get a 30 we will get this 32 blocks if we divide this 4096 then what will happen by doing this thing?
44:13So if we just divide this one vertical vertically then we can use a multiple GPU to speed up the process.
44:21So this kind of thing is called multi head attention.
44:27So what else can we do?
44:29We have a big matrix like uh as I have mentioned that ostrich algorithm.
44:36So our main problem is sizing.
44:40So what we what we can do is that instead of having all those 32 uh 32 vertical blocks we will throw away uh 31 blocks and we will assume that one block is sufficient enough
44:53that all the queries uh can handle those blocks.
44:57Our loss will be almost negligible and we come up with this algorithm uh which uh and this algorithm is called a multiquery attention.
45:08So as we can see right now we are at two spectrum.
45:12One is multi head attention where we split it into 32 blocks and use different uh different uh GPUs or do some parallel processing and at the same time we are just throwing 31 blocks and
45:28uh we are calling this is as a multi-query attention.
45:32So uh so at both extreme we should be come up with a middle ground like something we can say that instead of throwing all the 31 uh maybe we can group we can group
45:46we can group some of the blocks together so that uh uh and we can assume that uh similar blocks will attend to a um similar kind of uh queries.
45:58So this kind of technique comes under grouped query attention which is very popular right now.
46:04Uh even in uh even in mistral or in other models this grouped query attention works.
46:11So right now we have understand that we have a big matrix uh we can divide it the way we want and doing some mathematical calculation prove that loss is almost negligible kind of thing.
46:23So what else we can do?
46:26So after that uh after this grouped query attention uh see uh we have a big matrix uh one is one is key and one is value.
46:42Let's compress that matrix into a latent vector and then come up with some algorithm to uh reconstruct from latent vector uh to our original matrix.
46:55So this kind of a strategy comes under this one um multi head latent latent attention but again it has some problems with rope because rope is position dependent and uh and it is position independent kind of thing.
47:10So yeah one needs to also include some uh index for keys also so that one can map it.
47:17But again main problem is that why why we are why we are multiplying all those big matrices.
47:24So because that's how this attention mechanism works that each token will pay attention to every token.
47:34So how about let's don't pay attention to all the previous token only pay attention to the important tokens uh which is important for us.
47:43So this is a kind of uh this kind of field is uh evolving.
47:47So this comes under sparse uh deepseek sparse attention.
47:51So uh yeah and yeah yeah yeah so okay next yeah so next one is flash attention.
48:02So uh so in flash attention so main pro so main problem is that uh uh so so currently so so currently not currently so right now almost everyone uses flash attention but way in 2022 or 2023
48:20uh so that's how it works that's how it works is that uh so uh this Q K query and A and key matrices they were in HBM.
48:35Uh it loads uh it uh first uh it loads into uh this one uh tensor core and it do some uh it do some calculation and then it will uh write it back to
48:47uh HBM and then uh this process goes on multiple times.
48:51So in flash attention uh what they did is that uh is that instead of multiplying the whole matrices so they just divided it into like our world cup algorithm divided the
49:04bigger matrices into a small tile and only put those small tiles uh into a SBM so that uh it can process multiplication fast and just uh keep uh keeping track of this some three variables
49:18so that they can calculate this online softmax.
49:23Yeah.
49:25Next one.
49:26So, yeah.
49:27So, so this is just mathematics.
49:29So, if we have a multi head attention if it is 524 uh KV uh then it depends upon how much how much grouping we want and so if instead of 32 KV head we only want to use uh 8 KV heads.
49:47So uh so so we can get a compression of 4x times and this multi head latent attention this formula depends on the model to model how many layers your model have.
50:00So in the original deepseek paper uh I think they have some 128 dimension 128 d 12 I don't remember the exact dimension but according to that uh they have used uh this one latent vector in which they have used 512
50:18as a dimension and some 64 for for rope index.
50:24So and then they show that it is a 50x 56x uh more compressed than multi head attention.
50:37Okay.
50:39Yeah.
50:40So uh so yeah so this is uh so this is the uh this is the trade-off uh trade-off diagram.
50:46So here I think we have not talked about this linear attention or mamba.
50:50So main problem is just all this m Matrix multiplication.
50:55Right now everyone is using attention.
50:57Suppose in future uh if we don't want to use attention or rather than generating tokens sequentially just use maybe diffusion models where we can generate everything simultaneously.
51:11So all these algorithms will change also.
51:14But here I think they have two more.
51:16One is linear attention and one is mamba.
51:19So according to uh this slide so if we are not compressing anything so MHA is just we are parallelizing the process so there is no quality loss so it's a good and then this
51:33uh grouped query attention which is I think almost uh every model is using uh just GQA and DSA kind of thing or yeah I think same thing we are providing in the attention mechanism
51:48scorecard So uh so I think uh this one mha quality is good throughput is uh throughput is okay and for grouped query attention it depends upon your use case also though yeah though
52:04quality is almost similar to uh multi head attention but use case also matters a lot yeah multi-query attention is just one extreme we are I don't know why but we are just assuming that we only need one block
52:20and all the queries will attend to that smaller smaller block.
52:24So, so quality is not that great for M for MQA and this multi head latent attention.
52:32So yeah if you have tried some this deep seat models so I think uh they are doing great job yeah in in quality wise besides that sliding window so all these are sub techniques which
52:48yeah yeah all these are some techniques like I just slide the windows all those things and instead of yeah instead of multiplying everything so linear attention is just saying that sum summarize everything first
53:02uh and then look up into it and then mamba this is just a state space model.
53:08Yeah, I can cover that.
53:13Okay.
53:15Uh cool.
53:16Uh thank you T. So for the model like optimizations we also have like the two notebooks here.
53:28So there will be I have to go to this.
53:42Okay.
53:43Uh so for the quantization uh like the demo uh this is is this already run?
53:52No. Let me just run this.
54:03Okay.
54:03So we are loading the model which is like uh ML 7B.
54:12Uh so this one is like with the FP16 baseline.
54:29Wait.
54:31Uh, did it run?
54:35Okay.
54:36So, it's uh two millisecond run.
54:40Did this run?
54:41Okay.
54:42So, yeah, this time it's fetching that model with the FP16 precision.
54:57the Wi-Fi.
55:03It's going to take time.
55:07Okay.
55:11Yeah, it because it's downloading the weights from the hugging face.
55:17Huh.
55:20Yeah.
55:20So, MOLAB is like running online.
55:23Yes.
55:26because it needs to make the network call through to the hugging phase and like it fetching I don't know like but it's taking time to download probably Okay.
55:55So good.
55:56Okay.
55:57So here we see like the memory size is like 15 GB around approximately with the FP16 precision.
56:05We are trying to do the 2x compression as Tmet talked about with the int8.
56:13Let's download.
56:15Okay.
56:15So we do see like your memory size is now like 7.5 GB.
56:21What that means is now you have a more s more memory for your KV to basically grow.
56:27That means you can either serve higher context limit or you can serve the higher concurrent users there.
56:36If you do the like in your basic you are doing the 4x compression so that with the 4x compression it would be more lower.
56:45It would be I think around 3 to 4 GB.
56:51Yeah.
56:514.5 GB and Yep.
56:57So this is Wait.
57:02So this is just a basic plot of like so these are the like theoretical numbers.
57:09uh we are not doing the like any throughput test here but uh usually you would see like your memory increases so pro you would also have like a bit of higher
57:19uh throughput.
57:21Uh from some of the benchmarks that we studied we saw like the intate uh compression it does have like a lower throughput.
57:32Okay.
57:34And then there is like a demo on the like the attention mechanisms.
57:42So for the attention okay I have to run this.
57:58Uh okay so it has run.
58:03Oh, wait.
58:04Why does it say no GPU detected?
58:09It should say the GPU should be detected.
58:18Oh, okay.
58:37Wait, but this is surprising.
58:57Yeah, I guess it's not like able to detect the GPU for some reason.
59:04Uh we do have like a GPU here.
59:11Uh okay, never mind.
59:14Yeah.
59:14Yeah.
59:14So, but the like basic idea here was more like as you try to move towards like compressing the computation like by using different attention mechanisms like moving from the multi head to the grouped query attention and then to the MLA
59:34you would start seeing some optimizations.
59:38Um I think yesterday night we were doing some benchmarking.
59:42Uh I wanted to correct this part.
59:45Uh so it wasn't like 50 56x it was 14x.
59:50Uh basically the demo had a mistake of like a computation uh where it did not multiply the number of layers.
60:01Uh yeah so apologies for that.
60:03Uh so this MLA is like a 14x savings work in comparison to like your multi head attention.
60:14Uh so now that we have understanding of the pain points, the foundations, the one side of the optimizations which is the model optimizations, we want to talk about what can you do on the
60:30like the serving side.
60:33So the first thing is we saw like when you perform like a simple decode step you are pulling it you are basically pulling the model weights and then you are recomputing the key and the value vectors
60:48for all the previous tokens even though you already computed the those vectors for the tokens.
60:56So there is definitely like a lot of compute wastage.
61:01Uh and if you kind of analyze the time complexity of it, it would come out to be O of N².
61:08Uh and the way to resolve that is like a classic trade-off against the memory.
61:13You can maintain a memory of those vectors against the tokens and you can reference that memory.
61:20So that memory was called as like KV cache.
61:24uh and the like the flow looks something like this and then based on this KV cache there were like four optimizations that were really possible.
61:36Um the first one is about the page detention.
61:42So what's the different what's the problem today?
61:45So when you send like multiple requests as the input to the GPU these requests are in a batch uh every request is allocated like a continuous memory storage let's say of
62:01I'm just taking an example like let's set uh 2 KB however like your request needed only let's say uh 1 KB so there is like u 50% of that memory fragmentation.
62:19Uh and this fragmentation basically leads to the memory wastage.
62:24That means there was a space in the memory where you could have served more requests but you could not because you were looking for that contigious block of the memory.
62:36So an inspiration to was being taken from like how the OS works like you maintain a logical memory and you basically have a physical memory.
62:47So in the logical memory it would still feel like uh that the KV vector for the like every token is like a contiguous but it will be mapping to a different physical address.
63:06So that really helped like saving a lot of memory.
63:11Uh and it was only possible because you they considered like memory as a set of blocks and you would be dynamically allocating those blocks as the request need as the like new tokens comes in and they need that
63:27kind of memory.
63:30The another lever is like when you are sending multiple requests in the batch GPU is like taking those requests but it does not accepts the new batch unless all the requests in that batch gets completed.
63:49So the diagram looks more like a page retention but here it is more about like when is GPU available to take the next batch.
63:59So there is a time period where GPU is like sitting really idle and you want to like resolve for that and for that like the idea was like okay let's do that continuous batching.
64:17So the continuous batching also really helped with like throughput because now you can ship more requests pretty quickly.
64:24Keep making sure like GPU always uh get is always like occupied and it's not like uh sitting idle.
64:33So you are saving on that compute.
64:36The third is the like prefix caching.
64:39So you remember like the KV cache helped you save the computation for a single request across the tokens.
64:47But what if like you have the same tokens across multiple requests?
64:53How do you basically save against that?
64:56So the prefix caching uh which was introduced by VLM exactly counters that and then the third is like we talked about the fourth actually.
65:11So we talked about quantizing the model but you could also you can also like quantize the KV weights.
65:21So that means now you you need like a lesser space for your key and the value vectors.
65:28That means you can serve more key and the value vectors in the memory.
65:32And that means like you can serve more tokens.
65:34That means you can serve more context context limit.
65:38And that means like you can serve more model quality and all of this is like uh already present in the VLM.
65:51You don't really need to reinvent that wheel uh and you can like deploy this VLM in production and you could see that basically growth.
66:05So next we have like a benchmark that we did.
66:09So this benchmark was let me see if I have that here the demos.
66:23So doing this benchmark takes like around 1 hour because you have to continuously stop and like restart the VLM servers and you have to load the models and all.
66:35So it does take a lot of time in doing the testing but I can like really tell you here what we are doing.
66:42So we have kept the model as same like the Mistful 7B.
66:48Uh and then we have like the set of input questions that we are sending.
66:54Uh consider them as the prompts.
66:57Then we have couple of helper functions here like checking the server is up or not.
67:02The server is the VLM server.
67:05Then there are helper functions to get the VLM metrics.
67:10uh and I will talk about like what those metrics are.
67:14Uh then there are like lot of the benchmarks and all and then you have to measure uh the KV usage and all.
67:23So these are the like helper functions.
67:25So the baseline is very simple like we have a hugging phase baseline.
67:30Uh this is the raw like sending the text to the LLM getting back the response.
67:36We see some results here.
67:38We saw like hugging phase has a throughput of like around 51 tokens per second.
67:44Time to first token was like 54 and then the inter token latency was 19.
67:50Uh this bas uh this was all run on the h100.
67:55Uh and then we start like a very default VLM server.
68:00So by default VLM provides you the page detention, continuous batching and the KV caching.
68:08So three things are present by default and when you try to compare those benchmarks you see your throughput is like almost 15x you are able to serve more tokens per second
68:25then your time to the first token uh that also rises and then your v the inter token latency kind goes down and then your KV versus users and the versus context rate increases for sure.
68:47Now when you apply the prefix caching to it so with the prefix caching you see like your throughput increases more your TDF decreases your inter token latency is approximately
69:02same uh and then your KV cache usage versus the users it's kind of going down the vers context it's not going down it's approximately same I think this is also approximately
69:16same it's like not that uh big of a deal when you apply the like KV quantization on top of it.
69:27So it becomes like so so you see like your throughput is like almost similar.
69:34Your time to first token is similar.
69:36Your token latency is similar but then your KV usage actually goes down.
69:42And this is because like you have quantized your key value space.
69:48Uh and then there is a concept of speculative decoding that TME will talk about.
69:54Uh so when you try to benchmark those so you also see like there is a uh like a bit of like the less KV usage there although like the results are approximately
70:08same.
70:15So yeah, I mean overall like these are the like the metrics across probably I should zoom out.
70:26Okay, it's not zoom out.
70:27It's not working.
70:30Great.
70:31So yeah, this is the like VLM benchmarks.
70:35Um it's your production default by the way.
70:38uh we will also share that decision tree uh when we try to talk about like the other engines.
70:49So yeah, so we should talk about like what are some of the other inference optimizations we can do on top of it and what were some of the other solutions that came out.
71:05Uh so I would like to again invite Tanme.
71:08He's going to talk about like some of these optimizations.
71:18Oh, sorry.
71:19Uh I'm so sorry.
71:20Uh I didn't enable the slides.
71:25Uh what was the Okay, great.
71:30Perfect.
71:30Which one?
71:31The speculative.
71:32Yeah.
71:33Thank you, Hersel.
71:34Yeah.
71:36So, so all these are like speculative decoding all these are the uh so so what we say uh different flavors of same kind of soda.
71:47So this uh this technique comes under decoding accelerator.
71:52So first one so we are only talking about this speculative decoding but there are other variants like self speculative eagle medusa I only like I think uh this one eagle algorithm
72:07personally I don't think speculative decoding works because main problem is alignment okay so let's start with what is uh speculative decoding main problem is that in transformer architecture All these tokens are generated
72:23sequentially one by one by one.
72:27How about just use a smaller model and let a smaller model to generate maybe let's say four or five tokens and this teacher model or we can say according to our world cup algorithm we can say referee.
72:43So referee will decide how many uh tokens it accept and this loop keeps on going on and our assumption is that there are certain domain where this kind of things will work like maybe in decode
73:01maybe in coding or where almost there is no creativity uh each uh code or syntax is almost similar.
73:09So maybe it can help it.
73:11But uh based on personal testing, I didn't find this speculative decoding useful at all.
73:19But other techniques like uh self speculative decoding where teacher model also have one head auxiliary head and it will do same similar kind of things what this base model or small model is doing it.
73:35But then this eagle came Eagle 1 2 3 I don't know how many version versions are but it is just saying that instead of creating instead of generating tokens uh let's uh train a small model inside
73:50train a small model and just take a features from one of its uh one of main models layer so that instead of generating token uh it will generate uh this features so
74:03so uh so eagle is uh Eagle is better compared to this other kind of technologies and then another one is Medusa which is just saying that just generate all the tokens parallelly.
74:18Uh okay.
74:19So here so here in this slide
74:23yeah the next slide.
74:25Okay.
74:29Okay.
74:29Yeah.
74:31Okay.
74:31Now we come to uh now we will come to this one prefix caching.
74:35So I don't know whether people are using this one static prefix caching or not but thing is that main problem with prefix caching is that sometimes we type and make a small kind of mistake
74:48and this standard static prefix caching is basically it takes a prompt do some hashing and then next time when user asks similar kind of question it will try to match the hash.
74:59So if hash is uh if hash is equal then then it will instead of recomputing all those K and B it will just uh take it from from the storage
75:10but you know that sometimes we make a mistake or maybe we can just change a word or letter something like that then we have a very higher uh cache uh hit cache
75:20miss hit rate so that's why uh this one uh radics tree so radics tree is becoming very popular and also also because of agent.
75:31So I think almost everyone is doing agent and most of the computation is going during TT during test time inference kind of thing where we keep on asking same kind of questions and prompt for example
75:44you are an expert software engineer multiply by 200 times.
75:49This kind of loop keeps on going inside this uh agentic agentic kind of things where it is al necessary to keep uh or store similar kind of things in a radics tree.
76:03So radics tree is just so so radic tree is just advanced version of this prefix tree where where we will just where we will just collapse a node if it does not have a does not have any branch
76:18and for this kind of work where keep on repeating same thing this uh red x tree helps a lot and st lang uh use this kind of algorithm for prefix caching.
76:37Okay.
76:38Yeah.
76:38Then there is another thing.
76:39One is tensor RT LLM.
76:42This is very confusing.
76:43When I first started, I was I was just confused.
76:46What is tensor RTLM?
76:49So yeah.
76:50So tensor RT is just a uh it's just a standard uh SDK kind of thing.
76:56Tensor RTLM is just an inference engine just like VLM, SG lang.
77:02But problem is that it is related to Nvidia.
77:06They optimized each and every layer and every problem as I mentioned in our world cup algorithm.
77:13They just break everything and optimized everything at hardware level also.
77:19So uh yeah.
77:22So okay next.
77:27Yeah.
77:27So for this workshop we also uh did some benchmarking like which is best uh so our setup was something similar was so so we did two kind of testing.
77:39First one is without uh without agentic testing where we just so we use this shared GPT uh this one data set and uh just ask those questions uh using VLM
77:54and SG lang.
77:57Okay.
78:06Yeah.
78:06Okay.
78:09And let me just zoom it up.
78:13Okay, great.
78:14Okay.
78:15Yeah.
78:15So, yeah, for this workshop, we used H100 and of our first testing was that uh we just uh we just asked uh we take questions from shared GPT and put it into
78:27VLM, SG lang and we found that actually there's no statistical difference between which one is better.
78:35So both have almost similar kind.
78:37So both are fulfilling similar kind of request per second uh TTFT and latency.
78:44So but only difference we have seen during agentic uh agentic branching.
78:50So uh what we did was that we asked that similar kind of question that you are the best this one software engineer in the world just solve the problem of traffic congestion in this city kind of thing.
79:05Then we put this into LLM.
79:08LLM generates some output.
79:10Then we did another uh round two also.
79:14So once this LLM generates this output, then in round two we have specially mentioned that uh provide uh review the proposal and give ratings from 1 to 10.
79:28So this uh two turns we did uh and this loop keeps on uh repeating it.
79:35Uh what we found is that for this kind of uh workflow where everything is standard all those prompts and context engineering comes into the picture.
79:47If we do proper this agentic branching then I think uh this HG lang is three to four times better.
79:55But again this depends upon the different setup maybe uh if you do it uh you may get different results.
80:03Okay.
80:04Yeah.
80:04So I think uh did we uploaded it on GitHub?
80:09Okay.
80:10Yeah.
80:10So the PDF is like also in the drive.
80:15Uh it's the same link as the slides.
80:19So a quick summary here.
80:23So on a standard API workload throughput you would see like a VLM and the SG lang would behave same.
80:32So if you don't have if you have like a standard workload definitely go with VLM.
80:36It's the production default anyways.
80:39But what Tanme was also saying is when you try to like make it like agentic workloads that is where like your SG link really shines uh and uh it kind of like provides you all the benefits.
80:57So yeah keep like VLM as a default but if you have agentic workloads probably try to move as the towards the SG lang.
81:05if you're not happy with DB LLM.
81:08Uh but uh okay.
81:13Uh let me Okay.
81:26And then like there is like the like a comparison that is done at the 120 billion like for the GPTO OSS 120 billion.
81:36Um this is a benchmark that was prepared by clarify.
81:42So there is like a blog link here.
81:46Oh nice.
81:48Okay.
81:50Yeah.
81:50So they did the similar benchmark and they included like a tensor RT LLM in it.
81:58Definitely you can always go through these benchmarks and try to understand which basically suits your use case.
82:05As we mentioned like Tensor RT they try to optimize the hardware side as well having the peak hardware performance.
82:17Wait uh this is okay.
82:26Yeah.
82:27And then like in terms of when you want to dep pick like your engines once you figure out like between VLM, SG lang tenserati so that there are some new engines that are popping up
82:42Nvidia Dynamo for sure.
82:44Uh so they are also for the agentic uh session routing.
82:49Uh hugging phase is always there.
82:51It's a simple no server.
82:54Then there is like an MSAR engine that was recently proposed by Stanford.
83:01They are for like the multimodel.
83:05Uh so definitely you could explore those and when you try to basically just to like give a quick summary uh you we start with like a baseline we try to find
83:19what model could fit our use cases.
83:22Um, so you could pick like uh Deep Seek, you could pick like don't pick like a Mistral 7B.
83:31I mean, it's not good.
83:33Uh, but yeah, so you pick your model and you want to like have a smaller memory and you want to try to fit that bigger model into smaller memory
83:44so that you could save cost on the GPU cost.
83:48So you can do like all those com quantization then you can apply all those serving optimizations by using the right serving engine under the hood.
83:58So that can really provide you that throughut that you really want.
84:07And now something that you can do uh after going back home pro because we cannot like actually go over all the material here uh is definitely reading about some of the source informations
84:23like different attention mechanisms different like these engines like try to just read the different benchmarks which are present online as well and then there are a lot of like
84:37in-depth guides or the next phases of it which is like learning about some KV eviction strategies.
84:45So world is moving towards having a separate KV cache engineering domain.
84:50So you want to understand what's going on in there.
84:53So KV cache KV eviction cache compressions hybrid memories.
84:58So there are like lot of solutions that are happening around there.
85:01So always try to stick to those foundations or like the fundamentals or the first principles and try to see like which solution basically solves what problem and whether you actually need that
85:15problem to be solved for your use case and then there is like distributed LLM inference which is like a different painoint altogether.
85:26Uh you would probably need like a two-hour workshop there as well.
85:31uh to like go over like all the internals do all the hands-on.
85:41Yes.
85:41And this is something we are trying to propose for the AI engineer New York session uh which is to like dive deeper into the advanced sections of the LLM inference.
85:51So this workshop was more for the like beginner and the intermediate level.
85:56Um so in this form we do have like a feedback as well plus also the interest.
86:02Um if uh you think like we need certain improvements on certain sections definitely give that feedback as well and if you want to see this workshop in like New York
86:14uh fair you I mean definitely feel free to enroll your interest.
86:22Uhhuh.
86:24How is it possible?
86:28Well, let me just check.
86:46Huh?
86:48URL works, right?
86:50Not the QR code.
86:51Okay.
86:52Probably I forgot to link those two together.
87:00Z G A five.
87:11Okay, cool.
87:13Yes.
87:13So if you can give that feedback let me just okay that will be fine um and yeah I think we would like to wrap this workshop then I'm sure like lot of you would be having a lot of questions so we can take all those like offline
87:34uh we can meet uh and we can uh like talk about those questions.
87:39Yeah sure.
87:41Uh thank you everyone.
87:42Thanks for joining.
87:44Uh I think it was really meaningful and all of you like came here.
87:49Uh thanks a lot.
87:51Yeah.
87:51Thanks.