Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher
https://www.youtube.com/watch?v=y2W4FNAuPEA
AI Engineer · 88 min · 736 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:13] Uh so good afternoon everyone.
[00:16] Um my name is Hershel Jan and he is Tanisha.
[00:20] Uh and we would like to welcome you all in this two hours workshop on the LLM inference.
[00:26] Uh so the goal of this workshop is to understand this domain from the first principles uh dive deeper into it and like understand what's going on throughout the industry.
[00:41] Uh a bit of background about us.
[00:44] So I am a senior software engineer at Audible.
[00:48] uh have been building MLA data platforms for the past five years and on the sides I have been writing this opensource handbook on LLM inference and Tanme he is the
[01:02] senior quantitative modeler at XAN cup bank corporation he recently completed his PhD and he has been actively doing research in the agent verifiers and the world models Uh so a quick show of hands here.
[01:21] Uh Vu here is like brand new to the LLM inference.
[01:27] Okay, great.
[01:28] And Vu here has like deployed these models in production.
[01:33] They have been tuning it.
[01:34] They have been serving the production traffic.
[01:38] Okay, great.
[01:41] So this workshop is targeted towards the beginner and the intermediate level.
[01:47] U and all of the slides and exercises they are in the repo.
[01:50] I will share that soon.
[01:55] Here is the quick agenda for the workshop.
[01:58] We will start with the problem statement.
[02:00] We will try to understand few of the pain points around LLM inference.
[02:06] uh then we understand what causes those pain points and build our foundations from there.
[02:12] Then we will dive into like two kind of the optimizations that we do like the model optimizations and the serving optimizations.
[02:21] uh and then we start learning about different serving engines that are available to deploy our LLM inference solutions in production and we will showcase some benchmarks and the decision chart on like which engine to use.
[02:40] Cool.
[02:40] So to understand the pain points first we need to know what is like LLM inference.
[02:47] So, and probably a lot of us already know this.
[02:51] Um, but yeah, anything that you ask your AI to do like whether it be generate a video, audio, analyze any text, uh, analyze your medical reports or like your tax bills,
[03:04] all of that is like an LLM inference.
[03:08] And this market is like approximately $23 billion today.
[03:15] uh semi analysis recently shared that if you want to model like a Google search queries with LLMs, you need like a profit drain of like $36 billion and query cost
[03:29] has to be less than 0.5 cents to keep your search business profitable.
[03:35] On the other hand, the business insider mentioned like your AI has to be put on diet and everyone has to start auditing and budgeting their token usage and all of this is happening.
[03:48] Why?
[03:48] Because your hardware is limited, compute is expensive, your inference is expensive and with the growing need of like more and more AI usage, this inference cost is rising more and more.
[04:03] So this stat, it's an old stat from the open AI, but it's still it's still true.
[04:11] So if you look at the like training cost of the GPT3, it was like around $4.6 million.
[04:18] It was a one-time cost.
[04:20] But if you see the inference cost that has been like uh it's a recurring cost because it's a operating cost that scales with every user that comes in that every token that comes in every session that uh is being initiated on the like AI
[04:39] and there are only two ways to basically counter this.
[04:43] Uh one way is you reduce your token usage.
[04:47] um alternative is you should try to optimize your inference solutions as a inference service provider for your customers and for yourself.
[04:58] And so this um we have been seeing like lot and lot of like new solutions coming out every then and now.
[05:06] Um and so the idea would be like okay we will try to build those foundations that will help us understand and evaluate like whatever ships next.
[05:17] Um so yeah to get started like we will do a quick demo like it's a short demo of like what are the different pain points around inference u and so this is the repo
[05:31] uh I mean you can pull it or you can also open it on the GitHub uh it's called LLM inference at scale a bit of background here like four months back when I didn't knew anything on the LLM inference
[05:45] um I started learning it I saw like lot of resources were scattered.
[05:49] So we started putting it uh like all together in one place uh so that it could benefit people.
[05:56] Um yes.
[05:58] So let me actually get out of this slideshow mode and probably go into I will go to this extended mode.
[06:13] Um okay great.
[06:19] Uh yeah so in this uh repository if you see a readme file there is like a link to the slides.
[06:27] Uh so this it will be like this folder where you have like a pptx and there is like a benchmark report in there.
[06:36] uh you can always like download it and then for the demo purposes uh we have couple of Jupyter notebooks.
[06:44] Uh we have like collaborated with Moab who are the like Google collab alternative and what they basically provide you is like a free RTX 6000 GPU.
[06:54] It's a 100 GB V RAM GPU.
[06:58] So and we have like already set up these notebooks so that it becomes easy to like experiment with and like all of the assets and everything are preset for you.
[07:10] Uh so uh we will start with like a simple demo a probably let me just see Okay.
[07:31] Um yeah.
[07:33] So when it comes to the inference, you need to do an inference on a certain model, right?
[07:40] Uh for the workshop purposes, we are using a simple ML 7B model.
[07:45] Uh it's a small model of around 15GB in size.
[07:49] So we we are going to like load that into the GPU.
[07:53] So, and we would look like some of the GPU stats as well.
[07:58] So, we see like okay, we are working on the 6,000 Blackwell.
[08:01] Uh, and you might be thinking I'm not running the cells because I don't trust the Wi-Fi at conferences.
[08:08] So, yeah.
[08:09] So, I would probably be just going over the results uh that we kind of ran previously.
[08:18] Uh, yeah.
[08:19] So, we have like a GPU which is like 102GB.
[08:23] Uh now the first thing that comes to my mind is like what's my memory consumption looks like when I do the LLM inference.
[08:31] So I load this model uh and I see like okay I have like a 15 GB here.
[08:37] So I have roughly like 87.5GB.
[08:40] And now when I do the like inference here uh what I notice is like the more the number of inputs I pass more is the memory that I need.
[08:53] uh and it's increasing slowly but it's still increasing.
[08:56] [snorts]
[08:56] So imagine like if you have a context length of like around 4,000 or 16,000 or 32,000 uh tokens.
[09:04] Uh so this memory could like really grow big and it you could actually get like all of those out of memory issues.
[09:13] Uh so definitely this is like your problem one like your memory increasing with the increase in tokens.
[09:19] So in form of like a simple visualization it looks like this.
[09:25] The second problem that you would see is like the time to your first token it's very very slow.
[09:32] Uh we measure it by a metric called TTFT.
[09:35] It's a short short form of it.
[09:38] uh and when you try to like measure the TTFT uh with the like input size you would see like longer the context you would see like this uh TTFT being slow.
[09:53] So now there are two problems.
[09:54] Your memory increases with the token size.
[09:57] Your TTFT increases with the token size.
[10:00] Uh sorry not the token size, the context Uh and then the third is the like throughput.
[10:09] The throughput is like how many tokens can you serve per second and then how many users can you serve per second.
[10:17] So if you take a very very vanilla implementation on your local system uh it would be like very sequential.
[10:24] So if you send like five requests all those five requests would be catered like sequentially rather than parallelly.
[10:32] Uh and so like your request basically takes more time to complete if you have like multiple users.
[10:41] So these are the like three problems.
[10:43] There is a fourth one.
[10:44] I haven't described it here.
[10:45] probably we will build that intuition as we move forward.
[10:50] Uh but let's remember like these are the three problems the memory TTFT and the throughput.
[10:58] Cool.
[10:58] Uh I will go back to the slides.
[11:07] Okay, perfect.
[11:15] Uh so it should be this.
[11:20] Uh is it visible?
[11:35] Oh, you you need the Oh, okay.
[11:39] Workshop.
[11:40] Okay.
[11:47] Yeah.
[11:48] So, within that repository, if you see a workshop folder, you see that readme and then the readme has all the links, the slides and the demos.
[12:04] Uh, does that work?
[12:09] Okay, perfect.
[12:16] Okay.
[12:17] uh so let's start working through the foundations like let's start understanding uh what are the reasons behind those pain points and for that like we have to look at this inference pipeline
[12:29] um so we get like an input text uh that text could have like any number of words you convert those into the tokens so for simplicity you can assume one word equal to one token
[12:47] uh then you kind of convert them into like the embeddings and then you send it to the like transformers uh like there are 32 layers of transformers but that's specific to the Mistl 7B different models have different kind number of layers
[13:02] uh and then you generate a new token and that token basically goes back to the input then you generate another token and that keeps on going now in this entire pipeline
[13:13] you would see like 95% of your compute is like taken by these transformer layers.
[13:18] So it's worth looking at like what goes within this transformer layer.
[13:25] Within this transformer layer you would have like more layers.
[13:28] You have like a normalization layer, you have an attention layer, you have a feed forward layer and all.
[13:35] And attention layer is the one I think that has been very very famous.
[13:40] Attention is all you need paper.
[13:42] I think that's very well known.
[13:44] So attention is the most compute intensive layer and we need to understand what goes within that attention layer.
[13:54] So what does attention do?
[13:57] Attention uh so if you have an input text it needs to find the attention scores of every token with respect to all of the previous tokens.
[14:06] And to do that what it needs to do is like it needs to project every token into like a key query and the value space.
[14:14] So in like in a simpler terms just u understand this like if you have 10 tokens then it needs like the 10 different query key and the value vectors.
[14:27] If there are 100 tokens you would need 100 key and the value vectors.
[14:31] If there are thousand tokens you would need thousand key value vectors.
[14:35] And so like your number of the key and the value vectors they increase as you increase the input size.
[14:44] Uh and if you calculate the like KV size per token uh for a mist 7B it comes out to be 131 KV.
[14:54] Uh this is because like uh you have two vectors K and V. You have to multiply the size uh one vector is like 128 dimensions.
[15:04] you have to multi multiply it by 32 transformer layers and then you have to multiply it by the KV heads for ML 7B it's gate KV heads it's not like 32
[15:17] because it uses a different kind of an attention mechanism u which we will talk about for sure but yeah so the KV size per token is like your 131 KB
[15:31] now imagine if you have 4K context uh So that size becomes like half a GB.
[15:37] Uh if you do like 16k context that size becomes 2.1 GB.
[15:42] Uh now multiplay by the users like assume you can serve multiple users together at the same time within that GPU you could have like 42GB with a 4K context and 80 users.
[15:58] Um and if your GPU is only like let's say 24GB, you are already running out of the memory.
[16:05] So you cannot serve that many users with that many context.
[16:12] To visualize this, look at a GPU memory.
[16:15] So the GPU memory has like a model weights which are pretty fixed.
[16:19] These are pre-trained weights.
[16:21] Uh there is like an overhead that is also fixed.
[16:25] that also like that changes but it does not change that much u overall you can assume it's fixed and then there is like a leftover memory so this leftover memory is what being used by your KB me like key and the value vectors
[16:41] so assume like you have a one user you can only serve that many key and the value vectors or that many tokens which can like fit in this entire 80GB
[16:57] uh like memory that is left.
[17:02] Uh so we can show this with a simple demo too.
[17:08] Uh okay, let me Okay, great.
[17:25] Uh, let me see if I can actually run this.
[17:31] What probably?
[17:38] Where the heck is this happening?
[17:49] Okay, great.
[17:52] Yeah.
[17:53] So you would see like the GPU is attached.
[18:07] So here we are just trying to confirm the like memory based on the maths and based on the intuition that we have built.
[18:14] So the model memory is like let's say if you have a 7 billion parameters you are doing a 16 bit precision your total memory comes out to be 14.6
[18:23] GB with you can basically verify that with the maths.
[18:28] So if you do all that math that comes out to be the 14.6GB.
[18:33] Uh now comes the KVK and the KV size.
[18:37] So this KV size is like your 131 KB per token.
[18:42] Uh and if you do that maths and you try to like visualize this.
[18:49] [sighs]
[18:52] Oh, sure.
[19:00] Wait.
[19:03] Okay.
[19:05] And then let's just visualize this.
[19:09] Okay.
[19:10] Great.
[19:11] Uh yeah so this is the like a memory chart.
[19:15] So if you see like as your context increases your memory keeps increasing.
[19:21] Then another thing to realize is like as your users increase uh then also your memory increases.
[19:29] So if you want to serve like 160 users uh on a GPU you can support like uh you can only support like a lesser context length.
[19:40] So there is always a tradeoff between what context length you can serve versus how much cost you can save by like putting your multiple uh users or the concurrent users into like a single GPU.
[19:55] So you have to always take that tradeoff and and we will go through that like uh in couple of more slides.
[20:06] Uh can you repeat please?
[20:14] Uh I'm sorry I'm cannot hear you.
[20:24] Yeah.
[20:33] Cool.
[20:35] Uh okay.
[20:36] Great.
[20:37] So let me pull back.
[20:41] So that was like memory.
[20:43] Uh we need to understand uh why we had like uh slower time to first token uh when we increase the context length.
[20:54] So for that like we need to understand the two phases of inference and those phases are like the prefill and the decode phase.
[21:01] I think you would all seen like a lot of articles but we just wanted to explain it.
[21:06] Uh so when you send like a lot of uh like when you send these input tokens what you want to do is uh you want to build those key and the value vectors that I mentioned for all the tokens.
[21:21] Then you want to compute the attention scores of every token with respect to the previous token.
[21:27] All this operation that you do it's a very very metricsh heavy uh it's a very very compute heavy operation and we all know like the GPUs they are like
[21:38] very well suited for a heavy compute workload so we call like a prefill to be like a compute bound uh and it does take some time to complete so whatever time that this phase takes to complete
[21:54] that's your time to the first token So if if you have like a more input tokens, you have to generate more key value vectors.
[22:03] You have to do lot more attention math and because of that your TTFT becomes more more slower.
[22:12] Where is if once you generate one token you need to keep doing this to generate another tokens sequentially one after another.
[22:21] But in that process every time you have to build the key and the value vectors of all the previous tokens which is same as prefill like you were building key value vectors there also here also
[22:34] but in decode phase you are only computing the attention math for the new token and that is why it's a very less it's lesser compute oriented and it's also called as memory bound.
[22:48] We will see it shortly why it's called as memory bound.
[22:54] So in a classic timeline you would see prefill and decode phase like this.
[23:00] So time taken by prefill that's your time to first token and then your time taken by every decode step that's your uh basically your inter token latency.
[23:13] So that's like the fourth metric uh that you need to worry about like what's the time being taken by your decode step.
[23:25] Okay, cool.
[23:27] Uh now why why does the like decode step or why does decode takes time and why it's being called as like a memory bound operation?
[23:38] Let's try to understand that.
[23:40] uh to understand that we need to look at how the metrics map basically works on the GPU on a high level.
[23:48] So GPU has two kind of memories.
[23:51] You have a high bandwidth memory.
[23:53] You have a shared memory.
[23:56] So the high bandwidth memory is a larger size but a lower me like lower bandwidth.
[24:02] By lower bandwidth I mean like you can transfer data out of it at a lower rate compared to the shared memory.
[24:09] So the shared memory is smaller in size but it has a very very high bandwidth.
[24:14] Uh that means you can transfer data in and out of it with a very first thing.
[24:19] So when a when you have to do a metric math so you have to pick the data in chunks from the high bandwidth memory you have to put it into the shared memory.
[24:31] Do that math write back the result into the high bandwidth memory.
[24:37] uh for the prefill phase when you have to do this you have to do this matrix math only once but for the decode phase you have to do this metric
[24:50] math uh like uh again and again because you're generating each and every token sequentially and so like you it doesn't matter like how fast is your decode because now you can transfer your data out of the
[25:09] high bandwidth memory into the S shared memory at a certain speed because you are limited by the high bandwidth memory bandwidth speed and so that governs your like token sealing like
[25:23] at what rate can you actually generate tokens out of the decode step.
[25:32] If you look at this in the roof line plot uh so there is a left section which is called to be a memory bound.
[25:43] Mathematically it's governed by the arithmetic intensity.
[25:47] Arithmetic intensity is the number of flip-flop operations that you perform per bite of data being transferred.
[25:55] So for the decode step uh decode step since you are transferring lot of data like the key and the value vectors of the all the previous tokens the model weights but you are doing the like less computation because you're computing attention math for only one token.
[26:14] Uh so it's arithmetic intensity is very low but for a prefill phase you are transferring the data once but then like you are doing this heavy computation and so it's arithmetic intensity is very high.
[26:30] So now you know like in terms of mathematics like why the computer like why the arithmetic intensity of prefill is very high compared to your decor.
[26:43] Uh
[26:43] [snorts]
[26:44] okay so this is like another small small demo.
[26:50] Uh every time I have to Okay.
[26:57] Okay, great.
[27:00] I hope this is already running.
[27:03] So yeah, again we are loading the model.
[27:07] Now this is the like the prefill cost.
[27:10] So what we are basically doing is uh we are getting the like um the input uh text and then we are trying to generate this um the prefill step
[27:21] the amount of time it takes.
[27:22] We see like as we increase the like size of the input tokens this prefill is increasing.
[27:29] So you and this is the reason why your TDF increases and then like your decode time.
[27:37] So the decode time is like on average it stays about the same.
[27:41] Uh and so if it is like assuming like you ignore the like cold start your decode time is like approximately around the average line.
[27:50] it it it it is still impacted by like u the input size.
[27:57] It's not like it's a constant uh time and it is because it still needs to pull the key and the value vectors from the memory for all the previous tokens.
[28:07] So there is still like uh that uh basically small increase in time that you would see with the decode step.
[28:15] And then this is the like classic uh roof line plot.
[28:21] Uh okay.
[28:27] Presentation.
[28:31] Okay.
[28:32] Five.
[28:33] Okay.
[28:33] Great.
[28:35] Okay.
[28:36] So now now let's try to understand like uh the throughput dimension.
[28:43] You want to understand how many users you can actually serve and I think we saw like a diagram of the GPU memory where we saw okay there is some memory that is free for the key and the value vectors to grow.
[28:57] So assume like you have just a single user uh what's the total KV size that you have you can basically support it's defined by your context limit.
[29:12] uh the max users that you can support is like whatever is your GPU uh availability like whatever is the memory that is available in the GPU you divide it by the key and the value size per user
[29:26] uh and when you do that like it comes out to be like your the concurrent users.
[29:33] Now assume like your GPU is fixed, your model is fixed.
[29:40] Uh so your KV size per token is fixed.
[29:46] There are only two dimensions that are left here which is context and your concurrent users.
[29:53] If you want to serve more concurrent users, you have to reduce the context length.
[29:57] If you reduce the context length, you could impact your quality.
[30:02] Uh so these are the two dimensions right now that we are trading off.
[30:09] Then if we but can you actually serve the like max number of concurrent users?
[30:17] Uh in an ideal world probably not because every business has like a latency SLO that we have to meet.
[30:28] So if you remember like in the decode step I said the time for the decode still increases if you have more inputs.
[30:38] It also increases if you have more users.
[30:41] So ultimately uh your inter token latency also gets impacted if you have like a higher batch size and your TTF also gets impacted.
[30:54] So now there is a third dimension you have to worry about which is like your latency.
[30:58] So the three dimensions that you have is like a quality latency and the throughput.
[31:04] So it comes out to be like this trade-off triangle where you have to choose between the two.
[31:11] So for a premium chat application you would want to prioritize definitely the quality and you want to prioritize the like the latency.
[31:21] You would not want your users to wait infinitely for the like or like not infinitely but probably for the larger latency.
[31:31] You can always sacrifice the number of users you can support on the GPU and probably take that costed being more customers in form of like and and like if you consider like an agent
[31:45] uh sorry the async agent workload you would want to like prioritize definitely quality and the throughput uh because these are the longunning tasks uh and you would want to like serve as many as concurrent tasks.
[32:00] fast as possible but with a very very higher quality.
[32:07] And often like we think like okay if the GPU is like a very expensive GPU uh that might not be a good fit for us.
[32:19] Uh but it turns out that could actually serve you the lowest cost per million uh tokens.
[32:27] Uh but you really have to trust your kind of calculations on the max users that you want and like uh you really have to make those estimations uh correctly.
[32:43] Uh so we do have like uh let me just Where is this?
[33:02] Okay, great.
[33:03] So, for the capacity calculator, uh there is like a link to the collab because I was facing certain issues with molab.
[33:12] I had to migrate out the wall widget library and I didn't have time.
[33:17] So, being lazy, I just picked collab there.
[33:21] Uh apologies to Moab.
[33:25] Uh so my VR is connected.
[33:35] Okay.
[33:46] Wi-Fi probably.
[33:51] Okay.
[33:52] Great.
[34:02] So what we have done over here is we have like shaded some like the GPUs with their V RAMs, bandwidths, the flip-flops and the cost per hours.
[34:12] Um then we kind of like built this simple uh like uh capacity calculator.
[34:19] This is just a KV visualizer uh where you kind of like when you increase the number of tokens uh you see like your KV size it increases and when you increase the number of users your size is like increasing at a much faster rate
[34:37] and then in this capacity calculator uh let it run.
[34:49] So we have like a model which we which is like a 7 billion parameter model that we selected.
[34:58] We set the like precision to be FP16.
[35:01] Uh now we decide the way we basically go by the GPU decision is you have to decide what's your like you have to fix one dimension first which you care about the most.
[35:15] for premium chat I mentioned like latency is definitely the one uh and then like for the async workloads the batch the minimum batch size that you want to serve
[35:27] for from like a single GPU that is the second dimension so you want to fix these first so I will go about like in a premium chat application uh
[35:39] so I can go ahead with like 10 milliseconds latency a minimum batch size I don't care like I can so I'm okay with like probably two uh okay so probably with the seven
[35:56] concurrent users on a single GPU and then like my context limit is very important to me because I want to focus on the quality as well uh and so like I do see like some of the GPUs
[36:10] so the H18GB it's like a $8 per hour but like am I 300x is it?
[36:20] Yeah.
[36:20] So it's like around $10 per hour but if you do all that throughput math that we shared in the mathematics before you could find like your cost per million dollar tokens
[36:33] that could be very very that could be like lesser.
[36:37] So you need to do such calculations by fixing those dimensions and you need to decide your GPU to like reduce your kind of inference cost.
[36:49] This is at least the first step that you can take towards optimizing the inference.
[36:56] Okay, cool.
[37:01] So the next slide.
[37:05] So let me Okay, great.
[37:10] And so like now the next thing is about the model optimization.
[37:14] So we are now basically have built that foundation where we understood some of the pain points, reason behind those pain points, why those were happening um how we could like
[37:26] address that GPU capacity thing.
[37:29] We need to understand what can we do like what can we further do about it.
[37:33] So it it is about the model optimization and I think I would like to invite Tan I he can talk more about these model optimizations provided he has worked uh on this like during his research times
[37:48] okay I can control yeah here okay hi everyone uh mic check am I audible at last yeah okay so hi I'm Tesha I work as a senior quant modeler and also I am an AI researcher.
[38:07] My work focuses on a agent verification and right now building world models.
[38:13] So for this one model optimization before we start model optimization so I created a research template so that it will be easy for us to understand all these complex things.
[38:25] I so our template is simple.
[38:27] First we will identify the problem.
[38:30] Second step we will solve the problem using two algorithms.
[38:34] These are just fake algorithms.
[38:35] So first algorithm is called ostrich algorithm.
[38:39] Whenever we see uh just like ostrich whenever we see a problem ostrich put their head into the sand.
[38:46] So same thing we will do whenever we face a problem we will just ignore it.
[38:51] So this is an important algorithm we should follow.
[38:54] Second one is created it is called world cup algorithm.
[38:59] For example, we don't know who will win this FIFA World Cup.
[39:03] So, what organizers did, they uh break the 48 teams into 12 groups, uh then round 32.
[39:12] So, round 32 right now is currently going on.
[39:15] Uh then round 16, then quarterfinals, uh then semi-finals and finals.
[39:21] So what they are doing is that uh they are breaking it into a smaller problems and the useful results are moving forward.
[39:30] So same analogy or same algorithm we will use uh to understand this model optimization all those things.
[39:38] So yeah let's start.
[39:41] So I have one H100 GPU.
[39:46] I have to use this open-source model what is called GPTOSS 120 billion parameter model.
[39:54] So right now I think it's so they have trained it on BF float 16 and weight is 240 GB.
[40:02] What should I do?
[40:04] This is the problem we have.
[40:06] So first thing what we have to deal do is that 240 GB and 80 uh GB H100.
[40:16] So and I have to fit only in one GPU or not in multiple GPU.
[40:21] So what can we do?
[40:22] I think simple step is that just compress it.
[40:27] But how should we compress it?
[40:29] Uh that's the another challenge.
[40:30] So if we compress BF BF float 16 to FP8 uh then it will be around 120 GB but our GPU H100 is still 80 GB.
[40:42] So what I think they did is that they compressed it into further MX uh MX FP4 and I think size is around 65 GB.
[40:53] So this is something we can do uh compress but question so and we will use over this ostrich algorithm we are assuming that uh there is no loss in compressing a bigger model into a
[41:09] a smaller size.
[41:10] Second thing in uh in this one okay yeah so in this one in this slide we have used this mistral 7B so 7 billion parameters so it's a small model 7 billion parameters so
[41:25] uh so if you multiply it by two bytes so it so weight of it's around is 14 14.5 GB which can easily fit into H100 or even a a40 so
[41:39] so Next uh what we can do is that like mistral 7B instead of compressing it a floating point 16 we can apply different techniques like int 8 or int4
[41:52] or nf4.
[41:53] So basically we have to just use ostrich algorithm and just believe that uh there is no quality loss kind of things but somehow we also have to mathematically prove that by doing some kind of test testing on some external benchmark that whether it is working or not.
[42:10] So the and this comes under post training quantization kind of thing.
[42:16] One can also do uh this one uh during finetuning one can also do this kind of quantization.
[42:22] This comes under a quant training kind of thing.
[42:25] So uh let's move to our next problem.
[42:31] So we have this huge matrices just just imagine imagine uh 1,000 by 1,000 uh dimension matrix A and another matrix matrix um 1,000 by 1,000.
[42:50] So if we multiply uh if we multiply by this two matrices so number of operations will be 1,000 raised to the power q and this is kind of a problem in terms of
[43:04] uh uh in terms of computing.
[43:06] So we wondered our matrix multiplication should be fast and it should save memory.
[43:12] So what should we do?
[43:15] We have a giant matrix.
[43:17] Okay, let's take this one.
[43:19] Uh, Mr. 4096 by 4096.
[43:22] What should we do uh to solve our problem of speeding up the things and saving the memory 4096 by 4096.
[43:33] So first thing is that we will use just our world cup algorithm.
[43:36] We can decide a random number just break the block vertically.
[43:42] It does not matter what you are choosing it.
[43:45] So you have so let's say uh we have 4096 uh columns we will break it uh we will break it into a group of 128 column each.
[43:56] So 128 128 128 128 uh vertical vert uh vertically so we will get a 30 we will get this 32 blocks if we divide this 4096 then what will happen by doing this thing?
[44:13] So if we just divide this one vertical vertically then we can use a multiple GPU to speed up the process.
[44:21] So this kind of thing is called multi head attention.
[44:27] So what else can we do?
[44:29] We have a big matrix like uh as I have mentioned that ostrich algorithm.
[44:36] So our main problem is sizing.
[44:40] So what we what we can do is that instead of having all those 32 uh 32 vertical blocks we will throw away uh 31 blocks and we will assume that one block is sufficient enough
[44:53] that all the queries uh can handle those blocks.
[44:57] Our loss will be almost negligible and we come up with this algorithm uh which uh and this algorithm is called a multiquery attention.
[45:08] So as we can see right now we are at two spectrum.
[45:12] One is multi head attention where we split it into 32 blocks and use different uh different uh GPUs or do some parallel processing and at the same time we are just throwing 31 blocks and
[45:28] uh we are calling this is as a multi-query attention.
[45:32] So uh so at both extreme we should be come up with a middle ground like something we can say that instead of throwing all the 31 uh maybe we can group we can group
[45:46] we can group some of the blocks together so that uh uh and we can assume that uh similar blocks will attend to a um similar kind of uh queries.
[45:58] So this kind of technique comes under grouped query attention which is very popular right now.
[46:04] Uh even in uh even in mistral or in other models this grouped query attention works.
[46:11] So right now we have understand that we have a big matrix uh we can divide it the way we want and doing some mathematical calculation prove that loss is almost negligible kind of thing.
[46:23] So what else we can do?
[46:26] So after that uh after this grouped query attention uh see uh we have a big matrix uh one is one is key and one is value.
[46:42] Let's compress that matrix into a latent vector and then come up with some algorithm to uh reconstruct from latent vector uh to our original matrix.
[46:55] So this kind of a strategy comes under this one um multi head latent latent attention but again it has some problems with rope because rope is position dependent and uh and it is position independent kind of thing.
[47:10] So yeah one needs to also include some uh index for keys also so that one can map it.
[47:17] But again main problem is that why why we are why we are multiplying all those big matrices.
[47:24] So because that's how this attention mechanism works that each token will pay attention to every token.
[47:34] So how about let's don't pay attention to all the previous token only pay attention to the important tokens uh which is important for us.
[47:43] So this is a kind of uh this kind of field is uh evolving.
[47:47] So this comes under sparse uh deepseek sparse attention.
[47:51] So uh yeah and yeah yeah yeah so okay next yeah so next one is flash attention.
[48:02] So uh so in flash attention so main pro so main problem is that uh uh so so currently so so currently not currently so right now almost everyone uses flash attention but way in 2022 or 2023
[48:20] uh so that's how it works that's how it works is that uh so uh this Q K query and A and key matrices they were in HBM.
[48:35] Uh it loads uh it uh first uh it loads into uh this one uh tensor core and it do some uh it do some calculation and then it will uh write it back to
[48:47] uh HBM and then uh this process goes on multiple times.
[48:51] So in flash attention uh what they did is that uh is that instead of multiplying the whole matrices so they just divided it into like our world cup algorithm divided the
[49:04] bigger matrices into a small tile and only put those small tiles uh into a SBM so that uh it can process multiplication fast and just uh keep uh keeping track of this some three variables
[49:18] so that they can calculate this online softmax.
[49:23] Yeah.
[49:25] Next one.
[49:26] So, yeah.
[49:27] So, so this is just mathematics.
[49:29] So, if we have a multi head attention if it is 524 uh KV uh then it depends upon how much how much grouping we want and so if instead of 32 KV head we only want to use uh 8 KV heads.
[49:47] So uh so so we can get a compression of 4x times and this multi head latent attention this formula depends on the model to model how many layers your model have.
[50:00] So in the original deepseek paper uh I think they have some 128 dimension 128 d 12 I don't remember the exact dimension but according to that uh they have used uh this one latent vector in which they have used 512
[50:18] as a dimension and some 64 for for rope index.
[50:24] So and then they show that it is a 50x 56x uh more compressed than multi head attention.
[50:37] Okay.
[50:39] Yeah.
[50:40] So uh so yeah so this is uh so this is the uh this is the trade-off uh trade-off diagram.
[50:46] So here I think we have not talked about this linear attention or mamba.
[50:50] So main problem is just all this m Matrix multiplication.
[50:55] Right now everyone is using attention.
[50:57] Suppose in future uh if we don't want to use attention or rather than generating tokens sequentially just use maybe diffusion models where we can generate everything simultaneously.
[51:11] So all these algorithms will change also.
[51:14] But here I think they have two more.
[51:16] One is linear attention and one is mamba.
[51:19] So according to uh this slide so if we are not compressing anything so MHA is just we are parallelizing the process so there is no quality loss so it's a good and then this
[51:33] uh grouped query attention which is I think almost uh every model is using uh just GQA and DSA kind of thing or yeah I think same thing we are providing in the attention mechanism
[51:48] scorecard So uh so I think uh this one mha quality is good throughput is uh throughput is okay and for grouped query attention it depends upon your use case also though yeah though
[52:04] quality is almost similar to uh multi head attention but use case also matters a lot yeah multi-query attention is just one extreme we are I don't know why but we are just assuming that we only need one block
[52:20] and all the queries will attend to that smaller smaller block.
[52:24] So, so quality is not that great for M for MQA and this multi head latent attention.
[52:32] So yeah if you have tried some this deep seat models so I think uh they are doing great job yeah in in quality wise besides that sliding window so all these are sub techniques which
[52:48] yeah yeah all these are some techniques like I just slide the windows all those things and instead of yeah instead of multiplying everything so linear attention is just saying that sum summarize everything first
[53:02] uh and then look up into it and then mamba this is just a state space model.
[53:08] Yeah, I can cover that.
[53:13] Okay.
[53:15] Uh cool.
[53:16] Uh thank you T. So for the model like optimizations we also have like the two notebooks here.
[53:28] So there will be I have to go to this.
[53:42] Okay.
[53:43] Uh so for the quantization uh like the demo uh this is is this already run?
[53:52] No. Let me just run this.
[54:03] Okay.
[54:03] So we are loading the model which is like uh ML 7B.
[54:12] Uh so this one is like with the FP16 baseline.
[54:29] Wait.
[54:31] Uh, did it run?
[54:35] Okay.
[54:36] So, it's uh two millisecond run.
[54:40] Did this run?
[54:41] Okay.
[54:42] So, yeah, this time it's fetching that model with the FP16 precision.
[54:57] the Wi-Fi.
[55:03] It's going to take time.
[55:07] Okay.
[55:11] Yeah, it because it's downloading the weights from the hugging face.
[55:17] Huh.
[55:20] Yeah.
[55:20] So, MOLAB is like running online.
[55:23] Yes.
[55:26] because it needs to make the network call through to the hugging phase and like it fetching I don't know like but it's taking time to download probably Okay.
[55:55] So good.
[55:56] Okay.
[55:57] So here we see like the memory size is like 15 GB around approximately with the FP16 precision.
[56:05] We are trying to do the 2x compression as Tmet talked about with the int8.
[56:13] Let's download.
[56:15] Okay.
[56:15] So we do see like your memory size is now like 7.5 GB.
[56:21] What that means is now you have a more s more memory for your KV to basically grow.
[56:27] That means you can either serve higher context limit or you can serve the higher concurrent users there.
[56:36] If you do the like in your basic you are doing the 4x compression so that with the 4x compression it would be more lower.
[56:45] It would be I think around 3 to 4 GB.
[56:51] Yeah.
[56:51] 4.5 GB and Yep.
[56:57] So this is Wait.
[57:02] So this is just a basic plot of like so these are the like theoretical numbers.
[57:09] uh we are not doing the like any throughput test here but uh usually you would see like your memory increases so pro you would also have like a bit of higher
[57:19] uh throughput.
[57:21] Uh from some of the benchmarks that we studied we saw like the intate uh compression it does have like a lower throughput.
[57:32] Okay.
[57:34] And then there is like a demo on the like the attention mechanisms.
[57:42] So for the attention okay I have to run this.
[57:58] Uh okay so it has run.
[58:03] Oh, wait.
[58:04] Why does it say no GPU detected?
[58:09] It should say the GPU should be detected.
[58:18] Oh, okay.
[58:37] Wait, but this is surprising.
[58:57] Yeah, I guess it's not like able to detect the GPU for some reason.
[59:04] Uh we do have like a GPU here.
[59:11] Uh okay, never mind.
[59:14] Yeah.
[59:14] Yeah.
[59:14] So, but the like basic idea here was more like as you try to move towards like compressing the computation like by using different attention mechanisms like moving from the multi head to the grouped query attention and then to the MLA
[59:34] you would start seeing some optimizations.
[59:38] Um I think yesterday night we were doing some benchmarking.
[59:42] Uh I wanted to correct this part.
[59:45] Uh so it wasn't like 50 56x it was 14x.
[59:50] Uh basically the demo had a mistake of like a computation uh where it did not multiply the number of layers.
[60:01] Uh yeah so apologies for that.
[60:03] Uh so this MLA is like a 14x savings work in comparison to like your multi head attention.
[60:14] Uh so now that we have understanding of the pain points, the foundations, the one side of the optimizations which is the model optimizations, we want to talk about what can you do on the
[60:30] like the serving side.
[60:33] So the first thing is we saw like when you perform like a simple decode step you are pulling it you are basically pulling the model weights and then you are recomputing the key and the value vectors
[60:48] for all the previous tokens even though you already computed the those vectors for the tokens.
[60:56] So there is definitely like a lot of compute wastage.
[61:01] Uh and if you kind of analyze the time complexity of it, it would come out to be O of N².
[61:08] Uh and the way to resolve that is like a classic trade-off against the memory.
[61:13] You can maintain a memory of those vectors against the tokens and you can reference that memory.
[61:20] So that memory was called as like KV cache.
[61:24] uh and the like the flow looks something like this and then based on this KV cache there were like four optimizations that were really possible.
[61:36] Um the first one is about the page detention.
[61:42] So what's the different what's the problem today?
[61:45] So when you send like multiple requests as the input to the GPU these requests are in a batch uh every request is allocated like a continuous memory storage let's say of
[62:01] I'm just taking an example like let's set uh 2 KB however like your request needed only let's say uh 1 KB so there is like u 50% of that memory fragmentation.
[62:19] Uh and this fragmentation basically leads to the memory wastage.
[62:24] That means there was a space in the memory where you could have served more requests but you could not because you were looking for that contigious block of the memory.
[62:36] So an inspiration to was being taken from like how the OS works like you maintain a logical memory and you basically have a physical memory.
[62:47] So in the logical memory it would still feel like uh that the KV vector for the like every token is like a contiguous but it will be mapping to a different physical address.
[63:06] So that really helped like saving a lot of memory.
[63:11] Uh and it was only possible because you they considered like memory as a set of blocks and you would be dynamically allocating those blocks as the request need as the like new tokens comes in and they need that
[63:27] kind of memory.
[63:30] The another lever is like when you are sending multiple requests in the batch GPU is like taking those requests but it does not accepts the new batch unless all the requests in that batch gets completed.
[63:49] So the diagram looks more like a page retention but here it is more about like when is GPU available to take the next batch.
[63:59] So there is a time period where GPU is like sitting really idle and you want to like resolve for that and for that like the idea was like okay let's do that continuous batching.
[64:17] So the continuous batching also really helped with like throughput because now you can ship more requests pretty quickly.
[64:24] Keep making sure like GPU always uh get is always like occupied and it's not like uh sitting idle.
[64:33] So you are saving on that compute.
[64:36] The third is the like prefix caching.
[64:39] So you remember like the KV cache helped you save the computation for a single request across the tokens.
[64:47] But what if like you have the same tokens across multiple requests?
[64:53] How do you basically save against that?
[64:56] So the prefix caching uh which was introduced by VLM exactly counters that and then the third is like we talked about the fourth actually.
[65:11] So we talked about quantizing the model but you could also you can also like quantize the KV weights.
[65:21] So that means now you you need like a lesser space for your key and the value vectors.
[65:28] That means you can serve more key and the value vectors in the memory.
[65:32] And that means like you can serve more tokens.
[65:34] That means you can serve more context context limit.
[65:38] And that means like you can serve more model quality and all of this is like uh already present in the VLM.
[65:51] You don't really need to reinvent that wheel uh and you can like deploy this VLM in production and you could see that basically growth.
[66:05] So next we have like a benchmark that we did.
[66:09] So this benchmark was let me see if I have that here the demos.
[66:23] So doing this benchmark takes like around 1 hour because you have to continuously stop and like restart the VLM servers and you have to load the models and all.
[66:35] So it does take a lot of time in doing the testing but I can like really tell you here what we are doing.
[66:42] So we have kept the model as same like the Mistful 7B.
[66:48] Uh and then we have like the set of input questions that we are sending.
[66:54] Uh consider them as the prompts.
[66:57] Then we have couple of helper functions here like checking the server is up or not.
[67:02] The server is the VLM server.
[67:05] Then there are helper functions to get the VLM metrics.
[67:10] uh and I will talk about like what those metrics are.
[67:14] Uh then there are like lot of the benchmarks and all and then you have to measure uh the KV usage and all.
[67:23] So these are the like helper functions.
[67:25] So the baseline is very simple like we have a hugging phase baseline.
[67:30] Uh this is the raw like sending the text to the LLM getting back the response.
[67:36] We see some results here.
[67:38] We saw like hugging phase has a throughput of like around 51 tokens per second.
[67:44] Time to first token was like 54 and then the inter token latency was 19.
[67:50] Uh this bas uh this was all run on the h100.
[67:55] Uh and then we start like a very default VLM server.
[68:00] So by default VLM provides you the page detention, continuous batching and the KV caching.
[68:08] So three things are present by default and when you try to compare those benchmarks you see your throughput is like almost 15x you are able to serve more tokens per second
[68:25] then your time to the first token uh that also rises and then your v the inter token latency kind goes down and then your KV versus users and the versus context rate increases for sure.
[68:47] Now when you apply the prefix caching to it so with the prefix caching you see like your throughput increases more your TDF decreases your inter token latency is approximately
[69:02] same uh and then your KV cache usage versus the users it's kind of going down the vers context it's not going down it's approximately same I think this is also approximately
[69:16] same it's like not that uh big of a deal when you apply the like KV quantization on top of it.
[69:27] So it becomes like so so you see like your throughput is like almost similar.
[69:34] Your time to first token is similar.
[69:36] Your token latency is similar but then your KV usage actually goes down.
[69:42] And this is because like you have quantized your key value space.
[69:48] Uh and then there is a concept of speculative decoding that TME will talk about.
[69:54] Uh so when you try to benchmark those so you also see like there is a uh like a bit of like the less KV usage there although like the results are approximately
[70:08] same.
[70:15] So yeah, I mean overall like these are the like the metrics across probably I should zoom out.
[70:26] Okay, it's not zoom out.
[70:27] It's not working.
[70:30] Great.
[70:31] So yeah, this is the like VLM benchmarks.
[70:35] Um it's your production default by the way.
[70:38] uh we will also share that decision tree uh when we try to talk about like the other engines.
[70:49] So yeah, so we should talk about like what are some of the other inference optimizations we can do on top of it and what were some of the other solutions that came out.
[71:05] Uh so I would like to again invite Tanme.
[71:08] He's going to talk about like some of these optimizations.
[71:18] Oh, sorry.
[71:19] Uh I'm so sorry.
[71:20] Uh I didn't enable the slides.
[71:25] Uh what was the Okay, great.
[71:30] Perfect.
[71:30] Which one?
[71:31] The speculative.
[71:32] Yeah.
[71:33] Thank you, Hersel.
[71:34] Yeah.
[71:36] So, so all these are like speculative decoding all these are the uh so so what we say uh different flavors of same kind of soda.
[71:47] So this uh this technique comes under decoding accelerator.
[71:52] So first one so we are only talking about this speculative decoding but there are other variants like self speculative eagle medusa I only like I think uh this one eagle algorithm
[72:07] personally I don't think speculative decoding works because main problem is alignment okay so let's start with what is uh speculative decoding main problem is that in transformer architecture All these tokens are generated
[72:23] sequentially one by one by one.
[72:27] How about just use a smaller model and let a smaller model to generate maybe let's say four or five tokens and this teacher model or we can say according to our world cup algorithm we can say referee.
[72:43] So referee will decide how many uh tokens it accept and this loop keeps on going on and our assumption is that there are certain domain where this kind of things will work like maybe in decode
[73:01] maybe in coding or where almost there is no creativity uh each uh code or syntax is almost similar.
[73:09] So maybe it can help it.
[73:11] But uh based on personal testing, I didn't find this speculative decoding useful at all.
[73:19] But other techniques like uh self speculative decoding where teacher model also have one head auxiliary head and it will do same similar kind of things what this base model or small model is doing it.
[73:35] But then this eagle came Eagle 1 2 3 I don't know how many version versions are but it is just saying that instead of creating instead of generating tokens uh let's uh train a small model inside
[73:50] train a small model and just take a features from one of its uh one of main models layer so that instead of generating token uh it will generate uh this features so
[74:03] so uh so eagle is uh Eagle is better compared to this other kind of technologies and then another one is Medusa which is just saying that just generate all the tokens parallelly.
[74:18] Uh okay.
[74:19] So here so here in this slide
[74:23] yeah the next slide.
[74:25] Okay.
[74:29] Okay.
[74:29] Yeah.
[74:31] Okay.
[74:31] Now we come to uh now we will come to this one prefix caching.
[74:35] So I don't know whether people are using this one static prefix caching or not but thing is that main problem with prefix caching is that sometimes we type and make a small kind of mistake
[74:48] and this standard static prefix caching is basically it takes a prompt do some hashing and then next time when user asks similar kind of question it will try to match the hash.
[74:59] So if hash is uh if hash is equal then then it will instead of recomputing all those K and B it will just uh take it from from the storage
[75:10] but you know that sometimes we make a mistake or maybe we can just change a word or letter something like that then we have a very higher uh cache uh hit cache
[75:20] miss hit rate so that's why uh this one uh radics tree so radics tree is becoming very popular and also also because of agent.
[75:31] So I think almost everyone is doing agent and most of the computation is going during TT during test time inference kind of thing where we keep on asking same kind of questions and prompt for example
[75:44] you are an expert software engineer multiply by 200 times.
[75:49] This kind of loop keeps on going inside this uh agentic agentic kind of things where it is al necessary to keep uh or store similar kind of things in a radics tree.
[76:03] So radics tree is just so so radic tree is just advanced version of this prefix tree where where we will just where we will just collapse a node if it does not have a does not have any branch
[76:18] and for this kind of work where keep on repeating same thing this uh red x tree helps a lot and st lang uh use this kind of algorithm for prefix caching.
[76:37] Okay.
[76:38] Yeah.
[76:38] Then there is another thing.
[76:39] One is tensor RT LLM.
[76:42] This is very confusing.
[76:43] When I first started, I was I was just confused.
[76:46] What is tensor RTLM?
[76:49] So yeah.
[76:50] So tensor RT is just a uh it's just a standard uh SDK kind of thing.
[76:56] Tensor RTLM is just an inference engine just like VLM, SG lang.
[77:02] But problem is that it is related to Nvidia.
[77:06] They optimized each and every layer and every problem as I mentioned in our world cup algorithm.
[77:13] They just break everything and optimized everything at hardware level also.
[77:19] So uh yeah.
[77:22] So okay next.
[77:27] Yeah.
[77:27] So for this workshop we also uh did some benchmarking like which is best uh so our setup was something similar was so so we did two kind of testing.
[77:39] First one is without uh without agentic testing where we just so we use this shared GPT uh this one data set and uh just ask those questions uh using VLM
[77:54] and SG lang.
[77:57] Okay.
[78:06] Yeah.
[78:06] Okay.
[78:09] And let me just zoom it up.
[78:13] Okay, great.
[78:14] Okay.
[78:15] Yeah.
[78:15] So, yeah, for this workshop, we used H100 and of our first testing was that uh we just uh we just asked uh we take questions from shared GPT and put it into
[78:27] VLM, SG lang and we found that actually there's no statistical difference between which one is better.
[78:35] So both have almost similar kind.
[78:37] So both are fulfilling similar kind of request per second uh TTFT and latency.
[78:44] So but only difference we have seen during agentic uh agentic branching.
[78:50] So uh what we did was that we asked that similar kind of question that you are the best this one software engineer in the world just solve the problem of traffic congestion in this city kind of thing.
[79:05] Then we put this into LLM.
[79:08] LLM generates some output.
[79:10] Then we did another uh round two also.
[79:14] So once this LLM generates this output, then in round two we have specially mentioned that uh provide uh review the proposal and give ratings from 1 to 10.
[79:28] So this uh two turns we did uh and this loop keeps on uh repeating it.
[79:35] Uh what we found is that for this kind of uh workflow where everything is standard all those prompts and context engineering comes into the picture.
[79:47] If we do proper this agentic branching then I think uh this HG lang is three to four times better.
[79:55] But again this depends upon the different setup maybe uh if you do it uh you may get different results.
[80:03] Okay.
[80:04] Yeah.
[80:04] So I think uh did we uploaded it on GitHub?
[80:09] Okay.
[80:10] Yeah.
[80:10] So the PDF is like also in the drive.
[80:15] Uh it's the same link as the slides.
[80:19] So a quick summary here.
[80:23] So on a standard API workload throughput you would see like a VLM and the SG lang would behave same.
[80:32] So if you don't have if you have like a standard workload definitely go with VLM.
[80:36] It's the production default anyways.
[80:39] But what Tanme was also saying is when you try to like make it like agentic workloads that is where like your SG link really shines uh and uh it kind of like provides you all the benefits.
[80:57] So yeah keep like VLM as a default but if you have agentic workloads probably try to move as the towards the SG lang.
[81:05] if you're not happy with DB LLM.
[81:08] Uh but uh okay.
[81:13] Uh let me Okay.
[81:26] And then like there is like the like a comparison that is done at the 120 billion like for the GPTO OSS 120 billion.
[81:36] Um this is a benchmark that was prepared by clarify.
[81:42] So there is like a blog link here.
[81:46] Oh nice.
[81:48] Okay.
[81:50] Yeah.
[81:50] So they did the similar benchmark and they included like a tensor RT LLM in it.
[81:58] Definitely you can always go through these benchmarks and try to understand which basically suits your use case.
[82:05] As we mentioned like Tensor RT they try to optimize the hardware side as well having the peak hardware performance.
[82:17] Wait uh this is okay.
[82:26] Yeah.
[82:27] And then like in terms of when you want to dep pick like your engines once you figure out like between VLM, SG lang tenserati so that there are some new engines that are popping up
[82:42] Nvidia Dynamo for sure.
[82:44] Uh so they are also for the agentic uh session routing.
[82:49] Uh hugging phase is always there.
[82:51] It's a simple no server.
[82:54] Then there is like an MSAR engine that was recently proposed by Stanford.
[83:01] They are for like the multimodel.
[83:05] Uh so definitely you could explore those and when you try to basically just to like give a quick summary uh you we start with like a baseline we try to find
[83:19] what model could fit our use cases.
[83:22] Um, so you could pick like uh Deep Seek, you could pick like don't pick like a Mistral 7B.
[83:31] I mean, it's not good.
[83:33] Uh, but yeah, so you pick your model and you want to like have a smaller memory and you want to try to fit that bigger model into smaller memory
[83:44] so that you could save cost on the GPU cost.
[83:48] So you can do like all those com quantization then you can apply all those serving optimizations by using the right serving engine under the hood.
[83:58] So that can really provide you that throughut that you really want.
[84:07] And now something that you can do uh after going back home pro because we cannot like actually go over all the material here uh is definitely reading about some of the source informations
[84:23] like different attention mechanisms different like these engines like try to just read the different benchmarks which are present online as well and then there are a lot of like
[84:37] in-depth guides or the next phases of it which is like learning about some KV eviction strategies.
[84:45] So world is moving towards having a separate KV cache engineering domain.
[84:50] So you want to understand what's going on in there.
[84:53] So KV cache KV eviction cache compressions hybrid memories.
[84:58] So there are like lot of solutions that are happening around there.
[85:01] So always try to stick to those foundations or like the fundamentals or the first principles and try to see like which solution basically solves what problem and whether you actually need that
[85:15] problem to be solved for your use case and then there is like distributed LLM inference which is like a different painoint altogether.
[85:26] Uh you would probably need like a two-hour workshop there as well.
[85:31] uh to like go over like all the internals do all the hands-on.
[85:41] Yes.
[85:41] And this is something we are trying to propose for the AI engineer New York session uh which is to like dive deeper into the advanced sections of the LLM inference.
[85:51] So this workshop was more for the like beginner and the intermediate level.
[85:56] Um so in this form we do have like a feedback as well plus also the interest.
[86:02] Um if uh you think like we need certain improvements on certain sections definitely give that feedback as well and if you want to see this workshop in like New York
[86:14] uh fair you I mean definitely feel free to enroll your interest.
[86:22] Uhhuh.
[86:24] How is it possible?
[86:28] Well, let me just check.
[86:46] Huh?
[86:48] URL works, right?
[86:50] Not the QR code.
[86:51] Okay.
[86:52] Probably I forgot to link those two together.
[87:00] Z G A five.
[87:11] Okay, cool.
[87:13] Yes.
[87:13] So if you can give that feedback let me just okay that will be fine um and yeah I think we would like to wrap this workshop then I'm sure like lot of you would be having a lot of questions so we can take all those like offline
[87:34] uh we can meet uh and we can uh like talk about those questions.
[87:39] Yeah sure.
[87:41] Uh thank you everyone.
[87:42] Thanks for joining.
[87:44] Uh I think it was really meaningful and all of you like came here.
[87:49] Uh thanks a lot.
[87:51] Yeah.
[87:51] Thanks.