WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=4n08JHb90hI

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/4n08JHb90hI.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.719 --> 00:00:15.040
I mentioned before my name is Dewan Lightoot.

s2
00:00:15.040 --> 00:00:19.600
I'm a developer AC advocate at Akami and this is my colleague.

s3
00:00:19.600 --> 00:00:20.560
I'm Kaja Omar.

s4
00:00:20.560 --> 00:00:23.920
I go by R. I'm a senior software engineer at Akami.

s5
00:00:23.920 --> 00:00:31.359
And we're going to be talking agents that own their own inference, building production AI agents on dedicated GPUs.

s6
00:00:31.359 --> 00:00:34.719
Now, why are we here?

s7
00:00:34.719 --> 00:00:36.239
Things are kind of changing.

s8
00:00:36.239 --> 00:00:41.120
And for me, I started building agents before tool calling.

s9
00:00:41.120 --> 00:00:46.079
Back in January 2023 is when I built my first agent.

s10
00:00:46.079 --> 00:00:59.359
I would take a prompt, that context was about 8,000 tokens at the time, and I would parse that prompt and then make a tool call with Python and then give the response back to an LLM,

s11
00:00:59.359 --> 00:01:00.160
right?

s12
00:01:00.160 --> 00:01:04.479
But as I was doing that, I found something interesting.

s13
00:01:04.479 --> 00:01:07.600
It can get really expensive, right?

s14
00:01:07.600 --> 00:01:12.479
And then I started using local models to figure out, okay, how can I do this?

s15
00:01:12.479 --> 00:01:14.400
And then I learned something else.

s16
00:01:14.400 --> 00:01:26.159
There's another challenge around that because if I'm using a smaller model, it may not be as capable and then I may be compute bound that memory in that GPU may not handle

s17
00:01:26.159 --> 00:01:27.280
the larger context.

s18
00:01:27.280 --> 00:01:31.200
So that's the inputs and outputs and then the multiple turns in the tool cost.

s19
00:01:31.200 --> 00:01:40.320
How do we get the most either the GPUs, the costs, the latency that we need in production or just for our own use cases, right?

s20
00:01:40.320 --> 00:01:51.360
So the reason why we're here, we're going to talk about agents that are often renting their intelligence and then how you can take back that control and own it.

s21
00:01:51.360 --> 00:01:53.759
When it comes to an agent, it's just a loop.

s22
00:01:53.759 --> 00:02:04.640
You invoke an LLM, it has access to tools, it has access to an environment, makes a decision, a reasoning decision on what actions to take once you invoke it or give it a prompt.

s23
00:02:04.640 --> 00:02:14.000
And then those actions are being performed by your tools, your Python code, your other programming languages and then given back to a model.

s24
00:02:14.000 --> 00:02:23.440
The model evaluates that the actions, the results and then updates the plan, takes additional actions or the goal is achieved, right?

s25
00:02:23.440 --> 00:02:29.840
But all that requires tokens to be generated, right?

s26
00:02:29.840 --> 00:02:32.800
And depending on your application can lead to latency challenges.

s27
00:02:32.800 --> 00:02:38.160
So we're going to talk about how you can actually man manage all of that on dedicated GPUs.

s28
00:02:38.720 --> 00:02:41.040
So what does it mean to own your inference?

s29
00:02:41.040 --> 00:02:47.440
First, when it comes to owning owning your inference, you actually own the GPU or rent the GPU.

s30
00:02:47.440 --> 00:03:01.120
So when you do that, that means that you have access to the GPU, the hardware for that AI to be ran on, for the models to live on, which means then you have access to control

s31
00:03:01.120 --> 00:03:01.840
where it goes.

s32
00:03:01.840 --> 00:03:08.480
So the data never leaves your cluster, never leaves your data center, never leaves your laptop, right?

s33
00:03:08.480 --> 00:03:12.959
You now own that intelligence and what's happening with the model.

s34
00:03:12.959 --> 00:03:21.920
And if you're in the cloud, you can also do something like private VPC endpoints or have it in the VPC so it never leaves your VPC.

s35
00:03:21.920 --> 00:03:22.400
Right?

s36
00:03:22.400 --> 00:03:27.440
I know we all have different environments, have different ways to set up our infrastructure.

s37
00:03:27.440 --> 00:03:30.400
We can talk about all of those.

s38
00:03:30.400 --> 00:03:35.519
Now what's the difference like when do you own when do you rent?

s39
00:03:35.519 --> 00:03:42.000
Well, first if your usage is high and steady, why is that?

s40
00:03:42.000 --> 00:03:44.000
Yes.

s41
00:03:44.319 --> 00:03:44.879
Yes.

s42
00:03:44.879 --> 00:03:49.519
At the end of the talk, I'll have a link to the slide and everything that we showed code, everything.

s43
00:03:49.519 --> 00:03:51.040
You can download it and take it with you.

s44
00:03:51.040 --> 00:03:52.400
So, thank you.

s45
00:03:52.400 --> 00:03:57.280
Now, when do you own if the usage is high?

s46
00:03:57.280 --> 00:03:57.599
Right?

s47
00:03:57.599 --> 00:04:04.000
If you're constantly have a workload running, because if you're paying for a GPU, you're renting a GPU.

s48
00:04:04.000 --> 00:04:05.680
What does that mean?

s49
00:04:05.680 --> 00:04:08.720
that GPU should always be working.

s50
00:04:08.720 --> 00:04:10.640
Otherwise, what are you paying for?

s51
00:04:10.640 --> 00:04:12.720
So, that's something you got to think about.

s52
00:04:12.720 --> 00:04:25.440
Then, when it comes to the data, if you need that data to stay private, if you need to own the data, you care about your intellectual property, or you have some governance and compliance reasons why you can't use a public provider when it comes to AI,

s53
00:04:25.440 --> 00:04:28.639
that's another reason you want to own.

s54
00:04:28.960 --> 00:04:35.199
And then the hardware, that's something we got to think about your hardware.

s55
00:04:35.199 --> 00:04:45.680
If you can afford the capacity and you have the capacity, why not run those models, right, for your use cases because then you have that full control.

s56
00:04:45.680 --> 00:04:47.360
Now, when do you rent?

s57
00:04:47.360 --> 00:04:56.160
If your traffic is low, bursty, you just need intelligence sporadically, it might not be efficient for you to own some GPUs.

s58
00:04:56.160 --> 00:04:59.759
Next, well, we start talking about intelligence.

s59
00:04:59.759 --> 00:05:04.639
The more capable models, they're going to cost for those GPUs.

s60
00:05:04.639 --> 00:05:12.320
And often times, those providers will provide that intelligence on the GPUs because they have the infrastructure for that.

s61
00:05:12.320 --> 00:05:17.360
So, it may make more sense to use those providers to get that intelligence, get those benefits.

s62
00:05:17.360 --> 00:05:28.560
And then the next part which is often a challenge is that your teams may not have the knowledge or the expertise to support dedicated or self-hosted AI.

s63
00:05:29.120 --> 00:05:38.400
Now today's agenda we're going to start out discussing some of the foundation you need to understand like the infrastructure stack we'll be using units and memory budget.

s64
00:05:38.400 --> 00:05:42.560
We'll look at prefield decode and KB cache and what that looks like.

s65
00:05:42.560 --> 00:05:53.440
And then from there we'll look at dis dense models verse mixture of experts so you understand what that actually looks like on the GPU and why you would pick one or the other

s66
00:05:53.440 --> 00:06:04.560
from there will discuss how to optimize your inference right once you figure out okay we want to do self-hosted how can we optimize this to get the most out of

s67
00:06:04.560 --> 00:06:12.400
our GPUs the inference engine that we run and just our entire workload and then finally from there.

s68
00:06:12.400 --> 00:06:19.039
We'll take that agent and deploy it into our own dedicated cluster.

s69
00:06:19.039 --> 00:06:21.680
With that being said, I'm going to pass it off to you, Omar.

s70
00:06:21.680 --> 00:06:22.160
Right.

s71
00:06:22.160 --> 00:06:23.840
Thank you.

s72
00:06:23.840 --> 00:06:30.160
So, next uh we'll talk about uh the environment that you each of you is going to get for this workshop.

s73
00:06:30.160 --> 00:06:36.240
Um we have a Kubernetes cluster with around 200 plus GPU nodes.

s74
00:06:36.240 --> 00:06:38.720
Each of you gets a single GPU.

s75
00:06:38.720 --> 00:06:50.720
Um and we have set it up in such a way that uh each of you gets sort of like your own namespace and within that namespace you will have like your Jupyter notebook running and then you will have your

s76
00:06:50.720 --> 00:06:54.479
VLM inference endpoint running as well that you'll be playing around with.

s77
00:06:54.479 --> 00:06:59.680
Um and then we're also going to be deploying uh the agent on there as well.

s78
00:06:59.680 --> 00:07:11.360
Uh the cool thing here is that the VLM endpoint that we have running in your namespace, you're going to be optimizing it uh as part of uh um after the first four modules.

s79
00:07:11.360 --> 00:07:13.759
And um I think that'll be pretty interesting.

s80
00:07:13.759 --> 00:07:17.599
We're going to look at different techniques that we're going to be using there uh to do so.

s81
00:07:17.599 --> 00:07:30.560
And you're going to redeploy this VLM endpoint over and over again until you have sort of optimized it to increase the throughput uh reduce the latency and uh make sure like you're meeting all your SLAs's.

s82
00:07:30.960 --> 00:07:34.960
And then the other thing I want to talk about is uh GPUs on LKE.

s83
00:07:34.960 --> 00:07:39.360
So I work on the Kubernetes side of things at Okami.

s84
00:07:39.360 --> 00:07:42.400
Um so this is sort of my wheelhouse.

s85
00:07:42.400 --> 00:07:47.360
So if anyone has any Kubernetes related questions, uh feel free to ask anytime.

s86
00:07:47.360 --> 00:07:49.520
But I want to talk about what Akami has.

s87
00:07:49.520 --> 00:07:54.400
So we have the uh we're going to be using the RTX 4000 ADA GPUs.

s88
00:07:54.400 --> 00:07:56.560
Uh we also have other GPUs available.

s89
00:07:56.560 --> 00:07:58.479
Uh but these are dedicated GPUs.

s90
00:07:58.479 --> 00:08:07.520
EGPU has 200 uh eGPU has 2 GB of uh VRAM with 260 GB per second memory bandwidth.

s91
00:08:07.520 --> 00:08:10.160
So the decode will be pretty quick because of that.

s92
00:08:10.160 --> 00:08:13.120
Uh we have high performance in Tflops.

s93
00:08:13.120 --> 00:08:14.800
Uh large amount of CUDA core.

s94
00:08:14.800 --> 00:08:17.599
So the pre-fill will be pretty quick as well because of that.

s95
00:08:17.599 --> 00:08:29.599
Uh currently for your workshop we will be using a single GPU but that doesn't mean when you want to deploy this at the end on your own you can have different plans that we offer as well where you can have up to four GPUs in a node.

s96
00:08:29.599 --> 00:08:33.120
So that goes up to 80 gigabytes of uh VRAM on a single node.

s97
00:08:33.120 --> 00:08:37.599
So you can deploy uh decently sized uh models on there as well.

s98
00:08:37.599 --> 00:08:40.959
Um and uh everything is sort of orchestrated by LKE.

s99
00:08:40.959 --> 00:08:45.839
So setting up the node, uh deploying the drivers, uh setting it all up for you.

s100
00:08:45.839 --> 00:08:48.320
So that way you don't have to do anything manually.

s101
00:08:48.320 --> 00:08:50.080
Uh everything is sort of like one click.

s102
00:08:50.080 --> 00:08:53.760
You click a button, everything gets deployed and you're off to races.

s103
00:08:53.760 --> 00:08:57.360
Um yeah, for more information, we have a link here as well.

s104
00:08:57.360 --> 00:09:00.240
Um and at the end, you'll definitely get all these slides.

s105
00:09:00.240 --> 00:09:08.640
So you'll be able to get to the pages and then um I'll give it back to Dwan.

s106
00:09:08.640 --> 00:09:08.959
All right.

s107
00:09:08.959 --> 00:09:12.640
So for the inference engine, we're going to be using V LLM.

s108
00:09:12.640 --> 00:09:22.880
Now, some of you may have heard of Lama, which is great for local, but VLM offers some benefits that are better for production.

s109
00:09:22.880 --> 00:09:28.640
One of those being being continuous batching out the box, the other being page detention.

s110
00:09:28.640 --> 00:09:33.120
Now, it also supports the Open AI compatible API.

s111
00:09:33.120 --> 00:09:41.760
So if you're using whatever SDK that supports Open AI, you can use it with VLM.

s112
00:09:41.760 --> 00:09:51.519
It also allows you to do some really interesting tweaks when you're setting up inference for your for your endpoints and for your applications.

s113
00:09:51.519 --> 00:09:53.600
Now, I mentioned continuous batching.

s114
00:09:53.600 --> 00:10:00.720
For those of you that don't know what it is and how it actually works, this happens at the prefill stage, right?

s115
00:10:00.720 --> 00:10:06.880
the prompts come in and normally with static batch, let's say you have a school bus full of kids.

s116
00:10:06.880 --> 00:10:19.760
Well, in order for you to drop get new kids with static batching, you have to drop off every kid one at a time and then once all the kids are dropped off, you go back and get the rest of the kids and then start the next batch.

s117
00:10:19.760 --> 00:10:24.959
But with continuous batching, you can drop off a kid and pick a kid all at the same time.

s118
00:10:24.959 --> 00:10:30.560
This is way more efficient when it comes to the KV caching that happens during the preill stage.

s119
00:10:30.560 --> 00:10:35.200
now or when it comes to page detention.

s120
00:10:35.200 --> 00:10:42.959
Paged detention is when you think about the KB cache and you think about what's actually happening there's memory space that happens.

s121
00:10:42.959 --> 00:10:48.079
So the prompt comes in and the longest prompt is going to take up memory space.

s122
00:10:48.079 --> 00:10:53.760
When you when page attention that memory space gets allocated even for smaller prompts, right?

s123
00:10:53.760 --> 00:11:10.880
But with page attention now that is all managed through c with the VLM that ensures that you can continuously pro process generate and batch those requests as they come through VLM.

s124
00:11:10.880 --> 00:11:15.440
It has auler that kind of manages the batching, but it makes this really efficient.

s125
00:11:15.440 --> 00:11:23.120
Now, there are other service engines that are out there, and we'll talk about some of those in this workshop.

s126
00:11:23.120 --> 00:11:28.720
With that being said, some of the metrics you're going to want to care about is going to be throughput, right?

s127
00:11:28.720 --> 00:11:34.880
To understand how your GPU is performing that time, the first token, understand that, okay, you sent a prompt.

s128
00:11:34.880 --> 00:11:37.760
How long does it take to get the first token back?

s129
00:11:37.760 --> 00:11:40.079
from there that KV cache utilization.

s130
00:11:40.079 --> 00:11:45.279
We're going to look at these metrics and show you how to get all of that out of VLM and then the QEP.

s131
00:11:45.279 --> 00:11:51.360
So these are some of the metrics as we're going through this workshop you'll you'll see and be able to better understand.

s132
00:11:51.360 --> 00:11:55.279
And with that being said, we're going to access the environment.

s133
00:11:55.279 --> 00:11:59.360
So if you can scan this QR code, pull out your laptops if you want to follow along.

s134
00:11:59.360 --> 00:12:03.200
If not, you can follow us as we go through the workshop.

s135
00:12:03.200 --> 00:12:14.639
In this Jupyter notebook, we have nine different labs that we'll cover and all of the resources and dependencies are loaded into this environment.

s136
00:12:14.639 --> 00:12:24.240
Now, if you want to look at the code and try this on your own at home, I do have the repo public on accommoders.

s137
00:12:24.240 --> 00:12:31.519
I'll have this link along with the presentation and everything that we shared available with one QR code that you can download.

s138
00:12:31.519 --> 00:12:35.200
And if you need me to make a short link, I will.

s139
00:12:36.399 --> 00:12:41.120
All right, with that being said, we're gonna start with the first module in the workshop.

s140
00:12:41.120 --> 00:12:43.920
If everyone is connected.

s141
00:12:56.399 --> 00:13:03.600
Yeah, it was kind of slow for me too, which is why I went to my own leak shortener because I kind of made that really fast with Russ.

s142
00:13:03.600 --> 00:13:06.720
So yeah,

s143
00:13:07.600 --> 00:13:11.120
hopefully like if you do have a hot spot, I would use your hotspot.

s144
00:13:11.120 --> 00:13:15.200
Um but we'll try to make the best out of the experience.

s145
00:13:16.560 --> 00:13:18.880
Sound good?

s146
00:13:19.040 --> 00:13:19.760
Awesome.

s147
00:13:19.760 --> 00:13:20.000
All right.

s148
00:13:20.000 --> 00:13:23.760
So in the first module module, we're going to talk about the inference stack.

s149
00:13:23.760 --> 00:13:29.440
We're going to connect to our VLM endpoint, understand it, see how it generates tokens.

s150
00:13:29.440 --> 00:13:32.399
We'll also look at building our first agent.

s151
00:13:32.399 --> 00:13:38.000
some things that we need to know, look at some met some metrics and then understand the model that we'll connect to.

s152
00:13:38.000 --> 00:13:49.839
The model that we'll use is going to be Quinn 3 4B, but we have several models that sit on a PVC that you can actually load and unload for this endpoint.

s153
00:13:49.839 --> 00:13:59.839
Now, when we talk about VL LM, one of the things about it is when you serve an endpoint, it just has one model, right?

s154
00:13:59.839 --> 00:14:05.440
So if you want to serve two models, you're going to need another V LMA endpoint, right?

s155
00:14:05.440 --> 00:14:10.639
Which means you may need some model routing, a gateway in front of it to kind of handle that.

s156
00:14:10.639 --> 00:14:15.760
Now, it does support authentication, but there's no authorization.

s157
00:14:15.760 --> 00:14:18.800
So you may need a gateway around that, right?

s158
00:14:18.800 --> 00:14:22.720
So agent gateway is often kerve is another one.

s159
00:14:22.720 --> 00:14:24.959
Um, any others?

s160
00:14:24.959 --> 00:14:26.160
There's LLMD.

s161
00:14:26.160 --> 00:14:29.920
There's there's all sorts of open source tooling out there that lets you do that.

s162
00:14:29.920 --> 00:14:33.680
Uh but the main but the main one is like Kserve is what we use a lot.

s163
00:14:33.680 --> 00:14:39.440
Um and it's it's pretty good and doing the scaling, doing the routing and the authentication as well, right?

s164
00:14:39.440 --> 00:14:42.480
Another thing it doesn't have is rate limiting, right?

s165
00:14:42.480 --> 00:14:45.360
If you're processing requests, you may want to rate limit.

s166
00:14:45.360 --> 00:14:52.639
So again, you want to find a gateway to put in front of it to manage those requests and also handle the routing.

s167
00:14:56.959 --> 00:15:06.079
I mentioned that when it comes to the runtime there are other service endpoints that or service runtimes that you can actually use for the server.

s168
00:15:06.079 --> 00:15:15.519
SG lang is one, tensor RT is another and llama CPP which we do go over this later with a little bit more in depth so you can kind of understand it.

s169
00:15:15.519 --> 00:15:22.800
Now in this environment you are connected to a Jupyter notebook that lives in the name space.

s170
00:15:22.800 --> 00:15:28.160
You also have the VLM server endpoint that you'll see here in a moment.

s171
00:15:28.160 --> 00:15:30.639
And all of this is in its own pod.

s172
00:15:30.639 --> 00:15:32.079
And then you'll have another pod.

s173
00:15:32.079 --> 00:15:34.880
Well, another pod that you can launch for the agent.

s174
00:15:34.880 --> 00:15:36.079
And we'll see that too.

s175
00:15:36.079 --> 00:15:39.680
So, let's go ahead and install our dependencies.

s176
00:15:40.240 --> 00:15:42.480
So, we'll install our dependencies.

s177
00:15:42.480 --> 00:15:46.000
I've kind of already ran through this, so some of this is already loaded in my runtime.

s178
00:15:46.000 --> 00:15:50.880
I figured we don't know how the internet will be, right?

s179
00:15:50.880 --> 00:15:55.360
But the good thing is once you send a request, it happens within the cluster.

s180
00:15:55.360 --> 00:16:08.079
So once we send our first request to understand how we're configured in VLM, you can notice the model that we have the name of your name space inside of

s181
00:16:08.079 --> 00:16:09.759
the cluster.

s182
00:16:09.759 --> 00:16:13.040
The coupube config is just for your namespace.

s183
00:16:13.040 --> 00:16:17.680
So that way nobody can access anyone else's namespace in the environment.

s184
00:16:17.680 --> 00:16:33.040
And then we don't have an API key set, but you can set one with V L. Now from here, we're going to use the open AI API compatible endpoint

s185
00:16:33.040 --> 00:16:41.440
to be able or API to connect to the VLM QN 34B 4 billion parameter model.

s186
00:16:41.440 --> 00:16:48.320
I believe we're using that at FP8, meaning that it actually sits at um eight billion parameters.

s187
00:16:48.320 --> 00:16:50.000
Is it eight billion parameters?

s188
00:16:50.000 --> 00:16:51.519
No, four billion parameters.

s189
00:16:51.519 --> 00:16:52.160
Four billion parameters.

s190
00:16:52.160 --> 00:16:53.600
We're using BF16.

s191
00:16:53.600 --> 00:16:54.560
Oh, BF, that's what it is.

s192
00:16:54.560 --> 00:16:55.360
Four precision.

s193
00:16:55.360 --> 00:16:55.519
Yeah.

s194
00:16:55.519 --> 00:16:57.680
So, we're using BF16 full position.

s195
00:16:57.680 --> 00:17:02.160
So, you times the bits or bytes or parameters by two.

s196
00:17:02.160 --> 00:17:08.240
So, it's 4 billion, which means it's eight gigs on the GPU.

s197
00:17:10.880 --> 00:17:16.959
Now once we invoke we got our response for the model and we can see what it was actually served by.

s198
00:17:16.959 --> 00:17:18.880
Now this was all within our cluster.

s199
00:17:18.880 --> 00:17:20.160
That was pretty fast, right?

s200
00:17:20.160 --> 00:17:22.240
It lives within our cluster.

s201
00:17:22.240 --> 00:17:28.079
So hopefully with this experience since we're not hitting the public API, we're already connected to this HTTP.

s202
00:17:28.079 --> 00:17:29.679
It's going to happen right inside of the cluster.

s203
00:17:29.679 --> 00:17:32.559
The response should come back a little faster.

s204
00:17:32.559 --> 00:17:43.360
Now we're going to trace send a trace in the end so we can actually understand how this actually looks like when it goes through the prefill the KB cache and the decode phase.

s205
00:17:45.360 --> 00:17:48.400
So here we have our stream that we're going to configure.

s206
00:17:48.400 --> 00:17:51.520
We're going to look at the chunks and do some calculations.

s207
00:17:51.520 --> 00:17:54.640
We'll get that time to first token so we can understand that.

s208
00:17:54.640 --> 00:17:58.880
We'll see how all of this is generated here in a moment.

s209
00:17:58.880 --> 00:18:05.360
So if we run this, I ran that one again.

s210
00:18:05.360 --> 00:18:08.240
We'll come down here and run that.

s211
00:18:08.240 --> 00:18:11.600
This may take a little bit to go through, but we should see time.

s212
00:18:11.600 --> 00:18:23.919
The first tokens took 37 milliseconds and time output token, the decode phase, which means every token after the after the time of first token, right?

s213
00:18:23.919 --> 00:18:27.600
That's going to be that inner in interle latency, right?

s214
00:18:27.600 --> 00:18:31.919
that inter latency that happens when those tokens are processed.

s215
00:18:31.919 --> 00:18:37.840
So, we're calculating that and we're getting about 40 tokens per second, which is probably going to be our cap.

s216
00:18:37.840 --> 00:18:42.559
But when we start talking throughput, that kind of changes because this is just one request.

s217
00:18:42.559 --> 00:18:53.520
But imagine if we had hundreds of requests or 16 requests, that throughput is going to be a lot higher, but they're still going to be capped at that 40 tokens per second, right?

s218
00:18:53.520 --> 00:18:57.280
Or it could go down, but we can still get more on that throughput.

s219
00:18:57.280 --> 00:19:00.880
And I believe Omar will go deeper into that.

s220
00:19:00.880 --> 00:19:06.799
So one of the things that I want to showcase is the question is around why don't you just run this on CPU?

s221
00:19:06.799 --> 00:19:17.760
Um I didn't want to do CPU on environment because this this um pod for this or this container for the Jupyter notebook is only one gig, right?

s222
00:19:17.760 --> 00:19:22.080
So trying to run a local model on this is gonna it's just going to crash.

s223
00:19:22.080 --> 00:19:24.400
So I kind of simulated what that would look like.

s224
00:19:24.400 --> 00:19:28.960
But if you're trying to run a local model on CPU, it's going to be a lot slower, right?

s225
00:19:28.960 --> 00:19:33.840
It can't do the complex multiplication in those matrices like a GPU can.

s226
00:19:33.840 --> 00:19:35.280
That's the purpose of a GPU.

s227
00:19:35.280 --> 00:19:40.320
It does all those all that math in parallel, which makes it a lot faster, right?

s228
00:19:40.320 --> 00:19:44.880
So, we simulate this on on the CPU.

s229
00:19:46.320 --> 00:19:53.280
And then from there, we actually look look at this on the GPU to see how that looks.

s230
00:19:55.600 --> 00:19:59.120
We got 45 tokens per second.

s231
00:19:59.440 --> 00:20:11.440
And one of the things about this and which is interesting to me is that I do a lot of I would say local models and running them.

s232
00:20:11.440 --> 00:20:19.200
But 45 tokens or 39 tokens per second isn't that bad for an application if you're running it for a specific use case.

s233
00:20:19.200 --> 00:20:19.520
Right?

s234
00:20:19.520 --> 00:20:27.760
If you have a specific use case and you don't care about the latency, you don't care about those like if you have a chatbot, you want that kind of that first token to come fast.

s235
00:20:27.760 --> 00:20:34.880
But for a standard application where you just kick off a job in the background, well, that 39 tokens per second may not be that bad, right?

s236
00:20:34.880 --> 00:20:46.240
So sometimes a lower token per second or a lower time to first token may not be as important depending on the application and the latency that you care about, right?

s237
00:20:46.240 --> 00:20:56.559
you may care more about the intelligence in the job getting done than you do actually how fast it actually happens especially like I said if it's a background process.

s238
00:20:59.280 --> 00:21:01.440
Now here we're going to build our first agent.

s239
00:21:01.440 --> 00:21:06.400
Now for those of you that when we talk about agents there are no tools in this agent.

s240
00:21:06.400 --> 00:21:10.880
So it's actually just a request but just the foundation of the agent.

s241
00:21:10.880 --> 00:21:15.919
We're giving it a system prompt so it knows it role when we do give it to us later on in the course.

s242
00:21:15.919 --> 00:21:34.559
But right now we're just giving it a single prompt and we're giving it a role as system as a solutions architect agent for Akami and you can see that it was able to generate the response.

s243
00:21:34.559 --> 00:21:38.640
Now, this response pretty fast.

s244
00:21:38.640 --> 00:21:40.320
It works.

s245
00:21:40.320 --> 00:21:53.600
But then the question is, if we're trying to do this at scale or we're trying to build an agent that does more than just take it take a prompt and generate a response,

s246
00:21:53.600 --> 00:21:56.799
it may need to make multiple tool calls, right?

s247
00:21:56.799 --> 00:22:01.679
So, that's when you start to think about things like the KB cache.

s248
00:22:01.679 --> 00:22:11.600
You start to think about how much context because when we started talking about that KB cache, that's actually how much context you can actually fit on that GPU, right?

s249
00:22:11.600 --> 00:22:17.280
And so with continuous batching, what happens is you send a prompt to the model.

s250
00:22:17.280 --> 00:22:22.240
Well, you may have 100 users trying to send prompts at the same time, right?

s251
00:22:22.240 --> 00:22:24.960
Well, continuous batch, they're all going to be different sizes.

s252
00:22:24.960 --> 00:22:30.720
So, a smaller GPU can actually handle a bunch of requests if you're using some with something like VLM.

s253
00:22:30.720 --> 00:22:38.080
Now, if you're using something like Olama, you'll see you'll get throttled a lot faster because it doesn't process those requests the same way.

s254
00:22:38.080 --> 00:22:44.240
But VLM actually handles that with theiruler a lot better.

s255
00:22:46.799 --> 00:22:54.559
And so here we're going to set up our our our agent and invoke it and we're going to actually going to look at the steps.

s256
00:22:54.559 --> 00:23:04.720
Now the code that we're using for this is inside of common and you can actually look at all the code that we actually are importing that is custom code.

s257
00:23:04.720 --> 00:23:15.840
So here we have this foundation.py that allows us to do some measures measurements from with Prometheus from VLM right we can look at those metrics.

s258
00:23:15.840 --> 00:23:26.159
Now speaking speak speaking of metrics you can use Prometheus and you can use graphana right at home I always use graphana also when I'm building my agents outside of just looking at the metrics

s259
00:23:26.159 --> 00:23:36.320
another thing that I always build one of the first things I build into my agents is observability right lanefuse that's something I commonly use how many of you have heard of lanefuse

s260
00:23:36.320 --> 00:23:45.600
now there are other tools out there but it's open source you can run that in your cluster as well right beside your agents and then have access that now your observability is private as

s261
00:23:45.600 --> 00:23:45.919
Right?

s262
00:23:45.919 --> 00:23:57.440
And you can see every step, every trace, every span in your agent with how many tokens, what was the latency, not just from the time the first token, but for every tool call, right?

s263
00:23:57.440 --> 00:24:05.120
If you're trying to troubleshoot or you're trying to evaluate now, you can see all of the latency steps in the spans.

s264
00:24:05.120 --> 00:24:10.240
But then you can see, okay, I'm using a smaller model and it's making way more tool calls.

s265
00:24:10.240 --> 00:24:12.640
It's taking way more time than it should.

s266
00:24:12.640 --> 00:24:13.440
Why is that?

s267
00:24:13.440 --> 00:24:16.880
Well, that's where evaluations come in and OM will talk more about that.

s268
00:24:16.880 --> 00:24:24.320
But it's important to have that visibility into your agents so you can understand when you're using a smaller model how to optimize.

s269
00:24:24.320 --> 00:24:31.600
And that could be something as simple as a prompt removing tools or making the agent just a little bit more specialized.

s270
00:24:33.200 --> 00:24:36.000
And so we'll run this.

s271
00:24:41.520 --> 00:24:43.039
And so we're looking at the steps.

s272
00:24:43.039 --> 00:24:48.080
We're going to look at the max tokens and we should see the time to first token.

s273
00:24:48.080 --> 00:24:56.080
So you can see as we increase the tokens, what happens to the time to first token in millisecond as we increase the pump, right?

s274
00:24:56.080 --> 00:25:00.799
As the prompts get larger, that time the first token takes longer, right?

s275
00:25:00.799 --> 00:25:03.440
So the question is what do you optimize on?

s276
00:25:03.440 --> 00:25:07.279
Do you optimize on the median or the largest?

s277
00:25:07.279 --> 00:25:07.440
Right?

s278
00:25:07.440 --> 00:25:09.520
The one that took the most time doing the latency.

s279
00:25:09.520 --> 00:25:14.720
Well, you're going to optimize on the one that takes the most time rather than the median, right?

s280
00:25:14.720 --> 00:25:19.039
And then that way you can increase your median by doing that.

s281
00:25:20.880 --> 00:25:20.927
Now,

s282
00:25:20.927 --> 00:25:21.120
[snorts]

s283
00:25:21.120 --> 00:25:23.520
here we talk about renting versus owning.

s284
00:25:23.520 --> 00:25:25.120
We're going to look at tokens.

s285
00:25:25.120 --> 00:25:26.400
How much are being generated?

s286
00:25:26.400 --> 00:25:36.647
This is where I I mentioned that if you're using bursty where you're not using invoking a model as much with a provider versus if you have a agent

s287
00:25:36.647 --> 00:25:36.960
[snorts]

s288
00:25:36.960 --> 00:25:42.960
or some agentic applications that are always working you can kind of compare that cost.

s289
00:25:45.600 --> 00:25:49.039
So here we generate a response just to get those tokens.

s290
00:25:49.039 --> 00:25:54.240
And now we're going to actually look at the billing and compare which one would actually be more efficient.

s291
00:25:54.240 --> 00:26:04.480
So, if you're processing, let's say, 200,000 tokens per day on a hosted bill, rent it, that's going to be about 1,500 per month, right?

s292
00:26:04.480 --> 00:26:08.799
But if you're renting a GPU, that's going to be about a,000 per month.

s293
00:26:08.799 --> 00:26:15.440
Now, it does matter the size of the GPU, the size of the model, the if it actually performs the way you need.

s294
00:26:15.440 --> 00:26:19.919
But these are some of the things you think about when you're trying to compare, should I rent versus own?

s295
00:26:19.919 --> 00:26:23.840
You can actually do the math on that to break that down.

s296
00:26:25.360 --> 00:26:35.360
And we already talked about these with the data resiliency, you know, the compliance, the security, the controls, and all of those good things when it comes to running locally.

s297
00:26:35.360 --> 00:26:48.000
Now, before we go, the runtime landscape, I mentioned that we're using VLL, but you can also use SE lane, which is good if you're using if you have requests that share long prefixes,

s298
00:26:48.000 --> 00:26:49.120
multi-turn chats.

s299
00:26:49.120 --> 00:26:53.520
So for agents a lot of organizations are using SG lane for that.

s300
00:26:53.520 --> 00:26:57.919
When it come to tensor well this is about that peak throughput on Nvidia hardware.

s301
00:26:57.919 --> 00:27:04.559
It really specializes in providing you that and then llama CPP for that edge that that really fast processing.

s302
00:27:04.559 --> 00:27:10.240
It can help you with those heavily quantized models like on the ARM server.

s303
00:27:11.600 --> 00:27:20.320
So we're going to invoke one more time here and we're looking at our client just to verify that we're still good.

s304
00:27:20.320 --> 00:27:22.640
We're verifying that we can connect.

s305
00:27:22.640 --> 00:27:32.240
We can see the model serve and we can see the ID and I believe I can print the entire so we can see how that require that request actually looks when it comes back.

s306
00:27:32.240 --> 00:27:35.200
So I'm going to print this whole thing as well.

s307
00:27:35.200 --> 00:27:39.679
So we'll and now you can see and actually let's pretty print it.

s308
00:27:39.679 --> 00:27:42.559
So we can see it even better.

s309
00:27:56.080 --> 00:27:56.799
There we go.

s310
00:27:56.799 --> 00:28:04.240
And so when that request comes back, you can see the ID, you can see the max model length, which the that context length.

s311
00:28:04.240 --> 00:28:06.399
You can control that, right?

s312
00:28:06.399 --> 00:28:15.120
You can tweak those parameters of course when you're working with a model but we can see everything about the model itself.

s313
00:28:18.159 --> 00:28:20.480
Now we're going to move on to the next session.

s314
00:28:20.480 --> 00:28:31.440
Um this will have some taking it further when you can explore the cluster explore vm you can feel free to do that in this notebook.

s315
00:28:31.440 --> 00:28:32.640
So I'm going to close that out.

s316
00:28:32.640 --> 00:28:37.840
We'll discard that and we'll move on to units and memory.

s317
00:28:39.279 --> 00:28:44.080
Now the next section and I'm g kind of go faster so Omar can has some time.

s318
00:28:44.080 --> 00:28:49.919
Now when it comes to units and memory, we're going to talk about the ceiling when it comes to token per second on using a model.

s319
00:28:49.919 --> 00:28:59.760
We're also going to look at the model's real config and understanding how a model actually is sits and how actually looks um when it comes to a GPU.

s320
00:28:59.760 --> 00:29:05.919
So here, like Omar mentioned, we are using and I'm going to make that a little smaller.

s321
00:29:05.919 --> 00:29:07.840
Let's see if we can get more of that on there.

s322
00:29:07.840 --> 00:29:08.159
All right.

s323
00:29:08.159 --> 00:29:20.960
So when we talk about the the model actually sitting on the GPU, you have the model weights for the 4 billion parameter model, that model weights at BF16 is going to be 8 gigs.

s324
00:29:20.960 --> 00:29:24.480
So we have 8 gigs and the rest of the room is going to be for KV cache.

s325
00:29:24.480 --> 00:29:27.919
Meaning how much context can you fit on that GPU?

s326
00:29:27.919 --> 00:29:29.760
because it all has to fit within there.

s327
00:29:29.760 --> 00:29:37.360
Now, when your prompt comes in, when we talk about prefill, the model is going to read that entire prompt and store it in the KV cache.

s328
00:29:37.360 --> 00:29:41.039
And then every time it generates a new token, it's going to update that prefill.

s329
00:29:41.039 --> 00:29:43.200
So, that prefill is going to grow, right?

s330
00:29:43.200 --> 00:29:52.240
So, just because your your your prompt that goes in is one size, as it goes through and is processed through the model, it's going to increase as it generates,

s331
00:29:52.240 --> 00:29:52.640
right?

s332
00:29:52.640 --> 00:30:03.039
And then imagine the tool calls, the system prompt, all of these go to the model and all of that has to fit on the KV cache in order for it to be able to handle your request efficiently.

s333
00:30:03.279 --> 00:30:17.440
And so we're also looking at um if we're using at FP8, which is going to talk about quantization and what that actually looks like to shrink a model and to still get the efficiencies that you're looking for.

s334
00:30:17.440 --> 00:30:31.919
Now we're going to install our dependencies and then from here we're going to connect to verify what we have loaded.

s335
00:30:31.919 --> 00:30:40.880
So again we have the Quinn 3 4 billion parameter model our workshop namespace the cluster API key.

s336
00:30:40.880 --> 00:30:54.240
Now when it comes to models we have the config.json JSON like what is actually that file when you download a model you get the config.json let's say you go to huggerface you download a model you're going to get a collection of files

s337
00:30:54.240 --> 00:31:00.399
all of those files are needed right you're going to get the config.json JSON you're going to get these save.tensors tensors.

s338
00:31:00.399 --> 00:31:02.960
Those tensor files are the model weights, right?

s339
00:31:02.960 --> 00:31:07.600
But the configuration is what tells VLM how to handle that model, right?

s340
00:31:07.600 --> 00:31:12.240
So you can actually look at this and explore those files to understand what they actually look like.

s341
00:31:12.240 --> 00:31:15.760
So here we're showing what that configuration file looks like.

s342
00:31:15.760 --> 00:31:18.399
So we have 128 layers.

s343
00:31:18.399 --> 00:31:21.440
The size is 1,024.

s344
00:31:21.440 --> 00:31:24.080
The attention head, there are 16 attention heads.

s345
00:31:24.080 --> 00:31:26.000
The values are eight.

s346
00:31:26.000 --> 00:31:28.480
Vocabulary size, right?

s347
00:31:28.480 --> 00:31:29.600
How does the model train?

s348
00:31:29.600 --> 00:31:31.840
What is the vocabulary size of that ma model?

s349
00:31:31.840 --> 00:31:32.080
Right?

s350
00:31:32.080 --> 00:31:34.000
You can actually look at these configurations.

s351
00:31:34.000 --> 00:31:36.799
Can you modify them?

s352
00:31:38.240 --> 00:31:39.919
No. Right?

s353
00:31:39.919 --> 00:31:41.679
These files cannot be modified.

s354
00:31:41.679 --> 00:31:43.279
But you can fine-tune.

s355
00:31:43.279 --> 00:31:43.519
Right?

s356
00:31:43.519 --> 00:31:44.559
You can use lower.

s357
00:31:44.559 --> 00:31:47.360
You can do all these things to adapt the model.

s358
00:31:47.360 --> 00:31:51.279
But to modify these files, if you try to do that, it's just going to crash.

s359
00:31:51.279 --> 00:31:53.840
It's not going to load.

s360
00:31:55.600 --> 00:32:08.159
Now here we're going to look at the bytes per weight and where it comes from and we calculate here we're looking at a smaller model at BF-16 for the

s361
00:32:08.159 --> 00:32:19.760
6 billion parameter model and the parameters you multiply by two 1.5 gigs and for the 8 billion parameter models what those params look like and it's gonna be eight gigs

s362
00:32:20.159 --> 00:32:33.440
and if we compare the BF16 versus FP8 Now, when we talk about pre precision, most models when they're trained, they're going to be at sometimes FP32, but you'll often get it at FP16.

s363
00:32:33.440 --> 00:32:38.159
Full precision, meaning that you multiply whatever the model size by what?

s364
00:32:38.159 --> 00:32:40.799
Four by four, right?

s365
00:32:40.799 --> 00:32:45.760
So, this 4 billion parameter will be 16 gigs at full precision, right?

s366
00:32:45.760 --> 00:33:01.760
on your GPU when you quantize it down to let's say FP8 well you kind of lose a little bit capabilities of the model but not much right but that's when you start to evaluate

s367
00:33:01.760 --> 00:33:16.320
right if your compute constraint you have this larger model maybe 70 billion parameters that maybe you can do it at FP8 or maybe with int4 right all the way down to where now

s368
00:33:16.320 --> 00:33:22.320
that 70 billion parameter models is rather than 32 is 16 gigs, right?

s369
00:33:22.320 --> 00:33:23.519
If that's possible, right?

s370
00:33:23.519 --> 00:33:27.440
I don't know how much you're going to lose for the position, but you'll have to evaluate that.

s371
00:33:27.440 --> 00:33:31.360
And so evaluation hardnesses are important.

s372
00:33:33.919 --> 00:33:38.159
So we'll look at the GPU where the weights live.

s373
00:33:38.640 --> 00:33:43.120
So here BF16, you multiply by two, FP8, you multiply by one.

s374
00:33:43.120 --> 00:33:44.880
We're looking at this for Quinn.

s375
00:33:44.880 --> 00:33:54.640
And then we have another model that we're going to use in this course where it's say at 8 billion parameters is five gigs on the GPU which you can see how much you save

s376
00:33:54.640 --> 00:34:02.080
with the still intelligence that works for the use cases that we're discussing if you decide to quantize down.

s377
00:34:08.000 --> 00:34:10.800
So we're looking at this card the gigabytes per second.

s378
00:34:10.800 --> 00:34:13.040
you take the where we're actually calculating.

s379
00:34:13.040 --> 00:34:23.280
Okay, we know how the speed of the GPU is 360 and we're showing how to calculate to get the tokens per second and where we're actually bound.

s380
00:34:23.280 --> 00:34:28.079
So, we're going to be bound by the speed of the actual GPU on how fast it can go.

s381
00:34:28.079 --> 00:34:30.560
And we're showing the math for the KV cache here.

s382
00:34:30.560 --> 00:34:36.399
And I'm trying to go a little faster so we can get into the quantization and optimization.

s383
00:34:36.480 --> 00:34:41.440
But if you go through these, you can read through it and understand how the KV cache formula works.

s384
00:34:41.440 --> 00:34:51.040
Oftent times we may not have to go through this and figure it out, but you will have to when you're trying to calculate, for example, you have a 100 users you want to serve.

s385
00:34:51.040 --> 00:34:52.480
You have a smaller GPU.

s386
00:34:52.480 --> 00:34:55.440
The question is I got this smaller GPU.

s387
00:34:55.440 --> 00:34:57.200
Let's say I'll give you a perfect example.

s388
00:34:57.200 --> 00:35:07.200
I had to do a workshop and I had to figure out how to serve 80 students and perform inference for all the students with about three or four GPUs that were about 20 gigs,

s389
00:35:07.200 --> 00:35:07.440
right?

s390
00:35:07.440 --> 00:35:08.880
Do I need that many?

s391
00:35:08.880 --> 00:35:10.079
Is it going to fit?

s392
00:35:10.079 --> 00:35:12.400
Well, how big is my context for the workload?

s393
00:35:12.400 --> 00:35:14.800
What if everybody is invoking at the same time?

s394
00:35:14.800 --> 00:35:17.040
Will the GPUs will the cluster support that?

s395
00:35:17.040 --> 00:35:18.480
Will continuous batching work?

s396
00:35:18.480 --> 00:35:26.560
So, you set up an evaluation to evaluate the entire workload and to determine, okay, is it going to fit or do you hit that roof line, right?

s397
00:35:26.560 --> 00:35:28.800
Do you have start to have spill over the CPU?

s398
00:35:28.800 --> 00:35:38.960
These is your application slowdown or the memory weights memory budget versus KV pool.

s399
00:35:38.960 --> 00:35:43.040
So we'll look at that and then here's how you can actually get that the math.

s400
00:35:43.040 --> 00:35:54.640
So here's our we can get 37,000 tokens on this GPU right on the Sega GPU this 8 gigs for the model weights.

s401
00:35:54.640 --> 00:35:58.960
And then the room that you have is going to be 37,000 tokens, right?

s402
00:35:58.960 --> 00:36:02.800
And so if we have multiple users sending requests, right?

s403
00:36:02.800 --> 00:36:07.040
Well, this will show how many users in their context list.

s404
00:36:07.040 --> 00:36:11.359
So if we have four users, they can all send 8,000 tokens, right?

s405
00:36:11.359 --> 00:36:19.040
For input output tool calls, this going to add up to 8,000 tokens at the same time on that GPU.

s406
00:36:20.720 --> 00:36:32.400
And now we're showing the tax as this increases the different requests and the generation of that request.

s407
00:36:32.400 --> 00:36:40.560
And this kind of shows how to do that math when you're trying to understand your compute constraint, your GPU, and then the workloads that you're performing.

s408
00:36:40.560 --> 00:36:51.119
So you can take all this code and apply it and be able to take your use cases and do the math on it so you can see if it fits on your GPUs.

s409
00:36:54.000 --> 00:36:59.680
So we will go to the prefill and decode.

s410
00:36:59.680 --> 00:37:05.920
So going a little bit deeper into the prefill and the decode in the KV cache.

s411
00:37:05.920 --> 00:37:09.920
Again, we're using the Quinn3 model, but we're actually going to dive in to see how this looks.

s412
00:37:09.920 --> 00:37:15.839
So we have the prefill, which like I mentioned is going to be focused on that time to first token, right?

s413
00:37:15.839 --> 00:37:20.640
from the time you press send, how long does the first token, if you're streaming, right?

s414
00:37:20.640 --> 00:37:23.599
Because if you're not streaming, you're just going to get the whole response back.

s415
00:37:23.599 --> 00:37:28.000
But if you're streaming, how long does it take to get that first token back, right?

s416
00:37:28.000 --> 00:37:31.040
And that's going to be that prefill stage.

s417
00:37:31.040 --> 00:37:36.079
And then the decode is going to be that token, every token after that, that's all measured.

s418
00:37:36.079 --> 00:37:47.440
And if you're looking at the metrics inside of VLM, you send that to Graphana, you can actually view that on the dashboard and you can watch, okay, as I increase my prompts,

s419
00:37:47.440 --> 00:37:50.880
you can watch it increase, you can watch it decrease, right?

s420
00:37:50.880 --> 00:37:51.599
Right.

s421
00:37:51.599 --> 00:37:57.520
And really understand what's actually happening on that GPU when you're sending requests.

s422
00:37:57.520 --> 00:37:58.640
So, we'll go through here.

s423
00:37:58.640 --> 00:38:01.599
We'll install our dependencies.

s424
00:38:01.599 --> 00:38:04.240
Then from there, we'll look at our environment.

s425
00:38:04.240 --> 00:38:06.800
Make sure we're connected.

s426
00:38:06.800 --> 00:38:07.119
Yes.

s427
00:38:07.119 --> 00:38:09.839
And we're still using the Quinn 34B.

s428
00:38:09.839 --> 00:38:15.359
And we're going to look at the self attention, meaning how does that KB cache get processed.

s429
00:38:15.359 --> 00:38:16.800
So this is the actual formula.

s430
00:38:16.800 --> 00:38:23.839
So you can look at that soft max to really understand how that KB cache is being handled within a model.

s431
00:38:23.839 --> 00:38:28.480
We're just doing it by hand just so you can see the math on that.

s432
00:38:28.480 --> 00:38:38.000
And so when it comes to the KB cache, there's the naive approach and there's the KB cache approach that I would say is handled.

s433
00:38:38.000 --> 00:38:48.320
So the naive approach is you take a prompt, the model looks at the whole prompt and then one by one stores that prompt in the KV cache and generates a response.

s434
00:38:48.320 --> 00:38:55.760
What KV cache does, it says, okay, I'm gonna take the prompt, I'm gonna learn from it, and I'm gonna store it right in parallel.

s435
00:38:55.760 --> 00:39:06.400
And so the KB cache allows you to learn from the prompt to be able to reference the information rather than just doing the fixed sequence of the naive caching approach

s436
00:39:06.400 --> 00:39:10.160
when it comes to handling that KB cache.

s437
00:39:10.160 --> 00:39:13.440
And so if we generate this,

s438
00:39:14.097 --> 00:39:15.760
[snorts]

s439
00:39:15.760 --> 00:39:33.200
we can look how it computes the reduction of how much if you use the KB cache approach rather than the naive approach and we're just showing all that by hand in code so you can really break it down and understand.

s440
00:39:33.200 --> 00:39:40.800
And now we're going to look okay using K KV caching and we'll measure the time to first token.

s441
00:39:40.800 --> 00:39:48.400
And so here we have 57 milliseconds and then the decode was 25 milliseconds.

s442
00:39:48.720 --> 00:39:56.079
And so we'll continue on and we can see what VLM is actually exposing.

s443
00:39:56.079 --> 00:40:03.920
So we can actually look at those metrics and we're just showing some of the metrics that are actually being exposed and how you would view that in VL.

s444
00:40:05.200 --> 00:40:17.280
Now we're going to invoke the prompt and we're going to increase it and you can watch that KB cache fill and you can see this is going to take a little while.

s445
00:40:17.280 --> 00:40:21.520
So we'll give it a couple minutes while this process we're increasing it.

s446
00:40:21.520 --> 00:40:27.119
So we can see the KB cache usage and we can look at the metrics.

s447
00:40:27.119 --> 00:40:33.359
So we have peak usage running and then the KB usage after it's finished.

s448
00:40:33.359 --> 00:40:36.160
And this will be the math on how you would process that.

s449
00:40:36.160 --> 00:40:42.320
Again the worker that we're using for this code is in the foundation.py.

s450
00:40:42.480 --> 00:40:46.880
So that's going to be the helper for this code.

s451
00:40:47.200 --> 00:40:52.960
Now we're going to look at the warm the code um against the warm prefix caching.

s452
00:40:52.960 --> 00:40:55.359
Now preface caching is an interesting thing.

s453
00:40:55.359 --> 00:41:00.160
Um don't confuse it with prompt caching for and that's something I should have mentioned before.

s454
00:41:00.160 --> 00:41:11.040
If you're using a provider and you're paying by the token, prompt caching, especially if you're do you have a continuous application that's running within every five minutes, prompt caching is a must, right?

s455
00:41:11.040 --> 00:41:15.520
You can you can decrease the cost by like up to 75% depending on who you're using.

s456
00:41:15.520 --> 00:41:18.880
So there is some ways you can use a provider and save on cost.

s457
00:41:18.880 --> 00:41:25.520
But when we're talking about prefix caching, this is caching that prefix for the KB cache.

s458
00:41:25.520 --> 00:41:30.720
And so when we do this, we can see how this actually is handled here.

s459
00:41:30.720 --> 00:41:37.520
So the code is going to be 220 milliseconds and it's down to 41 milliseconds.

s460
00:41:37.520 --> 00:41:40.079
And this all shows you how to do that in the code.

s461
00:41:40.079 --> 00:41:41.200
So you can go through that.

s462
00:41:41.200 --> 00:41:46.960
And I'm trying to go a little faster so Omar can get to his part and hopefully we can deploy.

s463
00:41:53.599 --> 00:41:53.920
All right.

s464
00:41:53.920 --> 00:42:02.319
So we'll go back and we'll go to the fourth section where we talk about dense model verse the mixture of experts on the GPU.

s465
00:42:02.319 --> 00:42:06.160
Now when it comes to a dense model, what is a dense model?

s466
00:42:06.160 --> 00:42:13.280
Well, a disc model when it sits on a GPU, that whole model is used to generate tokens, right?

s467
00:42:13.280 --> 00:42:18.480
So, if it's 30 billion parameters or let's let's just do the model we're using now.

s468
00:42:18.480 --> 00:42:26.319
So, the Quinn 3 4B at BF16 8 gigs on the actual GPU.

s469
00:42:26.319 --> 00:42:33.760
Well, when that actually generates tokens, when it actually is working, that whole entire model is working, right?

s470
00:42:33.760 --> 00:42:38.240
So it's going to be a little slower than if you use a mixture of experts.

s471
00:42:38.240 --> 00:42:40.400
Mixture of experts may be a larger model.

s472
00:42:40.400 --> 00:42:51.119
Maybe it's 30 gigs, but when it actually generates inside of the model, it takes a collection of experts and only that that section of experts does the generation.

s473
00:42:51.119 --> 00:42:56.720
So even if it's 30 gigs, it may be 1.5 gigs just to do the generation.

s474
00:42:56.720 --> 00:42:58.400
So it's going to be a lot faster.

s475
00:42:58.400 --> 00:42:59.599
So that's one of the benefits.

s476
00:42:59.599 --> 00:43:09.599
Even though it's a larger model when it's a mixture of experts, it's actually a can be performed a lot better within an application or a lot faster within an application.

s477
00:43:12.880 --> 00:43:23.119
And so here we kind of talk about memory and why decode is memory bound meaning that when we talk about the generation phase after that time to first token when that generation actually happens

s478
00:43:23.119 --> 00:43:24.800
that's actually going to be bound by the memory.

s479
00:43:24.800 --> 00:43:30.480
So when we looked at the speed of this GPU that's bound, you can't go any faster than that, right?

s480
00:43:30.480 --> 00:43:34.160
There's nothing we can do to tweak that to make it go any faster.

s481
00:43:34.160 --> 00:43:39.280
But there are some things we can do to help this at scale.

s482
00:43:39.280 --> 00:43:44.640
So we're going to look at our roof line and plot that so we can actually see it.

s483
00:43:44.640 --> 00:43:49.359
So we'll make sure that we have our model loaded.

s484
00:43:49.359 --> 00:43:53.599
So we had a quint 3 for billion parameter, right?

s485
00:43:53.599 --> 00:43:55.119
and it's going to be compute bound.

s486
00:43:55.119 --> 00:43:56.240
So it's going to hit that roof line.

s487
00:43:56.240 --> 00:43:57.680
So we're going to look at the flops.

s488
00:43:57.680 --> 00:43:58.400
We're going to look at this.

s489
00:43:58.400 --> 00:44:01.599
So the ridge line is 297 flops per bite, right?

s490
00:44:01.599 --> 00:44:04.960
And where this is bound by the memory.

s491
00:44:06.079 --> 00:44:10.079
Now if we look at the mixture of experts, now we're just going to look at this as code.

s492
00:44:10.079 --> 00:44:13.200
So we can actually understand it.

s493
00:44:14.560 --> 00:44:18.480
So at this 30 billion parameter models, right?

s494
00:44:18.480 --> 00:44:21.599
128 experts and only eight active per token.

s495
00:44:21.599 --> 00:44:25.760
That's 48 layers, 248 hidden.

s496
00:44:25.760 --> 00:44:29.280
And so now the expert mixer expert, this is the weights.

s497
00:44:29.280 --> 00:44:30.319
This is the share weight.

s498
00:44:30.319 --> 00:44:34.079
So this is how much it how much is being used when it generates.

s499
00:44:34.079 --> 00:44:39.359
That's a lot that would be a lot more efficient than using a smaller model.

s500
00:44:39.359 --> 00:44:44.240
It can actually be faster when it actually does the generation.

s501
00:44:45.680 --> 00:44:50.079
And so here we're looking at the the dense model, right?

s502
00:44:50.079 --> 00:44:53.839
is nine times fewer bytes than a dense model, right?

s503
00:44:53.839 --> 00:44:56.560
So it decodes a little bit faster.

s504
00:44:56.560 --> 00:45:02.720
So that's kind of breaking down the two when you're actually trying to figure out, okay, do I want to use something like Mistro, right?

s505
00:45:02.720 --> 00:45:10.160
Or do I want to use like a Quinn model, we compare the dent sizes.

s506
00:45:12.960 --> 00:45:16.880
So here we're going to look at the rate, we'll look at the bandwidth, the GPU serve.

s507
00:45:16.880 --> 00:45:24.640
So measured Quinn 4B is about 40 40 tokens per second, right?

s508
00:45:24.640 --> 00:45:33.359
The smaller model of the Quinn, the 0.6. Now, this isn't done on the CPU, but we're just showcasing 300 tokens per second, right?

s509
00:45:33.359 --> 00:45:34.560
But we're looking at the ceiling.

s510
00:45:34.560 --> 00:45:39.440
This is actually looking at the bandwidth and it's doing the math here to get that the max that we can get.

s511
00:45:39.440 --> 00:45:45.680
The ceiling that we have on this GPU is going to be 45 tokens per second.

s512
00:45:46.560 --> 00:45:53.040
Now, if we enable reasoning on that model and we can look at that.

s513
00:45:53.040 --> 00:46:06.319
Now, when it comes to these Quinn models, if you're wanting to do any type of agent and the model needs to reason, meaning that maybe the prompt comes in, you have tools and it needs to make multiple tool calls.

s514
00:46:06.319 --> 00:46:08.480
Well, you're going to enable thinking.

s515
00:46:08.480 --> 00:46:18.640
So now when it makes a tool call, if it gets the results and it needs to make another tool call or verify the results to make the accurate tool call, you're going to have to enable thinking on these models.

s516
00:46:18.640 --> 00:46:22.800
And that's just a parameter that you can adjust in the model, right?

s517
00:46:22.800 --> 00:46:26.160
But with thinking off, it just generated a response.

s518
00:46:26.160 --> 00:46:27.920
It was 35 tokens.

s519
00:46:27.920 --> 00:46:30.480
But with thinking on now, it had to think about it.

s520
00:46:30.480 --> 00:46:33.359
It cost about 600 tokens.

s521
00:46:33.359 --> 00:46:39.440
Right now with some of the models like provider models you can actually set a token budget for life of thinking.

s522
00:46:39.440 --> 00:46:45.440
So that's one of the things you can save on cost with a provider model and maybe some of the open models you can do that too.

s523
00:46:45.440 --> 00:46:46.240
I'm not sure.

s524
00:46:46.240 --> 00:46:46.960
Yeah, you can.

s525
00:46:46.960 --> 00:46:50.400
Yeah, you can also adjust that token budget on the thinking as well.

s526
00:46:50.400 --> 00:46:59.839
So more tokens on thinking isn't always good but sometimes it can help the model better troubleshoot and better get to that end result that you're looking for.

s527
00:47:02.000 --> 00:47:12.240
And so when we talk about inference, there's a lot of things to think about, but one of the things we want to think about is that memory bound limit for the actual card and then how to apply that for the agent.

s528
00:47:12.240 --> 00:47:26.240
So we look at all of the modules, we talk about that KB cache, the prefix caching, the attention heads as well as quantization which um is going to cover here in a minute and then from there that continuous batching we also discussed that.

s529
00:47:26.240 --> 00:47:34.000
So if we split the prefill from the decode, you have the prefill GPU pool and the decode which is memory bound.

s530
00:47:34.000 --> 00:47:36.000
So the prefill compute bound, right?

s531
00:47:36.000 --> 00:47:42.560
So it needs to go from your memory to the GPU and it needs to be cached, right?

s532
00:47:42.560 --> 00:47:54.240
There's [clears throat] that that phase you can you can do a lot of optimizations, but when it comes to the decode phase, there's nothing you can really do because you're bounded by the GPU.

s533
00:47:54.240 --> 00:48:00.723
And so we kind of show this in code rather than actually showing it because again we

s534
00:48:00.723 --> 00:48:00.880
[snorts]

s535
00:48:00.880 --> 00:48:05.119
only have so much limitation in this workshop.

s536
00:48:05.119 --> 00:48:12.000
Now with that being said I am going to make sure that we are good to go here.

s537
00:48:19.520 --> 00:48:21.359
Um I don't think I need to switch the mod.

s538
00:48:21.359 --> 00:48:22.160
I think you should be good.

s539
00:48:22.160 --> 00:48:22.640
Armor.

s540
00:48:22.640 --> 00:48:24.319
I'm gonna turn it over to you.

s541
00:48:24.319 --> 00:48:24.800
All right.

s542
00:48:24.800 --> 00:48:27.440
And then Yeah, we should be good to go.

s543
00:48:27.440 --> 00:48:28.880
You can talk about quantization.

s544
00:48:28.880 --> 00:48:30.400
Thank you.

s545
00:48:30.400 --> 00:48:30.720
All right.

s546
00:48:30.720 --> 00:48:31.599
Thank you, Dwan.

s547
00:48:31.599 --> 00:48:33.599
Yep.

s548
00:48:33.839 --> 00:48:34.400
All right.

s549
00:48:34.400 --> 00:48:36.800
So, let me close that.

s550
00:48:36.800 --> 00:48:37.040
Sure.

s551
00:48:37.040 --> 00:48:38.720
That That's yours, actually.

s552
00:48:38.720 --> 00:48:40.079
There you go.

s553
00:48:40.079 --> 00:48:40.480
Okay.

s554
00:48:40.480 --> 00:48:40.960
Cool.

s555
00:48:40.960 --> 00:48:48.960
So, I'll be talking about a whole different um uh optimization techniques I think you can use to deploy these models on Kubernetes.

s556
00:48:48.960 --> 00:48:53.119
Um, like I said, uh, my background is in Kubernetes.

s557
00:48:53.119 --> 00:48:59.440
I develop a lot of Kubernetes stuff at Akami and I work on the manage Kubernetes platform there.

s558
00:48:59.440 --> 00:49:05.280
So, a lot of the stuff that we do is we want to make sure that we're able to use GPUs on Kubernetes really well.

s559
00:49:05.280 --> 00:49:07.920
Uh, use all the different features that they have.

s560
00:49:07.920 --> 00:49:13.359
Um, and uh, we deploy a lot of these models using some of these techniques that I'm going to be talking about.

s561
00:49:13.359 --> 00:49:16.720
Um, so there's uh, different techniques.

s562
00:49:16.720 --> 00:49:18.160
So, we're going to talk about quantization.

s563
00:49:18.160 --> 00:49:20.160
I think this one's like the main big one.

s564
00:49:20.160 --> 00:49:21.680
Um, it's super important.

s565
00:49:21.680 --> 00:49:24.079
We're going to talk about speculative decoding.

s566
00:49:24.079 --> 00:49:26.960
Uh, this is a pretty cool technique.

s567
00:49:26.960 --> 00:49:32.400
It can give you quite a big uh performance boost in your uh throughput.

s568
00:49:32.400 --> 00:49:44.800
Um, we're going to talk about um like how do you uh you know uh tweak your LM uh VLM config to make sure that you're getting max performance out of it and

s569
00:49:44.800 --> 00:49:46.240
um you're meeting your SLAs's.

s570
00:49:46.240 --> 00:49:48.880
And then we're going to actually do some tuning and evaluating.

s571
00:49:48.880 --> 00:49:58.800
So this I'm going to leave you guys with some sort of framework that you can use to figure out what exact configuration you need to use for your specific workloads and specific use cases.

s572
00:49:58.800 --> 00:50:02.319
Um so I'll dive into quantization.

s573
00:50:02.319 --> 00:50:04.319
Um let's see.

s574
00:50:04.319 --> 00:50:11.280
And if we have time, we can definitely go uh we can have like a quick Q&A as well, but I think we have 30 minutes.

s575
00:50:11.280 --> 00:50:15.760
I'm going to run through it uh at a good pace.

s576
00:50:15.760 --> 00:50:22.160
So in quantization um there's a whole lot of different uh there's a lot of different types of quantizations you can do.

s577
00:50:22.160 --> 00:50:26.720
Um you know there's the BF-16 the the one that the model is going to start off with.

s578
00:50:26.720 --> 00:50:32.880
That's the full precision model that uh Quint provides you and then there's a lot of other conversations on top of that.

s579
00:50:32.880 --> 00:50:38.400
People have created FP8 that's one of the models that we're going to look at in this particular module.

s580
00:50:38.400 --> 00:50:40.240
Um there's INT4.

s581
00:50:40.240 --> 00:50:43.920
Uh there's AWQ GPQ uh GPTQ.

s582
00:50:43.920 --> 00:50:48.400
These are different types of uh ways you can quantize the model.

s583
00:50:48.400 --> 00:50:54.000
Um there's NVFP4 which is Nvidia's proprietary quantization technique that they use.

s584
00:50:54.000 --> 00:50:57.680
Uh it's hardware accelerated on black hole GPUs and more.

s585
00:50:57.680 --> 00:50:59.760
Uh that one is pretty cool.

s586
00:50:59.760 --> 00:51:02.720
Um but let's see.

s587
00:51:02.720 --> 00:51:03.920
Let's dive into it.

s588
00:51:03.920 --> 00:51:11.119
Um make this hopefully we can fit that in there.

s589
00:51:11.119 --> 00:51:11.520
All right.

s590
00:51:11.520 --> 00:51:17.440
So as you can see, you know, like if you have the BF16 weights, it's going to take a lot of your memory.

s591
00:51:17.440 --> 00:51:25.359
Uh in this particular use case, we have 20 gigs of VRAM and we sort of want to maximize how many requests we can serve with these 20 gigs.

s592
00:51:25.359 --> 00:51:28.960
Uh so we want to make sure we have enough space for KB cache.

s593
00:51:28.960 --> 00:51:36.880
Uh reason being the more requests uh the more KV cache you can have on the memory, the more requests you can process in parallel.

s594
00:51:36.880 --> 00:51:38.480
Uh thereby increasing your throughput.

s595
00:51:38.480 --> 00:51:39.839
So that's super important.

s596
00:51:39.839 --> 00:51:42.720
Uh so that's why we essentially want to do quantization.

s597
00:51:42.720 --> 00:51:48.640
We want to reduce the size of the model so that way you can fit more KV cache and essentially increase your throughput.

s598
00:51:48.640 --> 00:51:51.119
Um so that's one reason.

s599
00:51:51.119 --> 00:51:56.160
The other big reason I think is also because you're shrinking the weights essentially by in half.

s600
00:51:56.160 --> 00:52:03.839
You don't have to do that memory movement between the different uh memory hierarchies resulting in essentially faster decode as well.

s601
00:52:03.839 --> 00:52:12.880
So we will see you'll go from you know 40 tokens per second which is the base model will probably go up to like 60 tokens per second if not more.

s602
00:52:12.880 --> 00:52:16.079
Um so let's look into this.

s603
00:52:16.079 --> 00:52:25.760
Let's um install the dependencies and then I'm just going to make sure that the model is still available and everything is running.

s604
00:52:25.760 --> 00:52:32.400
So yeah, we do have the model running uh the same four billion parameter model and it's running in the namespace.

s605
00:52:32.400 --> 00:52:39.440
Again, as I said, um sort of like a high level overview, there's a whole different sorts of different techniques you can do quantization with.

s606
00:52:39.440 --> 00:52:47.680
Um for example, like um uh Llama CBP is a pretty popular one for running a lot of these models at a lower precision.

s607
00:52:47.680 --> 00:52:49.520
Uh that uses GGUF format.

s608
00:52:49.520 --> 00:52:50.880
Uh that format is pretty cool.

s609
00:52:50.880 --> 00:53:04.559
you can have um uh you know five bit, 3bit, two bit, one bit quantizations which is pretty interesting um and pretty cool but the the downside of that is you lose a lot of uh you lose quality

s610
00:53:04.559 --> 00:53:06.800
trying to get a lot of performance out of it.

s611
00:53:06.800 --> 00:53:08.480
So, there's a trade-off there.

s612
00:53:08.480 --> 00:53:16.160
Um, but for the most part, going from BF-16 to FP8 is a very negligible drop in quality and performance.

s613
00:53:16.160 --> 00:53:20.720
Um, and it will result in more performance while keeping the quality exactly the same.

s614
00:53:20.720 --> 00:53:24.160
Uh, so that's what we want to showcase and that's one of the assumptions we're making.

s615
00:53:24.160 --> 00:53:31.839
Obviously, like when you're doing this for your particular workload, depending on what you're using it for, I highly encourage people to have their own evals.

s616
00:53:31.839 --> 00:53:42.160
uh me so that way if you're switching from you know BF-16 your eval are saying okay everything is good everything's running well for our workload you switch to FP8 you might have issues where

s617
00:53:42.160 --> 00:53:54.000
some things are breaking here sometimes it's hallucinating sometimes it's going in a loop so a lot of those things can happen for very specific use cases and in those scenarios it's very important that you have your own eval set and you're running your own evals

s618
00:53:54.000 --> 00:54:08.079
to make sure any changes you make um are uh okay to make and also you don't have any degradation in performance Um again we'll run and make sure that we have the BF16

s619
00:54:08.079 --> 00:54:10.559
base and then we're going to measure the performance.

s620
00:54:10.559 --> 00:54:18.480
So uh we have a little load test uh utility package that we use to do some uh testing.

s621
00:54:18.480 --> 00:54:24.880
Essentially what we're doing if you look at the levels where these are like a at different concurrencies we're making this request.

s622
00:54:24.880 --> 00:54:33.839
Each request is going to have about 256 tokens and the output is roughly going to be about 128 tokens.

s623
00:54:33.839 --> 00:54:42.640
So this is sort of trying to simulate how many requests you can process at one time and we're sort of going to use this as a baseline to see what the baseline performance is.

s624
00:54:42.640 --> 00:54:51.440
Um so for 32 uh we're getting aggregate performance of 714 tokens per second which is pretty good.

s625
00:54:51.440 --> 00:54:56.079
uh at single concurrency we're getting 37.5 tokens per second.

s626
00:54:56.079 --> 00:54:58.640
Uh so that's important to uh note down.

s627
00:54:58.640 --> 00:55:05.760
And also we're getting um uh 61 milliseconds for time to first token.

s628
00:55:05.760 --> 00:55:08.640
And then you'll notice the time to first token goes up, right?

s629
00:55:08.640 --> 00:55:13.280
Like the jump between 32 and 64 is quite large.

s630
00:55:13.280 --> 00:55:19.040
We're going for 277 milliseconds to uh 1300 milliseconds.

s631
00:55:19.040 --> 00:55:19.920
That's a quite a big jump.

s632
00:55:19.920 --> 00:55:29.599
and you might not necessarily want to make that trade-off, meaning you might want to potentially stop at only 32 uh concurrent requests that you're processing and not go higher than that.

s633
00:55:29.599 --> 00:55:32.720
So that is an important thing to notice there.

s634
00:55:32.720 --> 00:55:37.119
Um again as I was saying um eval are extremely important.

s635
00:55:37.119 --> 00:55:51.520
Uh reason being again you can have performance degradation when you do quantization and you can lose uh some intelligence reasoning can uh not work as it should there there's going to be a lot of issues.

s636
00:55:51.520 --> 00:55:58.559
So you want to essentially I'm kind of simulating here like a uh at a very basic level what kind of eval you might want to think about.

s637
00:55:58.559 --> 00:56:08.799
Um, you know, you want to think about like if the JSON is being generated correctly, if the JSON is being corrupted when you reduce the quantization, you know, at uh BF16,

s638
00:56:08.799 --> 00:56:16.640
you might have good uh responses, but when you go to FPA, maybe potentially your functions function calls are failing.

s639
00:56:16.640 --> 00:56:17.200
Why is that?

s640
00:56:17.200 --> 00:56:19.520
It's probably because it's not generating the JSON correctly.

s641
00:56:19.520 --> 00:56:25.760
So, there's a lot of these um eval cases that are already out there that you can kind of use in the open source.

s642
00:56:25.760 --> 00:56:27.520
Hugging face has a lot of evals.

s643
00:56:27.520 --> 00:56:32.960
You can go on GitHub, you can find a lot of these, but you want to come up with your own for your own use case.

s644
00:56:32.960 --> 00:56:40.160
Um, but then so next thing what we're going to do is we're going to switch from BF16 to an FP8 model.

s645
00:56:40.160 --> 00:56:43.839
So we are going to essentially reduce the size of the model in half.

s646
00:56:43.839 --> 00:56:53.359
And once we do that, we'll kind of get to see um what the performance differences are and are they worth it and are we losing any quality over it.

s647
00:56:53.359 --> 00:57:02.079
Uh so to do this uh you will need to be in the root and then click on manifest and in there you will see vlm.yaml.

s648
00:57:02.079 --> 00:57:06.720
So um I'll kind of go over this.

s649
00:57:06.720 --> 00:57:15.680
So this is a typical manifest uh configuration file that you uh uh have for Kubernetes deployments.

s650
00:57:15.680 --> 00:57:20.640
Um how many of you have used Kubernetes before or have knowledge of Kubernetes?

s651
00:57:20.640 --> 00:57:21.440
Okay, quite a few.

s652
00:57:21.440 --> 00:57:21.920
Okay, great.

s653
00:57:21.920 --> 00:57:22.880
So this will be quick.

s654
00:57:22.880 --> 00:57:32.880
I don't have to go over all all the details, but essentially we have um a PBC which is going which is essentially our cache where all of our models are living

s655
00:57:32.880 --> 00:57:39.200
and we have a simple deployment with VLM with a lot of different arguments and flags here.

s656
00:57:39.200 --> 00:57:44.400
Um so you can see we're using the Quen uh 34 billion parameter model.

s657
00:57:44.400 --> 00:57:52.480
Um, and then we're going to switch it to this FP8 version from Red Hat.

s658
00:57:52.480 --> 00:58:00.799
Uh, let me one more quick thing I want to quickly mention is you see the dynamic at the end.

s659
00:58:00.799 --> 00:58:02.240
Uh, this is pretty important.

s660
00:58:02.240 --> 00:58:06.079
This means that not everything in this model is quantized to FP8.

s661
00:58:06.079 --> 00:58:18.160
In a lot of cases, in a lot of different quantization techniques, you might want to quantize the model uh only specific parts of the model and not the others to make sure the performance stays the same and you don't have a lot of uh issues there.

s662
00:58:18.160 --> 00:58:21.599
So that's one of the reasons why you'll see dynamic, you'll see all kinds of things.

s663
00:58:21.599 --> 00:58:29.440
You'll even see quantizations where certain heads of the model are going to be 4bit and then some are going to be 8 bit and some are going to be full precision.

s664
00:58:29.440 --> 00:58:35.680
And you know, it's it's sort of uh interesting that people are doing that and finding a lot of performance there.

s665
00:58:35.680 --> 00:58:39.280
They're they're finding the performance gain while making sure the quality stays the same.

s666
00:58:39.280 --> 00:58:46.960
So once this is uh changed, we're going to save it and then uh we'll just quickly go through and run it and apply it.

s667
00:58:46.960 --> 00:58:49.599
So this is just making sure that the diff is good.

s668
00:58:49.599 --> 00:58:53.680
Um we're not causing any issues here.

s669
00:58:53.680 --> 00:58:57.359
Uh that should be fine.

s670
00:58:57.520 --> 00:59:01.680
And then we'll just go ahead and where's apply?

s671
00:59:01.680 --> 00:59:02.319
Right here.

s672
00:59:02.319 --> 00:59:04.960
We'll just go ahead and apply and that'll roll out.

s673
00:59:04.960 --> 00:59:10.160
Um it's gonna take uh a little bit of time.

s674
00:59:10.160 --> 00:59:11.599
Let me save that.

s675
00:59:11.599 --> 00:59:15.599
But another thing you guys can also do is you can open up a terminal tab as well.

s676
00:59:15.599 --> 00:59:24.000
Like if you just click on the plus sign and then you'll see a terminal option and then that will also let you see sort of like what's in your nameace

s677
00:59:24.000 --> 00:59:26.160
and what you can do here.

s678
00:59:26.160 --> 00:59:29.920
So if you get you see the VLM endpoint is still coming up.

s679
00:59:29.920 --> 00:59:34.720
Uh this pod is essentially your Jupyter notebook running within your namespace.

s680
00:59:34.720 --> 00:59:41.839
Um let's see this will take a little bit of time to come up.

s681
00:59:45.520 --> 00:59:45.920
Yeah.

s682
00:59:45.920 --> 00:59:48.720
While this is coming up, does anyone have any questions?

s683
00:59:48.720 --> 00:59:53.040
We can probably have one question in

s684
00:59:53.920 --> 00:59:55.920
cont.

s685
00:59:57.119 --> 00:59:59.040
was pre-quantized already.

s686
00:59:59.040 --> 01:00:00.400
Yes, it was pre-quantized.

s687
01:00:00.400 --> 01:00:03.040
Yeah, you can quantize your own models.

s688
01:00:03.040 --> 01:00:04.799
There is LLM compressor.

s689
01:00:04.799 --> 01:00:09.440
Uh it's uh it's provided by VLM, I believe.

s690
01:00:09.440 --> 01:00:13.119
LM compressor.

s691
01:00:14.799 --> 01:00:19.760
So there there there is tooling out there for you to do this on your own if you would like.

s692
01:00:19.760 --> 01:00:26.720
Um, but yeah, the one that we're using currently is already pre-quantized that we downloaded off of Hugging Face.

s693
01:00:27.119 --> 01:00:30.000
All right, I think that should be up.

s694
01:00:30.000 --> 01:00:36.000
Again, it's taking a little bit of time, mainly because it's taking all the weights out of the VRAMm and loading the new weights in.

s695
01:00:36.000 --> 01:00:38.559
So, it's going to take a little bit of time to do that.

s696
01:00:38.559 --> 01:00:43.119
Um, anyone else have any question?

s697
01:00:51.200 --> 01:00:52.079
I'm sorry.

s698
01:00:52.079 --> 01:00:54.799
Look at your your VL your YAML file for your arcs.

s699
01:00:54.799 --> 01:00:59.520
Look Oh, looked like they had uh two flags on the model.

s700
01:00:59.520 --> 01:01:00.240
Oh, they did.

s701
01:01:00.240 --> 01:01:01.760
Oh, that's a good call out.

s702
01:01:01.760 --> 01:01:04.000
Thank you for that.

s703
01:01:04.000 --> 01:01:05.520
Did not see that.

s704
01:01:05.520 --> 01:01:09.359
All right, we're going to go through it again.

s705
01:01:09.599 --> 01:01:12.319
Let's rerun that.

s706
01:01:13.440 --> 01:01:18.480
Okay, that's probably why it didn't come up as quickly as it should.

s707
01:01:22.319 --> 01:01:25.040
Oh, I can't do that.

s708
01:01:31.119 --> 01:01:31.520
All right.

s709
01:01:31.520 --> 01:01:34.799
Any any other questions while this is going through?

s710
01:01:34.799 --> 01:01:35.520
Go ahead.

s711
01:01:35.520 --> 01:01:38.000
How do you choose between?

s712
01:01:38.400 --> 01:01:41.119
Uh, how do you choose between FPA and intake?

s713
01:01:41.119 --> 01:01:42.319
Uh that's a good question.

s714
01:01:42.319 --> 01:01:49.440
So it depends right like int 8 is going to be slightly less precision than FP8.

s715
01:01:49.440 --> 01:01:50.640
That's essentially the reason.

s716
01:01:50.640 --> 01:01:55.280
Uh the the difference is like FP8 is floating point so you get a little bit more precision out of it.

s717
01:01:55.280 --> 01:01:57.520
Uh and then int8 is integer.

s718
01:01:57.520 --> 01:01:59.440
So you could use either or.

s719
01:01:59.440 --> 01:02:02.079
Uh again these are all the other options that you have.

s720
01:02:02.079 --> 01:02:06.079
You can choose based on your workload and where your constraints are.

s721
01:02:06.079 --> 01:02:11.280
Um but yeah you could totally go into eight as well and it should be fine.

s722
01:02:13.119 --> 01:02:16.000
Um, where does it say that?

s723
01:02:17.200 --> 01:02:19.200
Good.

s724
01:02:22.480 --> 01:02:23.440
Stack.

s725
01:02:23.440 --> 01:02:24.000
Oh, yeah.

s726
01:02:24.000 --> 01:02:26.319
I mean, uh, oh, that's the other good good call.

s727
01:02:26.319 --> 01:02:28.400
Um, it depends on the GPU as well.

s728
01:02:28.400 --> 01:02:31.040
Some of the GPUs might not necessarily support it.

s729
01:02:31.040 --> 01:02:31.839
GPU support.

s730
01:02:31.839 --> 01:02:32.079
Yeah.

s731
01:02:32.079 --> 01:02:33.359
So, GPU support is important.

s732
01:02:33.359 --> 01:02:34.400
That's another good call out.

s733
01:02:34.400 --> 01:02:44.960
So some of the um uh GPU might support it but it might not be accelerated by the GPU meaning you might not get the full performance that you would be expecting out of it.

s734
01:02:44.960 --> 01:02:59.040
Um for example the GPUs that we're using we're using RTX 40008 they support FP8 and we're going to get the full uh precision uh full performance out of it and it's able to hardware accelerate all those calculations.

s735
01:02:59.040 --> 01:03:08.160
But if we let's just say use FP4, it won't be able to do that and we would actually not see a whole lot of performance gains and we actually see performance go down instead of it going up.

s736
01:03:08.160 --> 01:03:11.920
So that's another thing to keep in mind.

s737
01:03:12.559 --> 01:03:14.079
All right.

s738
01:03:14.079 --> 01:03:16.400
Where is it?

s739
01:03:19.440 --> 01:03:19.839
All right.

s740
01:03:19.839 --> 01:03:21.440
It has successfully rolled out.

s741
01:03:21.440 --> 01:03:23.839
So we'll uh take a quick look.

s742
01:03:23.839 --> 01:03:27.760
So now we can see it's actually using the FP8 model now.

s743
01:03:27.760 --> 01:03:30.720
And then we'll also kind of quickly go through and do another load test.

s744
01:03:30.720 --> 01:03:32.000
It's going to be the exact same.

s745
01:03:32.000 --> 01:03:40.799
We're going to do it at uh concurrency 1 uh four 8 and uh 8 16 and 32 I believe over here.

s746
01:03:40.799 --> 01:03:42.880
It's going to be exactly the same.

s747
01:03:42.880 --> 01:03:53.760
So and then once that is up we can compare and see the performance differences and why you would want to quantize.

s748
01:03:56.720 --> 01:03:56.960
Okay.

s749
01:03:56.960 --> 01:03:58.480
the performance is in.

s750
01:03:58.480 --> 01:04:04.079
And if we compare it before and after, you can see, let me make this a little bit bigger.

s751
01:04:04.079 --> 01:04:12.079
Uh, you can see even at concurrency one, you know, we're going for 37.5 tokens per second to 53.4 tokens per second.

s752
01:04:12.079 --> 01:04:14.880
So, we're seeing considerable jump in performance here.

s753
01:04:14.880 --> 01:04:22.640
Um, and then at peak throughput, we're seeing uh at 64 concurrency, we're going from 800 to,40.

s754
01:04:22.640 --> 01:04:33.760
So, that's about, you know, 1.4 four uh 42 times uh which is quite a big performance jump but again keep in mind the time to first token is going up considerably as well

s755
01:04:33.760 --> 01:04:46.559
uh compared to 32 and 64 um but then yeah you can and then again you want to run evals um we can quickly run through this and it probably won't have any differences

s756
01:04:46.559 --> 01:05:04.960
so yeah again BF16 versus FP8 um there's not a whole lot of uh performance degradation you should spec it's it's going to work out of the box for the most part but uh for specific use cases it might not but for simple eval cases that we had here it's it's working fine

s757
01:05:05.760 --> 01:05:19.920
I think that is it for this module where we'll jump into next is speculative decoding which I think is a pretty interesting one um let's see let's jump in here all right so how many of you have heard of

s758
01:05:19.920 --> 01:05:24.880
uh speculative decoding before right quite a few people that is good.

s759
01:05:24.880 --> 01:05:30.319
Uh for those who don't know what speculative decoding is, let me make this smaller.

s760
01:05:30.319 --> 01:05:47.760
Um it's essentially we're using another draft model which is going to be a much smaller model than the main model that we're using and it's going to generate multiple tokens ahead of time and then our main model which is the four billion parameter model is going to look at those generated tokens and it's going to either say

s761
01:05:47.760 --> 01:05:49.520
it's it looks good.

s762
01:05:49.520 --> 01:05:52.799
That's it's going to verify it and say it's going to look good or it's not going to look good.

s763
01:05:52.799 --> 01:06:02.160
Uh so in this like diagram you kind of see like um the draft model you know generated four tokens ahead of time and then the verifier model which is our four billion parameter model

s764
01:06:02.160 --> 01:06:10.880
is going to say oh only the first and second token make sense the other two don't so it's going to discard it and it's essentially going to do this each time it's generating new tokens.

s765
01:06:10.880 --> 01:06:14.000
So it's going to go in in a loop there.

s766
01:06:14.000 --> 01:06:30.799
uh and uh for for this uh workshop we're going to be using the 6 billion parameter model uh from Quen as a drafter model and see if that is going to either help us or not help us in a lot of cases

s767
01:06:30.799 --> 01:06:39.280
uh it can also not help you and reduce the performance um and there's multitude of reasons for that and we'll go into that shortly.

s768
01:06:40.079 --> 01:06:41.599
How are we doing on time?

s769
01:06:41.599 --> 01:06:43.599
Good.

s770
01:06:44.480 --> 01:06:52.960
uh just make sure we are good here target model.

s771
01:06:53.440 --> 01:07:00.240
Okay, before we get into it, I also want to talk briefly about all of the different ways you can do um speculative decoding.

s772
01:07:00.240 --> 01:07:05.920
Um there's a lot of different ways and there's more ways coming out uh every every few months.

s773
01:07:05.920 --> 01:07:09.280
Uh but the you know the draft model is one way.

s774
01:07:09.280 --> 01:07:17.039
Um, a lot of times, uh, a lot of new, uh, ways of doing it is actually using a diffusion drafter model, which I think is really cool.

s775
01:07:17.039 --> 01:07:20.720
Uh, I personally use that for one of my home lab machines.

s776
01:07:20.720 --> 01:07:23.520
Um, and, uh, that gives you a huge performance.

s777
01:07:23.520 --> 01:07:33.359
The difference between that and this is because it's a diffusion model, it's going to be generating a lot of tokens uh, at once, uh, rather than doing what we do sequentially with LLMs.

s778
01:07:33.359 --> 01:07:38.480
Um, and then the verifier model is able to verify X amount of tokens out of that.

s779
01:07:38.480 --> 01:07:44.160
Um but but yeah there's a there's a lot of different types of uh methods you can use.

s780
01:07:44.160 --> 01:07:45.920
You can use suffix decoding.

s781
01:07:45.920 --> 01:07:49.520
Uh there's a lot of times model would come with a built-in drafter model as well.

s782
01:07:49.520 --> 01:07:53.359
So those are like the native models that are trained along with the main model.

s783
01:07:53.359 --> 01:07:57.440
Uh so you don't have to uh import another model to do this.

s784
01:07:57.440 --> 01:08:06.880
Uh there's medusa eagle again like diffusion dlash is a big one that uh has kind of been going viral and it's working pretty well.

s785
01:08:06.880 --> 01:08:14.559
So there's a configuration you set for figuring out how long like how much it generates before the verifier comes in and verifies it.

s786
01:08:14.559 --> 01:08:18.640
So you can set that you can set it to two tokens, four tokens and five tokens.

s787
01:08:18.640 --> 01:08:34.159
And it's important that you set it correctly uh because if there are for example let's just say you set it at four and uh the model verifier comes in it's like okay I only accept two that's 50% acceptance rate and that is pretty bad.

s788
01:08:34.159 --> 01:08:46.560
that's actually going to have a negative impact because not only are you running you're running two models so you're spending compute to run two models and then you're only getting 50% acceptance rate and that's essentially going to result into

s789
01:08:46.560 --> 01:08:48.239
a decrease like throughput.

s790
01:08:48.239 --> 01:08:50.319
Uh so that's not something you want.

s791
01:08:50.319 --> 01:08:56.400
Um so that's something you have to like kind of tweak and play around with and see how many tokens do I want to generate ahead of time.

s792
01:08:56.400 --> 01:09:00.960
Um so yeah technically you could generate 15 tokens but you probably don't want to do that.

s793
01:09:00.960 --> 01:09:03.600
You probably want to do two maybe three maybe four.

s794
01:09:03.600 --> 01:09:07.920
It also depends what type of um what's it called?

s795
01:09:07.920 --> 01:09:09.440
Um let me go back.

s796
01:09:09.440 --> 01:09:10.799
Where is it?

s797
01:09:10.799 --> 01:09:12.719
What type of method you're using again?

s798
01:09:12.719 --> 01:09:16.239
Like uh I use deflash uh for a lot of my home labbing.

s799
01:09:16.239 --> 01:09:19.040
And uh that works pretty well if I set it to eight.

s800
01:09:19.040 --> 01:09:26.640
And if I set it more than eight, I actually get a lot uh a lot less acceptance rate and that causes my performance to go down or be very flaky.

s801
01:09:26.640 --> 01:09:35.040
Um but all right so we will next um

s802
01:09:35.040 --> 01:09:36.400
you got one more question

s803
01:09:36.400 --> 01:09:37.920
one more question.

s804
01:09:37.920 --> 01:09:51.040
Hey uh sorry uh this is from the last module but uh how does dynamic uh compression work uh in a sentence or two I guess

s805
01:09:51.040 --> 01:10:01.440
um so the way it works is if you do you understand um the different heads of the model uh different because there's different layers within an LLM.

s806
01:10:01.440 --> 01:10:10.400
So if you look at the model.json JSON, you'll actually see there's a lot of different parts to an LLM and there's a whole pipeline that sort of like pushes the tokens from one thing to another.

s807
01:10:10.400 --> 01:10:20.880
And a lot of a lot of times when you see a dynamic thing, it essentially is saying, okay, I'm going to only quantize uh the attention heads to FPA, but I'm going to leave everything else as is.

s808
01:10:20.880 --> 01:10:21.280
I see.

s809
01:10:21.280 --> 01:10:22.480
So, so that's the reasoning.

s810
01:10:22.480 --> 01:10:29.440
And if you remember from previous modules, you you saw that the FPA version is not exactly half in size.

s811
01:10:29.440 --> 01:10:32.320
It's not going four gigabytes, it's like 5.1 gigabytes.

s812
01:10:32.320 --> 01:10:36.320
That's because not every part of the model has been quantized to FPA.

s813
01:10:36.320 --> 01:10:38.400
So that's like sort of the dynamic part of it.

s814
01:10:38.400 --> 01:10:43.120
See, is the same math applicable to all the models or it depends model

s815
01:10:43.120 --> 01:10:44.800
depends model to model.

s816
01:10:44.800 --> 01:10:46.560
Every model has a different architecture.

s817
01:10:46.560 --> 01:10:51.280
So architecturally speaking, it depends you know what parts of the model you quantize.

s818
01:10:51.280 --> 01:10:59.520
uh there's people who quantize you know some parts of the model to uh in4 but leave other parts to FP8 and then leave some parts to like BF16.

s819
01:10:59.520 --> 01:10:59.840
So

s820
01:10:59.840 --> 01:11:00.320
thank you.

s821
01:11:00.320 --> 01:11:02.320
Yeah.

s822
01:11:05.920 --> 01:11:06.320
All right.

s823
01:11:06.320 --> 01:11:07.840
We'll run through this again.

s824
01:11:07.840 --> 01:11:09.920
I'm just confirming that our baseline is good.

s825
01:11:09.920 --> 01:11:11.920
We're using FP8 model here.

s826
01:11:11.920 --> 01:11:24.400
Um and then uh we'll do another load test quickly um sort of to get another baseline of um what it would look like before we implement and add speculative decoding

s827
01:11:24.400 --> 01:11:28.560
uh to this particular deployment in VLM.

s828
01:11:31.360 --> 01:11:40.239
And then while that is running, I'm just going to make sure uh to uh let me actually copy this correctly.

s829
01:11:45.040 --> 01:11:47.280
Not that.

s830
01:11:48.000 --> 01:11:49.040
Oh, never mind.

s831
01:11:49.040 --> 01:11:54.080
You can just so in here um we are setting the speculative config.

s832
01:11:54.080 --> 01:11:59.360
This so this is a particular flag you can pass in VLM to set the draft model.

s833
01:11:59.360 --> 01:12:04.719
So we're setting it to the uh 0.6 billion parameter model.

s834
01:12:04.719 --> 01:12:09.760
And then we're also uh setting how many tokens do we want to generate as draft tokens.

s835
01:12:09.760 --> 01:12:12.960
So in this particular scenario we're setting it to four.

s836
01:12:12.960 --> 01:12:15.920
Again this is something you have to do a lot of testing on.

s837
01:12:15.920 --> 01:12:18.320
Uh you have to try out do I want to set it to four?

s838
01:12:18.320 --> 01:12:19.760
Do I want to set it to two?

s839
01:12:19.760 --> 01:12:20.880
Do I want to set it higher?

s840
01:12:20.880 --> 01:12:25.679
It all depends on the acceptance rate and uh how much performance you're getting out of it.

s841
01:12:25.679 --> 01:12:29.760
Um and then yeah once you set this flag and we will redeploy this.

s842
01:12:29.760 --> 01:12:32.159
Um we go back in here.

s843
01:12:32.159 --> 01:12:34.719
We got our baseline.

s844
01:12:35.679 --> 01:12:40.000
And then let me make sure that it's good.

s845
01:12:40.000 --> 01:12:42.800
Okay, things look good here.

s846
01:12:42.800 --> 01:12:48.800
And then we'll go ahead and um apply and roll this out.

s847
01:12:51.600 --> 01:12:53.600
And this is probably going to take another couple minutes.

s848
01:12:53.600 --> 01:12:57.280
So, uh does anyone have any any questions?

s849
01:12:57.280 --> 01:12:59.520
Go ahead.

s850
01:13:01.440 --> 01:13:03.920
Oh, one second.

s851
01:13:10.080 --> 01:13:16.239
I I noticed that you set the GPU memory utilization to 7

s852
01:13:16.239 --> 01:13:23.280
which leaves a lot of your um essentially leaves 30% of the VRAM

s853
01:13:23.280 --> 01:13:24.480
unused.

s854
01:13:24.480 --> 01:13:27.520
Is there any particular logic to that choice?

s855
01:13:27.520 --> 01:13:32.560
Uh in this particular scenario um there could be a lot of reasons.

s856
01:13:32.560 --> 01:13:45.679
Uh sometimes you know you might have multiple deployments on the same GPU and you want to make sure that you have enough RAM for something else VRAM for something else and then you keep enough VRAM for this but in this particular scenario

s857
01:13:45.679 --> 01:13:46.239
not really.

s858
01:13:46.239 --> 01:13:49.120
We're going to bump it up in a following module actually.

s859
01:13:49.120 --> 01:13:55.040
Um it's it's one of those knobs right like you have to keep you have to make sure that you're setting the utilization correctly.

s860
01:13:55.040 --> 01:13:57.760
If you set it too low, you're leaving performance on the table.

s861
01:13:57.760 --> 01:14:04.159
If you set it too high, um you're going to um kill your node, and that's not good.

s862
01:14:04.880 --> 01:14:05.280
Yeah.

s863
01:14:05.280 --> 01:14:11.679
So, this

s864
01:14:14.880 --> 01:14:16.800
Okay, it's still coming up.

s865
01:14:16.800 --> 01:14:21.120
This is taking a little bit more time.

s866
01:14:22.080 --> 01:14:24.320
Go ahead.

s867
01:14:24.480 --> 01:14:26.480
from

s868
01:14:29.679 --> 01:14:31.520
sorry one second

s869
01:14:31.520 --> 01:14:40.000
uh the bump that we see from 60 32 to 64 parallelization it's because of the key value cache just blowing out or I mean

s870
01:14:40.000 --> 01:14:50.000
yeah yeah essentially the key value cache is blowing out and it sort of hit like its max limit and it can't go above that uh so there's a lot of reasons so there's that part and then there's also

s871
01:14:50.000 --> 01:15:02.159
uh the continuous batching and a lot of times times will happen if if you don't if you set the configuration to a level where the continuous batching is not optimal, it can also cause you to see like high tokens per sec.

s872
01:15:02.159 --> 01:15:06.880
Uh uh you can see high latency there as well.

s873
01:15:06.880 --> 01:15:14.719
Um but yeah, in this in this scenario, we're essentially hitting like sort of like a ceiling on us uh uh for us in terms of how much VRAM we can use there.

s874
01:15:14.719 --> 01:15:17.440
That's why we're seeing that.

s875
01:15:18.239 --> 01:15:24.640
But in the following modules, we'll see how we can improve that scenario and hopefully like make it better.

s876
01:15:24.640 --> 01:15:25.440
Uh, okay.

s877
01:15:25.440 --> 01:15:26.960
All right, we're rolled out.

s878
01:15:26.960 --> 01:15:31.520
So, we now have speculative decoding uh set up correctly.

s879
01:15:31.520 --> 01:15:34.800
So, what we'll do is we'll do a quick load test again.

s880
01:15:34.800 --> 01:15:45.920
It's the same exact test that we ran up here where, you know, just keep in mind we're getting 58.2 tokens per second for concurrency one and then 42 milliseconds

s881
01:15:45.920 --> 01:15:47.040
of latency.

s882
01:15:47.040 --> 01:15:51.280
So this is our sort of baseline before we do speculative decoding.

s883
01:15:51.280 --> 01:16:00.239
Um but what I want to showcase here is not a happy path where you will automatically get a crazy improvement in performance.

s884
01:16:00.239 --> 01:16:02.560
You'll actually see the performance went down drastically.

s885
01:16:02.560 --> 01:16:07.520
You're seeing 16% uh 16 tokens per second whereas the baseline was 58.

s886
01:16:07.520 --> 01:16:13.760
Um and then even the time to first token is sort of going uh up a little bit as well.

s887
01:16:13.760 --> 01:16:18.239
uh and you know we want to figure out like why is this happening?

s888
01:16:18.239 --> 01:16:28.960
So there's a there's a big difference and there's a big uh decrease in performance like even if you look at concurrency 16 we're going from 700 tokens per second to 240 tokens per second.

s889
01:16:28.960 --> 01:16:32.800
So that's a considerable decrease in our performance.

s890
01:16:32.800 --> 01:16:36.400
So the way to interpret this is there's a lot of different reasons.

s891
01:16:36.400 --> 01:16:41.840
Um one is again as I was talking about earlier it's the low acceptance rate.

s892
01:16:41.840 --> 01:16:56.640
uh if your acceptance rate is very low and the verifier model is just it's going to keep rejecting those tokens that are generating by draft model, you're wasting a lot of compute to generate those draft tokens and then the verifier is coming in and then saying hey I don't want to take this work like this is bad

s893
01:16:56.640 --> 01:16:57.440
work.

s894
01:16:57.440 --> 01:16:58.560
Um, so that's one.

s895
01:16:58.560 --> 01:17:05.600
Um, it could be because we set the speculative tokens too high, meaning in in our current config, we set it to four.

s896
01:17:05.600 --> 01:17:07.199
Maybe we need to set it down.

s897
01:17:07.199 --> 01:17:12.640
Uh, and see if that actually helps in improve the acceptance criteria or acceptance rate.

s898
01:17:12.640 --> 01:17:15.760
Uh, there's a lot of things with theuler as well with VLM.

s899
01:17:15.760 --> 01:17:22.080
Uh, depending on how it's scheduling these requests in, uh, that could be leaving some, uh, performance out on the table.

s900
01:17:22.080 --> 01:17:25.440
So, it also depends on compatibility and things like that.

s901
01:17:25.440 --> 01:17:31.280
Um and then yeah again same GPU contention you're using the same GPU uh to do this work.

s902
01:17:31.280 --> 01:17:37.840
So it could be uh resulting in u the resources not being used correctly.

s903
01:17:37.840 --> 01:17:43.679
Um so in a lot of cases usually this works but in our scenario it didn't work.

s904
01:17:43.679 --> 01:17:53.440
Uh the main particular reason for that is um this particular drapher model is not trained correctly for this particular model.

s905
01:17:53.440 --> 01:17:57.679
So that's the reason we're seeing a a very very low acceptance rate.

s906
01:17:57.679 --> 01:18:07.920
Um so when you are doing this for your own models and for your own deployments, you have to make sure that the model, the draft model that you're using is going to be compatible with the model that you want to use it for.

s907
01:18:07.920 --> 01:18:16.159
If you don't do that, you'll run into scenario that I'm showcasing here where like your performance is going to tank and um there's not a whole lot you can do about it.

s908
01:18:16.159 --> 01:18:21.280
I can tell you from the resource perspective, this GPU has enough headroom to support all of this.

s909
01:18:21.280 --> 01:18:28.239
But the reason for us seeing this decrease in performance is because the the models are not trained correctly.

s910
01:18:28.239 --> 01:18:31.520
And uh that is something you want to keep in mind and test for as well.

s911
01:18:31.520 --> 01:18:38.960
And you want to make sure the speculative tokens uh are set to the right uh number as well.

s912
01:18:38.960 --> 01:18:50.800
Um but yeah when you in your own time I would definitely implore everyone to play around with that number meaning change this to from four to two to six to one and see

s913
01:18:50.800 --> 01:19:10.320
uh what kind of performance gain you see what kind of performance decrease you see and why is that happening but yeah that that's that is it for speculative decoding it is actually not going to help us in this scenario so I'm just going to go ahead and uh comment that out

s914
01:19:10.320 --> 01:19:12.000
and then we'll redeploy this.

s915
01:19:12.000 --> 01:19:15.360
So we are back to our baseline.

s916
01:19:15.360 --> 01:19:17.280
Hey, you got one question.

s917
01:19:17.280 --> 01:19:18.159
One question.

s918
01:19:18.159 --> 01:19:18.640
Hey.

s919
01:19:18.640 --> 01:19:34.960
Um, so I know there's probably some nuance depending on model architecture, but in general from what you've seen, is there a massive hit to acceptance rate if you use like mixed precision essentially like run the main model at FP8

s920
01:19:34.960 --> 01:19:37.920
and then quantize the draft model down to Q2?

s921
01:19:37.920 --> 01:19:40.560
If I'm like really memory constrained,

s922
01:19:40.560 --> 01:19:43.760
is it often even worth doing that?

s923
01:19:43.760 --> 01:19:47.040
Uh, or you know, what have you seen in general?

s924
01:19:47.040 --> 01:19:48.159
like does

s925
01:19:48.159 --> 01:19:49.360
that's a good question.

s926
01:19:49.360 --> 01:19:54.400
Um a lot of times it you it doesn't matter as much.

s927
01:19:54.400 --> 01:20:00.239
Um most of the time what you want to do is you want to keep the draft model and the main model at the same precision.

s928
01:20:00.239 --> 01:20:05.199
But let's just say the draft model is at BF-16 and then the main model is FP8.

s929
01:20:05.199 --> 01:20:10.080
The mismatch can cause the acceptance rate to be very low.

s930
01:20:10.080 --> 01:20:16.640
So, and then the the main part of it is you have to train the draft model correctly so that way it's able to generate the right tokens.

s931
01:20:16.640 --> 01:20:20.400
In our particular scenario, the draft model was not trained correctly.

s932
01:20:20.400 --> 01:20:24.800
Um, and then it was also quantized down to FP8.

s933
01:20:24.800 --> 01:20:33.040
So, there was probably some issues with that recipe when they trained this model and which resulted in us actually taking a massive performance set.

s934
01:20:33.040 --> 01:20:40.159
Um, so a lot of times when you when uh a lab releases a model, they release uh a drafter model along with it.

s935
01:20:40.159 --> 01:20:43.760
So you want to use those drafter models uh that come with the model.

s936
01:20:43.760 --> 01:20:46.480
Uh there's also thirdparty labs that do their own drafter model.

s937
01:20:46.480 --> 01:20:49.920
So you can try out a lot of different options that are out there and see which one works for you.

s938
01:20:49.920 --> 01:21:03.679
And in some cases, depending on the use case, let's just say you're using for coding, the drafter model might work perfectly fine, but if you're switching that to maybe some English writing, very specific academic writing, and it might not necessarily work well, and you're going to see like decrease in acceptance rate.

s939
01:21:03.679 --> 01:21:07.840
So, it's also uh it also depends a lot on your use case as well.

s940
01:21:07.840 --> 01:21:10.000
So, it's not universally applicable.

s941
01:21:10.000 --> 01:21:10.800
Gotcha.

s942
01:21:10.800 --> 01:21:11.360
Cool.

s943
01:21:11.360 --> 01:21:11.920
Makes sense.

s944
01:21:11.920 --> 01:21:12.480
Thanks.

s945
01:21:12.480 --> 01:21:14.480
Yeah.

s946
01:21:17.040 --> 01:21:19.120
All right.

s947
01:21:19.600 --> 01:21:24.000
Let's jump on to the inference side of things.

s948
01:21:24.000 --> 01:21:28.719
But before you make another jump, any other questions?

s949
01:21:29.600 --> 01:21:30.159
Oh, yeah.

s950
01:21:30.159 --> 01:21:33.199
I can put the YAML on there.

s951
01:21:35.440 --> 01:21:48.560
Um, so the the fact that we have speculative decoding uh presumably is because um for for a model it's faster to verify rather than decode.

s952
01:21:48.560 --> 01:21:48.880
Yeah.

s953
01:21:48.880 --> 01:21:58.320
Can you just give a little bit of mental model for like what it's what a model does underneath the hood um to verify such that it's faster than decoding

s954
01:21:58.320 --> 01:22:02.320
you know it can't be decoding and then comparing against what the draft model produces.

s955
01:22:02.320 --> 01:22:03.040
So like

s956
01:22:03.040 --> 01:22:06.639
what is it doing exactly so so that it's faster.

s957
01:22:06.639 --> 01:22:06.880
Yeah.

s958
01:22:06.880 --> 01:22:23.120
So essentially uh when a typical LLM is you know decoding you have a set of vocabulary it's a huge array of bunch of numbers and it's representing um a confidence score for each of those items right so instead of

s959
01:22:23.120 --> 01:22:32.800
um instead of doing that one token at a time when you run with specular decoding you're running it for a lot of you you're running the model for parallel things.

s960
01:22:32.800 --> 01:22:37.040
So with continuous batching as well, you're running a lot of different requests in parallel.

s961
01:22:37.040 --> 01:22:41.199
Uh so you're doing the same uh you're doing the calculation for everything.

s962
01:22:41.199 --> 01:22:44.159
Uh and you're doing it at once for a lot of different requests.

s963
01:22:44.159 --> 01:22:46.719
In the same way, you're doing the same thing for speculative decoding.

s964
01:22:46.719 --> 01:22:53.679
You're checking the confidence score for this uh the speculated tokens and making sure that oh this is a high confidence, so we're going to go for it.

s965
01:22:53.679 --> 01:23:01.840
This is you know 0.95 and this one is like 0.9. And then you you accept that and then the other ones you you know uh you reject those.

s966
01:23:01.840 --> 01:23:12.880
That's sort of roughly the mental model is like you are in parallel running uh through the layers and making sure that each of the tokens that the draft model

s967
01:23:12.880 --> 01:23:17.040
uh outputs is has a high confidence.

s968
01:23:17.199 --> 01:23:20.880
Yeah, hopefully that answers your question.

s969
01:23:21.840 --> 01:23:23.440
Oh, one more question.

s970
01:23:23.440 --> 01:23:26.800
Do you have any examples that

s971
01:23:26.800 --> 01:23:28.000
uh yes I do.

s972
01:23:28.000 --> 01:23:30.000
That's a good call out.

s973
01:23:30.000 --> 01:23:33.679
Um there's actually I left a few links in here.

s974
01:23:33.679 --> 01:23:36.159
Um where did I leave those?

s975
01:23:36.159 --> 01:23:37.679
Interpreting the results.

s976
01:23:37.679 --> 01:23:37.920
Yeah.

s977
01:23:37.920 --> 01:23:42.320
So there's a lot of different uh writeups on speculative decoding in in general.

s978
01:23:42.320 --> 01:23:46.000
If you do it right, you do see quite a bit of increase in performance.

s979
01:23:46.000 --> 01:23:57.520
Uh VLM has actually done a big report on this and they've seen about 1.5x increase in token generation if you set it up correctly with the right model and with the right drafter and the right settings.

s980
01:23:57.520 --> 01:23:59.520
Um so yeah, this this does excel.

s981
01:23:59.520 --> 01:24:06.080
It's just a matter of getting it right and tuning it correctly and making sure you have the right um things in place for it.

s982
01:24:06.080 --> 01:24:18.400
Um but yeah, like from my personal experience, um for my home lab, I used the Quinn 3.6 35 billion model and along with that I use a Dlash um

s983
01:24:18.400 --> 01:24:28.639
um draft model and I get roughly around two to 3x token generation uh performance improvement uh depending on the use case.

s984
01:24:28.639 --> 01:24:30.159
So it it does work.

s985
01:24:30.159 --> 01:24:34.239
It's just a matter of getting the right models to work together.

s986
01:24:34.239 --> 01:24:37.440
How do you choose the model size

s987
01:24:38.960 --> 01:24:41.040
for the daffer?

s988
01:24:41.040 --> 01:24:53.199
Uh you want to you basically want to figure out uh the smallest model that you can run that doesn't take as much GPU power because you the main model is the model that has all the knowledge.

s989
01:24:53.199 --> 01:24:56.719
That's the model that you really care about that has everything that you need.

s990
01:24:56.719 --> 01:25:00.639
But you don't want draft model to take a lot of those resources away from the main model.

s991
01:25:00.639 --> 01:25:07.440
So you want it to be small but not so small that it's dumb but big enough where like it's going to actually give you

s992
01:25:07.440 --> 01:25:08.480
you experimented with that.

s993
01:25:08.480 --> 01:25:13.920
Did you see like acceptance rates like 3B model

s994
01:25:13.920 --> 01:25:14.960
3B is too big.

s995
01:25:14.960 --> 01:25:21.040
Uh the reason why is a lot of models like I was talking about qu 3.6 uh 35 billion.

s996
01:25:21.040 --> 01:25:25.920
So it has it's a mixture of experts model and each expert is about three billion parameters.

s997
01:25:25.920 --> 01:25:29.920
So the experts can be around that uh and you would get good performance out of it.

s998
01:25:29.920 --> 01:25:44.000
But for the draft model, you probably want to be less than 1 billion parameter just to make sure that it also depends on the main model you're using I guess right like if you're using if your main model is like 400 billion parameters you could probably get away with using a bigger drafter model to make sure that

s999
01:25:44.000 --> 01:25:46.080
you get higher acceptance rate.

s1000
01:25:46.080 --> 01:25:49.600
So it's you got to find the right balance essentially.

s1001
01:25:49.600 --> 01:25:51.600
Yeah.

s1002
01:25:55.600 --> 01:25:56.320
Cool.

s1003
01:25:56.320 --> 01:26:00.080
Um, do we have any other questions over there?

s1004
01:26:00.080 --> 01:26:04.480
One more before you jump in.

s1005
01:26:04.480 --> 01:26:05.840
Yeah, I just had a question.

s1006
01:26:05.840 --> 01:26:08.239
How is the draft model trained?

s1007
01:26:08.239 --> 01:26:10.639
Like is it distillation or something else?

s1008
01:26:10.639 --> 01:26:12.239
It's essentially distillation.

s1009
01:26:12.239 --> 01:26:17.120
Uh, so you take the big model and you know you distill all of its capabilities.

s1010
01:26:17.120 --> 01:26:21.600
You create a data set and then you train the uh drafter model using that.

s1011
01:26:21.600 --> 01:26:24.639
And a lot of times actually what's there's different techniques to do this.

s1012
01:26:24.639 --> 01:26:30.000
Uh the other techniques you'll see is the draft model is trained alongside the main model.

s1013
01:26:30.000 --> 01:26:36.719
Uh so that way that's another way of doing it and a lot of labs will actually give you a drafter model uh inside of the same package.

s1014
01:26:36.719 --> 01:26:39.840
So you don't have to essentially download a second model and run it.

s1015
01:26:39.840 --> 01:26:42.080
Um so there's a couple of different ways of doing it.

s1016
01:26:42.080 --> 01:26:50.239
Uh but probably the best way is train alongside it and then that way you don't have a lot of issues and acceptance rate sort of stays high.

s1017
01:26:50.480 --> 01:26:51.280
Cool.

s1018
01:26:51.280 --> 01:26:53.679
Um, we'll jump into module seven.

s1019
01:26:53.679 --> 01:27:03.920
So, here I want to talk about some of the different flags that you uh should care about when deploying using VLM and what and how you want to get the saturation to be as high as possible.

s1020
01:27:03.920 --> 01:27:09.040
Saturation meaning we want to get the GPU to work as hard as possible as long as possible.

s1021
01:27:09.040 --> 01:27:16.719
Uh, and because you want to get your money's worth if you're renting a GPU or if you're buying a GPU, you want to run it uh as much as possible.

s1022
01:27:16.719 --> 01:27:23.840
So uh we will stop here.

s1023
01:27:23.840 --> 01:27:32.560
So uh one of the concepts I want to talk about uh if you remember previously we were looking at the 34 uh 32 concurrency and 64 concurrency.

s1024
01:27:32.560 --> 01:27:47.840
We saw a steep increase in the latency and the reason for that is essentially you the throughput at some point is going to flatten out and but the uh but the latency can keep increasing if you keep increasing the

s1025
01:27:47.840 --> 01:27:48.960
uh concurrency.

s1026
01:27:48.960 --> 01:28:02.080
So you essentially want to find the sweet spot and uh that is something that VLM sort of lets you kind of tweak and make sure that you find that sweet spot and once you do find it you can set the right settings for it.

s1027
01:28:02.080 --> 01:28:15.199
Um but but essentially you want to find the inter not so not the intersection but essentially right before where the throughput gets to like the flat state but the latency is like still acceptable enough.

s1028
01:28:15.199 --> 01:28:18.560
It's sort of like the point you want to get to.

s1029
01:28:19.760 --> 01:28:28.719
Um we will do a setup and then let's make sure our settings are all good still.

s1030
01:28:28.719 --> 01:28:30.719
Serving model is up.

s1031
01:28:30.719 --> 01:28:34.560
Um, let me just double check if it's up.

s1032
01:28:36.320 --> 01:28:38.400
Should be.

s1033
01:28:38.400 --> 01:28:40.400
Okay.

s1034
01:28:41.840 --> 01:28:48.159
And then here we're again going to get um another we're going to do another load test.

s1035
01:28:48.159 --> 01:28:54.000
So that way we know the data and then we can based on that figure out where that uh where this point is.

s1036
01:28:54.000 --> 01:29:04.320
Essentially the we call it the knee um where things are flattening but also the concurren based on concurrency uh the time to first token is acceptable.

s1037
01:29:04.320 --> 01:29:10.960
Um once that is Oh, I think that should be fine.

s1038
01:29:18.880 --> 01:29:23.280
So you know the throughput is slowly increasing as concurrency is increasing.

s1039
01:29:23.280 --> 01:29:24.960
That is acceptable.

s1040
01:29:24.960 --> 01:29:27.520
Um and then you also see the latency go up.

s1041
01:29:27.520 --> 01:29:30.960
That is also acceptable but it's not going up drastically.

s1042
01:29:30.960 --> 01:29:39.920
Uh but we should shortly see the big jumps uh with more um concurrency here.

s1043
01:29:39.920 --> 01:29:44.560
But while this is running, any any other questions for anyone?

s1044
01:29:44.560 --> 01:29:46.719
Go ahead.

s1045
01:29:46.960 --> 01:29:47.760
Oh, over there.

s1046
01:29:47.760 --> 01:29:52.000
So my question was again on the previous speculative decoding.

s1047
01:29:52.000 --> 01:30:03.760
Uh just wanted to know based on your experience um with like drafter model uh what kind of use cases you have seen it to work really like for example with agent loops

s1048
01:30:03.760 --> 01:30:07.040
with tool call or with coding like what kind of like use

s1049
01:30:07.040 --> 01:30:16.080
uh agent loops it's really good at tool calling it's really good at u browser use is the one particular use case that I use it for I want to automate a lot of browser

s1050
01:30:16.080 --> 01:30:22.320
tasks um because there's still you know you don't necessarily get APIs for everything and so you want to bridge that gap.

s1051
01:30:22.320 --> 01:30:27.280
Uh and uh yeah, the drafting model helps you generate more tokens faster.

s1052
01:30:27.280 --> 01:30:31.280
So you see your browser agent just essentially run around and do things faster.

s1053
01:30:31.280 --> 01:30:33.360
Uh so in a lot of ways it does help.

s1054
01:30:33.360 --> 01:30:40.320
Uh I will say for browser use I probably see around 1.5x increase whereas for coding I see like closer to 3x.

s1055
01:30:40.320 --> 01:30:45.600
So depending on your use case you may see more or less and and things like that.

s1056
01:30:45.600 --> 01:30:47.600
Uh, and a followup on that one.

s1057
01:30:47.600 --> 01:30:52.480
Are there any benchmarks like more trust me bro like coding fire ship?

s1058
01:30:52.480 --> 01:30:54.480
There there's definitely benchmarks out there.

s1059
01:30:54.480 --> 01:30:57.840
I I'm I'm personally a little skeptical of benchmarks whenever I see them.

s1060
01:30:57.840 --> 01:30:59.840
I never get the same results.

s1061
01:30:59.840 --> 01:31:01.760
I don't know if people agree with that here.

s1062
01:31:01.760 --> 01:31:16.239
Um, so, um, yeah, for the for the most part, like you know, you want to do your own testing and see what works for you because it very much depends on your use case because the acceptance rate is going to be different depending on your use case, what you're using it for.

s1063
01:31:16.239 --> 01:31:18.480
Thank you.

s1064
01:31:20.080 --> 01:31:21.120
Um, yeah.

s1065
01:31:21.120 --> 01:31:23.280
Any other questions?

s1066
01:31:23.280 --> 01:31:23.840
Mic check.

s1067
01:31:23.840 --> 01:31:24.960
Mic check.

s1068
01:31:24.960 --> 01:31:26.159
How's it going?

s1069
01:31:26.159 --> 01:31:43.440
Um do like big models like codeex or cloud like do they use this type of the coding like have they published that they use these patterns or is this more like for open source if you're training your own stuff or running inference

s1070
01:31:43.440 --> 01:31:56.400
I don't know the question to if they do it or not um I would imagine they do sort of like free performance but uh for for the open source yeah this is particularly for the open source all the open source big models that you

s1071
01:31:56.400 --> 01:32:04.880
want to use all the Quinn models, the GLM models, Kimmy models, they all have specular decoding uh drafter models available that you want to use.

s1072
01:32:04.880 --> 01:32:11.520
Is there like a main drafting model that they use for example for like DeepSseek or GLM?

s1073
01:32:11.520 --> 01:32:16.080
Deepseek actually came out with a new uh thing yesterday I believe or the day before.

s1074
01:32:16.080 --> 01:32:17.120
It's called DSpark.

s1075
01:32:17.120 --> 01:32:21.840
It's like their own technique for doing specular decoding in a more dynamic way.

s1076
01:32:21.840 --> 01:32:33.199
Essentially they're figuring out how so here our configuration is static right like we say we're going to generate four tokens at a time uh but with deepseek's way of doing it they are making it dynamic meaning

s1077
01:32:33.199 --> 01:32:44.239
you know it on on the fly sort of decides oh I'm going to generate only two tokens or I'm going to generate four tokens or five tokens depending on the acceptance criteria depending on the GPU load and all all sorts of metrics.

s1078
01:32:44.239 --> 01:32:52.239
So yeah, every every model uh every lab has like their own way of doing it as well and you probably want to stick to that but you can also experiment.

s1079
01:32:52.239 --> 01:33:03.120
For example, the Quinn models uh Quinn lab doesn't release the Dlash models for drafting but these are released by uh a third party lab but they work quite well with them.

s1080
01:33:03.120 --> 01:33:05.600
So you sort of have to test it out and see.

s1081
01:33:05.600 --> 01:33:07.840
Thank you.

s1082
01:33:08.400 --> 01:33:10.880
Yeah, I think this is hanging right.

s1083
01:33:10.880 --> 01:33:13.440
Yeah, I would check the pot.

s1084
01:33:13.440 --> 01:33:19.120
Yeah, might have killed a cluster here.

s1085
01:33:19.280 --> 01:33:19.520
Uh

s1086
01:33:19.520 --> 01:33:21.040
oh, I can't type.

s1087
01:33:21.040 --> 01:33:21.280
Uh

s1088
01:33:21.280 --> 01:33:23.280
oh,

s1089
01:33:24.400 --> 01:33:26.719
this was working well.

s1090
01:33:26.719 --> 01:33:28.719
Refresh.

s1091
01:33:28.829 --> 01:33:30.829
[laughter]

s1092
01:33:31.120 --> 01:33:34.560
Let me create the tab again.

s1093
01:33:35.440 --> 01:33:36.639
Okay.

s1094
01:33:36.639 --> 01:33:40.719
Um well, something is wrong.

s1095
01:33:40.880 --> 01:33:43.600
It copied a URL and opened in a new tab.

s1096
01:33:43.600 --> 01:33:45.679
See what happens.

s1097
01:33:45.679 --> 01:33:48.320
Let me open a new tab.

s1098
01:33:48.320 --> 01:33:51.199
I wouldn't close that one.

s1099
01:33:55.840 --> 01:34:01.040
You know, in all the testing that has not happened, it happened today.

s1100
01:34:02.800 --> 01:34:06.320
Can I at least scroll through and go through different things?

s1101
01:34:06.320 --> 01:34:25.440
Okay, at least I can scroll scroll through um you know since uh the cluster decided to take uh a vacation uh I'll sort of go through the content that I was going to cover.

s1102
01:34:25.440 --> 01:34:32.800
Um the main point of running this benchmark here was to kind of show where that knee is meaning where you see the max throughput.

s1103
01:34:32.800 --> 01:34:44.880
Uh if we go back to this diagram where you see the max throughput while making sure you are seeing the lowest uh tokens uh lowest latency for token generation.

s1104
01:34:44.880 --> 01:34:50.560
Um and that that is that essentially limit was at 32 concurrency.

s1105
01:34:50.560 --> 01:34:57.600
Um because anything above 32 concurrency you'll see uh the token generation is going to be kind of the same.

s1106
01:34:57.600 --> 01:34:59.040
It's not going to change much.

s1107
01:34:59.040 --> 01:35:03.600
uh the throughput is going to be the same but but the latency is going to go drastically up and you don't want that.

s1108
01:35:03.600 --> 01:35:06.159
Uh so that's sort of like the limit you are hitting.

s1109
01:35:06.159 --> 01:35:13.360
Um and that is important information that we need to keep in mind and look at when we're deploying these things.

s1110
01:35:13.360 --> 01:35:16.239
Uh and if

s1111
01:35:16.560 --> 01:35:18.480
yeah you can go to 100.

s1112
01:35:18.480 --> 01:35:21.440
Go to look at the spreadsheet

s1113
01:35:21.440 --> 01:35:22.000
100.

s1114
01:35:22.000 --> 01:35:25.040
Yeah, go to 100 and then just continue from

s1115
01:35:25.040 --> 01:35:25.440
Yeah.

s1116
01:35:25.440 --> 01:35:27.920
seven or eight.

s1117
01:35:28.159 --> 01:35:30.560
Where is it?

s1118
01:35:35.120 --> 01:35:36.159
There you go.

s1119
01:35:36.159 --> 01:35:36.480
All right.

s1120
01:35:36.480 --> 01:35:37.520
Well, this one's working.

s1121
01:35:37.520 --> 01:35:39.840
That's good.

s1122
01:35:45.760 --> 01:35:46.239
All right.

s1123
01:35:46.239 --> 01:35:46.639
Cool.

s1124
01:35:46.639 --> 01:35:49.040
So, we'll jump into eight.

s1125
01:35:49.040 --> 01:35:52.480
Um, let me also

s1126
01:35:54.896 --> 01:35:55.120
[laughter]

s1127
01:35:55.120 --> 01:36:00.320
open up Who's doing it?

s1128
01:36:05.280 --> 01:36:17.760
Okay, so for module A, we're going to talk quickly through um what sort of flags and knobs you can tweak to make sure that you can sort of get above that limiting point that we were talking about at 32 concurrency.

s1129
01:36:17.760 --> 01:36:24.239
You're sort of seeing that limit and how do we kind of get beyond that and what things you need to keep in mind when thinking about this.

s1130
01:36:24.239 --> 01:36:40.560
Uh so for this one I'm sort of going to give you guys um a thinking model is essentially you want to measure and that's why with a lot of different cells I was I keep running load tests and it's super important to run those load tests to see what your performance is like and any changes you're making

s1131
01:36:40.560 --> 01:36:53.199
how is that changing the performance is it making better is it making it worse um and in what particular metric it's is it making worse right like if is it if the throughput keeps going up or stays the same but like the latency is changing

s1132
01:36:53.199 --> 01:36:55.040
something is either correct or wrong there.

s1133
01:36:55.040 --> 01:36:57.119
So, that's something we have to keep in mind.

s1134
01:36:57.119 --> 01:37:03.360
Um, and there's a bunch of different knobs you can sort of like uh tweak in VLM to make it better.

s1135
01:37:03.360 --> 01:37:06.960
But the main four ones are again the GPU memory utilization.

s1136
01:37:06.960 --> 01:37:10.639
You want to make sure that you're setting that correctly depending on your use case.

s1137
01:37:10.639 --> 01:37:16.239
Again, if you're running multiple models on the same GPU, you might want to keep it low and that's a trade-off you're making.

s1138
01:37:16.239 --> 01:37:25.360
But if you're only running a single model and you want to make sure that it's getting the best performance possible, you want to bump that up to 0.9 or point even 0.95.

s1139
01:37:25.440 --> 01:37:33.600
Uh the other thing is you want to make sure that uh the max numbum sequences uh is set to a good limit as well.

s1140
01:37:33.600 --> 01:37:35.520
This is something you have to figure out by testing.

s1141
01:37:35.520 --> 01:37:44.719
Uh in our particular deployment, if you look at the VLM.yaml, um we start with 32 sequences, right?

s1142
01:37:44.719 --> 01:37:51.920
So what that is doing is it's telling VLM that it can only process 32 requests at a time uh and not more than that.

s1143
01:37:51.920 --> 01:37:55.760
And if you do more than that, you're going to see some performance uh degradation there.

s1144
01:37:55.760 --> 01:38:01.360
Um so there's a lot of different flags, but those are like the key ones that you want to keep in mind.

s1145
01:38:01.360 --> 01:38:05.520
You want to make sure that you're setting the model length to the correct length as well.

s1146
01:38:05.520 --> 01:38:11.199
Uh and then you know you want to make sure you're the batch tokens are also set to the correct length.

s1147
01:38:11.199 --> 01:38:16.159
um you don't want to you want to keep it bigger if you want to do prefill faster if your GPU can handle it.

s1148
01:38:16.159 --> 01:38:21.679
You want to do it smaller if your GPU is a on a little slower side and you can't handle it.

s1149
01:38:21.679 --> 01:38:25.360
Um but yeah, the idea is you want to change one flag at a time.

s1150
01:38:25.360 --> 01:38:27.199
You don't want to change multiple things.

s1151
01:38:27.199 --> 01:38:34.639
Um and when you change things, run the uh run the benchmarks and see uh what performance you're seeing.

s1152
01:38:34.639 --> 01:38:38.639
See measure it and then make assumptions on what you should do next.

s1153
01:38:38.639 --> 01:38:42.480
Um, kind of go through it quickly here.

s1154
01:38:42.480 --> 01:38:46.320
We have about what 10 minutes left.

s1155
01:38:49.040 --> 01:38:51.280
Uh, oh,

s1156
01:38:54.400 --> 01:38:57.040
I think we lost the P PVC.

s1157
01:38:57.040 --> 01:38:58.000
I think that's what happened.

s1158
01:38:58.000 --> 01:38:58.320
I got

s1159
01:38:58.320 --> 01:38:59.360
Oh, it got the different one.

s1160
01:38:59.360 --> 01:39:00.960
Yeah, I need to actually change this.

s1161
01:39:00.960 --> 01:39:03.199
Rerun this, I think.

s1162
01:39:03.199 --> 01:39:10.880
Um, let me actually just change this in here.

s1163
01:39:11.040 --> 01:39:15.199
This was supposed to be like sequential, but other Jupyter notebook died.

s1164
01:39:15.199 --> 01:39:19.119
So, change that and then that should work.

s1165
01:39:19.119 --> 01:39:24.960
Um, again, like I talked about, there's different knobs uh that you want to tweak um and see what happens.

s1166
01:39:24.960 --> 01:39:27.119
But we'll get sort of the measure.

s1167
01:39:27.119 --> 01:39:31.679
We'll measure the baseline and then we'll change a couple flags and see what happens next.

s1168
01:39:31.679 --> 01:39:36.159
Um, does it say work?

s1169
01:39:36.639 --> 01:39:37.360
Okay, cool.

s1170
01:39:37.360 --> 01:39:38.000
There you go.

s1171
01:39:38.000 --> 01:39:40.320
It's working.

s1172
01:39:40.639 --> 01:39:46.480
While this load test is running, any other questions anyone has?

s1173
01:39:46.480 --> 01:39:48.880
Excuse me.

s1174
01:39:52.000 --> 01:39:57.040
I noticed that there's a definite fall-off when we're optimizing the model.

s1175
01:39:57.040 --> 01:40:00.480
Like here, the 32 to 64, huge falloff.

s1176
01:40:00.480 --> 01:40:10.560
Uh is that when you would decide that maybe we should scale the hardware or is that something that you're confident you can still tune it further?

s1177
01:40:10.560 --> 01:40:14.480
Uh essentially is this is this the ceiling?

s1178
01:40:14.480 --> 01:40:17.440
Yeah, you figure out the ceiling by doing mathematics.

s1179
01:40:17.440 --> 01:40:26.800
I think in some of the previous modules Dwan covered uh like what the max ceiling can look like for tokens per second what the throughput looks like and um

s1180
01:40:26.800 --> 01:40:39.360
you know what sort of expectations you should have for the deployment and if you are like seeing those max expectations that's probably the maximum limit of your hardware and then you probably have to get additional hardware or better hardware to do so.

s1181
01:40:39.360 --> 01:40:47.600
Uh so yeah there there is sort of a limit you know there's only so much you can tune but you're trying to squeeze as much out of the GPU as possible.

s1182
01:40:48.480 --> 01:40:52.960
You want to share the slides before you continue?

s1183
01:40:54.400 --> 01:40:57.360
Share the slides real quick.

s1184
01:41:02.480 --> 01:41:07.119
The server I can leave it up for a little bit, couple hours.

s1185
01:41:07.119 --> 01:41:11.040
I'm not going to leave it up just because you do have access to GPUs.

s1186
01:41:11.040 --> 01:41:22.880
Um but what what I am going to give you if you scan this QR code you'll get access to the PowerPoint as well as the repo if you want to spin this up in your own AI cloud account

s1187
01:41:22.880 --> 01:41:25.280
or if you want to run on your laptop you can do that.

s1188
01:41:25.280 --> 01:41:35.199
So if you scan this QR code it'll take you to a site that has all the resources and if you don't have a cloud account you can sign up and you get $300 in credits

s1189
01:41:35.199 --> 01:41:36.719
with a free account.

s1190
01:41:36.719 --> 01:41:38.320
So that's everything you need.

s1191
01:41:38.320 --> 01:41:42.400
And if you all have questions, both of our contact information is here.

s1192
01:41:42.400 --> 01:41:43.920
Omar, I'm Dwan Lightoot.

s1193
01:41:43.920 --> 01:41:44.239
Yeah.

s1194
01:41:44.239 --> 01:41:49.280
And yeah, with that being said, if you all scan the QR code, I'll let Omar finish it up.

s1195
01:41:49.280 --> 01:41:52.239
I just want to make sure you all get this before people started to leave.

s1196
01:41:52.239 --> 01:41:54.480
Thank you.

s1197
01:41:54.560 --> 01:42:00.400
We'll leave it up there for a couple more minutes gets it.

s1198
01:42:01.920 --> 01:42:04.320
Yeah, go ahead.

s1199
01:42:18.400 --> 01:42:19.840
personal assistant.

s1200
01:42:19.840 --> 01:42:22.480
Um, so

s1201
01:42:23.920 --> 01:42:25.679
so you can use llamas.

s1202
01:42:25.679 --> 01:42:26.719
It depends on your hardware.

s1203
01:42:26.719 --> 01:42:28.800
I think that's like the first thing I would look at.

s1204
01:42:28.800 --> 01:42:30.480
Um,

s1205
01:42:30.480 --> 01:42:31.119
no problem.

s1206
01:42:31.119 --> 01:42:31.760
Thank you.

s1207
01:42:31.760 --> 01:42:33.119
Oh, you have both.

s1208
01:42:33.119 --> 01:42:35.360
for the sake of the argument, let's just say you have the best GPU.

s1209
01:42:35.360 --> 01:42:38.800
Let's just say you have um

s1210
01:42:38.880 --> 01:42:39.440
Yeah.

s1211
01:42:39.440 --> 01:42:39.679
Yeah.

s1212
01:42:39.679 --> 01:42:41.760
You have the latest Blackwill GPU.

s1213
01:42:41.760 --> 01:42:53.119
Um you know, I think for testing and doing like figuring out what model you want to use for your use case, I think it's good to use Olama and Llama CPP to quickly spin it up and try it out and see what happens.

s1214
01:42:53.119 --> 01:43:03.360
Uh but once you have a model that you know I'm going to use for sure for this use case and this this is working great you probably want to switch to uh VLM sg lang or one of these other model

s1215
01:43:03.360 --> 01:43:04.320
uh runtimes.

s1216
01:43:04.320 --> 01:43:10.320
Uh reason being they allow you to run at higher concurrencies and give you more performance out of it.

s1217
01:43:10.320 --> 01:43:13.920
So if you have a single agent running I think it's fine to use Olama.

s1218
01:43:13.920 --> 01:43:17.199
It's not a big deal because you only have one request going in at a time.

s1219
01:43:17.199 --> 01:43:26.800
But if you let's just say you're running like 20 agents and you want to run it parallel uh Yeah.

s1220
01:43:26.800 --> 01:43:29.920
Yeah, you can set up speculative decoding with VLM.

s1221
01:43:29.920 --> 01:43:34.320
I don't think Olma supports speculative decoding as far as I know, but I could be wrong.

s1222
01:43:34.320 --> 01:43:41.920
Um, but yeah, it when you want to get the max performance out of your hardware, you definitely want to do that.

s1223
01:43:44.639 --> 01:43:46.639
Yeah,

s1224
01:43:46.960 --> 01:43:50.719
I actually don't know if there's a diffusion uh LLM.

s1225
01:43:50.719 --> 01:43:54.960
Is there an open source diffusion model right now available?

s1226
01:43:56.400 --> 01:43:57.840
Yeah, Lana.

s1227
01:43:57.840 --> 01:43:59.840
Yeah, I mean I think VLM supports it.

s1228
01:43:59.840 --> 01:44:02.560
So I think you you would be able to run it on VLM.

s1229
01:44:02.560 --> 01:44:05.920
Uh Olama, if supports it, you can as well.

s1230
01:44:05.920 --> 01:44:10.239
Um yeah, that's definitely possible.

s1231
01:44:10.239 --> 01:44:15.040
Okay, we'll switch back to the module and kind of go through it.

s1232
01:44:15.040 --> 01:44:25.920
Um, again like I was saying, at 32 you're seeing, you know, good performance, but at 64, uh, we're going at, you know, the the throughput is sort of staying the same, if not going up,

s1233
01:44:25.920 --> 01:44:30.480
but we're seeing, um, a significant increase in latency.

s1234
01:44:30.480 --> 01:44:35.920
And to kind of get around it, um, one of the things I want to talk about is the max num sequence.

s1235
01:44:35.920 --> 01:44:41.679
Again, this is sort of, uh, how many requests VLM is able to process at one time.

s1236
01:44:41.679 --> 01:44:43.280
It's sort of setting that flag.

s1237
01:44:43.280 --> 01:44:50.560
If you want to have a lot of uh requests being processed in parallel uh then you want to sort of increase this amount.

s1238
01:44:50.560 --> 01:44:54.159
So currently I believe it's set up at 32.

s1239
01:44:54.159 --> 01:45:05.440
So we can bump this up to 128 and then we can run this deploy it and see how that changes the performance.

s1240
01:45:09.440 --> 01:45:12.800
Okay, it changed a lot of things here.

s1241
01:45:17.360 --> 01:45:17.547
Yeah,

s1242
01:45:17.547 --> 01:45:18.480
[snorts]

s1243
01:45:18.480 --> 01:45:32.400
once this comes up, uh, we can do another test and, um, and you you'll you'll see the the limitations that we're seeing here at 32 is going to go away and you'll actually see the latency be manageable again

s1244
01:45:32.400 --> 01:45:34.639
um, at higher concurrency levels as well.

s1245
01:45:34.639 --> 01:45:36.639
So, this is sort of like the loop you want to go in.

s1246
01:45:36.639 --> 01:45:40.080
You want to find the right flags for wherever the bottleneck is.

s1247
01:45:40.080 --> 01:45:47.440
find the bottleneck, figure out why that is happening, what flag you can tweak uh to make sure that you're kind of overcoming that bottleneck.

s1248
01:45:47.440 --> 01:45:52.000
Um it's come up.

s1249
01:45:52.480 --> 01:45:54.560
That's going to take a little bit of time.

s1250
01:45:54.560 --> 01:45:56.320
Um any questions?

s1251
01:45:56.320 --> 01:45:58.960
One more over there.

s1252
01:46:02.719 --> 01:46:05.600
Potentially bursty traffic

s1253
01:46:05.600 --> 01:46:06.719
um at times.

s1254
01:46:06.719 --> 01:46:07.040
Um

s1255
01:46:07.040 --> 01:46:07.679
Oh yeah.

s1256
01:46:07.679 --> 01:46:10.719
Is there such thing as like bursting to serverless?

s1257
01:46:10.719 --> 01:46:14.880
Is that the domain of you know model routing layer on top?

s1258
01:46:14.880 --> 01:46:17.600
Like what is the pattern for

s1259
01:46:17.600 --> 01:46:18.000
Yeah.

s1260
01:46:18.000 --> 01:46:27.040
Um I mean autoscaling is one of the patterns like specifically I'll tell you in LKE uh you can have uh dynamic like autoscaling.

s1261
01:46:27.040 --> 01:46:31.760
So it can scale up uh depending on your needs depending on bursty traffic.

s1262
01:46:31.760 --> 01:46:46.080
it can scale up to you know maybe use a node that has four GPUs instead of one because maybe that handles it better right so you can automate a lot of those things uh to make sure that you're doing it well and the other part of it is like you can also build like some statistical models to

s1263
01:46:46.080 --> 01:46:58.800
make sure that you kind of understand your traffic patterns over a long horizon and then the uh the cluster can also dynamically like sort of predict your burstiness in some ways and then can also scale based on that too

s1264
01:46:58.800 --> 01:47:01.119
if you got your browser I can show you where it's at

s1265
01:47:01.119 --> 01:47:04.239
browser you open your browser, I'll take it to you.

s1266
01:47:04.239 --> 01:47:07.040
Okay, so this one is successfully rolled out.

s1267
01:47:07.040 --> 01:47:17.360
Um, and then we'll do a quick load test and once that is run, we can see more.

s1268
01:47:18.639 --> 01:47:19.360
All right.

s1269
01:47:19.360 --> 01:47:21.360
Uh, any other questions?

s1270
01:47:21.360 --> 01:47:23.119
Most folks are gone, but

s1271
01:47:23.119 --> 01:47:29.040
yeah, like foot H.

s1272
01:47:29.679 --> 01:47:30.560
Yeah.

s1273
01:47:30.560 --> 01:47:31.280
32.

s1274
01:47:31.280 --> 01:47:34.239
That's where we saw the limit last time.

s1275
01:47:34.239 --> 01:47:39.520
Once we get to 64, we should see uh that it should improve.

s1276
01:47:39.520 --> 01:47:40.080
Okay.

s1277
01:47:40.080 --> 01:47:42.000
So, there was a big drastic improvement.

s1278
01:47:42.000 --> 01:47:44.239
The throughput actually went up quite a bit.

s1279
01:47:44.239 --> 01:47:47.440
Um so, it's at 1100 tokens per second.

s1280
01:47:47.440 --> 01:47:56.560
Whereas previously, if we look at it over here at 64, we were at 800 tokens per second and then the latency was uh 2,000 milliseconds.

s1281
01:47:56.560 --> 01:48:07.280
But now after the change that we made um it is uh 1100 tokens per second and then the latency is uh 800 milliseconds.

s1282
01:48:07.280 --> 01:48:12.400
So it's gone down the latency has gone down quite drastically and then we have also increased the throughput quite drastically.

s1283
01:48:12.400 --> 01:48:21.280
So the these are the sort of the tuning you can do uh over time to make sure that things are running well uh once you have it up and running and you can do this for your particular use case.

s1284
01:48:21.280 --> 01:48:28.080
Maybe you're not going to have uh six uh you know 64 concurrent requests coming in for a particular use case but maybe you are.

s1285
01:48:28.080 --> 01:48:32.000
So um you have to decide based on your use cases

s1286
01:48:32.000 --> 01:48:37.040
your concurrency one higher than before like was that a trade-off you just did

s1287
01:48:37.040 --> 01:48:40.719
concurrency one I mean it can be a little higher or lower.

s1288
01:48:40.719 --> 01:48:48.159
It's not going to be the same all the time but for example before it was 37 but now it's 35.

s1289
01:48:48.159 --> 01:48:56.960
So it's a little bit of deviation that happens with running models, but it should be plus minus two sec two tokens there.

s1290
01:48:56.960 --> 01:48:59.520
Um we're also running the full precision model now.

s1291
01:48:59.520 --> 01:49:02.719
We're not we're no longer running the FPA because we had to switch the notebook.

s1292
01:49:02.719 --> 01:49:08.400
So that's the other reason why you're seeing it go from like 50 something tokens to 30 something tokens.

s1293
01:49:08.400 --> 01:49:12.320
Uh so that's the performance that we're seeing there.

s1294
01:49:14.000 --> 01:49:15.600
But but yeah, that is all I have.

s1295
01:49:15.600 --> 01:49:17.040
That is one flag you can tweak.

s1296
01:49:17.040 --> 01:49:20.560
There's also all sorts of other flags that I implore everyone to take a look at.

s1297
01:49:20.560 --> 01:49:23.679
The workshop kind of goes through it as well.

s1298
01:49:23.679 --> 01:49:30.000
But I think we're on time and I'll give it back to Dorne.

s1299
01:49:30.000 --> 01:49:30.400
Yeah.

s1300
01:49:30.400 --> 01:49:33.040
Thank you all for attending our session today.

s1301
01:49:33.040 --> 01:49:40.800
If you have not scanned the QR code, the code, the repo, the PowerPoints are available on this site.

s1302
01:49:40.800 --> 01:49:43.760
Feel free to start a free AMA cloud account.

s1303
01:49:43.760 --> 01:49:49.040
My name is Dwan Lightoot and I'm joined by my colleague Omar and we thank you all for joining us today.

s1304
01:49:49.040 --> 01:49:51.760
Feel free to ask us questions after this session.

s1305
01:49:51.760 --> 01:49:54.000
Thank you.

s1306
01:50:07.716 --> 01:50:09.716
[music]