Agents That Own Their Inference — Du'an Lightfoot & Khaja Omer, Akamai Technologies

AI Engineer · 110 min · 1306 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:12I mentioned before my name is Dewan Lightoot.
2. 00:15I'm a developer AC advocate at Akami and this is my colleague.
3. 00:19I'm Kaja Omar.
4. 00:20I go by R. I'm a senior software engineer at Akami.
5. 00:23And we're going to be talking agents that own their own inference, building production AI agents on dedicated GPUs.
6. 00:31Now, why are we here?
7. 00:34Things are kind of changing.
8. 00:36And for me, I started building agents before tool calling.
9. 00:41Back in January 2023 is when I built my first agent.
10. 00:46I would take a prompt, that context was about 8,000 tokens at the time, and I would parse that prompt and then make a tool call with Python and then give the response back to an LLM,
11. 00:59right?
12. 01:00But as I was doing that, I found something interesting.
13. 01:04It can get really expensive, right?
14. 01:07And then I started using local models to figure out, okay, how can I do this?
15. 01:12And then I learned something else.
16. 01:14There's another challenge around that because if I'm using a smaller model, it may not be as capable and then I may be compute bound that memory in that GPU may not handle
17. 01:26the larger context.
18. 01:27So that's the inputs and outputs and then the multiple turns in the tool cost.
19. 01:31How do we get the most either the GPUs, the costs, the latency that we need in production or just for our own use cases, right?
20. 01:40So the reason why we're here, we're going to talk about agents that are often renting their intelligence and then how you can take back that control and own it.
21. 01:51When it comes to an agent, it's just a loop.
22. 01:53You invoke an LLM, it has access to tools, it has access to an environment, makes a decision, a reasoning decision on what actions to take once you invoke it or give it a prompt.
23. 02:04And then those actions are being performed by your tools, your Python code, your other programming languages and then given back to a model.
24. 02:14The model evaluates that the actions, the results and then updates the plan, takes additional actions or the goal is achieved, right?
25. 02:23But all that requires tokens to be generated, right?
26. 02:29And depending on your application can lead to latency challenges.
27. 02:32So we're going to talk about how you can actually man manage all of that on dedicated GPUs.
28. 02:38So what does it mean to own your inference?
29. 02:41First, when it comes to owning owning your inference, you actually own the GPU or rent the GPU.
30. 02:47So when you do that, that means that you have access to the GPU, the hardware for that AI to be ran on, for the models to live on, which means then you have access to control
31. 03:01where it goes.
32. 03:01So the data never leaves your cluster, never leaves your data center, never leaves your laptop, right?
33. 03:08You now own that intelligence and what's happening with the model.
34. 03:12And if you're in the cloud, you can also do something like private VPC endpoints or have it in the VPC so it never leaves your VPC.
35. 03:21Right?
36. 03:22I know we all have different environments, have different ways to set up our infrastructure.
37. 03:27We can talk about all of those.
38. 03:30Now what's the difference like when do you own when do you rent?
39. 03:35Well, first if your usage is high and steady, why is that?
40. 03:42Yes.
41. 03:44Yes.
42. 03:44At the end of the talk, I'll have a link to the slide and everything that we showed code, everything.
43. 03:49You can download it and take it with you.
44. 03:51So, thank you.
45. 03:52Now, when do you own if the usage is high?
46. 03:57Right?
47. 03:57If you're constantly have a workload running, because if you're paying for a GPU, you're renting a GPU.
48. 04:04What does that mean?
49. 04:05that GPU should always be working.
50. 04:08Otherwise, what are you paying for?
51. 04:10So, that's something you got to think about.
52. 04:12Then, when it comes to the data, if you need that data to stay private, if you need to own the data, you care about your intellectual property, or you have some governance and compliance reasons why you can't use a public provider when it comes to AI,
53. 04:25that's another reason you want to own.
54. 04:28And then the hardware, that's something we got to think about your hardware.
55. 04:35If you can afford the capacity and you have the capacity, why not run those models, right, for your use cases because then you have that full control.
56. 04:45Now, when do you rent?
57. 04:47If your traffic is low, bursty, you just need intelligence sporadically, it might not be efficient for you to own some GPUs.
58. 04:56Next, well, we start talking about intelligence.
59. 04:59The more capable models, they're going to cost for those GPUs.
60. 05:04And often times, those providers will provide that intelligence on the GPUs because they have the infrastructure for that.
61. 05:12So, it may make more sense to use those providers to get that intelligence, get those benefits.
62. 05:17And then the next part which is often a challenge is that your teams may not have the knowledge or the expertise to support dedicated or self-hosted AI.
63. 05:29Now today's agenda we're going to start out discussing some of the foundation you need to understand like the infrastructure stack we'll be using units and memory budget.
64. 05:38We'll look at prefield decode and KB cache and what that looks like.
65. 05:42And then from there we'll look at dis dense models verse mixture of experts so you understand what that actually looks like on the GPU and why you would pick one or the other
66. 05:53from there will discuss how to optimize your inference right once you figure out okay we want to do self-hosted how can we optimize this to get the most out of
67. 06:04our GPUs the inference engine that we run and just our entire workload and then finally from there.
68. 06:12We'll take that agent and deploy it into our own dedicated cluster.
69. 06:19With that being said, I'm going to pass it off to you, Omar.
70. 06:21Right.
71. 06:22Thank you.
72. 06:23So, next uh we'll talk about uh the environment that you each of you is going to get for this workshop.
73. 06:30Um we have a Kubernetes cluster with around 200 plus GPU nodes.
74. 06:36Each of you gets a single GPU.
75. 06:38Um and we have set it up in such a way that uh each of you gets sort of like your own namespace and within that namespace you will have like your Jupyter notebook running and then you will have your
76. 06:50VLM inference endpoint running as well that you'll be playing around with.
77. 06:54Um and then we're also going to be deploying uh the agent on there as well.
78. 06:59Uh the cool thing here is that the VLM endpoint that we have running in your namespace, you're going to be optimizing it uh as part of uh um after the first four modules.
79. 07:11And um I think that'll be pretty interesting.
80. 07:13We're going to look at different techniques that we're going to be using there uh to do so.
81. 07:17And you're going to redeploy this VLM endpoint over and over again until you have sort of optimized it to increase the throughput uh reduce the latency and uh make sure like you're meeting all your SLAs's.
82. 07:30And then the other thing I want to talk about is uh GPUs on LKE.
83. 07:34So I work on the Kubernetes side of things at Okami.
84. 07:39Um so this is sort of my wheelhouse.
85. 07:42So if anyone has any Kubernetes related questions, uh feel free to ask anytime.
86. 07:47But I want to talk about what Akami has.
87. 07:49So we have the uh we're going to be using the RTX 4000 ADA GPUs.
88. 07:54Uh we also have other GPUs available.
89. 07:56Uh but these are dedicated GPUs.
90. 07:58EGPU has 200 uh eGPU has 2 GB of uh VRAM with 260 GB per second memory bandwidth.
91. 08:07So the decode will be pretty quick because of that.
92. 08:10Uh we have high performance in Tflops.
93. 08:13Uh large amount of CUDA core.
94. 08:14So the pre-fill will be pretty quick as well because of that.
95. 08:17Uh currently for your workshop we will be using a single GPU but that doesn't mean when you want to deploy this at the end on your own you can have different plans that we offer as well where you can have up to four GPUs in a node.
96. 08:29So that goes up to 80 gigabytes of uh VRAM on a single node.
97. 08:33So you can deploy uh decently sized uh models on there as well.
98. 08:37Um and uh everything is sort of orchestrated by LKE.
99. 08:40So setting up the node, uh deploying the drivers, uh setting it all up for you.
100. 08:45So that way you don't have to do anything manually.
101. 08:48Uh everything is sort of like one click.
102. 08:50You click a button, everything gets deployed and you're off to races.
103. 08:53Um yeah, for more information, we have a link here as well.
104. 08:57Um and at the end, you'll definitely get all these slides.
105. 09:00So you'll be able to get to the pages and then um I'll give it back to Dwan.
106. 09:08All right.
107. 09:08So for the inference engine, we're going to be using V LLM.
108. 09:12Now, some of you may have heard of Lama, which is great for local, but VLM offers some benefits that are better for production.
109. 09:22One of those being being continuous batching out the box, the other being page detention.
110. 09:28Now, it also supports the Open AI compatible API.
111. 09:33So if you're using whatever SDK that supports Open AI, you can use it with VLM.
112. 09:41It also allows you to do some really interesting tweaks when you're setting up inference for your for your endpoints and for your applications.
113. 09:51Now, I mentioned continuous batching.
114. 09:53For those of you that don't know what it is and how it actually works, this happens at the prefill stage, right?
115. 10:00the prompts come in and normally with static batch, let's say you have a school bus full of kids.
116. 10:06Well, in order for you to drop get new kids with static batching, you have to drop off every kid one at a time and then once all the kids are dropped off, you go back and get the rest of the kids and then start the next batch.
117. 10:19But with continuous batching, you can drop off a kid and pick a kid all at the same time.
118. 10:24This is way more efficient when it comes to the KV caching that happens during the preill stage.
119. 10:30now or when it comes to page detention.
120. 10:35Paged detention is when you think about the KB cache and you think about what's actually happening there's memory space that happens.
121. 10:42So the prompt comes in and the longest prompt is going to take up memory space.
122. 10:48When you when page attention that memory space gets allocated even for smaller prompts, right?
123. 10:53But with page attention now that is all managed through c with the VLM that ensures that you can continuously pro process generate and batch those requests as they come through VLM.
124. 11:10It has auler that kind of manages the batching, but it makes this really efficient.
125. 11:15Now, there are other service engines that are out there, and we'll talk about some of those in this workshop.
126. 11:23With that being said, some of the metrics you're going to want to care about is going to be throughput, right?
127. 11:28To understand how your GPU is performing that time, the first token, understand that, okay, you sent a prompt.
128. 11:34How long does it take to get the first token back?
129. 11:37from there that KV cache utilization.
130. 11:40We're going to look at these metrics and show you how to get all of that out of VLM and then the QEP.
131. 11:45So these are some of the metrics as we're going through this workshop you'll you'll see and be able to better understand.
132. 11:51And with that being said, we're going to access the environment.
133. 11:55So if you can scan this QR code, pull out your laptops if you want to follow along.
134. 11:59If not, you can follow us as we go through the workshop.
135. 12:03In this Jupyter notebook, we have nine different labs that we'll cover and all of the resources and dependencies are loaded into this environment.
136. 12:14Now, if you want to look at the code and try this on your own at home, I do have the repo public on accommoders.
137. 12:24I'll have this link along with the presentation and everything that we shared available with one QR code that you can download.
138. 12:31And if you need me to make a short link, I will.
139. 12:36All right, with that being said, we're gonna start with the first module in the workshop.
140. 12:41If everyone is connected.
141. 12:56Yeah, it was kind of slow for me too, which is why I went to my own leak shortener because I kind of made that really fast with Russ.
142. 13:03So yeah,
143. 13:07hopefully like if you do have a hot spot, I would use your hotspot.
144. 13:11Um but we'll try to make the best out of the experience.
145. 13:16Sound good?
146. 13:19Awesome.
147. 13:19All right.
148. 13:20So in the first module module, we're going to talk about the inference stack.
149. 13:23We're going to connect to our VLM endpoint, understand it, see how it generates tokens.
150. 13:29We'll also look at building our first agent.
151. 13:32some things that we need to know, look at some met some metrics and then understand the model that we'll connect to.
152. 13:38The model that we'll use is going to be Quinn 3 4B, but we have several models that sit on a PVC that you can actually load and unload for this endpoint.
153. 13:49Now, when we talk about VL LM, one of the things about it is when you serve an endpoint, it just has one model, right?
154. 13:59So if you want to serve two models, you're going to need another V LMA endpoint, right?
155. 14:05Which means you may need some model routing, a gateway in front of it to kind of handle that.
156. 14:10Now, it does support authentication, but there's no authorization.
157. 14:15So you may need a gateway around that, right?
158. 14:18So agent gateway is often kerve is another one.
159. 14:22Um, any others?
160. 14:24There's LLMD.
161. 14:26There's there's all sorts of open source tooling out there that lets you do that.
162. 14:29Uh but the main but the main one is like Kserve is what we use a lot.
163. 14:33Um and it's it's pretty good and doing the scaling, doing the routing and the authentication as well, right?
164. 14:39Another thing it doesn't have is rate limiting, right?
165. 14:42If you're processing requests, you may want to rate limit.
166. 14:45So again, you want to find a gateway to put in front of it to manage those requests and also handle the routing.
167. 14:56I mentioned that when it comes to the runtime there are other service endpoints that or service runtimes that you can actually use for the server.
168. 15:06SG lang is one, tensor RT is another and llama CPP which we do go over this later with a little bit more in depth so you can kind of understand it.
169. 15:15Now in this environment you are connected to a Jupyter notebook that lives in the name space.
170. 15:22You also have the VLM server endpoint that you'll see here in a moment.
171. 15:28And all of this is in its own pod.
172. 15:30And then you'll have another pod.
173. 15:32Well, another pod that you can launch for the agent.
174. 15:34And we'll see that too.
175. 15:36So, let's go ahead and install our dependencies.
176. 15:40So, we'll install our dependencies.
177. 15:42I've kind of already ran through this, so some of this is already loaded in my runtime.
178. 15:46I figured we don't know how the internet will be, right?
179. 15:50But the good thing is once you send a request, it happens within the cluster.
180. 15:55So once we send our first request to understand how we're configured in VLM, you can notice the model that we have the name of your name space inside of
181. 16:08the cluster.
182. 16:09The coupube config is just for your namespace.
183. 16:13So that way nobody can access anyone else's namespace in the environment.
184. 16:17And then we don't have an API key set, but you can set one with V L. Now from here, we're going to use the open AI API compatible endpoint
185. 16:33to be able or API to connect to the VLM QN 34B 4 billion parameter model.
186. 16:41I believe we're using that at FP8, meaning that it actually sits at um eight billion parameters.
187. 16:48Is it eight billion parameters?
188. 16:50No, four billion parameters.
189. 16:51Four billion parameters.
190. 16:52We're using BF16.
191. 16:53Oh, BF, that's what it is.
192. 16:54Four precision.
193. 16:55Yeah.
194. 16:55So, we're using BF16 full position.
195. 16:57So, you times the bits or bytes or parameters by two.
196. 17:02So, it's 4 billion, which means it's eight gigs on the GPU.
197. 17:10Now once we invoke we got our response for the model and we can see what it was actually served by.
198. 17:16Now this was all within our cluster.
199. 17:18That was pretty fast, right?
200. 17:20It lives within our cluster.
201. 17:22So hopefully with this experience since we're not hitting the public API, we're already connected to this HTTP.
202. 17:28It's going to happen right inside of the cluster.
203. 17:29The response should come back a little faster.
204. 17:32Now we're going to trace send a trace in the end so we can actually understand how this actually looks like when it goes through the prefill the KB cache and the decode phase.
205. 17:45So here we have our stream that we're going to configure.
206. 17:48We're going to look at the chunks and do some calculations.
207. 17:51We'll get that time to first token so we can understand that.
208. 17:54We'll see how all of this is generated here in a moment.
209. 17:58So if we run this, I ran that one again.
210. 18:05We'll come down here and run that.
211. 18:08This may take a little bit to go through, but we should see time.
212. 18:11The first tokens took 37 milliseconds and time output token, the decode phase, which means every token after the after the time of first token, right?
213. 18:23That's going to be that inner in interle latency, right?
214. 18:27that inter latency that happens when those tokens are processed.
215. 18:31So, we're calculating that and we're getting about 40 tokens per second, which is probably going to be our cap.
216. 18:37But when we start talking throughput, that kind of changes because this is just one request.
217. 18:42But imagine if we had hundreds of requests or 16 requests, that throughput is going to be a lot higher, but they're still going to be capped at that 40 tokens per second, right?
218. 18:53Or it could go down, but we can still get more on that throughput.
219. 18:57And I believe Omar will go deeper into that.
220. 19:00So one of the things that I want to showcase is the question is around why don't you just run this on CPU?
221. 19:06Um I didn't want to do CPU on environment because this this um pod for this or this container for the Jupyter notebook is only one gig, right?
222. 19:17So trying to run a local model on this is gonna it's just going to crash.
223. 19:22So I kind of simulated what that would look like.
224. 19:24But if you're trying to run a local model on CPU, it's going to be a lot slower, right?
225. 19:28It can't do the complex multiplication in those matrices like a GPU can.
226. 19:33That's the purpose of a GPU.
227. 19:35It does all those all that math in parallel, which makes it a lot faster, right?
228. 19:40So, we simulate this on on the CPU.
229. 19:46And then from there, we actually look look at this on the GPU to see how that looks.
230. 19:55We got 45 tokens per second.
231. 19:59And one of the things about this and which is interesting to me is that I do a lot of I would say local models and running them.
232. 20:11But 45 tokens or 39 tokens per second isn't that bad for an application if you're running it for a specific use case.
233. 20:19Right?
234. 20:19If you have a specific use case and you don't care about the latency, you don't care about those like if you have a chatbot, you want that kind of that first token to come fast.
235. 20:27But for a standard application where you just kick off a job in the background, well, that 39 tokens per second may not be that bad, right?
236. 20:34So sometimes a lower token per second or a lower time to first token may not be as important depending on the application and the latency that you care about, right?
237. 20:46you may care more about the intelligence in the job getting done than you do actually how fast it actually happens especially like I said if it's a background process.
238. 20:59Now here we're going to build our first agent.
239. 21:01Now for those of you that when we talk about agents there are no tools in this agent.
240. 21:06So it's actually just a request but just the foundation of the agent.
241. 21:10We're giving it a system prompt so it knows it role when we do give it to us later on in the course.
242. 21:15But right now we're just giving it a single prompt and we're giving it a role as system as a solutions architect agent for Akami and you can see that it was able to generate the response.
243. 21:34Now, this response pretty fast.
244. 21:38It works.
245. 21:40But then the question is, if we're trying to do this at scale or we're trying to build an agent that does more than just take it take a prompt and generate a response,
246. 21:53it may need to make multiple tool calls, right?
247. 21:56So, that's when you start to think about things like the KB cache.
248. 22:01You start to think about how much context because when we started talking about that KB cache, that's actually how much context you can actually fit on that GPU, right?
249. 22:11And so with continuous batching, what happens is you send a prompt to the model.
250. 22:17Well, you may have 100 users trying to send prompts at the same time, right?
251. 22:22Well, continuous batch, they're all going to be different sizes.
252. 22:24So, a smaller GPU can actually handle a bunch of requests if you're using some with something like VLM.
253. 22:30Now, if you're using something like Olama, you'll see you'll get throttled a lot faster because it doesn't process those requests the same way.
254. 22:38But VLM actually handles that with theiruler a lot better.
255. 22:46And so here we're going to set up our our our agent and invoke it and we're going to actually going to look at the steps.
256. 22:54Now the code that we're using for this is inside of common and you can actually look at all the code that we actually are importing that is custom code.
257. 23:04So here we have this foundation.py that allows us to do some measures measurements from with Prometheus from VLM right we can look at those metrics.
258. 23:15Now speaking speak speaking of metrics you can use Prometheus and you can use graphana right at home I always use graphana also when I'm building my agents outside of just looking at the metrics
259. 23:26another thing that I always build one of the first things I build into my agents is observability right lanefuse that's something I commonly use how many of you have heard of lanefuse
260. 23:36now there are other tools out there but it's open source you can run that in your cluster as well right beside your agents and then have access that now your observability is private as
261. 23:45Right?
262. 23:45And you can see every step, every trace, every span in your agent with how many tokens, what was the latency, not just from the time the first token, but for every tool call, right?
263. 23:57If you're trying to troubleshoot or you're trying to evaluate now, you can see all of the latency steps in the spans.
264. 24:05But then you can see, okay, I'm using a smaller model and it's making way more tool calls.
265. 24:10It's taking way more time than it should.
266. 24:12Why is that?
267. 24:13Well, that's where evaluations come in and OM will talk more about that.
268. 24:16But it's important to have that visibility into your agents so you can understand when you're using a smaller model how to optimize.
269. 24:24And that could be something as simple as a prompt removing tools or making the agent just a little bit more specialized.
270. 24:33And so we'll run this.
271. 24:41And so we're looking at the steps.
272. 24:43We're going to look at the max tokens and we should see the time to first token.
273. 24:48So you can see as we increase the tokens, what happens to the time to first token in millisecond as we increase the pump, right?
274. 24:56As the prompts get larger, that time the first token takes longer, right?
275. 25:00So the question is what do you optimize on?
276. 25:03Do you optimize on the median or the largest?
277. 25:07Right?
278. 25:07The one that took the most time doing the latency.
279. 25:09Well, you're going to optimize on the one that takes the most time rather than the median, right?
280. 25:14And then that way you can increase your median by doing that.
281. 25:20Now,
282. 25:20[snorts]
283. 25:21here we talk about renting versus owning.
284. 25:23We're going to look at tokens.
285. 25:25How much are being generated?
286. 25:26This is where I I mentioned that if you're using bursty where you're not using invoking a model as much with a provider versus if you have a agent
287. 25:36[snorts]
288. 25:36or some agentic applications that are always working you can kind of compare that cost.
289. 25:45So here we generate a response just to get those tokens.
290. 25:49And now we're going to actually look at the billing and compare which one would actually be more efficient.
291. 25:54So, if you're processing, let's say, 200,000 tokens per day on a hosted bill, rent it, that's going to be about 1,500 per month, right?
292. 26:04But if you're renting a GPU, that's going to be about a,000 per month.
293. 26:08Now, it does matter the size of the GPU, the size of the model, the if it actually performs the way you need.
294. 26:15But these are some of the things you think about when you're trying to compare, should I rent versus own?
295. 26:19You can actually do the math on that to break that down.
296. 26:25And we already talked about these with the data resiliency, you know, the compliance, the security, the controls, and all of those good things when it comes to running locally.
297. 26:35Now, before we go, the runtime landscape, I mentioned that we're using VLL, but you can also use SE lane, which is good if you're using if you have requests that share long prefixes,
298. 26:48multi-turn chats.
299. 26:49So for agents a lot of organizations are using SG lane for that.
300. 26:53When it come to tensor well this is about that peak throughput on Nvidia hardware.
301. 26:57It really specializes in providing you that and then llama CPP for that edge that that really fast processing.
302. 27:04It can help you with those heavily quantized models like on the ARM server.
303. 27:11So we're going to invoke one more time here and we're looking at our client just to verify that we're still good.
304. 27:20We're verifying that we can connect.
305. 27:22We can see the model serve and we can see the ID and I believe I can print the entire so we can see how that require that request actually looks when it comes back.
306. 27:32So I'm going to print this whole thing as well.
307. 27:35So we'll and now you can see and actually let's pretty print it.
308. 27:39So we can see it even better.
309. 27:56There we go.
310. 27:56And so when that request comes back, you can see the ID, you can see the max model length, which the that context length.
311. 28:04You can control that, right?
312. 28:06You can tweak those parameters of course when you're working with a model but we can see everything about the model itself.
313. 28:18Now we're going to move on to the next session.
314. 28:20Um this will have some taking it further when you can explore the cluster explore vm you can feel free to do that in this notebook.
315. 28:31So I'm going to close that out.
316. 28:32We'll discard that and we'll move on to units and memory.
317. 28:39Now the next section and I'm g kind of go faster so Omar can has some time.
318. 28:44Now when it comes to units and memory, we're going to talk about the ceiling when it comes to token per second on using a model.
319. 28:49We're also going to look at the model's real config and understanding how a model actually is sits and how actually looks um when it comes to a GPU.
320. 28:59So here, like Omar mentioned, we are using and I'm going to make that a little smaller.
321. 29:05Let's see if we can get more of that on there.
322. 29:07All right.
323. 29:08So when we talk about the the model actually sitting on the GPU, you have the model weights for the 4 billion parameter model, that model weights at BF16 is going to be 8 gigs.
324. 29:20So we have 8 gigs and the rest of the room is going to be for KV cache.
325. 29:24Meaning how much context can you fit on that GPU?
326. 29:27because it all has to fit within there.
327. 29:29Now, when your prompt comes in, when we talk about prefill, the model is going to read that entire prompt and store it in the KV cache.
328. 29:37And then every time it generates a new token, it's going to update that prefill.
329. 29:41So, that prefill is going to grow, right?
330. 29:43So, just because your your your prompt that goes in is one size, as it goes through and is processed through the model, it's going to increase as it generates,
331. 29:52right?
332. 29:52And then imagine the tool calls, the system prompt, all of these go to the model and all of that has to fit on the KV cache in order for it to be able to handle your request efficiently.
333. 30:03And so we're also looking at um if we're using at FP8, which is going to talk about quantization and what that actually looks like to shrink a model and to still get the efficiencies that you're looking for.
334. 30:17Now we're going to install our dependencies and then from here we're going to connect to verify what we have loaded.
335. 30:31So again we have the Quinn 3 4 billion parameter model our workshop namespace the cluster API key.
336. 30:40Now when it comes to models we have the config.json JSON like what is actually that file when you download a model you get the config.json let's say you go to huggerface you download a model you're going to get a collection of files
337. 30:54all of those files are needed right you're going to get the config.json JSON you're going to get these save.tensors tensors.
338. 31:00Those tensor files are the model weights, right?
339. 31:02But the configuration is what tells VLM how to handle that model, right?
340. 31:07So you can actually look at this and explore those files to understand what they actually look like.
341. 31:12So here we're showing what that configuration file looks like.
342. 31:15So we have 128 layers.
343. 31:18The size is 1,024.
344. 31:21The attention head, there are 16 attention heads.
345. 31:24The values are eight.
346. 31:26Vocabulary size, right?
347. 31:28How does the model train?
348. 31:29What is the vocabulary size of that ma model?
349. 31:31Right?
350. 31:32You can actually look at these configurations.
351. 31:34Can you modify them?
352. 31:38No. Right?
353. 31:39These files cannot be modified.
354. 31:41But you can fine-tune.
355. 31:43Right?
356. 31:43You can use lower.
357. 31:44You can do all these things to adapt the model.
358. 31:47But to modify these files, if you try to do that, it's just going to crash.
359. 31:51It's not going to load.
360. 31:55Now here we're going to look at the bytes per weight and where it comes from and we calculate here we're looking at a smaller model at BF-16 for the
361. 32:086 billion parameter model and the parameters you multiply by two 1.5 gigs and for the 8 billion parameter models what those params look like and it's gonna be eight gigs
362. 32:20and if we compare the BF16 versus FP8 Now, when we talk about pre precision, most models when they're trained, they're going to be at sometimes FP32, but you'll often get it at FP16.
363. 32:33Full precision, meaning that you multiply whatever the model size by what?
364. 32:38Four by four, right?
365. 32:40So, this 4 billion parameter will be 16 gigs at full precision, right?
366. 32:45on your GPU when you quantize it down to let's say FP8 well you kind of lose a little bit capabilities of the model but not much right but that's when you start to evaluate
367. 33:01right if your compute constraint you have this larger model maybe 70 billion parameters that maybe you can do it at FP8 or maybe with int4 right all the way down to where now
368. 33:16that 70 billion parameter models is rather than 32 is 16 gigs, right?
369. 33:22If that's possible, right?
370. 33:23I don't know how much you're going to lose for the position, but you'll have to evaluate that.
371. 33:27And so evaluation hardnesses are important.
372. 33:33So we'll look at the GPU where the weights live.
373. 33:38So here BF16, you multiply by two, FP8, you multiply by one.
374. 33:43We're looking at this for Quinn.
375. 33:44And then we have another model that we're going to use in this course where it's say at 8 billion parameters is five gigs on the GPU which you can see how much you save
376. 33:54with the still intelligence that works for the use cases that we're discussing if you decide to quantize down.
377. 34:08So we're looking at this card the gigabytes per second.
378. 34:10you take the where we're actually calculating.
379. 34:13Okay, we know how the speed of the GPU is 360 and we're showing how to calculate to get the tokens per second and where we're actually bound.
380. 34:23So, we're going to be bound by the speed of the actual GPU on how fast it can go.
381. 34:28And we're showing the math for the KV cache here.
382. 34:30And I'm trying to go a little faster so we can get into the quantization and optimization.
383. 34:36But if you go through these, you can read through it and understand how the KV cache formula works.
384. 34:41Oftent times we may not have to go through this and figure it out, but you will have to when you're trying to calculate, for example, you have a 100 users you want to serve.
385. 34:51You have a smaller GPU.
386. 34:52The question is I got this smaller GPU.
387. 34:55Let's say I'll give you a perfect example.
388. 34:57I had to do a workshop and I had to figure out how to serve 80 students and perform inference for all the students with about three or four GPUs that were about 20 gigs,
389. 35:07right?
390. 35:07Do I need that many?
391. 35:08Is it going to fit?
392. 35:10Well, how big is my context for the workload?
393. 35:12What if everybody is invoking at the same time?
394. 35:14Will the GPUs will the cluster support that?
395. 35:17Will continuous batching work?
396. 35:18So, you set up an evaluation to evaluate the entire workload and to determine, okay, is it going to fit or do you hit that roof line, right?
397. 35:26Do you have start to have spill over the CPU?
398. 35:28These is your application slowdown or the memory weights memory budget versus KV pool.
399. 35:38So we'll look at that and then here's how you can actually get that the math.
400. 35:43So here's our we can get 37,000 tokens on this GPU right on the Sega GPU this 8 gigs for the model weights.
401. 35:54And then the room that you have is going to be 37,000 tokens, right?
402. 35:58And so if we have multiple users sending requests, right?
403. 36:02Well, this will show how many users in their context list.
404. 36:07So if we have four users, they can all send 8,000 tokens, right?
405. 36:11For input output tool calls, this going to add up to 8,000 tokens at the same time on that GPU.
406. 36:20And now we're showing the tax as this increases the different requests and the generation of that request.
407. 36:32And this kind of shows how to do that math when you're trying to understand your compute constraint, your GPU, and then the workloads that you're performing.
408. 36:40So you can take all this code and apply it and be able to take your use cases and do the math on it so you can see if it fits on your GPUs.
409. 36:54So we will go to the prefill and decode.
410. 36:59So going a little bit deeper into the prefill and the decode in the KV cache.
411. 37:05Again, we're using the Quinn3 model, but we're actually going to dive in to see how this looks.
412. 37:09So we have the prefill, which like I mentioned is going to be focused on that time to first token, right?
413. 37:15from the time you press send, how long does the first token, if you're streaming, right?
414. 37:20Because if you're not streaming, you're just going to get the whole response back.
415. 37:23But if you're streaming, how long does it take to get that first token back, right?
416. 37:28And that's going to be that prefill stage.
417. 37:31And then the decode is going to be that token, every token after that, that's all measured.
418. 37:36And if you're looking at the metrics inside of VLM, you send that to Graphana, you can actually view that on the dashboard and you can watch, okay, as I increase my prompts,
419. 37:47you can watch it increase, you can watch it decrease, right?
420. 37:50Right.
421. 37:51And really understand what's actually happening on that GPU when you're sending requests.
422. 37:57So, we'll go through here.
423. 37:58We'll install our dependencies.
424. 38:01Then from there, we'll look at our environment.
425. 38:04Make sure we're connected.
426. 38:06Yes.
427. 38:07And we're still using the Quinn 34B.
428. 38:09And we're going to look at the self attention, meaning how does that KB cache get processed.
429. 38:15So this is the actual formula.
430. 38:16So you can look at that soft max to really understand how that KB cache is being handled within a model.
431. 38:23We're just doing it by hand just so you can see the math on that.
432. 38:28And so when it comes to the KB cache, there's the naive approach and there's the KB cache approach that I would say is handled.
433. 38:38So the naive approach is you take a prompt, the model looks at the whole prompt and then one by one stores that prompt in the KV cache and generates a response.
434. 38:48What KV cache does, it says, okay, I'm gonna take the prompt, I'm gonna learn from it, and I'm gonna store it right in parallel.
435. 38:55And so the KB cache allows you to learn from the prompt to be able to reference the information rather than just doing the fixed sequence of the naive caching approach
436. 39:06when it comes to handling that KB cache.
437. 39:10And so if we generate this,
438. 39:14[snorts]
439. 39:15we can look how it computes the reduction of how much if you use the KB cache approach rather than the naive approach and we're just showing all that by hand in code so you can really break it down and understand.
440. 39:33And now we're going to look okay using K KV caching and we'll measure the time to first token.
441. 39:40And so here we have 57 milliseconds and then the decode was 25 milliseconds.
442. 39:48And so we'll continue on and we can see what VLM is actually exposing.
443. 39:56So we can actually look at those metrics and we're just showing some of the metrics that are actually being exposed and how you would view that in VL.
444. 40:05Now we're going to invoke the prompt and we're going to increase it and you can watch that KB cache fill and you can see this is going to take a little while.
445. 40:17So we'll give it a couple minutes while this process we're increasing it.
446. 40:21So we can see the KB cache usage and we can look at the metrics.
447. 40:27So we have peak usage running and then the KB usage after it's finished.
448. 40:33And this will be the math on how you would process that.
449. 40:36Again the worker that we're using for this code is in the foundation.py.
450. 40:42So that's going to be the helper for this code.
451. 40:47Now we're going to look at the warm the code um against the warm prefix caching.
452. 40:52Now preface caching is an interesting thing.
453. 40:55Um don't confuse it with prompt caching for and that's something I should have mentioned before.
454. 41:00If you're using a provider and you're paying by the token, prompt caching, especially if you're do you have a continuous application that's running within every five minutes, prompt caching is a must, right?
455. 41:11You can you can decrease the cost by like up to 75% depending on who you're using.
456. 41:15So there is some ways you can use a provider and save on cost.
457. 41:18But when we're talking about prefix caching, this is caching that prefix for the KB cache.
458. 41:25And so when we do this, we can see how this actually is handled here.
459. 41:30So the code is going to be 220 milliseconds and it's down to 41 milliseconds.
460. 41:37And this all shows you how to do that in the code.
461. 41:40So you can go through that.
462. 41:41And I'm trying to go a little faster so Omar can get to his part and hopefully we can deploy.
463. 41:53All right.
464. 41:53So we'll go back and we'll go to the fourth section where we talk about dense model verse the mixture of experts on the GPU.
465. 42:02Now when it comes to a dense model, what is a dense model?
466. 42:06Well, a disc model when it sits on a GPU, that whole model is used to generate tokens, right?
467. 42:13So, if it's 30 billion parameters or let's let's just do the model we're using now.
468. 42:18So, the Quinn 3 4B at BF16 8 gigs on the actual GPU.
469. 42:26Well, when that actually generates tokens, when it actually is working, that whole entire model is working, right?
470. 42:33So it's going to be a little slower than if you use a mixture of experts.
471. 42:38Mixture of experts may be a larger model.
472. 42:40Maybe it's 30 gigs, but when it actually generates inside of the model, it takes a collection of experts and only that that section of experts does the generation.
473. 42:51So even if it's 30 gigs, it may be 1.5 gigs just to do the generation.
474. 42:56So it's going to be a lot faster.
475. 42:58So that's one of the benefits.
476. 42:59Even though it's a larger model when it's a mixture of experts, it's actually a can be performed a lot better within an application or a lot faster within an application.
477. 43:12And so here we kind of talk about memory and why decode is memory bound meaning that when we talk about the generation phase after that time to first token when that generation actually happens
478. 43:23that's actually going to be bound by the memory.
479. 43:24So when we looked at the speed of this GPU that's bound, you can't go any faster than that, right?
480. 43:30There's nothing we can do to tweak that to make it go any faster.
481. 43:34But there are some things we can do to help this at scale.
482. 43:39So we're going to look at our roof line and plot that so we can actually see it.
483. 43:44So we'll make sure that we have our model loaded.
484. 43:49So we had a quint 3 for billion parameter, right?
485. 43:53and it's going to be compute bound.
486. 43:55So it's going to hit that roof line.
487. 43:56So we're going to look at the flops.
488. 43:57We're going to look at this.
489. 43:58So the ridge line is 297 flops per bite, right?
490. 44:01And where this is bound by the memory.
491. 44:06Now if we look at the mixture of experts, now we're just going to look at this as code.
492. 44:10So we can actually understand it.
493. 44:14So at this 30 billion parameter models, right?
494. 44:18128 experts and only eight active per token.
495. 44:21That's 48 layers, 248 hidden.
496. 44:25And so now the expert mixer expert, this is the weights.
497. 44:29This is the share weight.
498. 44:30So this is how much it how much is being used when it generates.
499. 44:34That's a lot that would be a lot more efficient than using a smaller model.
500. 44:39It can actually be faster when it actually does the generation.
501. 44:45And so here we're looking at the the dense model, right?
502. 44:50is nine times fewer bytes than a dense model, right?
503. 44:53So it decodes a little bit faster.
504. 44:56So that's kind of breaking down the two when you're actually trying to figure out, okay, do I want to use something like Mistro, right?
505. 45:02Or do I want to use like a Quinn model, we compare the dent sizes.
506. 45:12So here we're going to look at the rate, we'll look at the bandwidth, the GPU serve.
507. 45:16So measured Quinn 4B is about 40 40 tokens per second, right?
508. 45:24The smaller model of the Quinn, the 0.6. Now, this isn't done on the CPU, but we're just showcasing 300 tokens per second, right?
509. 45:33But we're looking at the ceiling.
510. 45:34This is actually looking at the bandwidth and it's doing the math here to get that the max that we can get.
511. 45:39The ceiling that we have on this GPU is going to be 45 tokens per second.
512. 45:46Now, if we enable reasoning on that model and we can look at that.
513. 45:53Now, when it comes to these Quinn models, if you're wanting to do any type of agent and the model needs to reason, meaning that maybe the prompt comes in, you have tools and it needs to make multiple tool calls.
514. 46:06Well, you're going to enable thinking.
515. 46:08So now when it makes a tool call, if it gets the results and it needs to make another tool call or verify the results to make the accurate tool call, you're going to have to enable thinking on these models.
516. 46:18And that's just a parameter that you can adjust in the model, right?
517. 46:22But with thinking off, it just generated a response.
518. 46:26It was 35 tokens.
519. 46:27But with thinking on now, it had to think about it.
520. 46:30It cost about 600 tokens.
521. 46:33Right now with some of the models like provider models you can actually set a token budget for life of thinking.
522. 46:39So that's one of the things you can save on cost with a provider model and maybe some of the open models you can do that too.
523. 46:45I'm not sure.
524. 46:46Yeah, you can.
525. 46:46Yeah, you can also adjust that token budget on the thinking as well.
526. 46:50So more tokens on thinking isn't always good but sometimes it can help the model better troubleshoot and better get to that end result that you're looking for.
527. 47:02And so when we talk about inference, there's a lot of things to think about, but one of the things we want to think about is that memory bound limit for the actual card and then how to apply that for the agent.
528. 47:12So we look at all of the modules, we talk about that KB cache, the prefix caching, the attention heads as well as quantization which um is going to cover here in a minute and then from there that continuous batching we also discussed that.
529. 47:26So if we split the prefill from the decode, you have the prefill GPU pool and the decode which is memory bound.
530. 47:34So the prefill compute bound, right?
531. 47:36So it needs to go from your memory to the GPU and it needs to be cached, right?
532. 47:42There's [clears throat] that that phase you can you can do a lot of optimizations, but when it comes to the decode phase, there's nothing you can really do because you're bounded by the GPU.
533. 47:54And so we kind of show this in code rather than actually showing it because again we
534. 48:00[snorts]
535. 48:00only have so much limitation in this workshop.
536. 48:05Now with that being said I am going to make sure that we are good to go here.
537. 48:19Um I don't think I need to switch the mod.
538. 48:21I think you should be good.
539. 48:22Armor.
540. 48:22I'm gonna turn it over to you.
541. 48:24All right.
542. 48:24And then Yeah, we should be good to go.
543. 48:27You can talk about quantization.
544. 48:28Thank you.
545. 48:30All right.
546. 48:30Thank you, Dwan.
547. 48:31Yep.
548. 48:33All right.
549. 48:34So, let me close that.
550. 48:36Sure.
551. 48:37That That's yours, actually.
552. 48:38There you go.
553. 48:40Okay.
554. 48:40Cool.
555. 48:40So, I'll be talking about a whole different um uh optimization techniques I think you can use to deploy these models on Kubernetes.
556. 48:48Um, like I said, uh, my background is in Kubernetes.
557. 48:53I develop a lot of Kubernetes stuff at Akami and I work on the manage Kubernetes platform there.
558. 48:59So, a lot of the stuff that we do is we want to make sure that we're able to use GPUs on Kubernetes really well.
559. 49:05Uh, use all the different features that they have.
560. 49:07Um, and uh, we deploy a lot of these models using some of these techniques that I'm going to be talking about.
561. 49:13Um, so there's uh, different techniques.
562. 49:16So, we're going to talk about quantization.
563. 49:18I think this one's like the main big one.
564. 49:20Um, it's super important.
565. 49:21We're going to talk about speculative decoding.
566. 49:24Uh, this is a pretty cool technique.
567. 49:26It can give you quite a big uh performance boost in your uh throughput.
568. 49:32Um, we're going to talk about um like how do you uh you know uh tweak your LM uh VLM config to make sure that you're getting max performance out of it and
569. 49:44um you're meeting your SLAs's.
570. 49:46And then we're going to actually do some tuning and evaluating.
571. 49:48So this I'm going to leave you guys with some sort of framework that you can use to figure out what exact configuration you need to use for your specific workloads and specific use cases.
572. 49:58Um so I'll dive into quantization.
573. 50:02Um let's see.
574. 50:04And if we have time, we can definitely go uh we can have like a quick Q&A as well, but I think we have 30 minutes.
575. 50:11I'm going to run through it uh at a good pace.
576. 50:15So in quantization um there's a whole lot of different uh there's a lot of different types of quantizations you can do.
577. 50:22Um you know there's the BF-16 the the one that the model is going to start off with.
578. 50:26That's the full precision model that uh Quint provides you and then there's a lot of other conversations on top of that.
579. 50:32People have created FP8 that's one of the models that we're going to look at in this particular module.
580. 50:38Um there's INT4.
581. 50:40Uh there's AWQ GPQ uh GPTQ.
582. 50:43These are different types of uh ways you can quantize the model.
583. 50:48Um there's NVFP4 which is Nvidia's proprietary quantization technique that they use.
584. 50:54Uh it's hardware accelerated on black hole GPUs and more.
585. 50:57Uh that one is pretty cool.
586. 50:59Um but let's see.
587. 51:02Let's dive into it.
588. 51:03Um make this hopefully we can fit that in there.
589. 51:11All right.
590. 51:11So as you can see, you know, like if you have the BF16 weights, it's going to take a lot of your memory.
591. 51:17Uh in this particular use case, we have 20 gigs of VRAM and we sort of want to maximize how many requests we can serve with these 20 gigs.
592. 51:25Uh so we want to make sure we have enough space for KB cache.
593. 51:28Uh reason being the more requests uh the more KV cache you can have on the memory, the more requests you can process in parallel.
594. 51:36Uh thereby increasing your throughput.
595. 51:38So that's super important.
596. 51:39Uh so that's why we essentially want to do quantization.
597. 51:42We want to reduce the size of the model so that way you can fit more KV cache and essentially increase your throughput.
598. 51:48Um so that's one reason.
599. 51:51The other big reason I think is also because you're shrinking the weights essentially by in half.
600. 51:56You don't have to do that memory movement between the different uh memory hierarchies resulting in essentially faster decode as well.
601. 52:03So we will see you'll go from you know 40 tokens per second which is the base model will probably go up to like 60 tokens per second if not more.
602. 52:12Um so let's look into this.
603. 52:16Let's um install the dependencies and then I'm just going to make sure that the model is still available and everything is running.
604. 52:25So yeah, we do have the model running uh the same four billion parameter model and it's running in the namespace.
605. 52:32Again, as I said, um sort of like a high level overview, there's a whole different sorts of different techniques you can do quantization with.
606. 52:39Um for example, like um uh Llama CBP is a pretty popular one for running a lot of these models at a lower precision.
607. 52:47Uh that uses GGUF format.
608. 52:49Uh that format is pretty cool.
609. 52:50you can have um uh you know five bit, 3bit, two bit, one bit quantizations which is pretty interesting um and pretty cool but the the downside of that is you lose a lot of uh you lose quality
610. 53:04trying to get a lot of performance out of it.
611. 53:06So, there's a trade-off there.
612. 53:08Um, but for the most part, going from BF-16 to FP8 is a very negligible drop in quality and performance.
613. 53:16Um, and it will result in more performance while keeping the quality exactly the same.
614. 53:20Uh, so that's what we want to showcase and that's one of the assumptions we're making.
615. 53:24Obviously, like when you're doing this for your particular workload, depending on what you're using it for, I highly encourage people to have their own evals.
616. 53:31uh me so that way if you're switching from you know BF-16 your eval are saying okay everything is good everything's running well for our workload you switch to FP8 you might have issues where
617. 53:42some things are breaking here sometimes it's hallucinating sometimes it's going in a loop so a lot of those things can happen for very specific use cases and in those scenarios it's very important that you have your own eval set and you're running your own evals
618. 53:54to make sure any changes you make um are uh okay to make and also you don't have any degradation in performance Um again we'll run and make sure that we have the BF16
619. 54:08base and then we're going to measure the performance.
620. 54:10So uh we have a little load test uh utility package that we use to do some uh testing.
621. 54:18Essentially what we're doing if you look at the levels where these are like a at different concurrencies we're making this request.
622. 54:24Each request is going to have about 256 tokens and the output is roughly going to be about 128 tokens.
623. 54:33So this is sort of trying to simulate how many requests you can process at one time and we're sort of going to use this as a baseline to see what the baseline performance is.
624. 54:42Um so for 32 uh we're getting aggregate performance of 714 tokens per second which is pretty good.
625. 54:51uh at single concurrency we're getting 37.5 tokens per second.
626. 54:56Uh so that's important to uh note down.
627. 54:58And also we're getting um uh 61 milliseconds for time to first token.
628. 55:05And then you'll notice the time to first token goes up, right?
629. 55:08Like the jump between 32 and 64 is quite large.
630. 55:13We're going for 277 milliseconds to uh 1300 milliseconds.
631. 55:19That's a quite a big jump.
632. 55:19and you might not necessarily want to make that trade-off, meaning you might want to potentially stop at only 32 uh concurrent requests that you're processing and not go higher than that.
633. 55:29So that is an important thing to notice there.
634. 55:32Um again as I was saying um eval are extremely important.
635. 55:37Uh reason being again you can have performance degradation when you do quantization and you can lose uh some intelligence reasoning can uh not work as it should there there's going to be a lot of issues.
636. 55:51So you want to essentially I'm kind of simulating here like a uh at a very basic level what kind of eval you might want to think about.
637. 55:58Um, you know, you want to think about like if the JSON is being generated correctly, if the JSON is being corrupted when you reduce the quantization, you know, at uh BF16,
638. 56:08you might have good uh responses, but when you go to FPA, maybe potentially your functions function calls are failing.
639. 56:16Why is that?
640. 56:17It's probably because it's not generating the JSON correctly.
641. 56:19So, there's a lot of these um eval cases that are already out there that you can kind of use in the open source.
642. 56:25Hugging face has a lot of evals.
643. 56:27You can go on GitHub, you can find a lot of these, but you want to come up with your own for your own use case.
644. 56:32Um, but then so next thing what we're going to do is we're going to switch from BF16 to an FP8 model.
645. 56:40So we are going to essentially reduce the size of the model in half.
646. 56:43And once we do that, we'll kind of get to see um what the performance differences are and are they worth it and are we losing any quality over it.
647. 56:53Uh so to do this uh you will need to be in the root and then click on manifest and in there you will see vlm.yaml.
648. 57:02So um I'll kind of go over this.
649. 57:06So this is a typical manifest uh configuration file that you uh uh have for Kubernetes deployments.
650. 57:15Um how many of you have used Kubernetes before or have knowledge of Kubernetes?
651. 57:20Okay, quite a few.
652. 57:21Okay, great.
653. 57:21So this will be quick.
654. 57:22I don't have to go over all all the details, but essentially we have um a PBC which is going which is essentially our cache where all of our models are living
655. 57:32and we have a simple deployment with VLM with a lot of different arguments and flags here.
656. 57:39Um so you can see we're using the Quen uh 34 billion parameter model.
657. 57:44Um, and then we're going to switch it to this FP8 version from Red Hat.
658. 57:52Uh, let me one more quick thing I want to quickly mention is you see the dynamic at the end.
659. 58:00Uh, this is pretty important.
660. 58:02This means that not everything in this model is quantized to FP8.
661. 58:06In a lot of cases, in a lot of different quantization techniques, you might want to quantize the model uh only specific parts of the model and not the others to make sure the performance stays the same and you don't have a lot of uh issues there.
662. 58:18So that's one of the reasons why you'll see dynamic, you'll see all kinds of things.
663. 58:21You'll even see quantizations where certain heads of the model are going to be 4bit and then some are going to be 8 bit and some are going to be full precision.
664. 58:29And you know, it's it's sort of uh interesting that people are doing that and finding a lot of performance there.
665. 58:35They're they're finding the performance gain while making sure the quality stays the same.
666. 58:39So once this is uh changed, we're going to save it and then uh we'll just quickly go through and run it and apply it.
667. 58:46So this is just making sure that the diff is good.
668. 58:49Um we're not causing any issues here.
669. 58:53Uh that should be fine.
670. 58:57And then we'll just go ahead and where's apply?
671. 59:01Right here.
672. 59:02We'll just go ahead and apply and that'll roll out.
673. 59:04Um it's gonna take uh a little bit of time.
674. 59:10Let me save that.
675. 59:11But another thing you guys can also do is you can open up a terminal tab as well.
676. 59:15Like if you just click on the plus sign and then you'll see a terminal option and then that will also let you see sort of like what's in your nameace
677. 59:24and what you can do here.
678. 59:26So if you get you see the VLM endpoint is still coming up.
679. 59:29Uh this pod is essentially your Jupyter notebook running within your namespace.
680. 59:34Um let's see this will take a little bit of time to come up.
681. 59:45Yeah.
682. 59:45While this is coming up, does anyone have any questions?
683. 59:48We can probably have one question in
684. 59:53cont.
685. 59:57was pre-quantized already.
686. 59:59Yes, it was pre-quantized.
687. 60:00Yeah, you can quantize your own models.
688. 60:03There is LLM compressor.
689. 60:04Uh it's uh it's provided by VLM, I believe.
690. 60:09LM compressor.
691. 60:14So there there there is tooling out there for you to do this on your own if you would like.
692. 60:19Um, but yeah, the one that we're using currently is already pre-quantized that we downloaded off of Hugging Face.
693. 60:27All right, I think that should be up.
694. 60:30Again, it's taking a little bit of time, mainly because it's taking all the weights out of the VRAMm and loading the new weights in.
695. 60:36So, it's going to take a little bit of time to do that.
696. 60:38Um, anyone else have any question?
697. 60:51I'm sorry.
698. 60:52Look at your your VL your YAML file for your arcs.
699. 60:54Look Oh, looked like they had uh two flags on the model.
700. 60:59Oh, they did.
701. 61:00Oh, that's a good call out.
702. 61:01Thank you for that.
703. 61:04Did not see that.
704. 61:05All right, we're going to go through it again.
705. 61:09Let's rerun that.
706. 61:13Okay, that's probably why it didn't come up as quickly as it should.
707. 61:22Oh, I can't do that.
708. 61:31All right.
709. 61:31Any any other questions while this is going through?
710. 61:34Go ahead.
711. 61:35How do you choose between?
712. 61:38Uh, how do you choose between FPA and intake?
713. 61:41Uh that's a good question.
714. 61:42So it depends right like int 8 is going to be slightly less precision than FP8.
715. 61:49That's essentially the reason.
716. 61:50Uh the the difference is like FP8 is floating point so you get a little bit more precision out of it.
717. 61:55Uh and then int8 is integer.
718. 61:57So you could use either or.
719. 61:59Uh again these are all the other options that you have.
720. 62:02You can choose based on your workload and where your constraints are.
721. 62:06Um but yeah you could totally go into eight as well and it should be fine.
722. 62:13Um, where does it say that?
723. 62:17Good.
724. 62:22Stack.
725. 62:23Oh, yeah.
726. 62:24I mean, uh, oh, that's the other good good call.
727. 62:26Um, it depends on the GPU as well.
728. 62:28Some of the GPUs might not necessarily support it.
729. 62:31GPU support.
730. 62:31Yeah.
731. 62:32So, GPU support is important.
732. 62:33That's another good call out.
733. 62:34So some of the um uh GPU might support it but it might not be accelerated by the GPU meaning you might not get the full performance that you would be expecting out of it.
734. 62:44Um for example the GPUs that we're using we're using RTX 40008 they support FP8 and we're going to get the full uh precision uh full performance out of it and it's able to hardware accelerate all those calculations.
735. 62:59But if we let's just say use FP4, it won't be able to do that and we would actually not see a whole lot of performance gains and we actually see performance go down instead of it going up.
736. 63:08So that's another thing to keep in mind.
737. 63:12All right.
738. 63:14Where is it?
739. 63:19All right.
740. 63:19It has successfully rolled out.
741. 63:21So we'll uh take a quick look.
742. 63:23So now we can see it's actually using the FP8 model now.
743. 63:27And then we'll also kind of quickly go through and do another load test.
744. 63:30It's going to be the exact same.
745. 63:32We're going to do it at uh concurrency 1 uh four 8 and uh 8 16 and 32 I believe over here.
746. 63:40It's going to be exactly the same.
747. 63:42So and then once that is up we can compare and see the performance differences and why you would want to quantize.
748. 63:56Okay.
749. 63:56the performance is in.
750. 63:58And if we compare it before and after, you can see, let me make this a little bit bigger.
751. 64:04Uh, you can see even at concurrency one, you know, we're going for 37.5 tokens per second to 53.4 tokens per second.
752. 64:12So, we're seeing considerable jump in performance here.
753. 64:14Um, and then at peak throughput, we're seeing uh at 64 concurrency, we're going from 800 to,40.
754. 64:22So, that's about, you know, 1.4 four uh 42 times uh which is quite a big performance jump but again keep in mind the time to first token is going up considerably as well
755. 64:33uh compared to 32 and 64 um but then yeah you can and then again you want to run evals um we can quickly run through this and it probably won't have any differences
756. 64:46so yeah again BF16 versus FP8 um there's not a whole lot of uh performance degradation you should spec it's it's going to work out of the box for the most part but uh for specific use cases it might not but for simple eval cases that we had here it's it's working fine
757. 65:05I think that is it for this module where we'll jump into next is speculative decoding which I think is a pretty interesting one um let's see let's jump in here all right so how many of you have heard of
758. 65:19uh speculative decoding before right quite a few people that is good.
759. 65:24Uh for those who don't know what speculative decoding is, let me make this smaller.
760. 65:30Um it's essentially we're using another draft model which is going to be a much smaller model than the main model that we're using and it's going to generate multiple tokens ahead of time and then our main model which is the four billion parameter model is going to look at those generated tokens and it's going to either say
761. 65:47it's it looks good.
762. 65:49That's it's going to verify it and say it's going to look good or it's not going to look good.
763. 65:52Uh so in this like diagram you kind of see like um the draft model you know generated four tokens ahead of time and then the verifier model which is our four billion parameter model
764. 66:02is going to say oh only the first and second token make sense the other two don't so it's going to discard it and it's essentially going to do this each time it's generating new tokens.
765. 66:10So it's going to go in in a loop there.
766. 66:14uh and uh for for this uh workshop we're going to be using the 6 billion parameter model uh from Quen as a drafter model and see if that is going to either help us or not help us in a lot of cases
767. 66:30uh it can also not help you and reduce the performance um and there's multitude of reasons for that and we'll go into that shortly.
768. 66:40How are we doing on time?
769. 66:41Good.
770. 66:44uh just make sure we are good here target model.
771. 66:53Okay, before we get into it, I also want to talk briefly about all of the different ways you can do um speculative decoding.
772. 67:00Um there's a lot of different ways and there's more ways coming out uh every every few months.
773. 67:05Uh but the you know the draft model is one way.
774. 67:09Um, a lot of times, uh, a lot of new, uh, ways of doing it is actually using a diffusion drafter model, which I think is really cool.
775. 67:17Uh, I personally use that for one of my home lab machines.
776. 67:20Um, and, uh, that gives you a huge performance.
777. 67:23The difference between that and this is because it's a diffusion model, it's going to be generating a lot of tokens uh, at once, uh, rather than doing what we do sequentially with LLMs.
778. 67:33Um, and then the verifier model is able to verify X amount of tokens out of that.
779. 67:38Um but but yeah there's a there's a lot of different types of uh methods you can use.
780. 67:44You can use suffix decoding.
781. 67:45Uh there's a lot of times model would come with a built-in drafter model as well.
782. 67:49So those are like the native models that are trained along with the main model.
783. 67:53Uh so you don't have to uh import another model to do this.
784. 67:57Uh there's medusa eagle again like diffusion dlash is a big one that uh has kind of been going viral and it's working pretty well.
785. 68:06So there's a configuration you set for figuring out how long like how much it generates before the verifier comes in and verifies it.
786. 68:14So you can set that you can set it to two tokens, four tokens and five tokens.
787. 68:18And it's important that you set it correctly uh because if there are for example let's just say you set it at four and uh the model verifier comes in it's like okay I only accept two that's 50% acceptance rate and that is pretty bad.
788. 68:34that's actually going to have a negative impact because not only are you running you're running two models so you're spending compute to run two models and then you're only getting 50% acceptance rate and that's essentially going to result into
789. 68:46a decrease like throughput.
790. 68:48Uh so that's not something you want.
791. 68:50Um so that's something you have to like kind of tweak and play around with and see how many tokens do I want to generate ahead of time.
792. 68:56Um so yeah technically you could generate 15 tokens but you probably don't want to do that.
793. 69:00You probably want to do two maybe three maybe four.
794. 69:03It also depends what type of um what's it called?
795. 69:07Um let me go back.
796. 69:09Where is it?
797. 69:10What type of method you're using again?
798. 69:12Like uh I use deflash uh for a lot of my home labbing.
799. 69:16And uh that works pretty well if I set it to eight.
800. 69:19And if I set it more than eight, I actually get a lot uh a lot less acceptance rate and that causes my performance to go down or be very flaky.
801. 69:26Um but all right so we will next um
802. 69:35you got one more question
803. 69:36one more question.
804. 69:37Hey uh sorry uh this is from the last module but uh how does dynamic uh compression work uh in a sentence or two I guess
805. 69:51um so the way it works is if you do you understand um the different heads of the model uh different because there's different layers within an LLM.
806. 70:01So if you look at the model.json JSON, you'll actually see there's a lot of different parts to an LLM and there's a whole pipeline that sort of like pushes the tokens from one thing to another.
807. 70:10And a lot of a lot of times when you see a dynamic thing, it essentially is saying, okay, I'm going to only quantize uh the attention heads to FPA, but I'm going to leave everything else as is.
808. 70:20I see.
809. 70:21So, so that's the reasoning.
810. 70:22And if you remember from previous modules, you you saw that the FPA version is not exactly half in size.
811. 70:29It's not going four gigabytes, it's like 5.1 gigabytes.
812. 70:32That's because not every part of the model has been quantized to FPA.
813. 70:36So that's like sort of the dynamic part of it.
814. 70:38See, is the same math applicable to all the models or it depends model
815. 70:43depends model to model.
816. 70:44Every model has a different architecture.
817. 70:46So architecturally speaking, it depends you know what parts of the model you quantize.
818. 70:51uh there's people who quantize you know some parts of the model to uh in4 but leave other parts to FP8 and then leave some parts to like BF16.
819. 70:59So
820. 70:59thank you.
821. 71:00Yeah.
822. 71:05All right.
823. 71:06We'll run through this again.
824. 71:07I'm just confirming that our baseline is good.
825. 71:09We're using FP8 model here.
826. 71:11Um and then uh we'll do another load test quickly um sort of to get another baseline of um what it would look like before we implement and add speculative decoding
827. 71:24uh to this particular deployment in VLM.
828. 71:31And then while that is running, I'm just going to make sure uh to uh let me actually copy this correctly.
829. 71:45Not that.
830. 71:48Oh, never mind.
831. 71:49You can just so in here um we are setting the speculative config.
832. 71:54This so this is a particular flag you can pass in VLM to set the draft model.
833. 71:59So we're setting it to the uh 0.6 billion parameter model.
834. 72:04And then we're also uh setting how many tokens do we want to generate as draft tokens.
835. 72:09So in this particular scenario we're setting it to four.
836. 72:12Again this is something you have to do a lot of testing on.
837. 72:15Uh you have to try out do I want to set it to four?
838. 72:18Do I want to set it to two?
839. 72:19Do I want to set it higher?
840. 72:20It all depends on the acceptance rate and uh how much performance you're getting out of it.
841. 72:25Um and then yeah once you set this flag and we will redeploy this.
842. 72:29Um we go back in here.
843. 72:32We got our baseline.
844. 72:35And then let me make sure that it's good.
845. 72:40Okay, things look good here.
846. 72:42And then we'll go ahead and um apply and roll this out.
847. 72:51And this is probably going to take another couple minutes.
848. 72:53So, uh does anyone have any any questions?
849. 72:57Go ahead.
850. 73:01Oh, one second.
851. 73:10I I noticed that you set the GPU memory utilization to 7
852. 73:16which leaves a lot of your um essentially leaves 30% of the VRAM
853. 73:23unused.
854. 73:24Is there any particular logic to that choice?
855. 73:27Uh in this particular scenario um there could be a lot of reasons.
856. 73:32Uh sometimes you know you might have multiple deployments on the same GPU and you want to make sure that you have enough RAM for something else VRAM for something else and then you keep enough VRAM for this but in this particular scenario
857. 73:45not really.
858. 73:46We're going to bump it up in a following module actually.
859. 73:49Um it's it's one of those knobs right like you have to keep you have to make sure that you're setting the utilization correctly.
860. 73:55If you set it too low, you're leaving performance on the table.
861. 73:57If you set it too high, um you're going to um kill your node, and that's not good.
862. 74:04Yeah.
863. 74:05So, this
864. 74:14Okay, it's still coming up.
865. 74:16This is taking a little bit more time.
866. 74:22Go ahead.
867. 74:24from
868. 74:29sorry one second
869. 74:31uh the bump that we see from 60 32 to 64 parallelization it's because of the key value cache just blowing out or I mean
870. 74:40yeah yeah essentially the key value cache is blowing out and it sort of hit like its max limit and it can't go above that uh so there's a lot of reasons so there's that part and then there's also
871. 74:50uh the continuous batching and a lot of times times will happen if if you don't if you set the configuration to a level where the continuous batching is not optimal, it can also cause you to see like high tokens per sec.
872. 75:02Uh uh you can see high latency there as well.
873. 75:06Um but yeah, in this in this scenario, we're essentially hitting like sort of like a ceiling on us uh uh for us in terms of how much VRAM we can use there.
874. 75:14That's why we're seeing that.
875. 75:18But in the following modules, we'll see how we can improve that scenario and hopefully like make it better.
876. 75:24Uh, okay.
877. 75:25All right, we're rolled out.
878. 75:26So, we now have speculative decoding uh set up correctly.
879. 75:31So, what we'll do is we'll do a quick load test again.
880. 75:34It's the same exact test that we ran up here where, you know, just keep in mind we're getting 58.2 tokens per second for concurrency one and then 42 milliseconds
881. 75:45of latency.
882. 75:47So this is our sort of baseline before we do speculative decoding.
883. 75:51Um but what I want to showcase here is not a happy path where you will automatically get a crazy improvement in performance.
884. 76:00You'll actually see the performance went down drastically.
885. 76:02You're seeing 16% uh 16 tokens per second whereas the baseline was 58.
886. 76:07Um and then even the time to first token is sort of going uh up a little bit as well.
887. 76:13uh and you know we want to figure out like why is this happening?
888. 76:18So there's a there's a big difference and there's a big uh decrease in performance like even if you look at concurrency 16 we're going from 700 tokens per second to 240 tokens per second.
889. 76:28So that's a considerable decrease in our performance.
890. 76:32So the way to interpret this is there's a lot of different reasons.
891. 76:36Um one is again as I was talking about earlier it's the low acceptance rate.
892. 76:41uh if your acceptance rate is very low and the verifier model is just it's going to keep rejecting those tokens that are generating by draft model, you're wasting a lot of compute to generate those draft tokens and then the verifier is coming in and then saying hey I don't want to take this work like this is bad
893. 76:56work.
894. 76:57Um, so that's one.
895. 76:58Um, it could be because we set the speculative tokens too high, meaning in in our current config, we set it to four.
896. 77:05Maybe we need to set it down.
897. 77:07Uh, and see if that actually helps in improve the acceptance criteria or acceptance rate.
898. 77:12Uh, there's a lot of things with theuler as well with VLM.
899. 77:15Uh, depending on how it's scheduling these requests in, uh, that could be leaving some, uh, performance out on the table.
900. 77:22So, it also depends on compatibility and things like that.
901. 77:25Um and then yeah again same GPU contention you're using the same GPU uh to do this work.
902. 77:31So it could be uh resulting in u the resources not being used correctly.
903. 77:37Um so in a lot of cases usually this works but in our scenario it didn't work.
904. 77:43Uh the main particular reason for that is um this particular drapher model is not trained correctly for this particular model.
905. 77:53So that's the reason we're seeing a a very very low acceptance rate.
906. 77:57Um so when you are doing this for your own models and for your own deployments, you have to make sure that the model, the draft model that you're using is going to be compatible with the model that you want to use it for.
907. 78:07If you don't do that, you'll run into scenario that I'm showcasing here where like your performance is going to tank and um there's not a whole lot you can do about it.
908. 78:16I can tell you from the resource perspective, this GPU has enough headroom to support all of this.
909. 78:21But the reason for us seeing this decrease in performance is because the the models are not trained correctly.
910. 78:28And uh that is something you want to keep in mind and test for as well.
911. 78:31And you want to make sure the speculative tokens uh are set to the right uh number as well.
912. 78:38Um but yeah when you in your own time I would definitely implore everyone to play around with that number meaning change this to from four to two to six to one and see
913. 78:50uh what kind of performance gain you see what kind of performance decrease you see and why is that happening but yeah that that's that is it for speculative decoding it is actually not going to help us in this scenario so I'm just going to go ahead and uh comment that out
914. 79:10and then we'll redeploy this.
915. 79:12So we are back to our baseline.
916. 79:15Hey, you got one question.
917. 79:17One question.
918. 79:18Hey.
919. 79:18Um, so I know there's probably some nuance depending on model architecture, but in general from what you've seen, is there a massive hit to acceptance rate if you use like mixed precision essentially like run the main model at FP8
920. 79:34and then quantize the draft model down to Q2?
921. 79:37If I'm like really memory constrained,
922. 79:40is it often even worth doing that?
923. 79:43Uh, or you know, what have you seen in general?
924. 79:47like does
925. 79:48that's a good question.
926. 79:49Um a lot of times it you it doesn't matter as much.
927. 79:54Um most of the time what you want to do is you want to keep the draft model and the main model at the same precision.
928. 80:00But let's just say the draft model is at BF-16 and then the main model is FP8.
929. 80:05The mismatch can cause the acceptance rate to be very low.
930. 80:10So, and then the the main part of it is you have to train the draft model correctly so that way it's able to generate the right tokens.
931. 80:16In our particular scenario, the draft model was not trained correctly.
932. 80:20Um, and then it was also quantized down to FP8.
933. 80:24So, there was probably some issues with that recipe when they trained this model and which resulted in us actually taking a massive performance set.
934. 80:33Um, so a lot of times when you when uh a lab releases a model, they release uh a drafter model along with it.
935. 80:40So you want to use those drafter models uh that come with the model.
936. 80:43Uh there's also thirdparty labs that do their own drafter model.
937. 80:46So you can try out a lot of different options that are out there and see which one works for you.
938. 80:49And in some cases, depending on the use case, let's just say you're using for coding, the drafter model might work perfectly fine, but if you're switching that to maybe some English writing, very specific academic writing, and it might not necessarily work well, and you're going to see like decrease in acceptance rate.
939. 81:03So, it's also uh it also depends a lot on your use case as well.
940. 81:07So, it's not universally applicable.
941. 81:10Gotcha.
942. 81:10Cool.
943. 81:11Makes sense.
944. 81:11Thanks.
945. 81:12Yeah.
946. 81:17All right.
947. 81:19Let's jump on to the inference side of things.
948. 81:24But before you make another jump, any other questions?
949. 81:29Oh, yeah.
950. 81:30I can put the YAML on there.
951. 81:35Um, so the the fact that we have speculative decoding uh presumably is because um for for a model it's faster to verify rather than decode.
952. 81:48Yeah.
953. 81:48Can you just give a little bit of mental model for like what it's what a model does underneath the hood um to verify such that it's faster than decoding
954. 81:58you know it can't be decoding and then comparing against what the draft model produces.
955. 82:02So like
956. 82:03what is it doing exactly so so that it's faster.
957. 82:06Yeah.
958. 82:06So essentially uh when a typical LLM is you know decoding you have a set of vocabulary it's a huge array of bunch of numbers and it's representing um a confidence score for each of those items right so instead of
959. 82:23um instead of doing that one token at a time when you run with specular decoding you're running it for a lot of you you're running the model for parallel things.
960. 82:32So with continuous batching as well, you're running a lot of different requests in parallel.
961. 82:37Uh so you're doing the same uh you're doing the calculation for everything.
962. 82:41Uh and you're doing it at once for a lot of different requests.
963. 82:44In the same way, you're doing the same thing for speculative decoding.
964. 82:46You're checking the confidence score for this uh the speculated tokens and making sure that oh this is a high confidence, so we're going to go for it.
965. 82:53This is you know 0.95 and this one is like 0.9. And then you you accept that and then the other ones you you know uh you reject those.
966. 83:01That's sort of roughly the mental model is like you are in parallel running uh through the layers and making sure that each of the tokens that the draft model
967. 83:12uh outputs is has a high confidence.
968. 83:17Yeah, hopefully that answers your question.
969. 83:21Oh, one more question.
970. 83:23Do you have any examples that
971. 83:26uh yes I do.
972. 83:28That's a good call out.
973. 83:30Um there's actually I left a few links in here.
974. 83:33Um where did I leave those?
975. 83:36Interpreting the results.
976. 83:37Yeah.
977. 83:37So there's a lot of different uh writeups on speculative decoding in in general.
978. 83:42If you do it right, you do see quite a bit of increase in performance.
979. 83:46Uh VLM has actually done a big report on this and they've seen about 1.5x increase in token generation if you set it up correctly with the right model and with the right drafter and the right settings.
980. 83:57Um so yeah, this this does excel.
981. 83:59It's just a matter of getting it right and tuning it correctly and making sure you have the right um things in place for it.
982. 84:06Um but yeah, like from my personal experience, um for my home lab, I used the Quinn 3.6 35 billion model and along with that I use a Dlash um
983. 84:18um draft model and I get roughly around two to 3x token generation uh performance improvement uh depending on the use case.
984. 84:28So it it does work.
985. 84:30It's just a matter of getting the right models to work together.
986. 84:34How do you choose the model size
987. 84:38for the daffer?
988. 84:41Uh you want to you basically want to figure out uh the smallest model that you can run that doesn't take as much GPU power because you the main model is the model that has all the knowledge.
989. 84:53That's the model that you really care about that has everything that you need.
990. 84:56But you don't want draft model to take a lot of those resources away from the main model.
991. 85:00So you want it to be small but not so small that it's dumb but big enough where like it's going to actually give you
992. 85:07you experimented with that.
993. 85:08Did you see like acceptance rates like 3B model
994. 85:133B is too big.
995. 85:14Uh the reason why is a lot of models like I was talking about qu 3.6 uh 35 billion.
996. 85:21So it has it's a mixture of experts model and each expert is about three billion parameters.
997. 85:25So the experts can be around that uh and you would get good performance out of it.
998. 85:29But for the draft model, you probably want to be less than 1 billion parameter just to make sure that it also depends on the main model you're using I guess right like if you're using if your main model is like 400 billion parameters you could probably get away with using a bigger drafter model to make sure that
999. 85:44you get higher acceptance rate.
1000. 85:46So it's you got to find the right balance essentially.
1001. 85:49Yeah.
1002. 85:55Cool.
1003. 85:56Um, do we have any other questions over there?
1004. 86:00One more before you jump in.
1005. 86:04Yeah, I just had a question.
1006. 86:05How is the draft model trained?
1007. 86:08Like is it distillation or something else?
1008. 86:10It's essentially distillation.
1009. 86:12Uh, so you take the big model and you know you distill all of its capabilities.
1010. 86:17You create a data set and then you train the uh drafter model using that.
1011. 86:21And a lot of times actually what's there's different techniques to do this.
1012. 86:24Uh the other techniques you'll see is the draft model is trained alongside the main model.
1013. 86:30Uh so that way that's another way of doing it and a lot of labs will actually give you a drafter model uh inside of the same package.
1014. 86:36So you don't have to essentially download a second model and run it.
1015. 86:39Um so there's a couple of different ways of doing it.
1016. 86:42Uh but probably the best way is train alongside it and then that way you don't have a lot of issues and acceptance rate sort of stays high.
1017. 86:50Cool.
1018. 86:51Um, we'll jump into module seven.
1019. 86:53So, here I want to talk about some of the different flags that you uh should care about when deploying using VLM and what and how you want to get the saturation to be as high as possible.
1020. 87:03Saturation meaning we want to get the GPU to work as hard as possible as long as possible.
1021. 87:09Uh, and because you want to get your money's worth if you're renting a GPU or if you're buying a GPU, you want to run it uh as much as possible.
1022. 87:16So uh we will stop here.
1023. 87:23So uh one of the concepts I want to talk about uh if you remember previously we were looking at the 34 uh 32 concurrency and 64 concurrency.
1024. 87:32We saw a steep increase in the latency and the reason for that is essentially you the throughput at some point is going to flatten out and but the uh but the latency can keep increasing if you keep increasing the
1025. 87:47uh concurrency.
1026. 87:48So you essentially want to find the sweet spot and uh that is something that VLM sort of lets you kind of tweak and make sure that you find that sweet spot and once you do find it you can set the right settings for it.
1027. 88:02Um but but essentially you want to find the inter not so not the intersection but essentially right before where the throughput gets to like the flat state but the latency is like still acceptable enough.
1028. 88:15It's sort of like the point you want to get to.
1029. 88:19Um we will do a setup and then let's make sure our settings are all good still.
1030. 88:28Serving model is up.
1031. 88:30Um, let me just double check if it's up.
1032. 88:36Should be.
1033. 88:38Okay.
1034. 88:41And then here we're again going to get um another we're going to do another load test.
1035. 88:48So that way we know the data and then we can based on that figure out where that uh where this point is.
1036. 88:54Essentially the we call it the knee um where things are flattening but also the concurren based on concurrency uh the time to first token is acceptable.
1037. 89:04Um once that is Oh, I think that should be fine.
1038. 89:18So you know the throughput is slowly increasing as concurrency is increasing.
1039. 89:23That is acceptable.
1040. 89:24Um and then you also see the latency go up.
1041. 89:27That is also acceptable but it's not going up drastically.
1042. 89:30Uh but we should shortly see the big jumps uh with more um concurrency here.
1043. 89:39But while this is running, any any other questions for anyone?
1044. 89:44Go ahead.
1045. 89:46Oh, over there.
1046. 89:47So my question was again on the previous speculative decoding.
1047. 89:52Uh just wanted to know based on your experience um with like drafter model uh what kind of use cases you have seen it to work really like for example with agent loops
1048. 90:03with tool call or with coding like what kind of like use
1049. 90:07uh agent loops it's really good at tool calling it's really good at u browser use is the one particular use case that I use it for I want to automate a lot of browser
1050. 90:16tasks um because there's still you know you don't necessarily get APIs for everything and so you want to bridge that gap.
1051. 90:22Uh and uh yeah, the drafting model helps you generate more tokens faster.
1052. 90:27So you see your browser agent just essentially run around and do things faster.
1053. 90:31Uh so in a lot of ways it does help.
1054. 90:33Uh I will say for browser use I probably see around 1.5x increase whereas for coding I see like closer to 3x.
1055. 90:40So depending on your use case you may see more or less and and things like that.
1056. 90:45Uh, and a followup on that one.
1057. 90:47Are there any benchmarks like more trust me bro like coding fire ship?
1058. 90:52There there's definitely benchmarks out there.
1059. 90:54I I'm I'm personally a little skeptical of benchmarks whenever I see them.
1060. 90:57I never get the same results.
1061. 90:59I don't know if people agree with that here.
1062. 91:01Um, so, um, yeah, for the for the most part, like you know, you want to do your own testing and see what works for you because it very much depends on your use case because the acceptance rate is going to be different depending on your use case, what you're using it for.
1063. 91:16Thank you.
1064. 91:20Um, yeah.
1065. 91:21Any other questions?
1066. 91:23Mic check.
1067. 91:23Mic check.
1068. 91:24How's it going?
1069. 91:26Um do like big models like codeex or cloud like do they use this type of the coding like have they published that they use these patterns or is this more like for open source if you're training your own stuff or running inference
1070. 91:43I don't know the question to if they do it or not um I would imagine they do sort of like free performance but uh for for the open source yeah this is particularly for the open source all the open source big models that you
1071. 91:56want to use all the Quinn models, the GLM models, Kimmy models, they all have specular decoding uh drafter models available that you want to use.
1072. 92:04Is there like a main drafting model that they use for example for like DeepSseek or GLM?
1073. 92:11Deepseek actually came out with a new uh thing yesterday I believe or the day before.
1074. 92:16It's called DSpark.
1075. 92:17It's like their own technique for doing specular decoding in a more dynamic way.
1076. 92:21Essentially they're figuring out how so here our configuration is static right like we say we're going to generate four tokens at a time uh but with deepseek's way of doing it they are making it dynamic meaning
1077. 92:33you know it on on the fly sort of decides oh I'm going to generate only two tokens or I'm going to generate four tokens or five tokens depending on the acceptance criteria depending on the GPU load and all all sorts of metrics.
1078. 92:44So yeah, every every model uh every lab has like their own way of doing it as well and you probably want to stick to that but you can also experiment.
1079. 92:52For example, the Quinn models uh Quinn lab doesn't release the Dlash models for drafting but these are released by uh a third party lab but they work quite well with them.
1080. 93:03So you sort of have to test it out and see.
1081. 93:05Thank you.
1082. 93:08Yeah, I think this is hanging right.
1083. 93:10Yeah, I would check the pot.
1084. 93:13Yeah, might have killed a cluster here.
1085. 93:19Uh
1086. 93:19oh, I can't type.
1087. 93:21Uh
1088. 93:21oh,
1089. 93:24this was working well.
1090. 93:26Refresh.
1091. 93:28[laughter]
1092. 93:31Let me create the tab again.
1093. 93:35Okay.
1094. 93:36Um well, something is wrong.
1095. 93:40It copied a URL and opened in a new tab.
1096. 93:43See what happens.
1097. 93:45Let me open a new tab.
1098. 93:48I wouldn't close that one.
1099. 93:55You know, in all the testing that has not happened, it happened today.
1100. 94:02Can I at least scroll through and go through different things?
1101. 94:06Okay, at least I can scroll scroll through um you know since uh the cluster decided to take uh a vacation uh I'll sort of go through the content that I was going to cover.
1102. 94:25Um the main point of running this benchmark here was to kind of show where that knee is meaning where you see the max throughput.
1103. 94:32Uh if we go back to this diagram where you see the max throughput while making sure you are seeing the lowest uh tokens uh lowest latency for token generation.
1104. 94:44Um and that that is that essentially limit was at 32 concurrency.
1105. 94:50Um because anything above 32 concurrency you'll see uh the token generation is going to be kind of the same.
1106. 94:57It's not going to change much.
1107. 94:59uh the throughput is going to be the same but but the latency is going to go drastically up and you don't want that.
1108. 95:03Uh so that's sort of like the limit you are hitting.
1109. 95:06Um and that is important information that we need to keep in mind and look at when we're deploying these things.
1110. 95:13Uh and if
1111. 95:16yeah you can go to 100.
1112. 95:18Go to look at the spreadsheet
1113. 95:21100.
1114. 95:22Yeah, go to 100 and then just continue from
1115. 95:25Yeah.
1116. 95:25seven or eight.
1117. 95:28Where is it?
1118. 95:35There you go.
1119. 95:36All right.
1120. 95:36Well, this one's working.
1121. 95:37That's good.
1122. 95:45All right.
1123. 95:46Cool.
1124. 95:46So, we'll jump into eight.
1125. 95:49Um, let me also
1126. 95:54[laughter]
1127. 95:55open up Who's doing it?
1128. 96:05Okay, so for module A, we're going to talk quickly through um what sort of flags and knobs you can tweak to make sure that you can sort of get above that limiting point that we were talking about at 32 concurrency.
1129. 96:17You're sort of seeing that limit and how do we kind of get beyond that and what things you need to keep in mind when thinking about this.
1130. 96:24Uh so for this one I'm sort of going to give you guys um a thinking model is essentially you want to measure and that's why with a lot of different cells I was I keep running load tests and it's super important to run those load tests to see what your performance is like and any changes you're making
1131. 96:40how is that changing the performance is it making better is it making it worse um and in what particular metric it's is it making worse right like if is it if the throughput keeps going up or stays the same but like the latency is changing
1132. 96:53something is either correct or wrong there.
1133. 96:55So, that's something we have to keep in mind.
1134. 96:57Um, and there's a bunch of different knobs you can sort of like uh tweak in VLM to make it better.
1135. 97:03But the main four ones are again the GPU memory utilization.
1136. 97:06You want to make sure that you're setting that correctly depending on your use case.
1137. 97:10Again, if you're running multiple models on the same GPU, you might want to keep it low and that's a trade-off you're making.
1138. 97:16But if you're only running a single model and you want to make sure that it's getting the best performance possible, you want to bump that up to 0.9 or point even 0.95.
1139. 97:25Uh the other thing is you want to make sure that uh the max numbum sequences uh is set to a good limit as well.
1140. 97:33This is something you have to figure out by testing.
1141. 97:35Uh in our particular deployment, if you look at the VLM.yaml, um we start with 32 sequences, right?
1142. 97:44So what that is doing is it's telling VLM that it can only process 32 requests at a time uh and not more than that.
1143. 97:51And if you do more than that, you're going to see some performance uh degradation there.
1144. 97:55Um so there's a lot of different flags, but those are like the key ones that you want to keep in mind.
1145. 98:01You want to make sure that you're setting the model length to the correct length as well.
1146. 98:05Uh and then you know you want to make sure you're the batch tokens are also set to the correct length.
1147. 98:11um you don't want to you want to keep it bigger if you want to do prefill faster if your GPU can handle it.
1148. 98:16You want to do it smaller if your GPU is a on a little slower side and you can't handle it.
1149. 98:21Um but yeah, the idea is you want to change one flag at a time.
1150. 98:25You don't want to change multiple things.
1151. 98:27Um and when you change things, run the uh run the benchmarks and see uh what performance you're seeing.
1152. 98:34See measure it and then make assumptions on what you should do next.
1153. 98:38Um, kind of go through it quickly here.
1154. 98:42We have about what 10 minutes left.
1155. 98:49Uh, oh,
1156. 98:54I think we lost the P PVC.
1157. 98:57I think that's what happened.
1158. 98:58I got
1159. 98:58Oh, it got the different one.
1160. 98:59Yeah, I need to actually change this.
1161. 99:00Rerun this, I think.
1162. 99:03Um, let me actually just change this in here.
1163. 99:11This was supposed to be like sequential, but other Jupyter notebook died.
1164. 99:15So, change that and then that should work.
1165. 99:19Um, again, like I talked about, there's different knobs uh that you want to tweak um and see what happens.
1166. 99:24But we'll get sort of the measure.
1167. 99:27We'll measure the baseline and then we'll change a couple flags and see what happens next.
1168. 99:31Um, does it say work?
1169. 99:36Okay, cool.
1170. 99:37There you go.
1171. 99:38It's working.
1172. 99:40While this load test is running, any other questions anyone has?
1173. 99:46Excuse me.
1174. 99:52I noticed that there's a definite fall-off when we're optimizing the model.
1175. 99:57Like here, the 32 to 64, huge falloff.
1176. 100:00Uh is that when you would decide that maybe we should scale the hardware or is that something that you're confident you can still tune it further?
1177. 100:10Uh essentially is this is this the ceiling?
1178. 100:14Yeah, you figure out the ceiling by doing mathematics.
1179. 100:17I think in some of the previous modules Dwan covered uh like what the max ceiling can look like for tokens per second what the throughput looks like and um
1180. 100:26you know what sort of expectations you should have for the deployment and if you are like seeing those max expectations that's probably the maximum limit of your hardware and then you probably have to get additional hardware or better hardware to do so.
1181. 100:39Uh so yeah there there is sort of a limit you know there's only so much you can tune but you're trying to squeeze as much out of the GPU as possible.
1182. 100:48You want to share the slides before you continue?
1183. 100:54Share the slides real quick.
1184. 101:02The server I can leave it up for a little bit, couple hours.
1185. 101:07I'm not going to leave it up just because you do have access to GPUs.
1186. 101:11Um but what what I am going to give you if you scan this QR code you'll get access to the PowerPoint as well as the repo if you want to spin this up in your own AI cloud account
1187. 101:22or if you want to run on your laptop you can do that.
1188. 101:25So if you scan this QR code it'll take you to a site that has all the resources and if you don't have a cloud account you can sign up and you get $300 in credits
1189. 101:35with a free account.
1190. 101:36So that's everything you need.
1191. 101:38And if you all have questions, both of our contact information is here.
1192. 101:42Omar, I'm Dwan Lightoot.
1193. 101:43Yeah.
1194. 101:44And yeah, with that being said, if you all scan the QR code, I'll let Omar finish it up.
1195. 101:49I just want to make sure you all get this before people started to leave.
1196. 101:52Thank you.
1197. 101:54We'll leave it up there for a couple more minutes gets it.
1198. 102:01Yeah, go ahead.
1199. 102:18personal assistant.
1200. 102:19Um, so
1201. 102:23so you can use llamas.
1202. 102:25It depends on your hardware.
1203. 102:26I think that's like the first thing I would look at.
1204. 102:28Um,
1205. 102:30no problem.
1206. 102:31Thank you.
1207. 102:31Oh, you have both.
1208. 102:33for the sake of the argument, let's just say you have the best GPU.
1209. 102:35Let's just say you have um
1210. 102:38Yeah.
1211. 102:39Yeah.
1212. 102:39You have the latest Blackwill GPU.
1213. 102:41Um you know, I think for testing and doing like figuring out what model you want to use for your use case, I think it's good to use Olama and Llama CPP to quickly spin it up and try it out and see what happens.
1214. 102:53Uh but once you have a model that you know I'm going to use for sure for this use case and this this is working great you probably want to switch to uh VLM sg lang or one of these other model
1215. 103:03uh runtimes.
1216. 103:04Uh reason being they allow you to run at higher concurrencies and give you more performance out of it.
1217. 103:10So if you have a single agent running I think it's fine to use Olama.
1218. 103:13It's not a big deal because you only have one request going in at a time.
1219. 103:17But if you let's just say you're running like 20 agents and you want to run it parallel uh Yeah.
1220. 103:26Yeah, you can set up speculative decoding with VLM.
1221. 103:29I don't think Olma supports speculative decoding as far as I know, but I could be wrong.
1222. 103:34Um, but yeah, it when you want to get the max performance out of your hardware, you definitely want to do that.
1223. 103:44Yeah,
1224. 103:46I actually don't know if there's a diffusion uh LLM.
1225. 103:50Is there an open source diffusion model right now available?
1226. 103:56Yeah, Lana.
1227. 103:57Yeah, I mean I think VLM supports it.
1228. 103:59So I think you you would be able to run it on VLM.
1229. 104:02Uh Olama, if supports it, you can as well.
1230. 104:05Um yeah, that's definitely possible.
1231. 104:10Okay, we'll switch back to the module and kind of go through it.
1232. 104:15Um, again like I was saying, at 32 you're seeing, you know, good performance, but at 64, uh, we're going at, you know, the the throughput is sort of staying the same, if not going up,
1233. 104:25but we're seeing, um, a significant increase in latency.
1234. 104:30And to kind of get around it, um, one of the things I want to talk about is the max num sequence.
1235. 104:35Again, this is sort of, uh, how many requests VLM is able to process at one time.
1236. 104:41It's sort of setting that flag.
1237. 104:43If you want to have a lot of uh requests being processed in parallel uh then you want to sort of increase this amount.
1238. 104:50So currently I believe it's set up at 32.
1239. 104:54So we can bump this up to 128 and then we can run this deploy it and see how that changes the performance.
1240. 105:09Okay, it changed a lot of things here.
1241. 105:17Yeah,
1242. 105:17[snorts]
1243. 105:18once this comes up, uh, we can do another test and, um, and you you'll you'll see the the limitations that we're seeing here at 32 is going to go away and you'll actually see the latency be manageable again
1244. 105:32um, at higher concurrency levels as well.
1245. 105:34So, this is sort of like the loop you want to go in.
1246. 105:36You want to find the right flags for wherever the bottleneck is.
1247. 105:40find the bottleneck, figure out why that is happening, what flag you can tweak uh to make sure that you're kind of overcoming that bottleneck.
1248. 105:47Um it's come up.
1249. 105:52That's going to take a little bit of time.
1250. 105:54Um any questions?
1251. 105:56One more over there.
1252. 106:02Potentially bursty traffic
1253. 106:05um at times.
1254. 106:06Um
1255. 106:07Oh yeah.
1256. 106:07Is there such thing as like bursting to serverless?
1257. 106:10Is that the domain of you know model routing layer on top?
1258. 106:14Like what is the pattern for
1259. 106:17Yeah.
1260. 106:18Um I mean autoscaling is one of the patterns like specifically I'll tell you in LKE uh you can have uh dynamic like autoscaling.
1261. 106:27So it can scale up uh depending on your needs depending on bursty traffic.
1262. 106:31it can scale up to you know maybe use a node that has four GPUs instead of one because maybe that handles it better right so you can automate a lot of those things uh to make sure that you're doing it well and the other part of it is like you can also build like some statistical models to
1263. 106:46make sure that you kind of understand your traffic patterns over a long horizon and then the uh the cluster can also dynamically like sort of predict your burstiness in some ways and then can also scale based on that too
1264. 106:58if you got your browser I can show you where it's at
1265. 107:01browser you open your browser, I'll take it to you.
1266. 107:04Okay, so this one is successfully rolled out.
1267. 107:07Um, and then we'll do a quick load test and once that is run, we can see more.
1268. 107:18All right.
1269. 107:19Uh, any other questions?
1270. 107:21Most folks are gone, but
1271. 107:23yeah, like foot H.
1272. 107:29Yeah.
1273. 107:3032.
1274. 107:31That's where we saw the limit last time.
1275. 107:34Once we get to 64, we should see uh that it should improve.
1276. 107:39Okay.
1277. 107:40So, there was a big drastic improvement.
1278. 107:42The throughput actually went up quite a bit.
1279. 107:44Um so, it's at 1100 tokens per second.
1280. 107:47Whereas previously, if we look at it over here at 64, we were at 800 tokens per second and then the latency was uh 2,000 milliseconds.
1281. 107:56But now after the change that we made um it is uh 1100 tokens per second and then the latency is uh 800 milliseconds.
1282. 108:07So it's gone down the latency has gone down quite drastically and then we have also increased the throughput quite drastically.
1283. 108:12So the these are the sort of the tuning you can do uh over time to make sure that things are running well uh once you have it up and running and you can do this for your particular use case.
1284. 108:21Maybe you're not going to have uh six uh you know 64 concurrent requests coming in for a particular use case but maybe you are.
1285. 108:28So um you have to decide based on your use cases
1286. 108:32your concurrency one higher than before like was that a trade-off you just did
1287. 108:37concurrency one I mean it can be a little higher or lower.
1288. 108:40It's not going to be the same all the time but for example before it was 37 but now it's 35.
1289. 108:48So it's a little bit of deviation that happens with running models, but it should be plus minus two sec two tokens there.
1290. 108:56Um we're also running the full precision model now.
1291. 108:59We're not we're no longer running the FPA because we had to switch the notebook.
1292. 109:02So that's the other reason why you're seeing it go from like 50 something tokens to 30 something tokens.
1293. 109:08Uh so that's the performance that we're seeing there.
1294. 109:14But but yeah, that is all I have.
1295. 109:15That is one flag you can tweak.
1296. 109:17There's also all sorts of other flags that I implore everyone to take a look at.
1297. 109:20The workshop kind of goes through it as well.
1298. 109:23But I think we're on time and I'll give it back to Dorne.
1299. 109:30Yeah.
1300. 109:30Thank you all for attending our session today.
1301. 109:33If you have not scanned the QR code, the code, the repo, the PowerPoints are available on this site.
1302. 109:40Feel free to start a free AMA cloud account.
1303. 109:43My name is Dwan Lightoot and I'm joined by my colleague Omar and we thank you all for joining us today.
1304. 109:49Feel free to ask us questions after this session.
1305. 109:51Thank you.
1306. 110:07[music]