WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=XTpyNrEgJQ4

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/XTpyNrEgJQ4.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.480 --> 00:00:13.599
My name is Sheila.

s2
00:00:13.599 --> 00:00:27.119
I'm a DA at Akami and we're here today to discuss um the topic speculative decoding and how you can um figure out whether it's worth enabling for your workloads.

s3
00:00:27.760 --> 00:00:30.720
Okay, I'm a little anxious.

s4
00:00:36.079 --> 00:00:38.079
Okay.

s5
00:00:39.440 --> 00:00:44.000
So, typically when you ask LLM a question, there are usually two phases that happen under the hood.

s6
00:00:44.000 --> 00:00:52.640
The first is the model would read your prompt, process all the input tokens and then build what is known as a KV cache um which is essentially the model's working

s7
00:00:52.640 --> 00:00:55.840
um memory um for the rest of the request.

s8
00:00:55.840 --> 00:00:57.760
And this happens only once.

s9
00:00:57.760 --> 00:01:03.359
Um and then you have the decoding phase which is uh when the model starts to actually write the answer.

s10
00:01:03.359 --> 00:01:09.760
Um so because every token depends on the previous tokens um this process usually happens one at a time.

s11
00:01:09.760 --> 00:01:16.000
Um and so imagine you're deploying an application that uses a big model like llama 70 billion model.

s12
00:01:16.000 --> 00:01:29.360
Um typically like this would require like even up to hundreds like forward passes on the model and this is a very expensive process right um and it can cause a lot of latency for your user applications

s13
00:01:29.360 --> 00:01:38.159
which um brings the question of like what if you didn't have to use your bigger model to handle the token generation.

s14
00:01:38.159 --> 00:01:46.159
What if you use a smaller model to um do some of that guesswork to speculate on a couple tokens?

s15
00:01:46.159 --> 00:01:54.320
Um which is still an auto reggressive process, but the whole point is because it's smaller, there's it's a faster process, so it would be a cheaper process.

s16
00:01:54.320 --> 00:02:04.079
Um the output is still the same because um the target model, the main model that has the accuracy still has to verify the tokens and it does this in one forward pass.

s17
00:02:04.079 --> 00:02:07.600
Um so that's the idea behind speculative decoding.

s18
00:02:07.600 --> 00:02:11.200
Um I'll just leave this here for a second.

s19
00:02:11.200 --> 00:02:19.520
So as I said you have a small model um which generates token typically you can set it between three to five for each cycle.

s20
00:02:19.520 --> 00:02:26.319
The target model would handle the predictions um like be able to scale like basically approve or reject it.

s21
00:02:26.319 --> 00:02:35.760
Um and then for all the for the tokens that are rejected the target model would um recomputee the the token for that.

s22
00:02:35.760 --> 00:02:37.680
Okay.

s23
00:02:37.680 --> 00:02:39.920
So, um this process it's great.

s24
00:02:39.920 --> 00:02:44.800
It can save you a lot in terms of inference generation, but it's not free.

s25
00:02:44.800 --> 00:02:50.879
Um because you're hosting now two models, you have to account for the memory required to host that second model.

s26
00:02:50.879 --> 00:02:59.120
Not that much since it's a smaller model, but you also have to allocate extra space for um the KV cache for both models.

s27
00:02:59.120 --> 00:03:17.680
Um yeah so typically you would consider running speculative decoding when you have um extra space in GPU um for example if your workload is high concurrency your your GPUs are probably already busy trying to run all the requests so it wouldn't make sense to implement speculative decoding

s28
00:03:17.680 --> 00:03:29.440
but if you have extra GPU space it's worth considering but uh we'll talk about how to evaluate whether it's worth um and the first question is um choosing the draft model.

s29
00:03:29.440 --> 00:03:42.400
Um yeah, so typically when you're deciding whether it's worth enabling, um can the draft model predict as accurate as the target model?

s30
00:03:42.400 --> 00:03:46.159
And then the other question is whether um it's as fast.

s31
00:03:46.159 --> 00:03:54.560
So you would typically choose a model that's definitely like a fraction of the cost like um yeah, so these are the two main um decisions that you have to balance, right?

s32
00:03:54.560 --> 00:03:59.920
Um, I'm just going to leave this here.

s33
00:03:59.920 --> 00:04:04.159
Um, so for my situation, um, I will explain why I chose these two models.

s34
00:04:04.159 --> 00:04:10.080
For my demo setup, when I was running this, I have access to one GPU.

s35
00:04:10.080 --> 00:04:17.280
I was running on a Blackwell GPU and I wanted to run both the baseline and the speculative decoding setup in one.

s36
00:04:17.280 --> 00:04:23.440
Um so to be able to do this I had to choose a model that was small enough to be able to fit because I have to split the GPU

s37
00:04:23.440 --> 00:04:25.600
utilization between the both servers.

s38
00:04:25.600 --> 00:04:39.840
Um so this is what that looked like for instance you can the baseline alone um takes about 16 gigabytes of weight and then the draft model is 2.5. So that leaves a huge

s39
00:04:39.840 --> 00:04:43.600
amount and space for KB caching.

s40
00:04:43.600 --> 00:04:56.240
Um, and then when you're thinking about choosing the model, some of the things that you would usually consider is um, first of all, like I said, it has to be smaller, typically 10 to 50 times smaller than your target model.

s41
00:04:56.240 --> 00:04:59.120
Um, it has to have the same tokenizer.

s42
00:04:59.120 --> 00:05:07.600
Um, ideally from the same model family, otherwise you know, you would have to like manually translate between the different tokenizers.

s43
00:05:07.600 --> 00:05:10.560
Um, what's the other?

s44
00:05:12.240 --> 00:05:14.560
Yeah, those are the main things.

s45
00:05:14.560 --> 00:05:15.919
I think I'm forgetting one.

s46
00:05:15.919 --> 00:05:20.080
Um, oh yeah, and like I said, the cost and accuracy.

s47
00:05:20.080 --> 00:05:24.639
Uh, okay.

s48
00:05:24.639 --> 00:05:31.759
So, oops.

s49
00:05:36.160 --> 00:05:39.840
So when I was running this, there's three questions.

s50
00:05:39.840 --> 00:05:45.520
Um, speculative decoding, there's some work loads that would benefit from it versus others that wouldn't.

s51
00:05:45.520 --> 00:05:53.520
Um, typically because the goal here is to get a smaller model that can predict almost as accurately as the target model would.

s52
00:05:53.520 --> 00:06:02.720
It helps with some use cases when you have a workload that is more like highly structured like coding use cases.

s53
00:06:02.720 --> 00:06:05.680
Um, what's another one?

s54
00:06:05.680 --> 00:06:07.840
Like writing JSON or SQL prompts.

s55
00:06:07.840 --> 00:06:17.199
Um, it would probably like benefit more versus if you were using it for a highly creative use case like writing poetry or brainstorming or anything like that.

s56
00:06:17.199 --> 00:06:17.759
Yeah.

s57
00:06:17.759 --> 00:06:27.440
So, I think I have this is a quick application.

s58
00:06:27.440 --> 00:06:36.080
I'm not even sure if it's going to work, but um so here I do have for the first demo, I just have um two tabs here.

s59
00:06:36.080 --> 00:06:44.960
Um like I said, this is just to focus on looking at how the acceptance rate performs depending on the type of task that you have.

s60
00:06:44.960 --> 00:06:47.919
Um so the first tab here is a structured output.

s61
00:06:47.919 --> 00:06:52.880
Um when I click this, it will run both at the same time.

s62
00:06:53.919 --> 00:06:55.919
Whoa.

s63
00:06:57.840 --> 00:07:06.560
Oh god, I'm just waiting.

s64
00:07:24.000 --> 00:07:25.520
Technical difficulties.

s65
00:07:25.520 --> 00:07:27.520
I'm not sure why my demo is not running.

s66
00:07:27.520 --> 00:07:31.199
Um, let's see.

s67
00:07:34.560 --> 00:07:35.919
Oh, do I have to reconnect?

s68
00:07:35.919 --> 00:07:37.039
Um, just give me one second.

s69
00:07:37.039 --> 00:08:06.720
I think I might have to restart the Looks like I need to restart the VLM servers and my setup.

s70
00:08:06.720 --> 00:08:07.759
Give me just a few minutes.

s71
00:08:07.759 --> 00:08:10.080
Is this on?

s72
00:08:15.440 --> 00:08:54.160
LSU I Oh.

s73
00:08:54.160 --> 00:09:03.279
Um, should I Yeah, I think I need Oh, it's showing up.

s74
00:09:03.279 --> 00:09:04.720
Okay, there we go.

s75
00:09:04.720 --> 00:09:08.000
Um, so yeah, I was saying this is a demo.

s76
00:09:08.000 --> 00:09:12.240
So when I run this, I'll redo it.

s77
00:09:17.120 --> 00:09:28.080
So you can see um as expected with a speculative on the right side um it was able to generate much much faster um 1.6 times faster.

s78
00:09:28.080 --> 00:09:42.320
Um the key thing to note here is how high the acceptance rate is which is the rate of the tokens that were accepted by the the number of tokens that were accepted over the total generated by the draft model.

s79
00:09:42.320 --> 00:09:54.800
Um another metric here um you can see with the speculative enabled um you we are able to generate a lot more tokens per second um which improves the throughput.

s80
00:09:54.800 --> 00:10:00.240
And then if I run it for the other case.

s81
00:10:09.440 --> 00:10:13.760
So here it's another use case.

s82
00:10:14.240 --> 00:10:16.320
Um as expected same thing.

s83
00:10:16.320 --> 00:10:22.720
Um but in this case I guess the main metric to see here is the low acceptance rate.

s84
00:10:22.720 --> 00:10:31.839
And like I said, this is because this is a more um there's more variety in how the next tokens that would be generated because first of all, the temperature is set high.

s85
00:10:31.839 --> 00:10:33.279
It's a creative use case.

s86
00:10:33.279 --> 00:10:34.720
Um yeah.

s87
00:10:34.720 --> 00:10:36.160
Um what else?

s88
00:10:36.160 --> 00:10:43.920
I think that's pretty much for this um case.

s89
00:10:43.920 --> 00:11:00.480
And then the other situation where oops oh yeah this is what I was I should have put this up here um just to explain when it would benefit and then the other use case would be context length.

s90
00:11:00.480 --> 00:11:05.760
Um so as I mentioned earlier there's two parts of generating tokens.

s91
00:11:05.760 --> 00:11:12.560
Um with speculative decoding it's supposed to accelerate the generating part not the part where the model reads the prompt.

s92
00:11:12.560 --> 00:11:23.120
So typically if you have a use case where you need to provide the model with a lot of context like for rag applications or um analyzing lots of documents

s93
00:11:23.120 --> 00:11:40.399
um with those situations um the model will spend a lot more time with the portion of like building the KV caching um especially if you're using it to do like a quick where the the generated tokens are much less than the context like the input

s94
00:11:40.399 --> 00:11:42.320
um Yeah.

s95
00:11:42.320 --> 00:11:50.560
So ba I think the main point here is just how to think through like what is your application doing?

s96
00:11:50.560 --> 00:11:51.920
Um is it highly structured?

s97
00:11:51.920 --> 00:11:54.399
Do you have enough VRAM space to even consider it?

s98
00:11:54.399 --> 00:11:56.560
Are you doing more creative scenarios?

s99
00:11:56.560 --> 00:12:00.160
Are you running a lot of concurrency um or not?

s100
00:12:00.160 --> 00:12:03.040
Like smaller batch sizes would benefit from this.

s101
00:12:03.040 --> 00:12:09.040
Um yeah.

s102
00:12:11.360 --> 00:12:12.959
Yeah, I'm gonna leave this slide here.

s103
00:12:12.959 --> 00:12:17.279
This just basically summarizes uh what I just went through.

s104
00:12:17.279 --> 00:12:22.560
And if anyone has questions, I'll take one or two questions now.

s105
00:12:27.600 --> 00:12:30.560
Oh, I can't hear you.

s106
00:12:38.000 --> 00:12:38.639
Oh, yeah.

s107
00:12:38.639 --> 00:12:45.680
Um the question was are there particular tools that I recommend for speculative decoding?

s108
00:12:45.680 --> 00:12:50.160
So I implemented this with VLM um the serving engine.

s109
00:12:50.160 --> 00:12:54.240
Um they have really good documentation on how to get started with it.

s110
00:12:54.240 --> 00:12:57.920
This is the simpler one of the simple versions of speculative decoding.

s111
00:12:57.920 --> 00:13:05.600
There's a couple others like the engram medusa eagle that's more like the architecture beneath is like more advanced.

s112
00:13:05.600 --> 00:13:16.880
Um, but for getting started, yeah, I would say you can't go wrong with just reading up I when I was doing this, there's there's not much content out there on this, but I did find a few like blogs.

s113
00:13:16.880 --> 00:13:25.839
Uh, when you re when you type in circulative decoding, um, one of the first blogs that will come up by general compute talks about more in detail.

s114
00:13:25.839 --> 00:13:31.839
Um, there's also like two research papers that were published um, by Google and what's the other company?

s115
00:13:31.839 --> 00:13:34.800
Yeah, it but that's more like goes like more in depth.

s116
00:13:34.800 --> 00:13:35.040
Yeah.

s117
00:13:35.040 --> 00:13:43.519
Um, but I'm also trying to maybe like make more content around this, like maybe like videos.

s118
00:13:43.519 --> 00:13:44.800
It was kind of hard to go through this.

s119
00:13:44.800 --> 00:13:52.000
I I'm a little all over the place, but I would definitely love to maybe like publish a blog or make snippet videos.

s120
00:13:52.000 --> 00:13:57.519
Um, yeah, on like YouTube or somewhere, but you can definitely find the resources.

s121
00:14:03.839 --> 00:14:04.399
Okay.

s122
00:14:04.399 --> 00:14:04.880
Right.

s123
00:14:04.880 --> 00:14:06.240
I should have covered that.

s124
00:14:06.240 --> 00:14:11.519
Um, but I don't have the I should have put a QR.

s125
00:14:13.519 --> 00:14:16.639
Oh, um, as I said, I I'm from Akamay.

s126
00:14:16.639 --> 00:14:24.399
And if you're interested in like learning how to replicate this, you can find it on our website, our GitHub, GitHub.

s127
00:14:24.399 --> 00:14:27.600
Um, Akami developers.

s128
00:14:29.680 --> 00:14:30.880
Yeah.

s129
00:14:30.880 --> 00:14:32.079
Um, yeah.

s130
00:14:32.079 --> 00:14:35.440
So we have a a lot of like different resources on here.

s131
00:14:35.440 --> 00:14:38.959
Um I'm focusing on in inference optimizations.

s132
00:14:38.959 --> 00:14:47.839
Uh we have other people that are focusing on building AI agents um and how to leverage our like manage Kubernetes service, how to use Akami functions.

s133
00:14:47.839 --> 00:14:55.199
So if you have time, please go check out our GitHub pages and also we have a Discord channel that we're trying to build up now.

s134
00:14:55.199 --> 00:14:59.199
Um so it would be great to chat with some of you on the side.

s135
00:14:59.199 --> 00:15:01.360
Thank you.

s136
00:15:04.565 --> 00:15:06.565
[music]

s137
00:15:12.425 --> 00:15:14.425
[music]