Is Speculative Decoding Worth It? Profiling vLLM on NVIDIA Blackwell — Akamai

AI Engineer · 15 min · 137 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:12My name is Sheila.
  2. 00:13I'm a DA at Akami and we're here today to discuss um the topic speculative decoding and how you can um figure out whether it's worth enabling for your workloads.
  3. 00:27Okay, I'm a little anxious.
  4. 00:36Okay.
  5. 00:39So, typically when you ask LLM a question, there are usually two phases that happen under the hood.
  6. 00:44The first is the model would read your prompt, process all the input tokens and then build what is known as a KV cache um which is essentially the model's working
  7. 00:52um memory um for the rest of the request.
  8. 00:55And this happens only once.
  9. 00:57Um and then you have the decoding phase which is uh when the model starts to actually write the answer.
  10. 01:03Um so because every token depends on the previous tokens um this process usually happens one at a time.
  11. 01:09Um and so imagine you're deploying an application that uses a big model like llama 70 billion model.
  12. 01:16Um typically like this would require like even up to hundreds like forward passes on the model and this is a very expensive process right um and it can cause a lot of latency for your user applications
  13. 01:29which um brings the question of like what if you didn't have to use your bigger model to handle the token generation.
  14. 01:38What if you use a smaller model to um do some of that guesswork to speculate on a couple tokens?
  15. 01:46Um which is still an auto reggressive process, but the whole point is because it's smaller, there's it's a faster process, so it would be a cheaper process.
  16. 01:54Um the output is still the same because um the target model, the main model that has the accuracy still has to verify the tokens and it does this in one forward pass.
  17. 02:04Um so that's the idea behind speculative decoding.
  18. 02:07Um I'll just leave this here for a second.
  19. 02:11So as I said you have a small model um which generates token typically you can set it between three to five for each cycle.
  20. 02:19The target model would handle the predictions um like be able to scale like basically approve or reject it.
  21. 02:26Um and then for all the for the tokens that are rejected the target model would um recomputee the the token for that.
  22. 02:35Okay.
  23. 02:37So, um this process it's great.
  24. 02:39It can save you a lot in terms of inference generation, but it's not free.
  25. 02:44Um because you're hosting now two models, you have to account for the memory required to host that second model.
  26. 02:50Not that much since it's a smaller model, but you also have to allocate extra space for um the KV cache for both models.
  27. 02:59Um yeah so typically you would consider running speculative decoding when you have um extra space in GPU um for example if your workload is high concurrency your your GPUs are probably already busy trying to run all the requests so it wouldn't make sense to implement speculative decoding
  28. 03:17but if you have extra GPU space it's worth considering but uh we'll talk about how to evaluate whether it's worth um and the first question is um choosing the draft model.
  29. 03:29Um yeah, so typically when you're deciding whether it's worth enabling, um can the draft model predict as accurate as the target model?
  30. 03:42And then the other question is whether um it's as fast.
  31. 03:46So you would typically choose a model that's definitely like a fraction of the cost like um yeah, so these are the two main um decisions that you have to balance, right?
  32. 03:54Um, I'm just going to leave this here.
  33. 03:59Um, so for my situation, um, I will explain why I chose these two models.
  34. 04:04For my demo setup, when I was running this, I have access to one GPU.
  35. 04:10I was running on a Blackwell GPU and I wanted to run both the baseline and the speculative decoding setup in one.
  36. 04:17Um so to be able to do this I had to choose a model that was small enough to be able to fit because I have to split the GPU
  37. 04:23utilization between the both servers.
  38. 04:25Um so this is what that looked like for instance you can the baseline alone um takes about 16 gigabytes of weight and then the draft model is 2.5. So that leaves a huge
  39. 04:39amount and space for KB caching.
  40. 04:43Um, and then when you're thinking about choosing the model, some of the things that you would usually consider is um, first of all, like I said, it has to be smaller, typically 10 to 50 times smaller than your target model.
  41. 04:56Um, it has to have the same tokenizer.
  42. 04:59Um, ideally from the same model family, otherwise you know, you would have to like manually translate between the different tokenizers.
  43. 05:07Um, what's the other?
  44. 05:12Yeah, those are the main things.
  45. 05:14I think I'm forgetting one.
  46. 05:15Um, oh yeah, and like I said, the cost and accuracy.
  47. 05:20Uh, okay.
  48. 05:24So, oops.
  49. 05:36So when I was running this, there's three questions.
  50. 05:39Um, speculative decoding, there's some work loads that would benefit from it versus others that wouldn't.
  51. 05:45Um, typically because the goal here is to get a smaller model that can predict almost as accurately as the target model would.
  52. 05:53It helps with some use cases when you have a workload that is more like highly structured like coding use cases.
  53. 06:02Um, what's another one?
  54. 06:05Like writing JSON or SQL prompts.
  55. 06:07Um, it would probably like benefit more versus if you were using it for a highly creative use case like writing poetry or brainstorming or anything like that.
  56. 06:17Yeah.
  57. 06:17So, I think I have this is a quick application.
  58. 06:27I'm not even sure if it's going to work, but um so here I do have for the first demo, I just have um two tabs here.
  59. 06:36Um like I said, this is just to focus on looking at how the acceptance rate performs depending on the type of task that you have.
  60. 06:44Um so the first tab here is a structured output.
  61. 06:47Um when I click this, it will run both at the same time.
  62. 06:53Whoa.
  63. 06:57Oh god, I'm just waiting.
  64. 07:24Technical difficulties.
  65. 07:25I'm not sure why my demo is not running.
  66. 07:27Um, let's see.
  67. 07:34Oh, do I have to reconnect?
  68. 07:35Um, just give me one second.
  69. 07:37I think I might have to restart the Looks like I need to restart the VLM servers and my setup.
  70. 08:06Give me just a few minutes.
  71. 08:07Is this on?
  72. 08:15LSU I Oh.
  73. 08:54Um, should I Yeah, I think I need Oh, it's showing up.
  74. 09:03Okay, there we go.
  75. 09:04Um, so yeah, I was saying this is a demo.
  76. 09:08So when I run this, I'll redo it.
  77. 09:17So you can see um as expected with a speculative on the right side um it was able to generate much much faster um 1.6 times faster.
  78. 09:28Um the key thing to note here is how high the acceptance rate is which is the rate of the tokens that were accepted by the the number of tokens that were accepted over the total generated by the draft model.
  79. 09:42Um another metric here um you can see with the speculative enabled um you we are able to generate a lot more tokens per second um which improves the throughput.
  80. 09:54And then if I run it for the other case.
  81. 10:09So here it's another use case.
  82. 10:14Um as expected same thing.
  83. 10:16Um but in this case I guess the main metric to see here is the low acceptance rate.
  84. 10:22And like I said, this is because this is a more um there's more variety in how the next tokens that would be generated because first of all, the temperature is set high.
  85. 10:31It's a creative use case.
  86. 10:33Um yeah.
  87. 10:34Um what else?
  88. 10:36I think that's pretty much for this um case.
  89. 10:43And then the other situation where oops oh yeah this is what I was I should have put this up here um just to explain when it would benefit and then the other use case would be context length.
  90. 11:00Um so as I mentioned earlier there's two parts of generating tokens.
  91. 11:05Um with speculative decoding it's supposed to accelerate the generating part not the part where the model reads the prompt.
  92. 11:12So typically if you have a use case where you need to provide the model with a lot of context like for rag applications or um analyzing lots of documents
  93. 11:23um with those situations um the model will spend a lot more time with the portion of like building the KV caching um especially if you're using it to do like a quick where the the generated tokens are much less than the context like the input
  94. 11:40um Yeah.
  95. 11:42So ba I think the main point here is just how to think through like what is your application doing?
  96. 11:50Um is it highly structured?
  97. 11:51Do you have enough VRAM space to even consider it?
  98. 11:54Are you doing more creative scenarios?
  99. 11:56Are you running a lot of concurrency um or not?
  100. 12:00Like smaller batch sizes would benefit from this.
  101. 12:03Um yeah.
  102. 12:11Yeah, I'm gonna leave this slide here.
  103. 12:12This just basically summarizes uh what I just went through.
  104. 12:17And if anyone has questions, I'll take one or two questions now.
  105. 12:27Oh, I can't hear you.
  106. 12:38Oh, yeah.
  107. 12:38Um the question was are there particular tools that I recommend for speculative decoding?
  108. 12:45So I implemented this with VLM um the serving engine.
  109. 12:50Um they have really good documentation on how to get started with it.
  110. 12:54This is the simpler one of the simple versions of speculative decoding.
  111. 12:57There's a couple others like the engram medusa eagle that's more like the architecture beneath is like more advanced.
  112. 13:05Um, but for getting started, yeah, I would say you can't go wrong with just reading up I when I was doing this, there's there's not much content out there on this, but I did find a few like blogs.
  113. 13:16Uh, when you re when you type in circulative decoding, um, one of the first blogs that will come up by general compute talks about more in detail.
  114. 13:25Um, there's also like two research papers that were published um, by Google and what's the other company?
  115. 13:31Yeah, it but that's more like goes like more in depth.
  116. 13:34Yeah.
  117. 13:35Um, but I'm also trying to maybe like make more content around this, like maybe like videos.
  118. 13:43It was kind of hard to go through this.
  119. 13:44I I'm a little all over the place, but I would definitely love to maybe like publish a blog or make snippet videos.
  120. 13:52Um, yeah, on like YouTube or somewhere, but you can definitely find the resources.
  121. 14:03Okay.
  122. 14:04Right.
  123. 14:04I should have covered that.
  124. 14:06Um, but I don't have the I should have put a QR.
  125. 14:13Oh, um, as I said, I I'm from Akamay.
  126. 14:16And if you're interested in like learning how to replicate this, you can find it on our website, our GitHub, GitHub.
  127. 14:24Um, Akami developers.
  128. 14:29Yeah.
  129. 14:30Um, yeah.
  130. 14:32So we have a a lot of like different resources on here.
  131. 14:35Um I'm focusing on in inference optimizations.
  132. 14:38Uh we have other people that are focusing on building AI agents um and how to leverage our like manage Kubernetes service, how to use Akami functions.
  133. 14:47So if you have time, please go check out our GitHub pages and also we have a Discord channel that we're trying to build up now.
  134. 14:55Um so it would be great to chat with some of you on the side.
  135. 14:59Thank you.
  136. 15:04[music]
  137. 15:12[music]