WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=YXowceUKYJI

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/YXowceUKYJI.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.800 --> 00:00:13.440
All right.

s2
00:00:13.440 --> 00:00:17.520
Um, welcome everyone to yet another inference talk.

s3
00:00:17.520 --> 00:00:20.400
I hope you have had a good conference so far.

s4
00:00:20.400 --> 00:00:28.720
And u, so in this session, I mean I'm sure you people who have been in the room uh must have heard these terms many times by now.

s5
00:00:28.720 --> 00:00:40.719
So we're going to do a little bit more deep dive into the challenges of LLM deployments for agentic workloads and uh in this session we'll focus specifically on KV cache away routing and

s6
00:00:40.719 --> 00:00:55.600
uh PD disagregation um and also you know when you when you look at public inference uh benchmark results you are typically looking at very steady state isolated highly sanitized numbers and what those benchmarks actually don't show you

s7
00:00:55.600 --> 00:01:04.879
u is the chaotic reality of multi-turn interactions, massive context fluctuations which are very typical of agentic workloads.

s8
00:01:04.879 --> 00:01:09.360
So we'll also try to pull the curtain back on some of those complexities.

s9
00:01:09.360 --> 00:01:13.520
Um by by way of introduction uh my name is Ashish Kamra.

s10
00:01:13.520 --> 00:01:17.439
I'm a senior manager of performance engineering at Red Hat.

s11
00:01:17.439 --> 00:01:18.320
And with me

s12
00:01:18.320 --> 00:01:19.360
hi I'm Yuch Chen.

s13
00:01:19.360 --> 00:01:25.040
I'm the product manager at Red Hat Inference working closely with VLM and AMD core maintainers.

s14
00:01:25.040 --> 00:01:27.920
also a contributor myself.

s15
00:01:28.880 --> 00:01:31.920
So here is the agenda for the next 20 minutes or so.

s16
00:01:31.920 --> 00:01:41.040
Um Euchen will start with an analysis of inference behavior in the agentic era and some of the core characteristics and challenges.

s17
00:01:41.040 --> 00:01:48.720
Uh next we next you will walk us through the KV cache um utilization and management strategies.

s18
00:01:48.720 --> 00:02:01.040
I will break down the mechanics of pre-fill decode disagregation and walk you through some some results and then Euchen will again bring it all back together with our ongoing case study on our favorite

s19
00:02:01.040 --> 00:02:12.239
open coding model GLM 5.2. Um and just a couple of uh sources from our side if you are more interested in learning more about open source inference we have a free course

s20
00:02:12.239 --> 00:02:17.120
free course on deep learning.ai AI uh by Cedric and with Andrew Ning.

s21
00:02:17.120 --> 00:02:27.040
Um and the other is a series of blogs on the Red Hat developer portal on distributed inference concepts uh troubleshooting and deployment patterns.

s22
00:02:27.680 --> 00:02:39.120
Uh and for those who may not be aware since Red Hat is better known as the Linux company for enterprise Linux and uh the Kubernetes company for Open Shift

s23
00:02:39.120 --> 00:02:51.120
uh but more recently we are also a major player in open source AI inference with uh us being the top contributor in VLM LLMD and the case of projects and also

s24
00:02:51.120 --> 00:03:06.879
uh having incubated guide LLM for benchmarking LLM compressor for model quantization and speculators for uh speculative uh decoding models and we also bring it bring all of that together in a

s25
00:03:06.879 --> 00:03:12.800
optimized model hub on hugging phase under the Red Hat AI arc.

s26
00:03:12.800 --> 00:03:23.599
Um and we are also building the platform for the next wave of agentic inference workloads and with that I will hand over to you to uh walk you through more of it.

s27
00:03:25.040 --> 00:03:33.519
So we are currently um at this inflection point moving from the era of classic inference to the agentic era.

s28
00:03:33.519 --> 00:03:47.680
So when we look at the real world agentic work workloads such as uh sweet bench and also watrices from real world cloud code sessions they fundamentally break many assumptions we made with classic LM serving.

s29
00:03:47.680 --> 00:04:01.439
uh as you heard actually many times in previous sessions for example multi-turns and new standard we found from a few turns all the way to 3,000 turns and also because agent frequently reuse the uh system prompt and the total definitions

s30
00:04:01.439 --> 00:04:07.360
we usually see super high cash hit rate um oftentimes well exceeding 90%.

s31
00:04:07.360 --> 00:04:20.400
Uh another thing is input output ratio is uh is massive oftentimes over a 100 ratio and even higher and in many cases and on top of that the context management is is incredibly

s32
00:04:20.400 --> 00:04:32.560
complex due to this high variance because we can't just simply take the average and oftentimes we need to look at the distributions and the P90 numbers especially when you do uh capacity planning

s33
00:04:32.560 --> 00:04:41.040
and also we observe really interesting patterns like sub Asian panel which is which further complex uh complicates scheduling.

s34
00:04:41.040 --> 00:04:53.919
So to help communities study um this patterns we collaborate with Google thank you and also IBM our parent company to add uh a a trace replay tool in the inference perf you heard from earlier sessions

s35
00:04:53.919 --> 00:04:55.759
u from Ashoken and Jason.

s36
00:04:55.759 --> 00:05:00.400
Um so yeah feel free to check it out and the link is here.

s37
00:05:01.680 --> 00:05:03.040
Uh next slide.

s38
00:05:03.040 --> 00:05:08.960
Oh, so transition from the class uh the characteristics um we just saw for agentic workloads.

s39
00:05:08.960 --> 00:05:13.919
We're no longer chasing this um this this raw throughput in a steady state.

s40
00:05:13.919 --> 00:05:25.840
We often need to optimize uh for example interactive latency and they're very um highly volatile and client-driven context because user and you know client define the prompt structure.

s41
00:05:25.840 --> 00:05:28.479
So this introduced several critical challenges.

s42
00:05:28.479 --> 00:05:34.800
First of all, KV cache management becomes super volatile because the context is client determined as I said.

s43
00:05:34.800 --> 00:05:47.600
So oftentimes we face this like you know frequent evictions and rewrites and secondly we also need to tune um the engine like VM with upper layer uh scheduling and routing.

s44
00:05:47.600 --> 00:05:57.759
It needs that coordination such as prefix routing especially when latency becomes a primary uh scheduling matrix rather than like a secondary or afterthought.

s45
00:05:57.759 --> 00:06:00.320
And thirdly, we also need to rethink our metrics.

s46
00:06:00.320 --> 00:06:03.520
For example, we need to measure cats throughput separately.

s47
00:06:03.520 --> 00:06:03.919
Why?

s48
00:06:03.919 --> 00:06:08.000
Because on the right, it's really clear that economic stakes is very high.

s49
00:06:08.000 --> 00:06:10.639
So, this is the uh anthropic API pricing.

s50
00:06:10.639 --> 00:06:12.639
You also heard from earlier sessions.

s51
00:06:12.639 --> 00:06:16.400
There's 10x cost difference between cash and non-cash tokens.

s52
00:06:16.400 --> 00:06:23.440
So, 10x difference on your um token balance sheet is is pretty serious impact on your business.

s53
00:06:23.440 --> 00:06:29.360
So next let's let's look at how the KV cache is um both utilized and managed in LMD.

s54
00:06:29.360 --> 00:06:40.240
So LMD router has this really flexible um endpoint picker plugins we call the EP that can route the request to the optimal pods and that meet the KV cache locality

s55
00:06:40.240 --> 00:06:42.560
and also the load criteria.

s56
00:06:42.560 --> 00:06:52.960
So the EP continue probe each pods like VM pod matrix to score each pod on like the running for example running and waiting request and then the KV cache utilization

s57
00:06:52.960 --> 00:07:03.440
also prefix uh cache availability and so we can schedule requests to the optimal pod with the lowest load and also highest possibility to um to of a cache hit.

s58
00:07:03.440 --> 00:07:10.400
So um going down from to the KV cache management layer actually you also heard from earlier session right before this.

s59
00:07:10.400 --> 00:07:22.160
So for agentic sessions when you have u hot warm and cold cache our current effort focus on for example um more offloading tiers like NVME SSD and also uh file system XF

s60
00:07:22.160 --> 00:07:37.120
along with KV ccentric store um like uh moon cake and also implementing smarter and session a wire eviction policies such as priority and also session pinning to uh ensure this uh really important you know the the context persists

s61
00:07:37.120 --> 00:07:40.720
exactly when and where it's needed.

s62
00:07:41.520 --> 00:07:44.800
So, I'm gonna play this um video really quick.

s63
00:07:44.800 --> 00:07:46.880
Uh it's a it's a short demo.

s64
00:07:46.880 --> 00:07:48.400
Stand here so you can look at it.

s65
00:07:48.400 --> 00:07:50.400
Okay.

s66
00:07:52.000 --> 00:07:55.120
So, okay.

s67
00:07:55.120 --> 00:07:57.919
So, this is a example of a KV cache bar routing.

s68
00:07:57.919 --> 00:08:04.240
As you see, when we send the very first request and it populate the KV cache, it takes roughly 3 seconds.

s69
00:08:04.240 --> 00:08:11.680
And when we actually look at where it's you know the KV cache uh is going there's no KV cache hit because it's the very first turn.

s70
00:08:11.680 --> 00:08:20.160
And then when we have the second turn the request actually reuse a KV cache because as you see the system prompt is the same and this time takes about one seconds.

s71
00:08:20.160 --> 00:08:25.840
And then when you actually look at the uh pod address exactly the same because we define the KV cache.

s72
00:08:25.840 --> 00:08:29.360
Now going to the third turn a new request with different system prompt.

s73
00:08:29.360 --> 00:08:40.000
Now it takes about three uh seconds and as you see you know right now and we don't find any KV cache here because you can tell it's different pod address

s74
00:08:40.000 --> 00:08:50.880
and then if you just change the user prompt and keep the same system prompt and the next turn you you reuse the KB cache and in this in this time it takes roughly about uh one second.

s75
00:08:50.880 --> 00:08:51.120
Yeah.

s76
00:08:51.120 --> 00:08:58.880
So it's a pretty intuitive demo and um I'll turn it to Ashish to talk about the next side but before that what does problem does it solve?

s77
00:08:58.880 --> 00:09:13.440
So often times the prefix routing KB cache routing helps you solve the TTFD problem and of course you'll improve your lat uh your your throughput but oftentimes for agentic workload is not just a TTFT your throughput is about your inter token latency

s78
00:09:13.440 --> 00:09:25.839
how do we solve that so preview decode disagregation is a really uh powerful technique but there are times there work at times it doesn't work so I'll turn it to Ashish to give you a preview of um of the PD

s79
00:09:25.839 --> 00:09:33.519
uh disregation So before we dive into PD, let's just uh look at what LLMD is.

s80
00:09:33.519 --> 00:09:39.920
So LLMD is a high performance Kubernetes native and actually now works on non-cubernetes environments as well.

s81
00:09:39.920 --> 00:09:44.959
Distributed LM LLM inference framework hosted under the CNCF umbrella.

s82
00:09:44.959 --> 00:09:52.320
LLMD provides a unified intelligent control plane designed specifically for agentic era of inference workloads.

s83
00:09:52.320 --> 00:09:57.279
Well, Euchin already talked about the router and the EP at the top of the slide.

s84
00:09:57.279 --> 00:10:13.200
Um, the other aspects are workload APIs such as leader worker set and disagregated set that orchestrates complex multi- multi-node model execution and then autoscalers that monitors capacity bounds and real-time traffic mixes

s85
00:10:13.200 --> 00:10:19.040
to independently scale up and scale down uh your pods depending on the system load.

s86
00:10:19.040 --> 00:10:23.279
So now look now let's look at uh prefill decode disagregation in detail.

s87
00:10:23.279 --> 00:10:29.680
Um uh okay so why does PD exist in the first place?

s88
00:10:29.680 --> 00:10:37.680
So one of the most powerful patterns implemented by LLMD is prefill decode disagregation and you must have heard from some of the previous talks as well.

s89
00:10:37.680 --> 00:10:49.120
So what happens is in in a nonPD situation in aggregated serving one pod is responsible for optimizing both your time to first token and your inter token latencies.

s90
00:10:49.120 --> 00:10:54.399
Uh but in PD prefill and decode become independently scalable inference pods.

s91
00:10:54.399 --> 00:11:00.480
But to understand why we actually need this we have to look at the physics of LLM execution.

s92
00:11:00.480 --> 00:11:07.680
colloccating uh both prefill and decode tasks on the same GPU creates something called as phase interference.

s93
00:11:07.680 --> 00:11:12.800
Prefill phase is the phase that creates the KV caches for your initial prompt.

s94
00:11:12.800 --> 00:11:14.880
It wants high compute.

s95
00:11:14.880 --> 00:11:27.120
It's highly bursty uh utilizes GPUs at uh high flops and and thrives on large batch parallelism to process the prompts and builds the initial KV cache.

s96
00:11:27.120 --> 00:11:33.680
The decode phase on the other hand is generating one token at a time and it's more me memory bandwidth hungry.

s97
00:11:33.680 --> 00:11:38.880
It's highly latency sensitive and requires high heavy cache residency.

s98
00:11:38.880 --> 00:11:50.399
So in a in a in a traditional aggregated pod if you if there's a sudden influx of a long prefilled palm, it will completely stall the ongoing decode token generation process

s99
00:11:50.399 --> 00:11:56.000
causing massive problems and jitter in user streaming latency.

s100
00:11:56.560 --> 00:12:00.399
So, so how does PD actually work in practice in LMD?

s101
00:12:00.399 --> 00:12:05.600
So, LNMD uses um uh you know like okay, we'll start with step one.

s102
00:12:05.600 --> 00:12:18.399
A incoming request hits the gateway router which dynamically evaluates cluster states using something known as the endpoint picker you talked about and schedules the request to use PD disagregation

s103
00:12:18.399 --> 00:12:21.279
selecting the optimal prefill and decode workers.

s104
00:12:21.279 --> 00:12:25.920
The router then coordinates the transaction directly with the designated pre-fill worker.

s105
00:12:25.920 --> 00:12:34.720
The pre-fill worker processes the prompt, construct the initial KV cache of the prompt and outputs the standard KV transfer metadata.

s106
00:12:34.720 --> 00:12:49.680
Um, and the target decode worker actually pulls the computed KV caches um, uh, across the network fab fabric utilizing uh, the KV transfer metadata that the uh, uh, prefill pod had generated.

s107
00:12:49.680 --> 00:13:02.560
Um okay so with that yes that's kind of how uh PD is implemented in practice in LMD and next I would like to show you some uh experimental results on where PD actually shines.

s108
00:13:02.560 --> 00:13:19.600
So in this graph you can see that um uh in in in the standard aggregated deployment which is the top red line uh the P99 ITL uh hovers roughly around 900 milliseconds and you can you can see some fluctuations

s109
00:13:19.600 --> 00:13:34.880
um up and down and but the the bottom blue line is the P99 uh inter token latency on a PD deployment and you can see that it's drastically almost nine times better at 100 millconds and it's also much smoother

s110
00:13:34.880 --> 00:13:44.000
uh than the aggregated serving and uh this is some of our own internal results at Red Hat.

s111
00:13:44.000 --> 00:14:04.320
So for a GPOSS 12B model uh 16 H100s uh the aggregated config is four replicas tensor parallelism 4 and the disagregated is two prefilled 2D code all with tensor parallelism 4. It's a highly multi-turn workload with a 10,000 token prefix and 128 tokens

s112
00:14:04.320 --> 00:14:07.040
for every turn every turn.

s113
00:14:07.040 --> 00:14:17.839
So, so this is a great chart like you can see at the bottom most line is a standard aggregated config that's uh is doing the default Kubernetes scheduling and uh and it's aggregated.

s114
00:14:17.839 --> 00:14:33.199
So that's kind of our baseline and then the middle blue line is still aggregated but with the LLMD uh KV cache aware routing and you can almost see the gains just just based on the routing and the red line is actually the PD

s115
00:14:33.199 --> 00:14:43.440
uh the pre-fill decode config with two pre-fill and two decode workers and you can actually see that like it's very similar to the aggregated config at the lower concurrency regimes

s116
00:14:43.440 --> 00:15:02.320
and uh even and and very similar at the higher concurrency regimes but it's actually the middle part of the concurrency regime that PD actually shines and and these are some of the the classic parita curves that we see when you actually do PD and uh aggregated side by side.

s117
00:15:02.320 --> 00:15:17.680
So these results are again from the GPTOSS 12B model 64 H100s aggregated is eight replicas TP8 and this a is uh three prefilled 5D code again TP8 and a pre-filled heavy workload with like 5,000

s118
00:15:17.680 --> 00:15:28.480
average input sequence length and 500 output sequence length and you can actually see the blue line is the the PD curve and the red line is the aggregated curve and the PD curve kind of dominates

s119
00:15:28.480 --> 00:15:34.959
um uh the aggregate curve across the entire interactivity spectrum.

s120
00:15:37.040 --> 00:15:42.000
Okay, but I don't want to leave you guys that PD is the answer to everything and it's a magic bullet.

s121
00:15:42.000 --> 00:15:47.600
But um it's uh it's essentially a separation phase separation trade-off and not a magic bullet.

s122
00:15:47.600 --> 00:15:53.920
So we created this uh matrix to help you decide when PD might be uh good for you.

s123
00:15:53.920 --> 00:16:07.360
So if you're managing long context uh with high ISL OSL ratios and you if you have a large model that you're serving that can that you can apply rich model parallelism techniques

s124
00:16:07.360 --> 00:16:22.560
um you're facing that middle concurrency regime uh that I I showed you in the previous graphs and and the very important part is that if you want uh strict ITL streaming requirements like you want the you want the token generation to be uh much more smooth

s125
00:16:22.560 --> 00:16:30.240
um then you want to consider PD but we also saw that it requires transfer of KV caches from your pre-filled workers to your decode workers.

s126
00:16:30.240 --> 00:16:38.959
So you must pro process an advanced uh high-sp speeded network fabric like uh RDMMA or rocky to support that KV cache transfer.

s127
00:16:38.959 --> 00:16:53.120
And if you do not have such requirements, short moderate context, any model size, low concurrency regimes or uh if you have strict TTF requirements because you can actually tune them on an aggregate serving

s128
00:16:53.120 --> 00:16:58.399
um and you the biggest point is like if you don't have the network fabric to support those KV cache transfers.

s129
00:16:58.399 --> 00:17:01.839
So you might actually just want to stick with aggregated.

s130
00:17:02.480 --> 00:17:04.480
So here is my key takeaway from all of this.

s131
00:17:04.480 --> 00:17:14.480
So architecting this complex platform requires balancing a lot of u knobs and a highly multi-dimensional design space all of which is supported in LLMD.

s132
00:17:14.480 --> 00:17:27.520
As you saw the scheduler must support or constantly evaluate SLO targets uh QEPs KV cache locality metrics PD ratios and network topologies to be able to route the request to the optimal

s133
00:17:27.520 --> 00:17:28.160
FOD.

s134
00:17:28.160 --> 00:17:40.000
While in while the PD design space you you need dynamic PD rate matching to adapt to PD ratios because you know you can start with a static PD ratio but it needs to evolve with the autoscaler

s135
00:17:40.000 --> 00:17:56.080
as the traffic changes um and you need uh yeah autoscaling to scale PD pools independently um and constantly tweaking model parallelism techniques like tensor parallelism data parallelism uh to meet your SLOs's.

s136
00:17:56.080 --> 00:18:08.080
So um I think with these uh I will hand it over to Euchen to anchor some of the concepts that we showed with the real world case study of serving the GLM 5.2 model

s137
00:18:08.080 --> 00:18:11.520
uh which is uh still ongoing as we speak.

s138
00:18:11.520 --> 00:18:12.640
Yeah, still ongoing.

s139
00:18:12.640 --> 00:18:22.960
You probably have seen tons of uh impressive numbers of GLM 5.2 on B200 when we talk to our customers and they usually don't have you know the luxury of B200.

s140
00:18:22.960 --> 00:18:24.480
They have a lot of H200.

s141
00:18:24.480 --> 00:18:32.559
So we have to figure out how to like put all the knobs together and make GM 5.2 work really well for cluster of of H200.

s142
00:18:32.559 --> 00:18:35.120
So uh we let's anchor all the concept together.

s143
00:18:35.120 --> 00:18:40.160
Um we went through for example the uh KV cache routing PD disagregation.

s144
00:18:40.160 --> 00:18:54.160
We kind of call them a wildl path in LMD and also we combine with different parallelism strategies to so we can uh independently uh scale prefuel paths because for agentic workload is super uh long you know like heavy prefill.

s145
00:18:54.160 --> 00:19:07.360
So uh in this case we designed the prefuel pool using up to three workers optimized for uh high throughput uh with deep and then for decoup we use uh one dedicated worker and um that's optimized for for low latency.

s146
00:19:07.360 --> 00:19:18.400
So we use Nixo for efficient KV transfer between the pools and also with the each worker we have the leader worker set group uh with TP1 DP8 and also uh EP8

s147
00:19:18.400 --> 00:19:29.919
uh expert parallelism 8. So the architecture is just highly modular because you can uh actually scale the throughput by simply adding uh preview workers without reconfiguring and um the decoup.

s148
00:19:29.919 --> 00:19:38.080
So uh this highlights how AMD effectly effectively managed the complexity of combining like PB and DB and EPI scale.

s149
00:19:38.080 --> 00:19:42.000
And also we found some interesting fun fact actually a couple days ago.

s150
00:19:42.000 --> 00:19:44.320
Um B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B Bf6 uh BF16

s151
00:19:44.320 --> 00:19:50.000
KV cache actually is faster than using like FPA uh KV cache for longer preview.

s152
00:19:50.000 --> 00:19:58.559
Um this is also like we continue to explore and found like more interesting patterns, but more importantly uh we want to kind of just show the result really quick.

s153
00:19:58.559 --> 00:20:10.400
So um for this uh data set agentic workload data set the ISO OSL ratio is pretty high 45 to1 ratio preview is uh is really the constraint you can tell um with 2P even 1D

s154
00:20:10.400 --> 00:20:22.160
we have um 4x passer TDFT and also 60 uh% more requests and this is continuous like work in progress so the next step is we need to also put the upper layer

s155
00:20:22.160 --> 00:20:28.960
lower TTFT and also adding more more preview replicas so um I know we're running out of time really quick.

s156
00:20:28.960 --> 00:20:43.520
Uh we um the fundamental shift for agentic workload we're continuing to uh have this um uh agentic north uh northstar uh with session graph orchestration program award scheduling uh state reuse life cycle and also

s157
00:20:43.520 --> 00:20:46.880
the uh agentic benchmark um we're working on.

s158
00:20:46.880 --> 00:20:57.919
So you can find them uh in AMD upstream AMD and also you know feel free to join the SIG group and uh and contribute and um this is the very last slide.

s159
00:20:57.919 --> 00:21:03.360
So distri distributed inference is not challenge uh every single comp a single company can solve along.

s160
00:21:03.360 --> 00:21:16.000
We're proud to be uh building this uh future in the open alongside our incredible ecosystem collaborators uh core wave Google IBM Nvidia growing list of launch partners and industry adopters.

s161
00:21:16.000 --> 00:21:20.640
So if you're passionate about the future of opensource inference, we invite you to join us.

s162
00:21:20.640 --> 00:21:22.080
We do have a booth downstairs.

s163
00:21:22.080 --> 00:21:24.480
Feel free to stop by, ask us any questions.

s164
00:21:24.480 --> 00:21:27.744
And uh thank you so much for your time.

s165
00:21:27.744 --> 00:21:29.744
[applause]

s166
00:21:44.774 --> 00:21:46.774
[music]