WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=l1-D89bAuOA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/l1-D89bAuOA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.719 --> 00:00:20.480
Hi everyone, welcome to our talk on um our LLM performance benchmarks reliable.

s2
00:00:21.359 --> 00:00:22.560
A little bit about us.

s3
00:00:22.560 --> 00:00:23.920
I am Ashok Chandra Seeker.

s4
00:00:23.920 --> 00:00:26.160
I'm a staff software engineer at Google.

s5
00:00:26.160 --> 00:00:29.599
I work on inference performance evaluation and optimization.

s6
00:00:29.599 --> 00:00:32.079
Um and I lead a couple of open source projects.

s7
00:00:32.079 --> 00:00:41.280
One is called inference perf uh which is a benchmarking tool to do reliable performance benchmarks and uh I'm also the sig lead for LLMD benchmarking.

s8
00:00:41.280 --> 00:00:47.280
LLMD is a distributed inference framework um that makes production scale inference possible.

s9
00:00:47.280 --> 00:00:49.360
Hi everyone, I'm Jason Kroger.

s10
00:00:49.360 --> 00:00:51.680
I'm a software engineer at Google.

s11
00:00:51.680 --> 00:00:58.320
Uh I'm also a co-maintainer of inference Perf and a few of the sub projects that Ashok brought up.

s12
00:00:58.320 --> 00:01:02.239
uh and I work on inference performance and benchmarking.

s13
00:01:03.280 --> 00:01:04.640
Okay, let's get started.

s14
00:01:04.640 --> 00:01:08.479
Um let's look a little bit about the how the benchmark ecosystem looks like.

s15
00:01:08.479 --> 00:01:09.119
Right?

s16
00:01:09.119 --> 00:01:11.200
You have your model server frameworks.

s17
00:01:11.200 --> 00:01:18.080
Uh these are VLM, SGLAN, um and other model servers and all of these have some benchmark capability within them.

s18
00:01:18.080 --> 00:01:18.320
Right?

s19
00:01:18.320 --> 00:01:25.920
These are primarily Python scripts and are developer focused benchmarks to see how you can measure the performance of your model server itself.

s20
00:01:25.920 --> 00:01:28.400
And then you have your competitive analysis tools.

s21
00:01:28.400 --> 00:01:33.280
Uh these are MLPF, semi analysis, artificial analysis and so on.

s22
00:01:33.280 --> 00:01:34.079
Right?

s23
00:01:34.079 --> 00:01:39.920
They mainly aim for competitive performance benchmarks to compare like chip and accelerator performance.

s24
00:01:39.920 --> 00:01:42.240
Uh and then you have your typical web benchmarks.

s25
00:01:42.240 --> 00:01:46.479
Uh these are like locus, graphfana, ksix um and so on.

s26
00:01:46.479 --> 00:01:50.799
These mainly focus on highcale http benchmarks, right?

s27
00:01:50.799 --> 00:02:03.040
uh and then you have your um last segment which is the production scale LLM benchmarks right these are to actually benchmark uh your production inference serving stack and that is our focus today

s28
00:02:03.040 --> 00:02:18.400
right we'll be focusing mainly on inference perf and how we solve this uh production scale benchmark problem so if you have run a benchmark before uh it typically looks like this right um you have some sort of benchmark hardness

s29
00:02:18.400 --> 00:02:28.560
uh and then you specify what model you are bench benchmarking the number of prompts you want to run, what is the input output sequence length uh and the request rate or the load you want to send,

s30
00:02:28.560 --> 00:02:28.959
right?

s31
00:02:28.959 --> 00:02:32.160
Uh and your output looks something uh like what is on the right.

s32
00:02:32.160 --> 00:02:41.360
Uh this is basically your uh input token throughput, output token throughput, some latency metrics, time to first token, time per output token and so on.

s33
00:02:41.360 --> 00:02:46.560
Um so what are some issues with a simple benchmark like this?

s34
00:02:46.560 --> 00:02:47.200
Right?

s35
00:02:47.200 --> 00:02:59.200
So if you want to actually benchmark production scale workloads um here I have LLMD inference stack as an example right uh you can have uh online serving you can have batch workloads

s36
00:02:59.200 --> 00:03:09.440
and if you see the inference pool below uh there are like lot of servers that are running right and then you have like complex configurations like pre-fill decode disagregation

s37
00:03:09.440 --> 00:03:22.400
um and workload autoscaling and other things that are going on under the hood and usually the scale is much larger right so your um normal benchmark harnesses runs into issues when you try to benchmark a setup like this.

s38
00:03:22.400 --> 00:03:31.760
And if you look at like the key characteristics of what we want out of a production scale benchmark, we need to be able to do high load uh which is limited in a lot of tools out there.

s39
00:03:31.760 --> 00:03:34.720
We need to be able to simulate real world workloads, right?

s40
00:03:34.720 --> 00:03:41.760
Um what use it is if it is just some synthetic workload that is not accurately representing uh what your customers are going to run.

s41
00:03:41.760 --> 00:03:44.560
Uh and then metrics fidelity is very important, right?

s42
00:03:44.560 --> 00:03:48.560
Are the metrics accurate and how well they work?

s43
00:03:49.840 --> 00:03:54.959
Uh this is like the set of metrics that LLMD measures uh by default.

s44
00:03:54.959 --> 00:03:57.760
Uh I just pulled it from the website there.

s45
00:03:57.760 --> 00:04:02.000
Uh as you can see it's not just like a single QPS that you are running, right?

s46
00:04:02.000 --> 00:04:12.239
You are sweeping a a list of various loads and uh you try to measure what the baseline is and what optimizations you are making and what the difference there is.

s47
00:04:12.239 --> 00:04:23.360
um you need to find the right point where the server gets saturated so you know the right optimal uh point to run your servers on uh to maximize performance and to save costs

s48
00:04:23.360 --> 00:04:43.600
um and things like uh SLOs's become more important right what is your um time to first token P90 SLO and are you conformant to that SLO so when you run like um normal benchmark like we saw before what are some of the pitfalls that you run into right uh we have been uh running benchmarks for a couple

s49
00:04:43.600 --> 00:04:44.479
of years.

s50
00:04:44.479 --> 00:04:51.759
So we run into all sort of uh different results that people share and a lot of times we aren't able to reproduce the results that are shared by other people.

s51
00:04:51.759 --> 00:04:52.080
Right?

s52
00:04:52.080 --> 00:04:54.400
So that is what motivated this talk.

s53
00:04:54.400 --> 00:04:59.680
Um so these four common uh things that we see as an issue, right?

s54
00:04:59.680 --> 00:05:05.040
One is accurate uh metrics and two observability into your benchmark tool itself.

s55
00:05:05.040 --> 00:05:08.240
Um do you know if your benchmark harness is actually failing?

s56
00:05:08.240 --> 00:05:10.800
Is it not able to maintain the load?

s57
00:05:10.800 --> 00:05:13.120
Um and three reproducibility.

s58
00:05:13.120 --> 00:05:17.360
Uh there is some inherent uh randomness uh in like the data sets that you use.

s59
00:05:17.360 --> 00:05:23.039
So how do you make sure it is reproducible and four the data set quality itself.

s60
00:05:24.880 --> 00:05:26.639
So this is an experiment we ran.

s61
00:05:26.639 --> 00:05:33.520
Um we asked like different benchmark harness to generate 200 QPS and this was the result right.

s62
00:05:33.520 --> 00:05:36.080
Um so a couple of things I want to point out.

s63
00:05:36.080 --> 00:05:44.960
Um, Python has this global interpreter lock GIL if you have been working with Python, you know that which makes everything uh sort of single threaded.

s64
00:05:44.960 --> 00:05:54.160
So even when you have like a multiCPU mission, a lot of times you are limited by the performance of a single CPU uh when you're CPU bound especially, right?

s65
00:05:54.160 --> 00:06:03.840
Uh so this kind of shows a single process um benchmark harness and a multiprocess harness and how the QPS you are able to achieve differs based on it, right?

s66
00:06:03.840 --> 00:06:17.199
when you run with a really uh small shad core mission, you can see that even when you request 200 QPS, you are only getting 38 QPS and then you give it a bigger mission and then some of these uh single process harness they cap out at like 170 QPS, right?

s67
00:06:17.199 --> 00:06:19.520
This is a much more powerful machine.

s68
00:06:19.520 --> 00:06:25.280
Um but it is a problem because you ask for 200 QPS and then you don't know whether it actually delivered it.

s69
00:06:25.280 --> 00:06:27.520
It will just say I ran it these are the numbers.

s70
00:06:27.520 --> 00:06:33.280
So you think okay you ran 200 QPS but in fact you you have not.

s71
00:06:33.360 --> 00:06:49.280
Uh the other issue that comes out of it is the latency inflation right if your server is saying okay this is how much QPS I was able to run and this was the accurate numbers that is one thing but if your uh benchmark harness is actually inflating latency right because it's thrashing trying to collect all the streaming

s72
00:06:49.280 --> 00:06:55.039
token requests um in one of the tests we noticed like u the delay was up to 58 seconds.

s73
00:06:55.039 --> 00:07:03.840
So you might look at this and go oh my server is bottlenecked right it's not able to handle all the requests but in fact it's actually your benchmark client that is inflating the latency

s74
00:07:03.840 --> 00:07:13.680
right uh we ran like a th000 QPS test uh when your when your benchmark harness is actually able to scale out you can see there is very minimal latency right this

s75
00:07:13.680 --> 00:07:25.039
simulated server so there shouldn't be any latency at all um and like I said there are like other variables that go into it right in one of the benchmarks

s76
00:07:25.039 --> 00:07:28.720
um someone shared and they said, "Hey, we are getting 20% better throughput."

s77
00:07:28.720 --> 00:07:34.160
Then we looked into it and we found out the benchmark harness were setting the model temperature to zero, right?

s78
00:07:34.160 --> 00:07:42.800
Which means your model outputs are a lot more deterministic and it was able to turn out a higher throughput than what you would normally see in like a real workload, right?

s79
00:07:42.800 --> 00:07:53.039
Where your model temperature is somewhere around 0.7. Um another thing is like we used a a shar GPD data set the same data set across two different benchmark harness

s80
00:07:53.039 --> 00:08:08.240
and they produce different input tokens right this is because they sample them differently they truncate them uh differently so as a user you don't have insight into this right you run it you trust the numbers it produces but they are wildly different

s81
00:08:08.240 --> 00:08:16.879
uh and there is much more right do you actually force it to generate till the end of sequence are you looking at prefix cache rates how do you do mult multi-turn

s82
00:08:16.879 --> 00:08:35.519
uh replay via benchmarks and uh how how do you actually get high fidelity uh on the actual workload that would resemble your production workload right so the main thing I wanted to convey here is like a lot of times you diagnose it as a your server or inference stack problem but in a lot of cases it could

s83
00:08:35.519 --> 00:08:48.000
be your benchmark harness so what is the solution to this how do we actually do reproducible benchmarks Jason here will take you over Thanks Ashook.

s84
00:08:48.000 --> 00:08:52.880
Uh so yeah, how do you how do you solve these problems?

s85
00:08:52.880 --> 00:09:08.959
Uh we al together inferencepf uh the CNCF project spanned out of Kubernetes working group serving uh to provide like a standardized place for us to work with the community and

s86
00:09:08.959 --> 00:09:11.839
solve some of these issues together.

s87
00:09:11.839 --> 00:09:25.200
uh it enables the ability to have like a userdefined declarative configuration uh that allows you to have clear reproducibility across runs.

s88
00:09:25.200 --> 00:09:45.680
We also added a load generator uh that solves the GIL problem in Python uh across multiple processes and reports those client metrics back along with server metrics to ensure that you have the highest metric fidelity and you're able to actually observe when

s89
00:09:45.680 --> 00:09:51.440
your tool is having an issue versus your system under test.

s90
00:09:52.640 --> 00:10:09.040
So, first going over the load generator, uh you see that the main process actually cues requests based off of the planned time that they need to execute, which is based off of your configuration.

s91
00:10:09.040 --> 00:10:18.640
This may be in some poison process or constant rate or maintaining a constant number of concurrent requests.

s92
00:10:18.640 --> 00:10:32.959
This request Q channel is then spread across multiple processes which pull and ensure that they execute with minimum overhead but then also observability about when they execute versus their plan time.

s93
00:10:34.640 --> 00:10:36.560
And you can see this working at scale.

s94
00:10:36.560 --> 00:10:44.800
So this is a comparison across other tools uh some being the HTTP scaled tools like K6.

s95
00:10:44.800 --> 00:10:57.680
uh but you see that even at 5,000 QPS inference perf was able to keep up due to this architecture and most importantly actually report that it was able to keep up.

s96
00:11:00.079 --> 00:11:02.160
Other portion is configuration.

s97
00:11:02.160 --> 00:11:09.440
So earlier I show showed a brief example of how you might simply run a benchmarking tool.

s98
00:11:09.440 --> 00:11:15.680
And here on the left you can see a simple example running a random data set against an endpoint.

s99
00:11:15.680 --> 00:11:30.240
But the actual configuration that we have in front of inference perf is very detailed with a lot of knobs that allow you to accurately uh test your configuration off of your workloads.

s100
00:11:30.240 --> 00:11:40.640
You can see here on the right that this is uh configuration for conversation replay where you're able to configure not only like the input output length but their distributions

s101
00:11:40.640 --> 00:11:59.600
etc. And further beyond just the ability to configure a single run, we've actually worked together to have a published set of some of these workloads and configuration of inference perf that are tied to state-of-the-art

s102
00:11:59.600 --> 00:12:01.839
inference workloads.

s103
00:12:01.839 --> 00:12:17.279
For example, here in the workload catalog that we've put out, you're able to access standard multi-turn uh generation, tree of thought, agentic generation uh as well as batch summarization

s104
00:12:17.279 --> 00:12:19.519
and others.

s105
00:12:19.600 --> 00:12:29.760
Each one of these has a simple definition uh kind of in natural language that allows you to understand what the scenario is.

s106
00:12:29.760 --> 00:12:44.800
But beyond that there's also pretty detailed configuration metrics not only in inference perf configuration but in generic terms so that this can actually be shared across tools and have a place for standardization

s107
00:12:44.800 --> 00:12:47.279
of these workloads.

s108
00:12:50.399 --> 00:12:58.320
So kind of the culmination of these things uh leads us to actual results that we can clearly display.

s109
00:12:58.320 --> 00:13:08.320
Uh and here's a screenshot from Prism which is a UI we have for sharing not only those workloads that I showed before but also some benchmarking results.

s110
00:13:08.320 --> 00:13:13.120
Uh and Prism is a part of the LLMD project.

s111
00:13:13.120 --> 00:13:23.040
Here you can see a benchmark result for the agentic code generation workload that we showed before on TPUs.

s112
00:13:23.040 --> 00:13:39.920
Uh these three lines you see here show you The difference between combined optimizations is the green line and a baseline that is just a simple Kubernetes service instead of in front of multiple

s113
00:13:39.920 --> 00:13:41.839
model server replicas.

s114
00:13:41.839 --> 00:13:48.320
It's important to note here is that this is at production scale with eight replicas.

s115
00:13:48.320 --> 00:14:00.160
And you can see that the combined optimizations were measured to be much higher than the baseline scaling into almost hundreds of thousands of tokens per second.

s116
00:14:02.480 --> 00:14:11.920
So the takeaways kind of the principles for benchmarking validity based off of the pitfalls that Ashok brought up earlier.

s117
00:14:11.920 --> 00:14:21.920
At production scale, you need client concurrency and you need observability into your client's behavior and its ability to meet your configuration.

s118
00:14:21.920 --> 00:14:35.519
The metric fidelity allows you to actually observe your client's behavior as well as your system under test and understand that your scenario was accurately executed and your performance results were valid.

s119
00:14:35.519 --> 00:14:49.600
The stochcastic variables uh and non-determinism or determin determinism that you set amongst your run needs to reflect your real world demands uh for your workload.

s120
00:14:49.600 --> 00:14:54.480
And most importantly, your data sets do truly matter.

s121
00:14:54.480 --> 00:15:01.199
Your workloads uh need to be as close to what you are intending to test as possible.

s122
00:15:01.199 --> 00:15:06.079
And uh we have examples in the workload catalog.

s123
00:15:06.079 --> 00:15:12.079
So here we have three links to some of the things that we've presented on here before.

s124
00:15:12.079 --> 00:15:17.519
Inferencepf is our benchmarking tool and there's the git repo for it.

s125
00:15:17.519 --> 00:15:24.000
Llmd is a project that we work under and inferencepf under it in LLMD benchmark.

s126
00:15:24.000 --> 00:15:39.040
uh LLMD is for production scale inference and then LLMD Prism which was that UI we showed for the benchmarking results uh and the workload catalog that defines some of these workloads.

s127
00:15:41.360 --> 00:15:47.680
So that will answer questions after but we appreciate your time.

s128
00:15:49.105 --> 00:15:51.105
[applause]

s129
00:16:04.826 --> 00:16:06.826
[music]