Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google

AI Engineer · 16 min · 129 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:12Hi everyone, welcome to our talk on um our LLM performance benchmarks reliable.
2. 00:21A little bit about us.
3. 00:22I am Ashok Chandra Seeker.
4. 00:23I'm a staff software engineer at Google.
5. 00:26I work on inference performance evaluation and optimization.
6. 00:29Um and I lead a couple of open source projects.
7. 00:32One is called inference perf uh which is a benchmarking tool to do reliable performance benchmarks and uh I'm also the sig lead for LLMD benchmarking.
8. 00:41LLMD is a distributed inference framework um that makes production scale inference possible.
9. 00:47Hi everyone, I'm Jason Kroger.
10. 00:49I'm a software engineer at Google.
11. 00:51Uh I'm also a co-maintainer of inference Perf and a few of the sub projects that Ashok brought up.
12. 00:58uh and I work on inference performance and benchmarking.
13. 01:03Okay, let's get started.
14. 01:04Um let's look a little bit about the how the benchmark ecosystem looks like.
15. 01:08Right?
16. 01:09You have your model server frameworks.
17. 01:11Uh these are VLM, SGLAN, um and other model servers and all of these have some benchmark capability within them.
18. 01:18Right?
19. 01:18These are primarily Python scripts and are developer focused benchmarks to see how you can measure the performance of your model server itself.
20. 01:25And then you have your competitive analysis tools.
21. 01:28Uh these are MLPF, semi analysis, artificial analysis and so on.
22. 01:33Right?
23. 01:34They mainly aim for competitive performance benchmarks to compare like chip and accelerator performance.
24. 01:39Uh and then you have your typical web benchmarks.
25. 01:42Uh these are like locus, graphfana, ksix um and so on.
26. 01:46These mainly focus on highcale http benchmarks, right?
27. 01:50uh and then you have your um last segment which is the production scale LLM benchmarks right these are to actually benchmark uh your production inference serving stack and that is our focus today
28. 02:03right we'll be focusing mainly on inference perf and how we solve this uh production scale benchmark problem so if you have run a benchmark before uh it typically looks like this right um you have some sort of benchmark hardness
29. 02:18uh and then you specify what model you are bench benchmarking the number of prompts you want to run, what is the input output sequence length uh and the request rate or the load you want to send,
30. 02:28right?
31. 02:28Uh and your output looks something uh like what is on the right.
32. 02:32Uh this is basically your uh input token throughput, output token throughput, some latency metrics, time to first token, time per output token and so on.
33. 02:41Um so what are some issues with a simple benchmark like this?
34. 02:46Right?
35. 02:47So if you want to actually benchmark production scale workloads um here I have LLMD inference stack as an example right uh you can have uh online serving you can have batch workloads
36. 02:59and if you see the inference pool below uh there are like lot of servers that are running right and then you have like complex configurations like pre-fill decode disagregation
37. 03:09um and workload autoscaling and other things that are going on under the hood and usually the scale is much larger right so your um normal benchmark harnesses runs into issues when you try to benchmark a setup like this.
38. 03:22And if you look at like the key characteristics of what we want out of a production scale benchmark, we need to be able to do high load uh which is limited in a lot of tools out there.
39. 03:31We need to be able to simulate real world workloads, right?
40. 03:34Um what use it is if it is just some synthetic workload that is not accurately representing uh what your customers are going to run.
41. 03:41Uh and then metrics fidelity is very important, right?
42. 03:44Are the metrics accurate and how well they work?
43. 03:49Uh this is like the set of metrics that LLMD measures uh by default.
44. 03:54Uh I just pulled it from the website there.
45. 03:57Uh as you can see it's not just like a single QPS that you are running, right?
46. 04:02You are sweeping a a list of various loads and uh you try to measure what the baseline is and what optimizations you are making and what the difference there is.
47. 04:12um you need to find the right point where the server gets saturated so you know the right optimal uh point to run your servers on uh to maximize performance and to save costs
48. 04:23um and things like uh SLOs's become more important right what is your um time to first token P90 SLO and are you conformant to that SLO so when you run like um normal benchmark like we saw before what are some of the pitfalls that you run into right uh we have been uh running benchmarks for a couple
49. 04:43of years.
50. 04:44So we run into all sort of uh different results that people share and a lot of times we aren't able to reproduce the results that are shared by other people.
51. 04:51Right?
52. 04:52So that is what motivated this talk.
53. 04:54Um so these four common uh things that we see as an issue, right?
54. 04:59One is accurate uh metrics and two observability into your benchmark tool itself.
55. 05:05Um do you know if your benchmark harness is actually failing?
56. 05:08Is it not able to maintain the load?
57. 05:10Um and three reproducibility.
58. 05:13Uh there is some inherent uh randomness uh in like the data sets that you use.
59. 05:17So how do you make sure it is reproducible and four the data set quality itself.
60. 05:24So this is an experiment we ran.
61. 05:26Um we asked like different benchmark harness to generate 200 QPS and this was the result right.
62. 05:33Um so a couple of things I want to point out.
63. 05:36Um, Python has this global interpreter lock GIL if you have been working with Python, you know that which makes everything uh sort of single threaded.
64. 05:44So even when you have like a multiCPU mission, a lot of times you are limited by the performance of a single CPU uh when you're CPU bound especially, right?
65. 05:54Uh so this kind of shows a single process um benchmark harness and a multiprocess harness and how the QPS you are able to achieve differs based on it, right?
66. 06:03when you run with a really uh small shad core mission, you can see that even when you request 200 QPS, you are only getting 38 QPS and then you give it a bigger mission and then some of these uh single process harness they cap out at like 170 QPS, right?
67. 06:17This is a much more powerful machine.
68. 06:19Um but it is a problem because you ask for 200 QPS and then you don't know whether it actually delivered it.
69. 06:25It will just say I ran it these are the numbers.
70. 06:27So you think okay you ran 200 QPS but in fact you you have not.
71. 06:33Uh the other issue that comes out of it is the latency inflation right if your server is saying okay this is how much QPS I was able to run and this was the accurate numbers that is one thing but if your uh benchmark harness is actually inflating latency right because it's thrashing trying to collect all the streaming
72. 06:49token requests um in one of the tests we noticed like u the delay was up to 58 seconds.
73. 06:55So you might look at this and go oh my server is bottlenecked right it's not able to handle all the requests but in fact it's actually your benchmark client that is inflating the latency
74. 07:03right uh we ran like a th000 QPS test uh when your when your benchmark harness is actually able to scale out you can see there is very minimal latency right this
75. 07:13simulated server so there shouldn't be any latency at all um and like I said there are like other variables that go into it right in one of the benchmarks
76. 07:25um someone shared and they said, "Hey, we are getting 20% better throughput."
77. 07:28Then we looked into it and we found out the benchmark harness were setting the model temperature to zero, right?
78. 07:34Which means your model outputs are a lot more deterministic and it was able to turn out a higher throughput than what you would normally see in like a real workload, right?
79. 07:42Where your model temperature is somewhere around 0.7. Um another thing is like we used a a shar GPD data set the same data set across two different benchmark harness
80. 07:53and they produce different input tokens right this is because they sample them differently they truncate them uh differently so as a user you don't have insight into this right you run it you trust the numbers it produces but they are wildly different
81. 08:08uh and there is much more right do you actually force it to generate till the end of sequence are you looking at prefix cache rates how do you do mult multi-turn
82. 08:16uh replay via benchmarks and uh how how do you actually get high fidelity uh on the actual workload that would resemble your production workload right so the main thing I wanted to convey here is like a lot of times you diagnose it as a your server or inference stack problem but in a lot of cases it could
83. 08:35be your benchmark harness so what is the solution to this how do we actually do reproducible benchmarks Jason here will take you over Thanks Ashook.
84. 08:48Uh so yeah, how do you how do you solve these problems?
85. 08:52Uh we al together inferencepf uh the CNCF project spanned out of Kubernetes working group serving uh to provide like a standardized place for us to work with the community and
86. 09:08solve some of these issues together.
87. 09:11uh it enables the ability to have like a userdefined declarative configuration uh that allows you to have clear reproducibility across runs.
88. 09:25We also added a load generator uh that solves the GIL problem in Python uh across multiple processes and reports those client metrics back along with server metrics to ensure that you have the highest metric fidelity and you're able to actually observe when
89. 09:45your tool is having an issue versus your system under test.
90. 09:52So, first going over the load generator, uh you see that the main process actually cues requests based off of the planned time that they need to execute, which is based off of your configuration.
91. 10:09This may be in some poison process or constant rate or maintaining a constant number of concurrent requests.
92. 10:18This request Q channel is then spread across multiple processes which pull and ensure that they execute with minimum overhead but then also observability about when they execute versus their plan time.
93. 10:34And you can see this working at scale.
94. 10:36So this is a comparison across other tools uh some being the HTTP scaled tools like K6.
95. 10:44uh but you see that even at 5,000 QPS inference perf was able to keep up due to this architecture and most importantly actually report that it was able to keep up.
96. 11:00Other portion is configuration.
97. 11:02So earlier I show showed a brief example of how you might simply run a benchmarking tool.
98. 11:09And here on the left you can see a simple example running a random data set against an endpoint.
99. 11:15But the actual configuration that we have in front of inference perf is very detailed with a lot of knobs that allow you to accurately uh test your configuration off of your workloads.
100. 11:30You can see here on the right that this is uh configuration for conversation replay where you're able to configure not only like the input output length but their distributions
101. 11:40etc. And further beyond just the ability to configure a single run, we've actually worked together to have a published set of some of these workloads and configuration of inference perf that are tied to state-of-the-art
102. 11:59inference workloads.
103. 12:01For example, here in the workload catalog that we've put out, you're able to access standard multi-turn uh generation, tree of thought, agentic generation uh as well as batch summarization
104. 12:17and others.
105. 12:19Each one of these has a simple definition uh kind of in natural language that allows you to understand what the scenario is.
106. 12:29But beyond that there's also pretty detailed configuration metrics not only in inference perf configuration but in generic terms so that this can actually be shared across tools and have a place for standardization
107. 12:44of these workloads.
108. 12:50So kind of the culmination of these things uh leads us to actual results that we can clearly display.
109. 12:58Uh and here's a screenshot from Prism which is a UI we have for sharing not only those workloads that I showed before but also some benchmarking results.
110. 13:08Uh and Prism is a part of the LLMD project.
111. 13:13Here you can see a benchmark result for the agentic code generation workload that we showed before on TPUs.
112. 13:23Uh these three lines you see here show you The difference between combined optimizations is the green line and a baseline that is just a simple Kubernetes service instead of in front of multiple
113. 13:39model server replicas.
114. 13:41It's important to note here is that this is at production scale with eight replicas.
115. 13:48And you can see that the combined optimizations were measured to be much higher than the baseline scaling into almost hundreds of thousands of tokens per second.
116. 14:02So the takeaways kind of the principles for benchmarking validity based off of the pitfalls that Ashok brought up earlier.
117. 14:11At production scale, you need client concurrency and you need observability into your client's behavior and its ability to meet your configuration.
118. 14:21The metric fidelity allows you to actually observe your client's behavior as well as your system under test and understand that your scenario was accurately executed and your performance results were valid.
119. 14:35The stochcastic variables uh and non-determinism or determin determinism that you set amongst your run needs to reflect your real world demands uh for your workload.
120. 14:49And most importantly, your data sets do truly matter.
121. 14:54Your workloads uh need to be as close to what you are intending to test as possible.
122. 15:01And uh we have examples in the workload catalog.
123. 15:06So here we have three links to some of the things that we've presented on here before.
124. 15:12Inferencepf is our benchmarking tool and there's the git repo for it.
125. 15:17Llmd is a project that we work under and inferencepf under it in LLMD benchmark.
126. 15:24uh LLMD is for production scale inference and then LLMD Prism which was that UI we showed for the benchmarking results uh and the workload catalog that defines some of these workloads.
127. 15:41So that will answer questions after but we appreciate your time.
128. 15:49[applause]
129. 16:04[music]