Stop Guessing Your vLLM Config: Benchmarking LLMs on Runpod - Chris Vela, Runpod

AI Engineer · 16 min · 200 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12My name is Chris Bella.
00:13I work at RunPod.
00:14I'm a forward deploy engineer.
00:16Today I'm going to be talking about benchmarking VLMs.
00:20Kind of like the struggle that people have when like especially like a new model comes out.
00:24Just some like learnings that we've had like internally at RunPod and with like customers.
00:29So today I'm going to be showing you essentially like a dashboard that is built on top of like VLM.
00:36It's using our like pods actually.
00:37So to give you like a little more context of like what RunPod does is we're in the new cloud.
00:43We make it really really easy for you to rent GPUs and starting from like R&D to like more service production workflows.
00:52A lot of times what I see with customers especially with VLM is like someone wants you know the latest model.
00:57They don't know how to like spin it up or they they want to start going to production and they don't know the the knobs to like touch.
01:03They're they're just using whatever like the base config and I'm guessing a lot of you probably seen like VLM recipes before.
01:10You know it's a good start especially but like especially in this world we're always moving fast.
01:15So how do you get to more production level style of code and at the same time not just like vibing it.
01:22It's more like how do you actually have data?
01:24How do you show you know your manager like your staff engineer that hey this is actually better.
01:29This is actually working correctly.
01:32So really what I'm trying to say is like I've seen a lot everyone's just guessing.
01:38Like you know you're a lot of times I see people are using the wrong VLM version.
01:42They might they're probably not using like the nightly version.
01:45They have the wrong size for their context window.
01:48It really depends like I I think it always is a step back.
01:52It's like what are you trying to achieve with your your LM model?
01:56Are you is it going to be a coding agent?
01:57Is it more kind of like you're just um you know, doing summaries, like very very basic tasks.
02:03That's usually the first question I'd be asking before you you know, getting Kim K 3, having you know, a million long context.
02:11Like that's not always necessary.
02:13Uh and at the same time it's like how many users are going to be using this?
02:17Um a lot of times people are like going everyone's trying to always get like the the smallest amount of GPU for the cheapest price.
02:25But if you're going to be going to a more concurrent users, you need to start thinking about like how am I going to be utilizing the KB cache?
02:33Uh so all this is there's all these little like optimization knobs that you need to like figure out before you go to production.
02:39And a lot of times we're trying to move fast.
02:41So um Yeah, like like I said, like RunPod like we like have very much like, you know, we make it very very easy.
02:49It's like a VM.
02:50Uh it's dockerized.
02:51Uh you can bring your own image.
02:53But at the end of the day like um like who here has used like v l m
02:57before?
02:59Okay, so you know, like when you're looking for like like especially like Kim K 3, like if you you use like a version of like, you know, 0.27, it's not going to be working.
03:08Like like we're moving so fast the code is not getting as fast.
03:12Like you need to have the right version.
03:13And a lot of times that turns into kind of like a CD CI CD process that um especially if people who don't understand what's happening uh just start struggling.
03:23They they start seeing errors and they're like, "Why is this not working?"
03:25You know, it the the layers within it are not they haven't been written or they're it's very in development code.
03:32Uh and at the same time that's actually kind of a problem is like Kim K 3 comes out, you know, the next day I would not put that on production just yet.
03:40Uh you're going to be seeing a lot of probably hallucinations.
03:43Uh it's just not ready in that state, I would say.
03:47Um so what I propose um so uh I created this thing called it's called BenchPod.
03:52It's going to be a template.
03:53It hasn't come out yet.
03:55Um so it's essentially built on top of like uh v l m. Um it's a whole metric system.
04:01Uh you can benchmark very easy.
04:02And a lot of you who are familiar with like v l m you can benchmark already, but this kind of um centralizes everything, so you can be able to see everything, be able to change the config
04:13very quickly.
04:14Uh that's like a very important thing that I I notice is that uh especially with people who use like our uh our pods is you set up a config, but then you have to restart the image.
04:25You have to wait for cold starts to happen.
04:27It's all this loading, and you're not moving as fast.
04:29You're waiting like 30 minutes like for Kim A I 3 to like download the weights, load into GPU.
04:35Uh it's kind of wasted time.
04:37I mean, if you have an agent, you know, you really don't care, but um that is a pain point especially when we're trying to move fast.
04:44And at the same time it's like the cool thing about Runpod is you can find these machines.
04:49So, if you have that question, it's like I want to run Kim A I 3 on a B300, we have B300s, and you can start going from like not just like reading about it.
04:57You can actually start testing it and seeing if, you know, there's truth to all these metrics.
05:03So, yeah, so like bench like like I said, like bench pod is more like um it's it's built like a of a a docker image of um um v l m open a b i. We're going to be having every version already.
05:16Um we just have a little like like payload that like pretty much creates this whole fast a p i application.
05:22Uh you can use it for your agents.
05:24Uh the cool thing is you can serve uh you can pretty much specify which uh models you want to have.
05:29We have a whole watchdog, too, so it like self-heals, too.
05:32So, uh if you were actually starting to go to more production levels, uh you can actually uh you don't have to worry about it.
05:38And then like I will show you uh like this past Sunday uh Kim A I went down on our instance, and it uh self-healed.
05:46And, you know, we have we have right now like four nodes running it, so none of like our internal users saw any problem.
05:53It was a production ready um didn't have to like think about it too much.
05:57And at the end of day, yeah, like it's it it makes it much more sim- uh simple process to uh have production work workflows.
06:07So, I'm guessing a lot of people I'm I'm going to show you Kim and K3 because we've been using that a lot.
06:12Um I'm guessing a lot of people here are very familiar with Kim and K3.
06:16Um so, you know, 1 1 million contacts.
06:19Uh one thing that's interesting on 1B 300, uh can't get to the 1 million.
06:23You got to use half of it.
06:24But one one thing I've noticed especially with our team, like we're using it a lot for coding uh agents.
06:30Um that is very much um enough to do long task.
06:36Um It's utilizing KB cache uh very well, too.
06:40So, um in terms of speed of like time to first second, um it is very very very quick.
06:45And like in the demo, I'll show you too with like my coding agent running this Kim and uh model on RunPod.
06:51So, this is like our dashboard.
06:54Um It has a lot of metrics.
06:56You know, it can it look looks like a lot, but uh essentially, you know, we're always thinking about like the GPU, like, you know, have a lot of like GPU
07:04uh metrics, you know, and what CUDA version you're on, which is very very important.
07:07You know, if you're running Kim and it doesn't work on a 12 8 CUDA, it works on 13.0.
07:12Uh and really it's what we have is like the best we can we we can look at benchmarking, look at all our data, we can like playground with it.
07:19So, like before you even put it into your like coding agent or uh just hitting the OpenAI uh response, you can start testing real quick if this is working.
07:29And just a lot of like more CICD process of like, you know, how long are my cold starts happening?
07:35Um how are how are my metrics looking um through actual traffic?
07:40So, I'm going to show you guys the demo.
07:41So, So, really like I'm going to show you real quick like RunPod itself.
07:48So, um this will So, we don't have this set up yet.
07:52Like I said, um it'll be up probably at the end of the month.
07:56Uh but really all we have is just like a Docker image.
07:59We can be able to uh put that in right there.
08:03If it loads correctly.
08:05But uh Uh, yeah, so right now there's no B300s, but uh, I have one running right now.
08:09That's the fun thing about working in this industry.
08:12Uh, you know, uh, supply can be uh, dependent.
08:16Uh, but
08:17[laughter]
08:18But yeah, it's in in the sense of that like, you know, we make it really easy.
08:23You can put storage volume.
08:24Uh, you deploy it, you know, we have a part two so you can deploy when you're available.
08:28So, you know, if you were seeing the screen like this, it will deploy whenever it get it get started.
08:35Uh, so that's like literally the process.
08:36You click that button, benchpod like starts up.
08:40Um, and I can show you like right here.
08:46I have it on the wrong one, but I'll show you guys here.
08:49So, like this is like this is our actual like Kimi running currently.
08:54Um, so a cool thing is like you know, you you start like it has a whole air scan so you have actual data to know like what is happening.
09:03Um, cool thing is we have live metrics, too.
09:05As like things are running.
09:06Uh, all of this is also like backed up in like storage data.
09:09So, like it's all um, um, backed up no matter what.
09:13Um, it's all face fast API calls so your agent could easily just call all this data.
09:17You can actually just serve everything.
09:19Like you don't even have to use the UI.
09:21I I think I'm a big proponent of like as we go to agents it's like I still want to have like some awareness of what's happening and having this gives me a good like like eagle's eye view of like what is happening with my uh, Kimi K3 K3 instance.
09:35Uh, but yeah, like you can see here like I was going to show you like uh, this past weekend we had a failure with a GPU.
09:42Uh, we were able to fix it, but uh, being able to just do a cold start time like 9 minutes on it like K3s pretty pretty good.
09:50Um, and the cool thing here is that we have the serve so you can start out experiments and essentially like apply them.
09:59It'll restart real quick.
10:00Um, you can it's literally just like the config YAML file from like VLM.
10:04And um, so like for this like little demo what I tested was we were just changing the max number of batch tokens uh, that were sent through.
10:13And I kind of wanted to show you guys of like, you know, what what kind of like speed ups would happen with this.
10:19Um I'll go right here.
10:22So, I run these before because we're not going to spend like 20 minutes just looking at benchmarks.
10:29So, these benchmarks are running like concurrency.
10:31So, they're they're running um you know, starting with like just one concurrent user and then eight, 64, 192.
10:40And the cool thing is like we can be at we can actually compare multiple runs against each other.
10:46You can kind of see like even just increasing the match the the number of batch tokens doesn't really change that much the like time to first second.
10:56But, interesting thing that I did notice of increasing that is like the inner token latency actually decreases quite a bit when you increase that batch tokens.
11:07Which means it's like the request itself, if it's especially running like a long like job essentially, it's actually much faster finishing the job than the base like config that like you find on VLM recipe.
11:22The cool thing is is like you can add this to an agent and just start going crazy with it.
11:27And you can start getting those answers much much quicker, have the data, and move from this if you have it like on a pod, you can start moving to, you know, a multi-node
11:38cluster.
11:39You can move to serverless, too.
11:41So, if you're going to be like scaling up and down that that this is a perfect way of like actually getting data.
11:49And just to give you like a context on our serverless, like serverless a lot of people will do the same thing.
11:53They they just put whatever the base config is and then are struggling as like they're waiting for workers to start up.
12:01This actually gets you the real data.
12:03You're now ready to go to production.
12:05Makes it really really simple.
12:08And just gives It's like I always say, it's basing everything in facts.
12:13And I can like show you guys right here.
12:15Like this is actually like Kenny writing.
12:17So I was just like, what is a bench pod?
12:21I misspelled it.
12:22Sorry.
12:24But that this is little command instance running through it.
12:28Um yeah, and Um you know, it is pretty pretty quick.
12:32Like like I would say like there's like a probably like 50 users right now on our internal one and we most of us are getting off cloud and just using the and we have other instances with like GLM,
12:43uh deep seek, uh quant 3.8. Um but yeah, like I think in the future we're looking to more at like smart routers.
12:51Like how how do we host all this without being reliant on someone else?
12:56Um Yeah, so let me go back to our uh thing real quick.
13:04So some like important data like we were just talking about it.
13:08If anyone wants to screenshot that.
13:10But uh Like I said, like you can you can specify the experiment yourself too.
13:14So if you were like, I'm going to have like 500 people using this all the time.
13:18Like I want to like see how well this is going to be.
13:21Um But at the end of the day, like I said, it is how do we go from R&D to production?
13:28And I think this is where like bench pod will make it a lot more easier for people to use.
13:33Um So like at the end of the day, like what I want you guys to like know is like um you know, pick a pod, get like a GPU machine.
13:42You have your model, compare configs.
13:44Don't just start with like the VLN recipe.
13:47The VLN recipe is proven, but it's not production ready.
13:51Um and then test.
13:54Have your agent tested.
13:55And then whatever, you know, based on what you need, if it's ITL, if you need TTF, TPOT, like metric, like optimize on that.
14:03And then ship that winner.
14:05Go to service, scale out, and now you can like um Now you have a working LLM model that can be used internally, can be used for our Um Yeah, it's it just makes it very very simple.
14:21But yeah, one thing I would say especially with like Kimmy that I've noticed is um it's not always necessary for a team.
14:31Like like a big thing that I think like most people don't think about is that um you can use a smaller model.
14:40You know, it it's a B 200 right now like a node.
14:43That's like on our on our platform like that's going to cost you 40K a month essentially.
14:49And it might go up, it might go down a little, but you know, it's a very powerful machine and I think when you're thinking about like how you're going to be
14:57um serving stuff if you want to be doing this internally or for yourselves is like what makes sense the best.
15:05And just because it's the the frontier model, maybe it doesn't make sense.
15:10Um like I said like like I run a deep six on a 4 H 200s and does a really good job.
15:17Kimmy's more for like long tasks that I'm trying to solve something like very very detailed.
15:23Um but like I said, the other thing is like making sure your like tooling choice is like on there.
15:27That's a big thing.
15:29Context like I I'm not going to be going through too much about like context like wikis and all that, but the ability to have all that and like AI governance
15:39is the next step that we are going to be going with RunPod, but just something to be always be thinking about especially if you're going to be self-hosting your um
15:47your LLM models.
15:49But yeah, is there any questions?
15:50Like And we'll also be at the booths.
15:54So like downstairs.
15:56But yeah, add me on LinkedIn if you want to talk more.
15:58I'm always exploring every new model like every week.
16:02So always have some like gotchas that I've been learning through the through the time I've been doing all this work.
16:09And yeah, also scan the card QR code so you can get you get some credits and you can test out RunPod.
16:18And and said, the template will be out like at the end of the month.
16:20So you can go play around with it.