WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=P_0_Th_ZzIg

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/P_0_Th_ZzIg.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.560 --> 00:00:13.800
My name is Chris Bella.

s3
00:00:13.800 --> 00:00:14.440
I work at RunPod.

s4
00:00:14.440 --> 00:00:16.320
I'm a forward deploy engineer.

s5
00:00:16.320 --> 00:00:20.200
Today I'm going to be talking about benchmarking VLMs.

s6
00:00:20.200 --> 00:00:24.880
Kind of like the struggle that people have when like especially like a new model comes out.

s7
00:00:24.880 --> 00:00:29.520
Just some like learnings that we've had like internally at RunPod and with like customers.

s8
00:00:29.520 --> 00:00:36.240
So today I'm going to be showing you essentially like a dashboard that is built on top of like VLM.

s9
00:00:36.240 --> 00:00:37.880
It's using our like pods actually.

s10
00:00:37.880 --> 00:00:43.640
So to give you like a little more context of like what RunPod does is we're in the new cloud.

s11
00:00:43.640 --> 00:00:52.560
We make it really really easy for you to rent GPUs and starting from like R&D to like more service production workflows.

s12
00:00:52.560 --> 00:00:57.520
A lot of times what I see with customers especially with VLM is like someone wants you know the latest model.

s13
00:00:57.520 --> 00:01:03.240
They don't know how to like spin it up or they they want to start going to production and they don't know the the knobs to like touch.

s14
00:01:03.240 --> 00:01:10.160
They're they're just using whatever like the base config and I'm guessing a lot of you probably seen like VLM recipes before.

s15
00:01:10.160 --> 00:01:15.960
You know it's a good start especially but like especially in this world we're always moving fast.

s16
00:01:15.960 --> 00:01:22.880
So how do you get to more production level style of code and at the same time not just like vibing it.

s17
00:01:22.880 --> 00:01:24.640
It's more like how do you actually have data?

s18
00:01:24.640 --> 00:01:29.640
How do you show you know your manager like your staff engineer that hey this is actually better.

s19
00:01:29.640 --> 00:01:32.240
This is actually working correctly.

s20
00:01:32.240 --> 00:01:38.680
So really what I'm trying to say is like I've seen a lot everyone's just guessing.

s21
00:01:38.680 --> 00:01:42.280
Like you know you're a lot of times I see people are using the wrong VLM version.

s22
00:01:42.280 --> 00:01:45.920
They might they're probably not using like the nightly version.

s23
00:01:45.920 --> 00:01:48.920
They have the wrong size for their context window.

s24
00:01:48.920 --> 00:01:52.120
It really depends like I I think it always is a step back.

s25
00:01:52.120 --> 00:01:56.360
It's like what are you trying to achieve with your your LM model?

s26
00:01:56.360 --> 00:01:57.880
Are you is it going to be a coding agent?

s27
00:01:57.880 --> 00:02:03.612
Is it more kind of like you're just um you know, doing summaries, like very very basic tasks.

s28
00:02:03.612 --> 00:02:11.292
That's usually the first question I'd be asking before you you know, getting Kim K 3, having you know, a million long context.

s29
00:02:11.292 --> 00:02:13.772
Like that's not always necessary.

s30
00:02:13.772 --> 00:02:17.092
Uh and at the same time it's like how many users are going to be using this?

s31
00:02:17.092 --> 00:02:25.692
Um a lot of times people are like going everyone's trying to always get like the the smallest amount of GPU for the cheapest price.

s32
00:02:25.692 --> 00:02:33.132
But if you're going to be going to a more concurrent users, you need to start thinking about like how am I going to be utilizing the KB cache?

s33
00:02:33.132 --> 00:02:39.252
Uh so all this is there's all these little like optimization knobs that you need to like figure out before you go to production.

s34
00:02:39.252 --> 00:02:41.772
And a lot of times we're trying to move fast.

s35
00:02:41.772 --> 00:02:49.332
So um Yeah, like like I said, like RunPod like we like have very much like, you know, we make it very very easy.

s36
00:02:49.332 --> 00:02:50.572
It's like a VM.

s37
00:02:50.572 --> 00:02:51.972
Uh it's dockerized.

s38
00:02:51.972 --> 00:02:53.452
Uh you can bring your own image.

s39
00:02:53.452 --> 00:02:57.612
But at the end of the day like um like who here has used like v l m

s40
00:02:57.612 --> 00:02:59.052
before?

s41
00:02:59.052 --> 00:03:08.572
Okay, so you know, like when you're looking for like like especially like Kim K 3, like if you you use like a version of like, you know, 0.27, it's not going to be working.

s42
00:03:08.572 --> 00:03:12.092
Like like we're moving so fast the code is not getting as fast.

s43
00:03:12.092 --> 00:03:13.892
Like you need to have the right version.

s44
00:03:13.892 --> 00:03:23.092
And a lot of times that turns into kind of like a CD CI CD process that um especially if people who don't understand what's happening uh just start struggling.

s45
00:03:23.092 --> 00:03:25.252
They they start seeing errors and they're like, "Why is this not working?"

s46
00:03:25.252 --> 00:03:32.092
You know, it the the layers within it are not they haven't been written or they're it's very in development code.

s47
00:03:32.092 --> 00:03:40.252
Uh and at the same time that's actually kind of a problem is like Kim K 3 comes out, you know, the next day I would not put that on production just yet.

s48
00:03:40.252 --> 00:03:43.452
Uh you're going to be seeing a lot of probably hallucinations.

s49
00:03:43.452 --> 00:03:47.252
Uh it's just not ready in that state, I would say.

s50
00:03:47.252 --> 00:03:52.852
Um so what I propose um so uh I created this thing called it's called BenchPod.

s51
00:03:52.852 --> 00:03:53.572
It's going to be a template.

s52
00:03:53.572 --> 00:03:55.412
It hasn't come out yet.

s53
00:03:55.412 --> 00:04:01.052
Um so it's essentially built on top of like uh v l m. Um it's a whole metric system.

s54
00:04:01.052 --> 00:04:02.492
Uh you can benchmark very easy.

s55
00:04:02.492 --> 00:04:13.854
And a lot of you who are familiar with like v l m you can benchmark already, but this kind of um centralizes everything, so you can be able to see everything, be able to change the config

s56
00:04:13.854 --> 00:04:14.934
very quickly.

s57
00:04:14.934 --> 00:04:25.254
Uh that's like a very important thing that I I notice is that uh especially with people who use like our uh our pods is you set up a config, but then you have to restart the image.

s58
00:04:25.254 --> 00:04:27.494
You have to wait for cold starts to happen.

s59
00:04:27.494 --> 00:04:29.934
It's all this loading, and you're not moving as fast.

s60
00:04:29.934 --> 00:04:35.814
You're waiting like 30 minutes like for Kim A I 3 to like download the weights, load into GPU.

s61
00:04:35.814 --> 00:04:37.134
Uh it's kind of wasted time.

s62
00:04:37.134 --> 00:04:44.894
I mean, if you have an agent, you know, you really don't care, but um that is a pain point especially when we're trying to move fast.

s63
00:04:44.894 --> 00:04:49.334
And at the same time it's like the cool thing about Runpod is you can find these machines.

s64
00:04:49.334 --> 00:04:57.414
So, if you have that question, it's like I want to run Kim A I 3 on a B300, we have B300s, and you can start going from like not just like reading about it.

s65
00:04:57.414 --> 00:05:03.414
You can actually start testing it and seeing if, you know, there's truth to all these metrics.

s66
00:05:03.694 --> 00:05:16.214
So, yeah, so like bench like like I said, like bench pod is more like um it's it's built like a of a a docker image of um um v l m open a b i. We're going to be having every version already.

s67
00:05:16.214 --> 00:05:22.414
Um we just have a little like like payload that like pretty much creates this whole fast a p i application.

s68
00:05:22.414 --> 00:05:24.014
Uh you can use it for your agents.

s69
00:05:24.014 --> 00:05:29.894
Uh the cool thing is you can serve uh you can pretty much specify which uh models you want to have.

s70
00:05:29.894 --> 00:05:32.334
We have a whole watchdog, too, so it like self-heals, too.

s71
00:05:32.334 --> 00:05:38.374
So, uh if you were actually starting to go to more production levels, uh you can actually uh you don't have to worry about it.

s72
00:05:38.374 --> 00:05:46.054
And then like I will show you uh like this past Sunday uh Kim A I went down on our instance, and it uh self-healed.

s73
00:05:46.054 --> 00:05:53.014
And, you know, we have we have right now like four nodes running it, so none of like our internal users saw any problem.

s74
00:05:53.014 --> 00:05:57.214
It was a production ready um didn't have to like think about it too much.

s75
00:05:57.214 --> 00:06:07.575
And at the end of day, yeah, like it's it it makes it much more sim- uh simple process to uh have production work workflows.

s76
00:06:07.815 --> 00:06:12.695
So, I'm guessing a lot of people I'm I'm going to show you Kim and K3 because we've been using that a lot.

s77
00:06:12.695 --> 00:06:16.975
Um I'm guessing a lot of people here are very familiar with Kim and K3.

s78
00:06:16.975 --> 00:06:19.615
Um so, you know, 1 1 million contacts.

s79
00:06:19.615 --> 00:06:23.655
Uh one thing that's interesting on 1B 300, uh can't get to the 1 million.

s80
00:06:23.655 --> 00:06:24.855
You got to use half of it.

s81
00:06:24.855 --> 00:06:30.975
But one one thing I've noticed especially with our team, like we're using it a lot for coding uh agents.

s82
00:06:30.975 --> 00:06:36.495
Um that is very much um enough to do long task.

s83
00:06:36.495 --> 00:06:40.055
Um It's utilizing KB cache uh very well, too.

s84
00:06:40.055 --> 00:06:45.375
So, um in terms of speed of like time to first second, um it is very very very quick.

s85
00:06:45.375 --> 00:06:51.855
And like in the demo, I'll show you too with like my coding agent running this Kim and uh model on RunPod.

s86
00:06:51.855 --> 00:06:54.415
So, this is like our dashboard.

s87
00:06:54.415 --> 00:06:56.295
Um It has a lot of metrics.

s88
00:06:56.295 --> 00:07:04.295
You know, it can it look looks like a lot, but uh essentially, you know, we're always thinking about like the GPU, like, you know, have a lot of like GPU

s89
00:07:04.295 --> 00:07:07.615
uh metrics, you know, and what CUDA version you're on, which is very very important.

s90
00:07:07.615 --> 00:07:12.535
You know, if you're running Kim and it doesn't work on a 12 8 CUDA, it works on 13.0.

s91
00:07:12.535 --> 00:07:19.615
Uh and really it's what we have is like the best we can we we can look at benchmarking, look at all our data, we can like playground with it.

s92
00:07:19.615 --> 00:07:29.455
So, like before you even put it into your like coding agent or uh just hitting the OpenAI uh response, you can start testing real quick if this is working.

s93
00:07:29.455 --> 00:07:35.175
And just a lot of like more CICD process of like, you know, how long are my cold starts happening?

s94
00:07:35.175 --> 00:07:40.215
Um how are how are my metrics looking um through actual traffic?

s95
00:07:40.215 --> 00:07:41.855
So, I'm going to show you guys the demo.

s96
00:07:41.855 --> 00:07:48.895
So, So, really like I'm going to show you real quick like RunPod itself.

s97
00:07:48.895 --> 00:07:52.815
So, um this will So, we don't have this set up yet.

s98
00:07:52.815 --> 00:07:56.215
Like I said, um it'll be up probably at the end of the month.

s99
00:07:56.215 --> 00:07:59.255
Uh but really all we have is just like a Docker image.

s100
00:07:59.255 --> 00:08:03.535
We can be able to uh put that in right there.

s101
00:08:03.535 --> 00:08:05.015
If it loads correctly.

s102
00:08:05.015 --> 00:08:09.897
But uh Uh, yeah, so right now there's no B300s, but uh, I have one running right now.

s103
00:08:09.897 --> 00:08:12.737
That's the fun thing about working in this industry.

s104
00:08:12.737 --> 00:08:16.817
Uh, you know, uh, supply can be uh, dependent.

s105
00:08:16.817 --> 00:08:17.805
Uh, but

s106
00:08:17.805 --> 00:08:18.897
[laughter]

s107
00:08:18.897 --> 00:08:23.017
But yeah, it's in in the sense of that like, you know, we make it really easy.

s108
00:08:23.017 --> 00:08:24.577
You can put storage volume.

s109
00:08:24.577 --> 00:08:28.897
Uh, you deploy it, you know, we have a part two so you can deploy when you're available.

s110
00:08:28.897 --> 00:08:35.257
So, you know, if you were seeing the screen like this, it will deploy whenever it get it get started.

s111
00:08:35.257 --> 00:08:36.817
Uh, so that's like literally the process.

s112
00:08:36.817 --> 00:08:40.217
You click that button, benchpod like starts up.

s113
00:08:40.217 --> 00:08:44.417
Um, and I can show you like right here.

s114
00:08:46.857 --> 00:08:49.017
I have it on the wrong one, but I'll show you guys here.

s115
00:08:49.017 --> 00:08:54.337
So, like this is like this is our actual like Kimi running currently.

s116
00:08:54.337 --> 00:09:03.177
Um, so a cool thing is like you know, you you start like it has a whole air scan so you have actual data to know like what is happening.

s117
00:09:03.177 --> 00:09:05.377
Um, cool thing is we have live metrics, too.

s118
00:09:05.377 --> 00:09:06.617
As like things are running.

s119
00:09:06.617 --> 00:09:09.777
Uh, all of this is also like backed up in like storage data.

s120
00:09:09.777 --> 00:09:13.977
So, like it's all um, um, backed up no matter what.

s121
00:09:13.977 --> 00:09:17.657
Um, it's all face fast API calls so your agent could easily just call all this data.

s122
00:09:17.657 --> 00:09:19.497
You can actually just serve everything.

s123
00:09:19.497 --> 00:09:21.257
Like you don't even have to use the UI.

s124
00:09:21.257 --> 00:09:35.777
I I think I'm a big proponent of like as we go to agents it's like I still want to have like some awareness of what's happening and having this gives me a good like like eagle's eye view of like what is happening with my uh, Kimi K3 K3 instance.

s125
00:09:35.777 --> 00:09:42.817
Uh, but yeah, like you can see here like I was going to show you like uh, this past weekend we had a failure with a GPU.

s126
00:09:42.817 --> 00:09:50.817
Uh, we were able to fix it, but uh, being able to just do a cold start time like 9 minutes on it like K3s pretty pretty good.

s127
00:09:50.817 --> 00:09:59.377
Um, and the cool thing here is that we have the serve so you can start out experiments and essentially like apply them.

s128
00:09:59.377 --> 00:10:00.897
It'll restart real quick.

s129
00:10:00.897 --> 00:10:04.737
Um, you can it's literally just like the config YAML file from like VLM.

s130
00:10:04.737 --> 00:10:13.138
And um, so like for this like little demo what I tested was we were just changing the max number of batch tokens uh, that were sent through.

s131
00:10:13.138 --> 00:10:19.098
And I kind of wanted to show you guys of like, you know, what what kind of like speed ups would happen with this.

s132
00:10:19.098 --> 00:10:22.698
Um I'll go right here.

s133
00:10:22.698 --> 00:10:29.218
So, I run these before because we're not going to spend like 20 minutes just looking at benchmarks.

s134
00:10:29.218 --> 00:10:31.538
So, these benchmarks are running like concurrency.

s135
00:10:31.538 --> 00:10:40.138
So, they're they're running um you know, starting with like just one concurrent user and then eight, 64, 192.

s136
00:10:40.298 --> 00:10:46.658
And the cool thing is like we can be at we can actually compare multiple runs against each other.

s137
00:10:46.658 --> 00:10:56.018
You can kind of see like even just increasing the match the the number of batch tokens doesn't really change that much the like time to first second.

s138
00:10:56.018 --> 00:11:07.818
But, interesting thing that I did notice of increasing that is like the inner token latency actually decreases quite a bit when you increase that batch tokens.

s139
00:11:07.818 --> 00:11:22.938
Which means it's like the request itself, if it's especially running like a long like job essentially, it's actually much faster finishing the job than the base like config that like you find on VLM recipe.

s140
00:11:22.938 --> 00:11:27.858
The cool thing is is like you can add this to an agent and just start going crazy with it.

s141
00:11:27.858 --> 00:11:38.218
And you can start getting those answers much much quicker, have the data, and move from this if you have it like on a pod, you can start moving to, you know, a multi-node

s142
00:11:38.218 --> 00:11:39.818
cluster.

s143
00:11:39.818 --> 00:11:41.058
You can move to serverless, too.

s144
00:11:41.058 --> 00:11:49.378
So, if you're going to be like scaling up and down that that this is a perfect way of like actually getting data.

s145
00:11:49.378 --> 00:11:53.738
And just to give you like a context on our serverless, like serverless a lot of people will do the same thing.

s146
00:11:53.738 --> 00:12:01.098
They they just put whatever the base config is and then are struggling as like they're waiting for workers to start up.

s147
00:12:01.098 --> 00:12:03.338
This actually gets you the real data.

s148
00:12:03.338 --> 00:12:05.578
You're now ready to go to production.

s149
00:12:05.578 --> 00:12:08.098
Makes it really really simple.

s150
00:12:08.098 --> 00:12:13.139
And just gives It's like I always say, it's basing everything in facts.

s151
00:12:13.139 --> 00:12:15.219
And I can like show you guys right here.

s152
00:12:15.219 --> 00:12:17.379
Like this is actually like Kenny writing.

s153
00:12:17.379 --> 00:12:21.499
So I was just like, what is a bench pod?

s154
00:12:21.499 --> 00:12:22.259
I misspelled it.

s155
00:12:22.259 --> 00:12:24.259
Sorry.

s156
00:12:24.539 --> 00:12:28.139
But that this is little command instance running through it.

s157
00:12:28.139 --> 00:12:32.179
Um yeah, and Um you know, it is pretty pretty quick.

s158
00:12:32.179 --> 00:12:43.419
Like like I would say like there's like a probably like 50 users right now on our internal one and we most of us are getting off cloud and just using the and we have other instances with like GLM,

s159
00:12:43.419 --> 00:12:51.059
uh deep seek, uh quant 3.8. Um but yeah, like I think in the future we're looking to more at like smart routers.

s160
00:12:51.059 --> 00:12:56.099
Like how how do we host all this without being reliant on someone else?

s161
00:12:56.099 --> 00:13:04.059
Um Yeah, so let me go back to our uh thing real quick.

s162
00:13:04.059 --> 00:13:08.459
So some like important data like we were just talking about it.

s163
00:13:08.459 --> 00:13:10.139
If anyone wants to screenshot that.

s164
00:13:10.139 --> 00:13:14.659
But uh Like I said, like you can you can specify the experiment yourself too.

s165
00:13:14.659 --> 00:13:18.139
So if you were like, I'm going to have like 500 people using this all the time.

s166
00:13:18.139 --> 00:13:21.139
Like I want to like see how well this is going to be.

s167
00:13:21.139 --> 00:13:28.059
Um But at the end of the day, like I said, it is how do we go from R&D to production?

s168
00:13:28.059 --> 00:13:33.259
And I think this is where like bench pod will make it a lot more easier for people to use.

s169
00:13:33.259 --> 00:13:42.659
Um So like at the end of the day, like what I want you guys to like know is like um you know, pick a pod, get like a GPU machine.

s170
00:13:42.659 --> 00:13:44.819
You have your model, compare configs.

s171
00:13:44.819 --> 00:13:47.419
Don't just start with like the VLN recipe.

s172
00:13:47.419 --> 00:13:51.939
The VLN recipe is proven, but it's not production ready.

s173
00:13:51.939 --> 00:13:54.499
Um and then test.

s174
00:13:54.499 --> 00:13:55.819
Have your agent tested.

s175
00:13:55.819 --> 00:14:03.619
And then whatever, you know, based on what you need, if it's ITL, if you need TTF, TPOT, like metric, like optimize on that.

s176
00:14:03.619 --> 00:14:05.139
And then ship that winner.

s177
00:14:05.139 --> 00:14:20.261
Go to service, scale out, and now you can like um Now you have a working LLM model that can be used internally, can be used for our Um Yeah, it's it just makes it very very simple.

s178
00:14:21.381 --> 00:14:31.261
But yeah, one thing I would say especially with like Kimmy that I've noticed is um it's not always necessary for a team.

s179
00:14:31.261 --> 00:14:40.821
Like like a big thing that I think like most people don't think about is that um you can use a smaller model.

s180
00:14:40.821 --> 00:14:43.621
You know, it it's a B 200 right now like a node.

s181
00:14:43.621 --> 00:14:49.741
That's like on our on our platform like that's going to cost you 40K a month essentially.

s182
00:14:49.741 --> 00:14:57.901
And it might go up, it might go down a little, but you know, it's a very powerful machine and I think when you're thinking about like how you're going to be

s183
00:14:57.901 --> 00:15:05.261
um serving stuff if you want to be doing this internally or for yourselves is like what makes sense the best.

s184
00:15:05.261 --> 00:15:10.581
And just because it's the the frontier model, maybe it doesn't make sense.

s185
00:15:10.581 --> 00:15:17.341
Um like I said like like I run a deep six on a 4 H 200s and does a really good job.

s186
00:15:17.341 --> 00:15:23.021
Kimmy's more for like long tasks that I'm trying to solve something like very very detailed.

s187
00:15:23.021 --> 00:15:27.581
Um but like I said, the other thing is like making sure your like tooling choice is like on there.

s188
00:15:27.581 --> 00:15:29.101
That's a big thing.

s189
00:15:29.101 --> 00:15:39.501
Context like I I'm not going to be going through too much about like context like wikis and all that, but the ability to have all that and like AI governance

s190
00:15:39.501 --> 00:15:47.181
is the next step that we are going to be going with RunPod, but just something to be always be thinking about especially if you're going to be self-hosting your um

s191
00:15:47.181 --> 00:15:49.141
your LLM models.

s192
00:15:49.141 --> 00:15:50.821
But yeah, is there any questions?

s193
00:15:50.821 --> 00:15:54.581
Like And we'll also be at the booths.

s194
00:15:54.581 --> 00:15:56.061
So like downstairs.

s195
00:15:56.061 --> 00:15:58.941
But yeah, add me on LinkedIn if you want to talk more.

s196
00:15:58.941 --> 00:16:02.261
I'm always exploring every new model like every week.

s197
00:16:02.261 --> 00:16:09.541
So always have some like gotchas that I've been learning through the through the time I've been doing all this work.

s198
00:16:09.541 --> 00:16:18.061
And yeah, also scan the card QR code so you can get you get some credits and you can test out RunPod.

s199
00:16:18.061 --> 00:16:20.782
And and said, the template will be out like at the end of the month.

s200
00:16:20.782 --> 00:16:24.302
So you can go play around with it.