WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=byn9PURoBNY

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/byn9PURoBNY.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.880 --> 00:00:14.400
Hello everyone.

s3
00:00:14.400 --> 00:00:15.560
My name is Gabriel.

s4
00:00:15.560 --> 00:00:23.000
I work at Korea and I'll be talking about the infrastructure that allowed us to train K2 and also how we serve it.

s5
00:00:23.000 --> 00:00:24.320
So, what is K2?

s6
00:00:24.320 --> 00:00:35.600
K2 is our pre-trained from scratch model we just released like less than a month ago and the whole idea about training this model was because we were kind of bored of AI images.

s7
00:00:35.600 --> 00:00:38.800
They're quite, you know, soulless.

s8
00:00:38.800 --> 00:00:40.240
They have no spice.

s9
00:00:40.240 --> 00:00:52.680
And the whole idea was we want to give creatives tools to to explore out of distribution extremely interesting images, do composition and like actually give tools to creatives and that was the whole idea of the model.

s10
00:00:52.680 --> 00:00:54.360
The model was trained from scratch.

s11
00:00:54.360 --> 00:00:56.960
No base checkpoint, not anything.

s12
00:00:56.960 --> 00:01:11.400
Everything done in house and also as I said, built for exploration and right now this is what you can get out of Korea 2. Very different styles and, you know, like pixel art and and like

s13
00:01:11.400 --> 00:01:14.000
like photo real and like some silly stuff.

s14
00:01:14.000 --> 00:01:18.880
Uh, and the whole idea of the model, as I said, let's explore this medium.

s15
00:01:18.880 --> 00:01:20.680
Uh, Korea 2 is open source.

s16
00:01:20.680 --> 00:01:23.400
Right now, you can go play with it.

s17
00:01:23.400 --> 00:01:24.800
There's two checkpoints.

s18
00:01:24.800 --> 00:01:26.920
We also serve in production.

s19
00:01:26.920 --> 00:01:32.320
There's a raw checkpoint we just pre-trained so people can post-train and do whatever they wish to do with it.

s20
00:01:32.320 --> 00:01:35.640
And there's also the post-train version which is the turbo one which is very very fast.

s21
00:01:35.640 --> 00:01:38.640
You can get like an image in like, I don't know, less than a second.

s22
00:01:38.640 --> 00:01:42.040
And this is just like this type of images you can get in less than a second.

s23
00:01:42.040 --> 00:01:45.120
Uh, on Hugging Face GitHub, go go play with it.

s24
00:01:45.120 --> 00:01:48.840
Also, you can go in production on korea.ai and go play with it.

s25
00:01:48.840 --> 00:01:52.120
Uh, so let's talk about how we trained this model.

s26
00:01:52.120 --> 00:01:55.000
Uh, as I said, it's going to be how we train and how we serve.

s27
00:01:55.000 --> 00:01:59.160
Uh, first the model I was trained from scratch on thousands of GPUs.

s28
00:01:59.160 --> 00:02:03.880
Uh, we have a big cluster, one main cluster with a lot of GPUs, all Infiniband connected.

s29
00:02:03.880 --> 00:02:06.440
And you put those GPUs work and it train.

s30
00:02:06.440 --> 00:02:09.840
But like that that's I wish was that simple, but it's not.

s31
00:02:09.840 --> 00:02:17.280
Uh, so at the beginning we did a bunch of like small ablations on like like a small number of GPUs to see like how things will work.

s32
00:02:17.280 --> 00:02:22.280
So you want to test some hypotheses and you do a small number of GPUs and let it train for a little bit.

s33
00:02:22.280 --> 00:02:23.680
Oh, this works, it doesn't work.

s34
00:02:23.680 --> 00:02:24.720
Let's scale.

s35
00:02:24.720 --> 00:02:33.920
Uh, and as we are like training this model, the whole idea was to like kind of bridge the gap between like LLM research and diffusion transformers.

s36
00:02:33.920 --> 00:02:39.840
Uh, so my AI researchers they ported a lot of research from LLMs into into DiTs.

s37
00:02:39.840 --> 00:02:46.560
And the whole like like the whole like architecture of the model was meant to be extremely extremely simple.

s38
00:02:46.560 --> 00:02:51.760
And so like it is very very dumb, but like very effective.

s39
00:02:51.760 --> 00:02:54.520
Uh, and so let's start talking about numbers.

s40
00:02:54.520 --> 00:03:02.920
Uh, incredibly our maybe skill issue on our part, maybe our cluster uh, was very interesting as we were like scaling.

s41
00:03:02.920 --> 00:03:09.959
Uh, when you did like small experiments, experiments would like run for days and like even less than we would like to, but like they would still run fine.

s42
00:03:09.959 --> 00:03:19.560
Uh, and as we like start scaling, like getting like more and more and more GPUs, like 128, 256, 512, whatever number, and like you scale and scale, like things are crashing more.

s43
00:03:19.560 --> 00:03:20.800
That's expected, right?

s44
00:03:20.800 --> 00:03:25.400
Like there's more surface area for things to break and things going to go wrong.

s45
00:03:25.400 --> 00:03:28.040
And a lot of the times things would go wrong in silent ways.

s46
00:03:28.040 --> 00:03:33.560
I don't know nickel timeouts, like just crashes and you like the metrics are all good.

s47
00:03:33.560 --> 00:03:35.880
Uh, and it is extremely annoying stuff.

s48
00:03:35.880 --> 00:03:40.000
At the beginning we were like paranoid, swap node, change node, whatever, whatever, whatever.

s49
00:03:40.000 --> 00:03:42.519
And we learned that like sometimes you just let it crash.

s50
00:03:42.519 --> 00:03:52.000
It crashes, like it runs for like an hour, crash, runs for an hour, crash, and then runs again on the same set of machines, same code, same data for like 12 hours, 16 hours, 24 hours.

s51
00:03:52.000 --> 00:03:58.240
Uh, there is this paper from Meta that kind of gives you a like rough estimate of like how many failures you should expect.

s52
00:03:58.240 --> 00:04:02.640
Uh, we we kind of saw the same pattern but like not the same numbers.

s53
00:04:02.640 --> 00:04:05.640
Uh, our runs would last way way less than this.

s54
00:04:05.640 --> 00:04:12.320
Uh, so it was extremely annoying and you can imagine doing large-scale pre-training on runs that last less than 8 hours, it is a problem, right?

s55
00:04:12.320 --> 00:04:19.840
You want kept those GPUs fed and if things are crashing, you are not doing progress and losing time and models are going to be late.

s56
00:04:19.840 --> 00:04:26.080
Uh, so for us extremely important and like for at least from the infra side was to get metrics.

s57
00:04:26.080 --> 00:04:27.080
Metrics are everything.

s58
00:04:27.080 --> 00:04:28.800
That's how I can support my researchers.

s59
00:04:28.800 --> 00:04:30.960
That's how I have visibility in the system.

s60
00:04:30.960 --> 00:04:37.120
And like if you're doing large-scale pre-training, I highly highly recommend for you to invest heavily on metrics.

s61
00:04:37.120 --> 00:04:40.440
Uh, don't don't go blind because you're going to go you're going to go crazy.

s62
00:04:40.440 --> 00:04:43.400
So I'm going to share some of the metrics that like were important for us.

s63
00:04:43.400 --> 00:04:46.160
And like they're quite silly but like extremely effective.

s64
00:04:46.160 --> 00:04:47.800
Uh, first one like GPU temperature.

s65
00:04:47.800 --> 00:04:49.840
Uh, GPUs are very very annoying.

s66
00:04:49.840 --> 00:04:59.000
They if you're if you have a single GPU that is like a bit warmer than the others, it's going to start like throttling and slow down and and the training's going to be unstable

s67
00:04:59.000 --> 00:05:00.520
and you have weird problems.

s68
00:05:00.520 --> 00:05:05.440
So for us it was like if there is any GPUs above like 78°, you you remove them.

s69
00:05:05.440 --> 00:05:06.200
Don't don't think about it.

s70
00:05:06.200 --> 00:05:07.240
Don't try to fix.

s71
00:05:07.240 --> 00:05:08.520
Don't don't try to be smart.

s72
00:05:08.520 --> 00:05:10.000
You just remove the GPU.

s73
00:05:10.000 --> 00:05:15.680
Uh, you're going to save time and just ask your provider and like this GPU's hot.

s74
00:05:15.680 --> 00:05:17.040
Please replace it.

s75
00:05:17.040 --> 00:05:26.560
Uh and this there there is two metrics that at the beginning we did not fully understand and as we are like getting more and more used to GPUs and like how they work,

s76
00:05:26.560 --> 00:05:28.440
uh, there is GPU utilization which is a lie.

s77
00:05:28.440 --> 00:05:29.360
Uh, don't trust this.

s78
00:05:29.360 --> 00:05:30.640
Uh, this is dumb.

s79
00:05:30.640 --> 00:05:33.520
Uh, this tells you, oh, the GPU's doing work.

s80
00:05:33.520 --> 00:05:36.360
And this is amount of time GPU's doing work but like not good work.

s81
00:05:36.360 --> 00:05:38.120
Not how efficient the GPU's working.

s82
00:05:38.120 --> 00:05:41.520
So like as you can see during our pre-training, the GPUs are 100%.

s83
00:05:41.520 --> 00:05:42.080
This is not true.

s84
00:05:42.080 --> 00:05:43.760
We are not fully utilizing the GPU.

s85
00:05:43.760 --> 00:05:46.000
Uh, this is 100% a lie.

s86
00:05:46.000 --> 00:05:49.400
What we would use uh, as a proxy was tensor core utilization.

s87
00:05:49.400 --> 00:05:53.960
This is actually how how much of a tensor core you're using and like how effective they are being.

s88
00:05:53.960 --> 00:06:02.600
Uh and it was also very interesting as we're doing pre-training and then you go from pre-training to mid-training to post-training, you start like scaling the like the resolution of the images.

s89
00:06:02.600 --> 00:06:14.080
So like for example, pre-training we do like I think 128, 256, 512 1024 pixels as they scale and you could see the tensor core utilization go up as we like would scale on these these resolutions

s90
00:06:14.080 --> 00:06:17.280
cuz now you're doing more work on on on images.

s91
00:06:17.280 --> 00:06:21.600
And like also very very interesting was InfiniBand and NVLink metrics.

s92
00:06:21.600 --> 00:06:26.520
These by default not exported by uh the NVIDIA metrics that DCGM stuff.

s93
00:06:26.520 --> 00:06:28.960
Some NVLink stuff, yes, but no InfiniBand.

s94
00:06:28.960 --> 00:06:32.360
So if you don't have InfiniBand metrics, uh go go go get it.

s95
00:06:32.360 --> 00:06:41.280
Uh I'm telling you right now, if you're doing large-scale pre-training with a bunch of GPUs talking to each other between machines and you have no InfiniBand metrics, you're doing something wrong.

s96
00:06:41.320 --> 00:06:47.400
These was probably the most important stuff for us because most of our failures were like related to like cross-node communication.

s97
00:06:47.400 --> 00:06:58.800
So like for example, here you just have throughput, but like on our like reference dashboard, we have a bunch of stuff like from like wait like when you when a message sent on on on the fabric like how much time the the the message is waiting

s98
00:06:58.800 --> 00:07:08.880
or like the like number of errors and different types of errors and like number of packets and all of the things that like InfiniBand's exported uh we collect.

s99
00:07:08.880 --> 00:07:10.440
We had to build custom stuff to get this.

s100
00:07:10.440 --> 00:07:11.360
It was not hard.

s101
00:07:11.360 --> 00:07:12.520
Uh you can figure it out.

s102
00:07:12.520 --> 00:07:13.680
It's very very easy.

s103
00:07:13.680 --> 00:07:15.000
Same thing for NVLink.

s104
00:07:15.000 --> 00:07:20.760
Uh NVIDIA exports some stuff about NVLink, but like for example, NVLink NVLink errors, NVIDIA doesn't export this.

s105
00:07:20.760 --> 00:07:26.680
Uh so you can collect this and like as I said, InfiniBand was extremely important and NVLink was a little bit less.

s106
00:07:26.680 --> 00:07:31.400
Uh in some cases, these helped us catch some problems in like especially like cuz NVLink single node, right?

s107
00:07:31.400 --> 00:07:33.360
Like communication inside a node.

s108
00:07:33.360 --> 00:07:43.680
So sometimes a single node would have a weird failure where the GPUs seem to be fine, but like some weird error happens and then you can look at NVLink NVLink errors and like you see errors happening.

s109
00:07:43.680 --> 00:07:45.200
And then replace that machine.

s110
00:07:45.200 --> 00:07:47.400
So, go get this matrix.

s111
00:07:47.400 --> 00:07:52.960
They they're extremely important and without this we would not be able to to train at all.

s112
00:07:52.960 --> 00:08:02.000
Also, as I said, our our trains would crash constantly and a hacky way to do it to fix the problem just checkpoint.

s113
00:08:02.000 --> 00:08:04.000
Use and abuse the file system that you have.

s114
00:08:04.000 --> 00:08:05.600
At the beginning we use SEF.

s115
00:08:05.600 --> 00:08:07.720
SEF didn't work well.

s116
00:08:07.720 --> 00:08:08.600
Was very annoying.

s117
00:08:08.600 --> 00:08:09.600
It broke.

s118
00:08:09.600 --> 00:08:11.040
We lost trust in data.

s119
00:08:11.040 --> 00:08:17.360
So, I recommend if if you have the money go go with something paid and cuz you can trust your data.

s120
00:08:17.360 --> 00:08:18.040
You can see numbers.

s121
00:08:18.040 --> 00:08:19.720
This is our our worker cluster.

s122
00:08:19.720 --> 00:08:25.040
We can do like 1.8 terabytes of second of reads, almost a terabyte of writes.

s123
00:08:25.040 --> 00:08:27.080
And the file system would not choke on the training.

s124
00:08:27.080 --> 00:08:34.360
So, we could checkpoint every like 30 minutes, 20 minutes, produce like a terabyte of data in like less than 30 seconds.

s125
00:08:34.360 --> 00:08:36.080
So, this would not delay trainings.

s126
00:08:36.080 --> 00:08:43.120
That was like probably one of the like most important things that we need to to like recoup the loss and just checkpoint.

s127
00:08:43.120 --> 00:08:46.040
Don't don't don't don't forget about it.

s128
00:08:46.040 --> 00:08:47.960
And like how we serve.

s129
00:08:47.960 --> 00:08:51.760
This goes in connection on how the trains are launched.

s130
00:08:51.760 --> 00:08:55.720
Because at the beginning I don't want my researchers to think about GPUs.

s131
00:08:55.720 --> 00:09:01.000
I just want them to launch stuff and this goes into a queue and if we have GPUs we have GPUs.

s132
00:09:01.000 --> 00:09:03.760
If we don't have GPUs we don't have GPUs.

s133
00:09:03.760 --> 00:09:05.600
So, queue.

s134
00:09:05.600 --> 00:09:07.080
This is a open source project.

s135
00:09:07.080 --> 00:09:08.040
You can look it up.

s136
00:09:08.040 --> 00:09:09.160
It does gang scheduling.

s137
00:09:09.160 --> 00:09:14.040
Gang scheduling is important for for for trainings in general.

s138
00:09:14.040 --> 00:09:23.240
And this gives us a semantic of like two tiers of priority where you have a workload priority and this you can say like oh this training is more important than this one.

s139
00:09:23.240 --> 00:09:25.600
So, it skips on the queue in front of the queue.

s140
00:09:25.600 --> 00:09:31.160
And then after this we have the normal Kubernetes priority if you're used to Kubernetes.

s141
00:09:31.160 --> 00:09:35.920
And the way this system works is like the training pods they always have like high priority for everything.

s142
00:09:35.920 --> 00:09:38.200
So, like once they are submitted they going on a schedule.

s143
00:09:38.200 --> 00:09:41.280
If there's inference running on those machines, the inference gets kicked out.

s144
00:09:41.280 --> 00:09:42.480
And you'd say, "Oh, this is bad.

s145
00:09:42.480 --> 00:09:43.520
Production is going to go down."

s146
00:09:43.520 --> 00:09:47.280
No, you can build on top of that to to make production not go down, which is very, very cool.

s147
00:09:47.280 --> 00:09:51.560
Uh the only one of the problems with Q, which is annoying, you can automate that.

s148
00:09:51.560 --> 00:09:52.680
We have not.

s149
00:09:52.680 --> 00:09:54.840
It's just that you specify the queues.

s150
00:09:54.840 --> 00:10:00.120
The queues have like amount of resources, CPU, and like Nvidia GPUs, memory, whatever.

s151
00:10:00.120 --> 00:10:05.000
Uh but this is manually like manually like specified.

s152
00:10:05.000 --> 00:10:07.360
And at least our cluster is quite fluid.

s153
00:10:07.360 --> 00:10:09.640
Nodes phasing in and out of existence.

s154
00:10:09.640 --> 00:10:12.200
They go into to maintenance, whatever.

s155
00:10:12.200 --> 00:10:13.800
You lose a few nodes here and there.

s156
00:10:13.800 --> 00:10:15.440
Uh this number gets out of sync.

s157
00:10:15.440 --> 00:10:17.840
And sometimes these these would break gang scheduling.

s158
00:10:17.840 --> 00:10:20.280
Uh so, FYI, this this is a bit annoying.

s159
00:10:20.280 --> 00:10:24.560
You're going to face this if you use Q. Uh but yeah, very good project.

s160
00:10:24.560 --> 00:10:27.840
Kubernetes 1.15, uh we have not had the chance to play with it.

s161
00:10:27.840 --> 00:10:29.960
Has gang scheduling uh out of the box.

s162
00:10:29.960 --> 00:10:34.600
Something very similar to Q. Uh so, maybe you can use Kubernetes 1.15.

s163
00:10:34.600 --> 00:10:39.720
Uh and as I said, this this is the system that we built that allowed us to train using the whole cluster.

s164
00:10:39.720 --> 00:10:43.440
As I said, we have one big cluster that runs production and and trainings.

s165
00:10:43.440 --> 00:10:52.080
Uh so, I don't want my researchers to think about GPUs, and I don't want make the trainings and research be delayed because production is running, right?

s166
00:10:52.080 --> 00:10:53.839
Uh production is lower priority.

s167
00:10:53.839 --> 00:10:55.040
The site still needs to work.

s168
00:10:55.040 --> 00:10:56.760
People still need to be use the website.

s169
00:10:56.760 --> 00:11:04.240
But like the GPUs, they're like the value that we get off the GPUs doing trainings is more like higher than than we get out of production.

s170
00:11:04.240 --> 00:11:12.960
So, the whole system works by default where there is this magical system that I'm going to explain in in a bit that allows us to flip traffic between clusters magically.

s171
00:11:12.960 --> 00:11:16.839
And not just clusters, and like external providers, GPU rentals, whatever.

s172
00:11:16.839 --> 00:11:22.240
And you can see like the green like the the dark green is like inference running in cluster.

s173
00:11:22.240 --> 00:11:31.040
Uh and then someone launches a train, and then suddenly start flipping to the other cluster, and then training is done or whatever happens, it flips back.

s174
00:11:31.040 --> 00:11:32.360
So, we stop wasting money.

s175
00:11:32.360 --> 00:11:33.200
And this is seamless.

s176
00:11:33.200 --> 00:11:34.800
No one needs to think about it.

s177
00:11:34.800 --> 00:11:39.000
Uh the whole system like like handles itself.

s178
00:11:39.000 --> 00:11:45.360
And you get this this very nice pattern of like I'm going to use all the GPUs in my cluster uh for trainings.

s179
00:11:45.360 --> 00:11:46.480
Production is going to run somewhere else.

s180
00:11:46.480 --> 00:11:47.280
I don't need to think about it.

s181
00:11:47.280 --> 00:11:49.880
My users on production they not going to feel anything.

s182
00:11:49.880 --> 00:11:54.720
Uh research is going to be happy, and we can get values out of the GPUs.

s183
00:11:54.720 --> 00:11:56.280
Uh so, how does this work?

s184
00:11:56.280 --> 00:11:59.440
There's this very nice project called virtual kubelet, also open source.

s185
00:11:59.440 --> 00:12:00.640
Uh you can build on top of it.

s186
00:12:00.640 --> 00:12:02.600
It is a very nice code base.

s187
00:12:02.600 --> 00:12:05.720
Uh and this works by creating a fake machine in Kubernetes.

s188
00:12:05.720 --> 00:12:06.880
Uh Kubernetes has nodes.

s189
00:12:06.880 --> 00:12:10.640
This creates a fake machine that is like up to you to control how it works.

s190
00:12:10.640 --> 00:12:13.600
So, Kubernetes does normal scheduling as you would expect.

s191
00:12:13.600 --> 00:12:15.760
Things would go into into these nodes.

s192
00:12:15.760 --> 00:12:18.720
For example, here all the GPUs are in the cluster in use, right?

s193
00:12:18.720 --> 00:12:23.480
So, this pod goes into the virtual kubelet node, and in there you can do whatever.

s194
00:12:23.480 --> 00:12:24.880
This is the system that we built.

s195
00:12:24.880 --> 00:12:28.520
There is like you receive the pods back, and then you find a provider.

s196
00:12:28.520 --> 00:12:30.520
You can like This is up to you.

s197
00:12:30.520 --> 00:12:32.640
Let's say you have a deal with some provider that gives nice prices.

s198
00:12:32.640 --> 00:12:34.400
You integrate integrate into here.

s199
00:12:34.400 --> 00:12:38.720
We built like some nice interfaces to be to to like not leak things.

s200
00:12:38.720 --> 00:12:44.839
So, like we just implement a provider, and there is a a algo that decides which one it goes.

s201
00:12:44.839 --> 00:12:51.839
Uh you translate this back of the pods into into the provider stuff, and you deploy, and then you have something that reconciles between the both sides.

s202
00:12:51.839 --> 00:12:59.480
And what's extremely extremely nice, if you guys know about Kubernetes, Kubernetes has like the horizontal pod auto scaler, which like scales the number of of replicas.

s203
00:12:59.480 --> 00:13:04.160
Uh number of replicas uh Could you stop being annoying?

s204
00:13:04.160 --> 00:13:05.320
Thank you, sir.

s205
00:13:05.320 --> 00:13:06.480
I appreciate it.

s206
00:13:06.480 --> 00:13:07.280
There you go.

s207
00:13:07.280 --> 00:13:08.320
Let's go back.

s208
00:13:08.320 --> 00:13:10.760
Uh now we go back.

s209
00:13:10.760 --> 00:13:12.640
Uh back back.

s210
00:13:12.640 --> 00:13:14.839
There you go.

s211
00:13:15.040 --> 00:13:17.880
Like Kubernetes has the HPA, and the HPA scales the pods.

s212
00:13:17.880 --> 00:13:21.400
And so, if something fails, it is very interesting you don't need to handle the fail.

s213
00:13:21.400 --> 00:13:23.400
Uh the only thing you need to handle is like, "Oh, something failed."

s214
00:13:23.400 --> 00:13:24.800
You mark the pod as failed.

s215
00:13:24.800 --> 00:13:27.640
Kubernetes is it's to detect that something has failed and create a new one.

s216
00:13:27.640 --> 00:13:29.680
Uh you don't need to try to save the world.

s217
00:13:29.680 --> 00:13:33.600
Let Kubernetes handle it for you, which is extremely nice way of handling stuff.

s218
00:13:33.600 --> 00:13:40.160
If something breaks on your side, something breaks on the the other side, just mark it as failed, let Kubernetes handle it, create a new one, and things keep working.

s219
00:13:40.160 --> 00:13:41.920
Very, very nice way to handle stuff.

s220
00:13:41.920 --> 00:13:45.640
Uh and also very interesting, let's say you have GPUs on your cluster available, right?

s221
00:13:45.640 --> 00:13:46.640
You don't want to waste money.

s222
00:13:46.640 --> 00:13:47.960
This would be very, very bad.

s223
00:13:47.960 --> 00:13:52.440
So, the system works about like works with like using taints, Kubernetes taints.

s224
00:13:52.440 --> 00:13:55.200
They allow and disallow things to run.

s225
00:13:55.200 --> 00:13:56.960
Pods have tolerations for the taints.

s226
00:13:56.960 --> 00:14:02.880
And when we have GPUs in the cluster, uh if you look back, there is the taint system on the bottom.

s227
00:14:02.880 --> 00:14:09.080
This taint system, it is what would like by itself decide what you like if you have GPUs or not GPUs in the cluster.

s228
00:14:09.080 --> 00:14:14.080
Uh and this adds a taint into the into the node when we have a lot of GPUs.

s229
00:14:14.080 --> 00:14:18.080
So, like a lot of GPUs in the cluster, we taint the node, nothing can schedule on it.

s230
00:14:18.080 --> 00:14:20.720
Uh so, we stop wasting GPUs.

s231
00:14:20.720 --> 00:14:25.120
Like the pods, they go into into the GPUs in the cluster, we don't waste money.

s232
00:14:25.120 --> 00:14:27.200
Uh and then imagine someone launches a training, right?

s233
00:14:27.200 --> 00:14:31.480
This training is going to take all the GPUs in the cluster, it's going to hog all of the GPUs, no GPUs in the cluster.

s234
00:14:31.480 --> 00:14:34.839
The system detects this, removes the taint, new pod schedules there.

s235
00:14:34.839 --> 00:14:35.960
Very, very nice.

s236
00:14:35.960 --> 00:14:37.520
Uh you also don't think about it.

s237
00:14:37.520 --> 00:14:40.680
Uh and for us, for example, we use just some Prometheus metrics.

s238
00:14:40.680 --> 00:14:43.880
That's how we do it, very simple, uh but it works very, very, very well.

s239
00:14:43.880 --> 00:14:46.800
Uh you let the system run by itself.

s240
00:14:46.800 --> 00:14:47.920
Someone's going to launch stuff.

s241
00:14:47.920 --> 00:14:55.880
You're going to like the training SLI is going to kick out the pods, it's going to schedule, it's going to take the GPUs, the system is going to detect that, remove the taint, pod schedules there.

s242
00:14:55.880 --> 00:14:56.560
Very nice.

s243
00:14:56.560 --> 00:14:57.640
Someone finish the training.

s244
00:14:57.640 --> 00:15:00.640
Now we have pods running on the other side, you're wasting money.

s245
00:15:00.640 --> 00:15:02.080
Uh how do we fix this?

s246
00:15:02.080 --> 00:15:06.760
Uh same thing, you run something else that detects the system and removes things back.

s247
00:15:06.760 --> 00:15:08.520
Uh in this case, a descheduler.

s248
00:15:08.520 --> 00:15:15.920
Uh once the taint's added back, so like GPUs available, we add the taint, the descheduler says, "Oh, these pods, they don't tolerate the taints.

s249
00:15:15.920 --> 00:15:17.400
I'm going to migrate them back."

s250
00:15:17.400 --> 00:15:20.760
And you can ask, "Oh, why you don't use a no no no execute taint?"

s251
00:15:20.760 --> 00:15:23.400
No execute in Kubernetes would kick everything out at the same time.

s252
00:15:23.400 --> 00:15:27.440
So, as as like at moment you put the tanks, everything will be kicked out and that's bad.

s253
00:15:27.440 --> 00:15:28.560
Production would go down.

s254
00:15:28.560 --> 00:15:34.880
So, like this system my like slowly migrates the pods back, so production doesn't go down and we don't waste money.

s255
00:15:34.880 --> 00:15:38.840
Um it is a like a very like self-healing system.

s256
00:15:38.840 --> 00:15:40.120
You don't need to interfere with it.

s257
00:15:40.120 --> 00:15:41.720
Uh it just runs.

s258
00:15:41.720 --> 00:15:43.760
Uh yes, of course, there was bugs at the beginning.

s259
00:15:43.760 --> 00:15:44.960
Uh nothing's perfect.

s260
00:15:44.960 --> 00:15:51.320
But like once you calibrate it was was like very very well and like changed the way we do research cuz no one else needs to care about GPUs.

s261
00:15:51.320 --> 00:15:52.280
They just launch stuff.

s262
00:15:52.280 --> 00:15:53.440
If have GPUs, we have GPUs.

s263
00:15:53.440 --> 00:15:55.840
If not, we don't have GPUs, go into the queue.

s264
00:15:55.840 --> 00:15:57.960
And we fully utilize the cluster for trainings.

s265
00:15:57.960 --> 00:15:59.240
Production runs somewhere else.

s266
00:15:59.240 --> 00:16:01.520
And the GPUs are doing like useful work.

s267
00:16:01.520 --> 00:16:08.440
And also like if you're doing like diffusion transformers, they're not huge like LLMs that need like multi-node like inference.

s268
00:16:08.440 --> 00:16:11.000
Uh something that we learned like whatever GPU works.

s269
00:16:11.000 --> 00:16:13.520
Uh the GPU can be hot, falling out of the bus.

s270
00:16:13.520 --> 00:16:15.000
It can be exploding.

s271
00:16:15.000 --> 00:16:16.240
Uh inference still going to run.

s272
00:16:16.240 --> 00:16:17.520
It is very interesting.

s273
00:16:17.520 --> 00:16:19.920
So, like you can have very very bad GPUs for inference.

s274
00:16:19.920 --> 00:16:22.560
Uh and everyone's going to be happy.

s275
00:16:22.560 --> 00:16:23.840
Uh we are hiring.

s276
00:16:23.840 --> 00:16:33.839
Uh if you're interested in building sort of stuff, uh doing large-scale pre-training, uh build this sort of system for researchers, shoot me a message at gabriel plus hiring at cerebra.ai.

s277
00:16:33.839 --> 00:16:36.360
There's also jobs listing and that's it.

s278
00:16:36.360 --> 00:16:38.520
Thank you.

s279
00:16:51.986 --> 00:16:53.986
[music]