WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=bRGyYaE0lxI NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/bRGyYaE0lxI.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.639 --> 00:00:13.040 All right. s3 00:00:13.040 --> 00:00:13.920 Hello everyone. s4 00:00:13.920 --> 00:00:14.960 Uh, my name is Connor. s5 00:00:14.960 --> 00:00:19.760 I'm a developer advocate at Crusoe and I'm joined by my colleagues Nikil and Young. s6 00:00:19.760 --> 00:00:26.960 And we're going to be talking today about the infrastructure we've built at Crusoe and how it handles critical hardware errors. s7 00:00:26.960 --> 00:00:35.360 And so our customers when they're running extremely largecale training workloads across thousands of GPUs, GPU failures are inevitable. s8 00:00:35.360 --> 00:00:40.079 And so at scale, manual remediation is completely unsustainable. s9 00:00:40.079 --> 00:00:53.199 And so what we'll cover today is how by using both Slurm and Kubernetes, we can provide a robust platform with automated remediation that maintains ease of use for both machine learning engineers running jobs s10 00:00:53.199 --> 00:00:58.000 and the platform teams managing the underlying clusters. s11 00:00:58.879 --> 00:01:13.280 So to provide a little bit more context on what Cruso cloud is for those of you who haven't heard of us um we provide infrastructure as a service consisting of compute storage networking and the latest gen GPUs from Nvidia and AMD s12 00:01:13.280 --> 00:01:25.759 but the focus of this talk will be on the tooling that we've built on top of that to orchestrate these large fleets of GPUs including our managed Kubernetes service called CMK s13 00:01:25.759 --> 00:01:37.119 and our manage slurm service which is built on top of CMK and built off of Slinky which is SCADMD's official open source project for running slurm on top of Kubernetes s14 00:01:37.119 --> 00:01:50.079 and this allows us to get the high performance job scheduling of slurm with the infrastructure resilience features and observability of Kubernetes including one of the systems we've developed called autoclusters s15 00:01:50.079 --> 00:01:58.560 which is our system for automatically detecting and replacing failed GPU nodes when a GPU failure has been detected. s16 00:01:58.560 --> 00:02:10.879 And so to explain how and why we landed on this architecture, it's helpful to understand where traditional slurm both excels and falls short, which Young will get into now. s17 00:02:13.280 --> 00:02:14.319 Thanks, Connor. s18 00:02:14.319 --> 00:02:20.319 Um, traditionally, I mean, looking at it from the tra traditional perspective, s19 00:02:21.280 --> 00:02:22.640 sorry, can can you hear me now? s20 00:02:22.640 --> 00:02:23.280 Okay, cool. s21 00:02:23.280 --> 00:02:24.560 I I'll just do it like this. s22 00:02:24.560 --> 00:02:35.040 But basically, traditionally, slurm was built slurm was built by the researchers for the researchers, universities, labs over 20 years ago. s23 00:02:35.040 --> 00:02:47.760 Uh built for high performance computing uh which as we've observed with our customers and others that are using slurm today translates very well to modern AI specifically around training workloads. s24 00:02:47.760 --> 00:02:58.720 this idea that when you're running a multi-node training job, you need this idea of tight collective communication across the ranks that that you're using. s25 00:02:58.720 --> 00:03:04.640 Uh including, you know, specific network built for training workloads today. s26 00:03:04.640 --> 00:03:12.879 Um concepts like gang scheduling, topology awareness, uh prologue and epilog for cluster validations. s27 00:03:12.879 --> 00:03:20.239 These are all the features that are readily available with slurm today that helps researchers run their jobs. s28 00:03:20.239 --> 00:03:23.599 Uh essentially running their jobs with tools that are already familiar to them. s29 00:03:23.599 --> 00:03:31.120 Scripts, especi cluster today. s30 00:03:31.120 --> 00:03:41.440 Now, where we feel like this is a bit of a shortcoming or short falls a little bit short around the modern training is the things that you see on the screen right now. s31 00:03:41.440 --> 00:03:47.360 Um AI training workloads today are very dynamic and it doesn't just include training. s32 00:03:47.360 --> 00:04:02.000 Uh there's a lot of other aspects that your AI labs and other companies are running as part of your your AI workloads including training post-training eval inferences some of which may or may not even land on your slurm cluster today. s33 00:04:02.000 --> 00:04:13.680 But because of of what we've observed as as a GPU capacity constraints and other things kind of limits you to how you need to be able to be more dynamic around the resources that you have. s34 00:04:13.680 --> 00:04:20.799 Uh slurm traditionally is pretty static and it's partitioning and other other uh functionalities. s35 00:04:20.799 --> 00:04:26.720 The other thing I want to mention is this idea of health checks and maintaining your nodes. s36 00:04:26.720 --> 00:04:30.720 Uh you know you can make sure your nodes are healthy or not healthy. s37 00:04:30.720 --> 00:04:33.680 GPUs do fail, how do you manage that? s38 00:04:33.680 --> 00:04:45.680 Uh, SlurpM obviously has features around it to be able to, you know, reue the jobs, uh, detect bad nodes, uh, using things like prologue and other open source tools that are now available s39 00:04:45.680 --> 00:04:46.880 to be able to do this. s40 00:04:46.880 --> 00:04:52.639 It just adds an operational burden, uh, that we've seen from customers today. s41 00:04:52.639 --> 00:04:59.919 you know, this idea that you have to detect if nodes are bad, you have to drain the nodes when they're bad, you have to kind of work around that. s42 00:04:59.919 --> 00:05:05.680 That could either be manual or you need some sort of an automation or scripting to be able to do all of this. s43 00:05:05.680 --> 00:05:07.440 And the last thing is observability. s44 00:05:07.440 --> 00:05:09.840 Um, you know, jobs can fail. s45 00:05:09.840 --> 00:05:14.639 Jobs could also come with degraded performances for any number of reasons. s46 00:05:14.639 --> 00:05:16.800 I mean, GPU failure is just one of them. s47 00:05:16.800 --> 00:05:24.240 there's other things like network flaps and other things within your your switches that may be causing degraded or failed performance. s48 00:05:24.240 --> 00:05:26.320 Uh slurm knows when a job fails. s49 00:05:26.320 --> 00:05:30.160 It doesn't necessarily be able to tell you everything about why it failed. s50 00:05:30.160 --> 00:05:34.479 And that's where the observability comes into key uh key play. s51 00:05:34.479 --> 00:05:43.120 So all of this kind of encompasses the reason why we we built this product that Nikil is going to go over and sort of describe some of the specifics around s52 00:05:43.120 --> 00:05:45.520 manage slur. s53 00:05:49.280 --> 00:05:54.000 Um all right uh thanks young. s54 00:05:54.000 --> 00:06:08.560 So yeah, I'm going to be talking about our Cruso manage slurm product and uh like like we've mentioned before um there's a lot of benefits uh that slurm can bring to the uh training teams s55 00:06:08.560 --> 00:06:21.600 and uh researchers that are building the next best models but there are some issues with you know setting up slurm managing it keeping it reliable uh and on the other hand you have kubernetes s56 00:06:21.600 --> 00:06:26.880 which is the OS of the cloud It's widespread use and familiarity. s57 00:06:26.880 --> 00:06:37.120 It's a very mature ecosystem with lots of tools, extensions, and plugins for whatever you need to add, whether it be observability, networking, security. s58 00:06:37.120 --> 00:06:45.520 It helps uh maintain services with there, you know, built-in safe healing, uh load balancing and autoscaling that keeps everything running smoothly. s59 00:06:45.520 --> 00:06:57.440 And so we often see that uh customers will uh come with us starting with say inference running on Kubernetes and then they want to expand to training uh or vice versa. s60 00:06:57.440 --> 00:07:04.560 And so maybe a Kubernetes familiar team struggles with deploying and managing a slurm cluster. s61 00:07:04.560 --> 00:07:19.840 Um or uh the incoming slurm native training team is unfamiliar with the existing Kubernetes frameworks uh and adds you know additional friction that just slows down your team and with you know how fast everyone's trying to get to s62 00:07:19.840 --> 00:07:23.599 uh the next best model and next best product. s63 00:07:23.599 --> 00:07:34.000 um teams get set up with the tools they're familiar with and you end up with two separate infrastructure stacks which has its own host of extra operational burdens uh and extra costs. s64 00:07:34.000 --> 00:07:40.800 So we decided to build out our Cruso managed slurm uh on top of Kubernetes. s65 00:07:40.800 --> 00:07:48.639 So we have our Cruso slurm operator uh which coordinates with our Cruso managed Kubernetes platform and the broader Cruso cloud. s66 00:07:48.639 --> 00:07:58.000 So including storage, networking etc. And uh cso manages uh slurm users, slurm partitions, configurations and storage. s67 00:07:58.000 --> 00:07:59.599 So you don't have to do it peacemeal. s68 00:07:59.599 --> 00:08:06.240 You just you know one one command you create your slurm cluster and there it is existing in your kubernetes cluster. s69 00:08:06.240 --> 00:08:11.280 So the benefit is for the slurm users um it's just another slurm cluster. s70 00:08:11.280 --> 00:08:19.680 They just need uh an IP address they can SSH with their uh user and now to them it's a slurm cluster with all the bells and whistles that they need. s71 00:08:19.680 --> 00:08:22.879 They don't even need to know that Kubernetes is running underneath. s72 00:08:22.879 --> 00:08:30.960 For your platform team and your uh infrastructure team, uh the benefit is slurm is now just another service that plugs into your existing infrastructure. s73 00:08:30.960 --> 00:08:39.200 You don't need to set up an entirely new infrastructure stack, new observability tools, new you know on call teams, new runbooks, nothing like that. s74 00:08:39.200 --> 00:08:40.640 It's all built into the same thing. s75 00:08:40.640 --> 00:08:47.120 It runs alongside all your other services and um resources in slurps. s76 00:08:47.120 --> 00:09:01.839 any jobs make use of the uh built-in topology aware scheduling um and the platform team just sees it as another service um and now since all your GPUs and hardware are in one cluster they can move around seamlessly s77 00:09:01.839 --> 00:09:15.839 uh it enables uh more interesting uh tooling and dynamic availability like when you need to burst your uh inference service when you're reaching uh high uh users high number of users s78 00:09:15.839 --> 00:09:22.399 you can take away some some of your or even reallocate some of your hardware from your training cluster to your inference service or vice versa. s79 00:09:22.399 --> 00:09:28.080 You know, when your user uh load goes down, you have all these GPUs sitting idle. s80 00:09:28.080 --> 00:09:34.320 That's the perfect time to up your training job or do a a larger hero run across more GPUs. s81 00:09:34.320 --> 00:09:42.800 Um this is something that's only possible when you have everything in a single uh infrastructure stack like uh cru managed slurm. s82 00:09:42.800 --> 00:09:52.560 Um an additional benefit is uh now crucial managetorm can make use of our autoclusters product which is our auto remediation for additional hardware errors. s83 00:09:52.560 --> 00:09:59.680 So Connor is uh going to walk through that and show a demo of uh of how that works. s84 00:10:03.200 --> 00:10:10.160 So this is an overview of what exactly happens when autoclusters detects a critical hardware error. s85 00:10:10.160 --> 00:10:15.200 in this case an XID79 error where one of the GPUs is completely unusable. s86 00:10:15.200 --> 00:10:21.760 And so the first thing that happens is that the user is notified although there's no action required from them. s87 00:10:21.760 --> 00:10:28.160 Then the slurm operator sets that that specific node to down and cancels any running jobs. s88 00:10:28.160 --> 00:10:39.839 There's then a uh the process re receives a sig term signal before being terminated and you have up to two minutes to handle that sig term to either save your checkpoints or flush your logs s89 00:10:39.839 --> 00:10:42.640 before that node is eventually replaced. s90 00:10:42.640 --> 00:10:48.000 The cancel job is automatically reued and then autoclusters start. s91 00:10:48.000 --> 00:10:56.320 So first it verifies that no pods on the node have a label that disables automatic replacement. s92 00:10:56.320 --> 00:11:02.240 Assuming that node can be replaced, the Kubernetes node is then cordoned and drained. s93 00:11:02.240 --> 00:11:10.160 And then the unhealthy node is removed from the node pool and replaced with a new healthy node from spare capacity. s94 00:11:10.160 --> 00:11:15.200 And then a record of the alert and the remediation is logged for the user's reference. s95 00:11:15.200 --> 00:11:17.600 Uh but there's still no action required from them. s96 00:11:17.600 --> 00:11:22.800 And then once that healthy node comes back online, then the slurm job starts. s97 00:11:22.800 --> 00:11:29.519 Your application code runs to load your model and load your checkpoint and resume training. s98 00:11:30.240 --> 00:11:36.959 And so here's what that looks like um in a demo. s99 00:11:36.959 --> 00:11:42.560 And so here on the left, I'm just using sbatch to launch a PyTorch training script. s100 00:11:42.560 --> 00:11:49.920 On the right is the Cruso cloud console where I have a GPU node pool of two A100 nodes. s101 00:11:49.920 --> 00:11:53.680 And I'm going to launch this as normal. s102 00:11:53.680 --> 00:12:04.399 Um, and then what I'm going to do is run a script in to tell our backend internal monitoring system that there's been an XID79 error on one of the GPUs. s103 00:12:04.399 --> 00:12:06.639 And this will trigger autoclusters. s104 00:12:06.639 --> 00:12:11.279 And so we can see immediately the GPU utilization drops. s105 00:12:11.279 --> 00:12:18.079 And so we know that that that GPU has gone offline and autocluster's node replacement starts immediately. s106 00:12:18.079 --> 00:12:29.279 And the full process from detecting a critical hardware error to getting a healthy node back into the node pool takes roughly five minutes with autoclusters. s107 00:12:29.279 --> 00:12:32.399 The rest of the time is mainly application level code. s108 00:12:32.399 --> 00:12:38.480 And so in this example, it's a little under 15 minutes um from end to end. s109 00:12:38.480 --> 00:12:45.839 And here I'm just showing some of the alerts that uh the user receives when there's a critical hardware error like this. s110 00:12:45.839 --> 00:12:50.639 So as you'll see um there's no action required from the user. s111 00:12:50.639 --> 00:12:52.240 The slurm job will restart. s112 00:12:52.240 --> 00:12:56.160 It'll load the checkpoint and continue training where it left off. s113 00:12:56.160 --> 00:12:59.440 And we can see the GPU utilization returns back to normal. s114 00:12:59.440 --> 00:13:03.040 And the total downtime is less than 15 minutes. s115 00:13:03.040 --> 00:13:11.279 And this is um much better than having an engineer have to log on in the middle of the night and spend hours debugging something. s116 00:13:11.279 --> 00:13:15.040 All of this is handled automatically on our platform. s117 00:13:18.320 --> 00:13:25.519 Now, historically, creating this type of environment has been challenging to merge both Slurm and Kubernetes. s118 00:13:25.519 --> 00:13:37.200 And so we're really excited to announce what we call one-click slurm where everything that we've talked about in this presentation, this entire environment can be provisioned with with just a single command. s119 00:13:37.200 --> 00:13:45.680 And so this one command includes provisioning the underlying Kubernetes cluster, the slurm controller, login nodes and storage. s120 00:13:45.680 --> 00:13:51.120 And then adding a GPU node pool is only one command beyond that. s121 00:13:51.120 --> 00:14:00.480 And so this means that engineers can simply SSH in and start running jobs immediately and autoclusters is enabled by default. s122 00:14:02.079 --> 00:14:14.800 So wrapping up what we covered um starting with the limitations of traditional slurm it gives you the high performance job scheduling but not the automated the automated healing that modern AI infrastructure requires. s123 00:14:14.800 --> 00:14:25.120 And so by building slurm on top of Kubernetes, we can enable these advanced advanced features without having either team change uh their workflows. s124 00:14:25.120 --> 00:14:34.560 Then we covered the full node remediation flow from an error being detected to the slurm job resuming and the checkpoint and the training resuming. s125 00:14:34.560 --> 00:14:43.120 And so the key takeaways here first is that traditionally slurm and the infrastructure that it runs on are two separate systems. s126 00:14:43.120 --> 00:14:58.800 But together with Kubernetes, both systems stay in sync and so a failed GPU node can get cord in automatically and that signal will propagate directly to the slurm operator. s127 00:14:58.959 --> 00:15:06.800 Also, like I mentioned earlier, neither the platform teams nor the machine learning teams have to change their workloads. s128 00:15:06.800 --> 00:15:10.320 Researchers can still use the same commands to launch slurm. s129 00:15:10.320 --> 00:15:21.920 And although Kubernetes is running underneath, researchers never have to SSH into a pod or worry about containers, but the Kubernetes team still get access to the same level of s130 00:15:21.920 --> 00:15:27.600 telemetry and observability that they're used to. s131 00:15:27.600 --> 00:15:32.720 And then lastly, like Nicole mentioned, the GPU nodes are Kubernetes nodes first. s132 00:15:32.720 --> 00:15:37.440 And that means that slurm is just a workload running on top. s133 00:15:37.440 --> 00:15:44.000 And when a slurm job finishes, those GPU nodes don't have to sit idle waiting for another job. s134 00:15:44.000 --> 00:15:54.320 You can schedule inference pods on those immediately through Kubernetes because from the platform's perspective, they're still just nodes in a pool. s135 00:15:55.519 --> 00:16:00.160 And so the core idea here is that failures are inevitable. s136 00:16:00.160 --> 00:16:12.480 Um and so in today's AI landscape, the architecture of infrastructure should be designed in a way so that when there is a critical error, all the right actions are handled autonomously s137 00:16:12.480 --> 00:16:18.399 and engineers can focus on building their application and not worrying about their infrastructure. s138 00:16:20.240 --> 00:16:22.720 So hopefully this was an informative talk. s139 00:16:22.720 --> 00:16:30.959 Um, if you have any questions, feel free to um, connect with us at booth ug12 or on social at crusadev. s140 00:16:30.959 --> 00:16:34.560 And uh, thank you for listening and feel free to ask any questions. s141 00:16:34.560 --> 00:16:36.560 Thanks.