WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=XczFLWEntiE NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/XczFLWEntiE.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.400 --> 00:00:14.240 Hello, good afternoon everyone. s2 00:00:14.240 --> 00:00:26.320 I'm uh Dean Kinyanola from Runpod Flash and um I want to talk to you about how you might be able to deploy AI applications without Docker. s3 00:00:26.320 --> 00:00:31.760 So currently this is how you deploy a serverless application. s4 00:00:31.760 --> 00:00:34.000 Well, especially AI applications. s5 00:00:34.000 --> 00:00:41.680 You write code, you build your docker file, you run a bunch of commands, docker build, all that. s6 00:00:41.680 --> 00:00:44.640 And then you push the image to docker hub. s7 00:00:44.640 --> 00:00:46.079 Then you execute it. s8 00:00:46.079 --> 00:00:48.480 Then maybe you'll get what you want. s9 00:00:48.480 --> 00:00:56.000 If not, you have to change your code again, rebuild it, repush it, execute. s10 00:00:56.079 --> 00:01:03.680 Um, in my experience, sometimes this one loop takes about 15 minutes depending on the changes you're doing. s11 00:01:03.680 --> 00:01:05.439 So, my name is Dean Kinyanola. s12 00:01:05.439 --> 00:01:12.880 I am a staff engineer at RunPod and I have an entire team that's working on making that easy. s13 00:01:12.880 --> 00:01:15.200 So, why did we build flash? s14 00:01:15.200 --> 00:01:18.960 development iteration loop requires a makeover. s15 00:01:18.960 --> 00:01:20.640 It's really a hassle. s16 00:01:20.640 --> 00:01:27.840 A lot of you guys are software engineers or ML engineers who really don't want to deal with all the DevOps. s17 00:01:27.840 --> 00:01:33.520 So, as you're working on your ideas or as you're working on your experiments, you want the results. s18 00:01:33.520 --> 00:01:35.840 So, no more need for Docker. s19 00:01:35.840 --> 00:01:39.040 Run code like it's local, as if you have the GPU. s20 00:01:39.040 --> 00:01:45.680 Uh test your changes instantly and debug quickly if you need to. s21 00:01:45.759 --> 00:01:48.960 So this is how we're thinking about keeping it simple. s22 00:01:48.960 --> 00:01:54.240 You write your code, you execute, you get the results. s23 00:01:54.240 --> 00:01:57.040 Rewrite your code, execute, you get the results. s24 00:01:57.040 --> 00:01:57.439 Right? s25 00:01:57.439 --> 00:01:59.200 Of course, there's the deployment part. s26 00:01:59.200 --> 00:02:02.079 I'll talk about that later. s27 00:02:02.640 --> 00:02:07.040 So it's so simple it feels like it's running locally. s28 00:02:07.040 --> 00:02:08.720 So what are the features that we have? s29 00:02:08.720 --> 00:02:10.959 So we are able to build and deploy faster. s30 00:02:10.959 --> 00:02:12.959 We optimize the resource utilization. s31 00:02:12.959 --> 00:02:16.480 There's a lot of tricks happening under the hood and it's intuitive. s32 00:02:16.480 --> 00:02:17.760 It's just Python code. s33 00:02:17.760 --> 00:02:24.080 It's really whatever you can run locally with a GPU, just do it on the cloud. s34 00:02:24.080 --> 00:02:26.319 So, what can you do with Flash today? s35 00:02:26.319 --> 00:02:28.239 We have serverless endpoints. s36 00:02:28.239 --> 00:02:32.800 It's mostly locked on serverless right now, but we're working towards doing it for pods. s37 00:02:32.800 --> 00:02:37.680 Uh we have Q endpoints for it and load balance endpoints that you can use it for. s38 00:02:37.680 --> 00:02:44.640 Here's the uh really cool feature that I put in an endpoint to endpoint communication. s39 00:02:44.640 --> 00:02:51.360 Sometimes you're building all these servers, but then you still have to know how to contact each server, right? s40 00:02:51.360 --> 00:02:55.280 You don't have to do that with this because it's just a code talking to another code. s41 00:02:55.280 --> 00:03:02.480 You execute a function and it looks like it's just local, but once you deploy it in different endpoints, it's all talking to each other. s42 00:03:02.480 --> 00:03:04.560 It's all happening under the hood. s43 00:03:04.560 --> 00:03:07.280 We also have network volume attachments. s44 00:03:07.280 --> 00:03:18.879 Um we have warm caching in network volume attachments so that if you ever have to scale up with your serverless endpoints then it has those cache carried over through the network volumes. s45 00:03:18.879 --> 00:03:21.760 And then the most important part is the live serless. s46 00:03:21.760 --> 00:03:27.519 Uh you are coding and testing as if it's all happening in your local system. s47 00:03:27.519 --> 00:03:32.879 And then you know this would just be a toy project if you can't deploy to production, right? s48 00:03:32.879 --> 00:03:34.560 So you want to deploy to production. s49 00:03:34.560 --> 00:03:38.400 So serve the inference, serve all the uh stuff that you're building. s50 00:03:38.400 --> 00:03:42.400 And we're making that possible all through the CLI. s51 00:03:42.400 --> 00:03:44.560 So let's get a demo. s52 00:03:44.560 --> 00:03:47.760 Hopefully I can do this with one hand. s53 00:03:48.640 --> 00:03:49.840 Okay. s54 00:03:49.840 --> 00:03:50.239 All right. s55 00:03:50.239 --> 00:03:56.879 So my demo here shows the the rapid iteration and evolution of a function. s56 00:03:56.879 --> 00:04:04.000 So the first function is just talking to torch getting what kind of GPU you have. s57 00:04:04.000 --> 00:04:06.080 So I'll do this live. s58 00:04:06.080 --> 00:04:10.560 I'll do flash dev. s59 00:04:10.560 --> 00:04:14.959 And what this does is it launches a fast API local host. s60 00:04:14.959 --> 00:04:17.280 So you can actually toy around with it. s61 00:04:17.280 --> 00:04:19.919 So now you want to try it out. s62 00:04:19.919 --> 00:04:21.359 Let's try it out. s63 00:04:21.359 --> 00:04:23.040 Execute. s64 00:04:23.040 --> 00:04:25.440 So it's provisioning this serverless endpoint. s65 00:04:25.440 --> 00:04:30.160 And if we go to runpod console, refresh this, you'll see this. s66 00:04:30.160 --> 00:04:32.880 This is running now. s67 00:04:33.120 --> 00:04:38.160 And as it's loading, let's let's show the logs. s68 00:04:38.160 --> 00:04:41.120 So you can actually see it. s69 00:04:44.880 --> 00:04:45.199 All right. s70 00:04:45.199 --> 00:04:47.759 Workers ready. s71 00:04:52.639 --> 00:04:54.639 Yep. s72 00:05:02.720 --> 00:05:05.280 Oh, John. s73 00:05:06.720 --> 00:05:07.759 Cold start. s74 00:05:07.759 --> 00:05:11.600 This is common with all serverless platforms. s75 00:05:13.199 --> 00:05:14.000 Oh, there it is. s76 00:05:14.000 --> 00:05:15.840 So, it's running. s77 00:05:15.840 --> 00:05:19.199 We got logs, container logs. s78 00:05:19.199 --> 00:05:24.160 So, it's testing all these things, making sure it's all healthy, and it ran the job. s79 00:05:24.160 --> 00:05:27.600 So, now if we go back here, you'll see the result. s80 00:05:27.600 --> 00:05:32.320 It ran on an Nvidia GeForce RTX 4090. s81 00:05:32.320 --> 00:05:34.800 You see all the RAM, GPU count. s82 00:05:34.800 --> 00:05:36.639 Now, here's the kicker. s83 00:05:36.639 --> 00:05:41.680 What if I want to change this function and just want to test it? s84 00:05:41.680 --> 00:05:44.880 So, here's a different iteration of this function. s85 00:05:44.880 --> 00:05:49.120 What we do is we add more code. s86 00:05:49.120 --> 00:05:53.680 And this code really just does generation of text. s87 00:05:53.680 --> 00:05:57.039 And I've added some dependencies, the Python dependencies here. s88 00:05:57.039 --> 00:06:01.759 Oh, actually, let's do this without the hugging face token first. s89 00:06:01.759 --> 00:06:03.120 Let's see what's happening. s90 00:06:03.120 --> 00:06:05.360 See, it refreshed everything. s91 00:06:05.360 --> 00:06:07.759 I refreshed this. s92 00:06:07.759 --> 00:06:10.240 Test it again. s93 00:06:11.039 --> 00:06:12.160 What prompt do we need? s94 00:06:12.160 --> 00:06:15.600 Uh, let's do string cheese. s95 00:06:15.600 --> 00:06:18.080 Is it real? s96 00:06:19.600 --> 00:06:21.840 Run it. s97 00:06:22.479 --> 00:06:25.280 You see it again here. s98 00:06:30.319 --> 00:06:31.280 Container. s99 00:06:31.280 --> 00:06:33.280 Okay. s100 00:06:33.680 --> 00:06:35.680 It's coming up. s101 00:06:35.680 --> 00:06:41.440 You'll see all the the logs are simpler here because you're not supposed to be inundated with so much stuff. s102 00:06:41.440 --> 00:06:54.960 But what it's doing is in on the fly installing the dependencies and it even gave me a warning about communicating with the hugging face server without the token and it should be finished. s103 00:06:54.960 --> 00:06:56.639 Okay, now it's finished. s104 00:06:56.639 --> 00:06:58.160 Let's see the result. s105 00:06:58.160 --> 00:07:01.120 I ask is string cheese real? s106 00:07:01.120 --> 00:07:04.560 I'm sorry but I cannot provide answers. s107 00:07:05.360 --> 00:07:08.319 All right, that's that's spicy. s108 00:07:08.639 --> 00:07:11.199 All right, let's let's change that up. s109 00:07:11.199 --> 00:07:15.199 Let's enable my HF token. s110 00:07:15.199 --> 00:07:17.840 That's just changing the code right now. s111 00:07:17.840 --> 00:07:19.680 We release this again. s112 00:07:19.680 --> 00:07:22.720 Let's change the uh prompt. s113 00:07:25.599 --> 00:07:31.759 Let's see who is Batman. s114 00:07:31.759 --> 00:07:34.400 Let's execute that. s115 00:07:35.199 --> 00:07:36.240 Ready? s116 00:07:36.240 --> 00:07:40.240 And let's see how's this working here. s117 00:07:41.360 --> 00:07:41.840 It's cool. s118 00:07:41.840 --> 00:07:44.880 I'm doing this with one hand. s119 00:07:46.240 --> 00:07:48.080 All right, let's finish. s120 00:07:48.080 --> 00:07:49.759 Now we see the result. s121 00:07:49.759 --> 00:07:50.319 Oh, there. s122 00:07:50.319 --> 00:07:50.720 There you go. s123 00:07:50.720 --> 00:07:52.240 Batman's a fictional character. s124 00:07:52.240 --> 00:07:54.319 All right, we knew that. s125 00:07:54.319 --> 00:08:00.879 Let's go for something that's more intensive than just generating text. s126 00:08:00.879 --> 00:08:07.199 Now, we'll do an image generation. s127 00:08:07.360 --> 00:08:10.160 All right, it refreshed. s128 00:08:10.160 --> 00:08:12.879 Let's refresh this local host. s129 00:08:12.879 --> 00:08:20.479 Now you see generate image and then we'll test this out with a prompt. s130 00:08:20.479 --> 00:08:28.479 Let's create an image of a string cheese Batman. s131 00:08:29.680 --> 00:08:34.880 All right, this is how creative AI could be. s132 00:08:34.959 --> 00:08:38.080 Let's look at what's happening here. s133 00:08:40.159 --> 00:08:44.080 Running logs. s134 00:08:44.480 --> 00:08:45.279 Oh, look at that. s135 00:08:45.279 --> 00:08:48.000 Roll out in progress. s136 00:08:55.519 --> 00:08:57.519 Okay. s137 00:08:59.519 --> 00:09:02.399 I got plenty of time. s138 00:09:07.200 --> 00:09:08.720 That's not it. s139 00:09:08.720 --> 00:09:09.279 All right. s140 00:09:09.279 --> 00:09:13.680 I just had to terminate that because it was running the the older version. s141 00:09:13.680 --> 00:09:14.000 All right. s142 00:09:14.000 --> 00:09:15.600 Now it's new. s143 00:09:15.600 --> 00:09:18.959 I didn't want to wait for that roll out. s144 00:09:20.320 --> 00:09:23.760 I just want to see string cheese Batman. s145 00:09:23.920 --> 00:09:24.240 All right. s146 00:09:24.240 --> 00:09:28.480 So, it's it responded with a base 64. s147 00:09:28.480 --> 00:09:29.360 Okay. s148 00:09:29.360 --> 00:09:33.120 Let me clean that up before I show that. s149 00:09:34.880 --> 00:09:41.519 So how you can render this is through the browser. s150 00:09:47.600 --> 00:09:52.880 All right, let's see what string cheese Batman looks like. s151 00:09:53.440 --> 00:09:57.519 That's not string cheese, but that proves it right. s152 00:09:57.519 --> 00:09:59.040 It did it. s153 00:09:59.040 --> 00:10:01.360 Now, what if you want to deploy this? s154 00:10:01.360 --> 00:10:02.240 So, check this out. s155 00:10:02.240 --> 00:10:05.279 When I cancel it, it actually deprovisioned that endpoint. s156 00:10:05.279 --> 00:10:09.360 So, you don't have to have that lying around. s157 00:10:09.360 --> 00:10:11.279 See, it's gone now. s158 00:10:11.279 --> 00:10:13.680 Now, what if we want to deploy this? s159 00:10:13.680 --> 00:10:16.480 So, let's deploy that. s160 00:10:18.160 --> 00:10:20.480 Flash deploy. s161 00:10:20.480 --> 00:10:27.120 That packages up the whole thing, builds it up, sends it to our servers. s162 00:10:27.120 --> 00:10:28.959 Again, no Docker. s163 00:10:28.959 --> 00:10:30.640 You're not dealing with Docker here. s164 00:10:30.640 --> 00:10:37.920 This is instantaneous packaging of your application, sending it out to our servers, and serving it. s165 00:10:38.160 --> 00:10:39.839 Now, here's the cool part. s166 00:10:39.839 --> 00:10:51.279 Once it's out there, I have prepared this prompt, but I just need to change this ID later because this is going to matter. s167 00:10:51.279 --> 00:10:51.600 All right. s168 00:10:51.600 --> 00:10:55.680 So, it's created the archive built flash. s169 00:10:55.680 --> 00:10:56.160 All right. s170 00:10:56.160 --> 00:10:56.800 Sending it out. s171 00:10:56.800 --> 00:10:58.160 Uploaded. s172 00:10:58.160 --> 00:10:59.600 Deploying. s173 00:10:59.600 --> 00:11:01.279 Now it's up. s174 00:11:01.279 --> 00:11:01.600 All right. s175 00:11:01.600 --> 00:11:04.560 Let's see the ID here. s176 00:11:07.360 --> 00:11:09.360 Okay. s177 00:11:09.440 --> 00:11:13.600 Now, let's get this going. s178 00:11:13.600 --> 00:11:18.959 So, my prompt is a cute baby sea otter with a bow tie. s179 00:11:19.440 --> 00:11:22.000 Generate that image. s180 00:11:22.000 --> 00:11:22.320 Oh, yeah. s181 00:11:22.320 --> 00:11:24.560 I want to show you where this is. s182 00:11:24.560 --> 00:11:26.240 Now this is fully servable, right? s183 00:11:26.240 --> 00:11:30.720 Like you can have your customers, you know, use your application that uses this back end. s184 00:11:30.720 --> 00:11:33.120 It no longer lives in my local host. s185 00:11:33.120 --> 00:11:35.920 This is being hosted. s186 00:11:37.519 --> 00:11:42.800 All right, let's see workers log. s187 00:11:45.040 --> 00:11:51.680 Okay, when it's done, it should print the output here. s188 00:11:51.760 --> 00:11:53.040 There we go. s189 00:11:53.040 --> 00:11:59.200 All right, let me wrap this so I could pull this up. s190 00:12:03.920 --> 00:12:06.720 This is crazy doing it in one hand. s191 00:12:06.720 --> 00:12:10.959 All right, let's load up the picture of a sea otter with a bow tie. s192 00:12:10.959 --> 00:12:13.760 There's no bow tie. s193 00:12:14.079 --> 00:12:16.160 But that's not the point. s194 00:12:16.160 --> 00:12:17.920 We All right. s195 00:12:17.920 --> 00:12:19.200 So, you get that. s196 00:12:19.200 --> 00:12:20.639 That's the demo. s197 00:12:20.639 --> 00:12:27.600 We had a hackathon yesterday and I just wanted to be to present this to I'm very proud of this guy uh Funka. s198 00:12:27.600 --> 00:12:43.760 He created a wrapper around our flash that enables you to upload a data sheet of all your experiments of some specifications of some models and run it instantly and it just fans out builds out all these endpoints s199 00:12:43.760 --> 00:12:47.920 runs your training and then reports back to you. s200 00:12:47.920 --> 00:12:52.320 And he did this within three to four hours. s201 00:12:52.320 --> 00:12:55.760 And he had no idea how to do runpod flash. s202 00:12:55.760 --> 00:12:58.240 He had to ask me questions if this was even possible. s203 00:12:58.240 --> 00:13:00.160 I'm like, let's see. s204 00:13:00.160 --> 00:13:10.560 And he was very surprised when he won yesterday because his his project was not even his best one apparently and he keeps losing to other hackathons. s205 00:13:10.560 --> 00:13:13.519 So just really proud of this guy. s206 00:13:13.519 --> 00:13:15.360 How do you get started with runpad flash? s207 00:13:15.360 --> 00:13:27.040 So, well, you sign up for an account on RunPod, scan that QR code, get some $10 uh credit now, and all you really have to do is just install it, write your code, s208 00:13:27.040 --> 00:13:39.040 log in to have your uh API key in there, initialize, it'll have some skeleton application if you want, flash, the local dev that you have, or flash deploy to production. s209 00:13:39.040 --> 00:13:41.040 And we have all these URLs. s210 00:13:41.040 --> 00:13:53.519 The best advice I have is visit the runbot flash examples on GitHub because we have all these cases and examples that you could kind of refer or if you like AI coding just s211 00:13:53.519 --> 00:13:58.480 run cloud code into that repository and it will learn everything. s212 00:13:58.480 --> 00:14:01.600 We also have skills if you're into that. s213 00:14:01.600 --> 00:14:04.399 And that is my presentation.