AI Apps in a Flash: Ship to GPUs Without Docker — Dean Quiñanola, Runpod

AI Engineer · 14 min · 213 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:12Hello, good afternoon everyone.
2. 00:14I'm uh Dean Kinyanola from Runpod Flash and um I want to talk to you about how you might be able to deploy AI applications without Docker.
3. 00:26So currently this is how you deploy a serverless application.
4. 00:31Well, especially AI applications.
5. 00:34You write code, you build your docker file, you run a bunch of commands, docker build, all that.
6. 00:41And then you push the image to docker hub.
7. 00:44Then you execute it.
8. 00:46Then maybe you'll get what you want.
9. 00:48If not, you have to change your code again, rebuild it, repush it, execute.
10. 00:56Um, in my experience, sometimes this one loop takes about 15 minutes depending on the changes you're doing.
11. 01:03So, my name is Dean Kinyanola.
12. 01:05I am a staff engineer at RunPod and I have an entire team that's working on making that easy.
13. 01:12So, why did we build flash?
14. 01:15development iteration loop requires a makeover.
15. 01:18It's really a hassle.
16. 01:20A lot of you guys are software engineers or ML engineers who really don't want to deal with all the DevOps.
17. 01:27So, as you're working on your ideas or as you're working on your experiments, you want the results.
18. 01:33So, no more need for Docker.
19. 01:35Run code like it's local, as if you have the GPU.
20. 01:39Uh test your changes instantly and debug quickly if you need to.
21. 01:45So this is how we're thinking about keeping it simple.
22. 01:48You write your code, you execute, you get the results.
23. 01:54Rewrite your code, execute, you get the results.
24. 01:57Right?
25. 01:57Of course, there's the deployment part.
26. 01:59I'll talk about that later.
27. 02:02So it's so simple it feels like it's running locally.
28. 02:07So what are the features that we have?
29. 02:08So we are able to build and deploy faster.
30. 02:10We optimize the resource utilization.
31. 02:12There's a lot of tricks happening under the hood and it's intuitive.
32. 02:16It's just Python code.
33. 02:17It's really whatever you can run locally with a GPU, just do it on the cloud.
34. 02:24So, what can you do with Flash today?
35. 02:26We have serverless endpoints.
36. 02:28It's mostly locked on serverless right now, but we're working towards doing it for pods.
37. 02:32Uh we have Q endpoints for it and load balance endpoints that you can use it for.
38. 02:37Here's the uh really cool feature that I put in an endpoint to endpoint communication.
39. 02:44Sometimes you're building all these servers, but then you still have to know how to contact each server, right?
40. 02:51You don't have to do that with this because it's just a code talking to another code.
41. 02:55You execute a function and it looks like it's just local, but once you deploy it in different endpoints, it's all talking to each other.
42. 03:02It's all happening under the hood.
43. 03:04We also have network volume attachments.
44. 03:07Um we have warm caching in network volume attachments so that if you ever have to scale up with your serverless endpoints then it has those cache carried over through the network volumes.
45. 03:18And then the most important part is the live serless.
46. 03:21Uh you are coding and testing as if it's all happening in your local system.
47. 03:27And then you know this would just be a toy project if you can't deploy to production, right?
48. 03:32So you want to deploy to production.
49. 03:34So serve the inference, serve all the uh stuff that you're building.
50. 03:38And we're making that possible all through the CLI.
51. 03:42So let's get a demo.
52. 03:44Hopefully I can do this with one hand.
53. 03:48Okay.
54. 03:49All right.
55. 03:50So my demo here shows the the rapid iteration and evolution of a function.
56. 03:56So the first function is just talking to torch getting what kind of GPU you have.
57. 04:04So I'll do this live.
58. 04:06I'll do flash dev.
59. 04:10And what this does is it launches a fast API local host.
60. 04:14So you can actually toy around with it.
61. 04:17So now you want to try it out.
62. 04:19Let's try it out.
63. 04:21Execute.
64. 04:23So it's provisioning this serverless endpoint.
65. 04:25And if we go to runpod console, refresh this, you'll see this.
66. 04:30This is running now.
67. 04:33And as it's loading, let's let's show the logs.
68. 04:38So you can actually see it.
69. 04:44All right.
70. 04:45Workers ready.
71. 04:52Yep.
72. 05:02Oh, John.
73. 05:06Cold start.
74. 05:07This is common with all serverless platforms.
75. 05:13Oh, there it is.
76. 05:14So, it's running.
77. 05:15We got logs, container logs.
78. 05:19So, it's testing all these things, making sure it's all healthy, and it ran the job.
79. 05:24So, now if we go back here, you'll see the result.
80. 05:27It ran on an Nvidia GeForce RTX 4090.
81. 05:32You see all the RAM, GPU count.
82. 05:34Now, here's the kicker.
83. 05:36What if I want to change this function and just want to test it?
84. 05:41So, here's a different iteration of this function.
85. 05:44What we do is we add more code.
86. 05:49And this code really just does generation of text.
87. 05:53And I've added some dependencies, the Python dependencies here.
88. 05:57Oh, actually, let's do this without the hugging face token first.
89. 06:01Let's see what's happening.
90. 06:03See, it refreshed everything.
91. 06:05I refreshed this.
92. 06:07Test it again.
93. 06:11What prompt do we need?
94. 06:12Uh, let's do string cheese.
95. 06:15Is it real?
96. 06:19Run it.
97. 06:22You see it again here.
98. 06:30Container.
99. 06:31Okay.
100. 06:33It's coming up.
101. 06:35You'll see all the the logs are simpler here because you're not supposed to be inundated with so much stuff.
102. 06:41But what it's doing is in on the fly installing the dependencies and it even gave me a warning about communicating with the hugging face server without the token and it should be finished.
103. 06:54Okay, now it's finished.
104. 06:56Let's see the result.
105. 06:58I ask is string cheese real?
106. 07:01I'm sorry but I cannot provide answers.
107. 07:05All right, that's that's spicy.
108. 07:08All right, let's let's change that up.
109. 07:11Let's enable my HF token.
110. 07:15That's just changing the code right now.
111. 07:17We release this again.
112. 07:19Let's change the uh prompt.
113. 07:25Let's see who is Batman.
114. 07:31Let's execute that.
115. 07:35Ready?
116. 07:36And let's see how's this working here.
117. 07:41It's cool.
118. 07:41I'm doing this with one hand.
119. 07:46All right, let's finish.
120. 07:48Now we see the result.
121. 07:49Oh, there.
122. 07:50There you go.
123. 07:50Batman's a fictional character.
124. 07:52All right, we knew that.
125. 07:54Let's go for something that's more intensive than just generating text.
126. 08:00Now, we'll do an image generation.
127. 08:07All right, it refreshed.
128. 08:10Let's refresh this local host.
129. 08:12Now you see generate image and then we'll test this out with a prompt.
130. 08:20Let's create an image of a string cheese Batman.
131. 08:29All right, this is how creative AI could be.
132. 08:34Let's look at what's happening here.
133. 08:40Running logs.
134. 08:44Oh, look at that.
135. 08:45Roll out in progress.
136. 08:55Okay.
137. 08:59I got plenty of time.
138. 09:07That's not it.
139. 09:08All right.
140. 09:09I just had to terminate that because it was running the the older version.
141. 09:13All right.
142. 09:14Now it's new.
143. 09:15I didn't want to wait for that roll out.
144. 09:20I just want to see string cheese Batman.
145. 09:23All right.
146. 09:24So, it's it responded with a base 64.
147. 09:28Okay.
148. 09:29Let me clean that up before I show that.
149. 09:34So how you can render this is through the browser.
150. 09:47All right, let's see what string cheese Batman looks like.
151. 09:53That's not string cheese, but that proves it right.
152. 09:57It did it.
153. 09:59Now, what if you want to deploy this?
154. 10:01So, check this out.
155. 10:02When I cancel it, it actually deprovisioned that endpoint.
156. 10:05So, you don't have to have that lying around.
157. 10:09See, it's gone now.
158. 10:11Now, what if we want to deploy this?
159. 10:13So, let's deploy that.
160. 10:18Flash deploy.
161. 10:20That packages up the whole thing, builds it up, sends it to our servers.
162. 10:27Again, no Docker.
163. 10:28You're not dealing with Docker here.
164. 10:30This is instantaneous packaging of your application, sending it out to our servers, and serving it.
165. 10:38Now, here's the cool part.
166. 10:39Once it's out there, I have prepared this prompt, but I just need to change this ID later because this is going to matter.
167. 10:51All right.
168. 10:51So, it's created the archive built flash.
169. 10:55All right.
170. 10:56Sending it out.
171. 10:56Uploaded.
172. 10:58Deploying.
173. 10:59Now it's up.
174. 11:01All right.
175. 11:01Let's see the ID here.
176. 11:07Okay.
177. 11:09Now, let's get this going.
178. 11:13So, my prompt is a cute baby sea otter with a bow tie.
179. 11:19Generate that image.
180. 11:22Oh, yeah.
181. 11:22I want to show you where this is.
182. 11:24Now this is fully servable, right?
183. 11:26Like you can have your customers, you know, use your application that uses this back end.
184. 11:30It no longer lives in my local host.
185. 11:33This is being hosted.
186. 11:37All right, let's see workers log.
187. 11:45Okay, when it's done, it should print the output here.
188. 11:51There we go.
189. 11:53All right, let me wrap this so I could pull this up.
190. 12:03This is crazy doing it in one hand.
191. 12:06All right, let's load up the picture of a sea otter with a bow tie.
192. 12:10There's no bow tie.
193. 12:14But that's not the point.
194. 12:16We All right.
195. 12:17So, you get that.
196. 12:19That's the demo.
197. 12:20We had a hackathon yesterday and I just wanted to be to present this to I'm very proud of this guy uh Funka.
198. 12:27He created a wrapper around our flash that enables you to upload a data sheet of all your experiments of some specifications of some models and run it instantly and it just fans out builds out all these endpoints
199. 12:43runs your training and then reports back to you.
200. 12:47And he did this within three to four hours.
201. 12:52And he had no idea how to do runpod flash.
202. 12:55He had to ask me questions if this was even possible.
203. 12:58I'm like, let's see.
204. 13:00And he was very surprised when he won yesterday because his his project was not even his best one apparently and he keeps losing to other hackathons.
205. 13:10So just really proud of this guy.
206. 13:13How do you get started with runpad flash?
207. 13:15So, well, you sign up for an account on RunPod, scan that QR code, get some $10 uh credit now, and all you really have to do is just install it, write your code,
208. 13:27log in to have your uh API key in there, initialize, it'll have some skeleton application if you want, flash, the local dev that you have, or flash deploy to production.
209. 13:39And we have all these URLs.
210. 13:41The best advice I have is visit the runbot flash examples on GitHub because we have all these cases and examples that you could kind of refer or if you like AI coding just
211. 13:53run cloud code into that repository and it will learn everything.
212. 13:58We also have skills if you're into that.
213. 14:01And that is my presentation.