WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=7BcQdtRA3Tc NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/7BcQdtRA3Tc.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:13.360 --> 00:00:15.280 Uh so let's get started. s3 00:00:15.280 --> 00:00:17.440 Uh hi, my name is Abdul. s4 00:00:17.440 --> 00:00:20.560 Uh uh I'm from modular. s5 00:00:20.560 --> 00:00:26.560 Uh today I'm going to be talking about what modular is and what value it brings to you. s6 00:00:26.560 --> 00:00:34.559 So as you probably know today there is a huge crunch uh for compute and there's like no alleviation in sight. s7 00:00:34.559 --> 00:00:39.200 A lot of these things are booked for the next few years. s8 00:00:39.760 --> 00:00:43.200 But there's also a kind of as a result you have an explosion of hardware. s9 00:00:43.200 --> 00:00:45.120 We call it like a Cambrian explosion. s10 00:00:45.120 --> 00:00:54.960 You have Nvidia providing you know a few generations of GPUs, the Intels, the ARMs, the Apples, everything uh is is present. s11 00:00:54.960 --> 00:00:58.559 They all have some niches uh that they cover. s12 00:00:58.559 --> 00:01:05.120 But the problem is that you have to ask is how do you program all of these uh systems. s13 00:01:06.640 --> 00:01:19.840 Now if you look at the current software as it is today um it's kind of a mess as well and that's because a lot of these hardware choices bubble up from the you know from down under s14 00:01:19.840 --> 00:01:22.400 uh into upwards into the stack. s15 00:01:22.400 --> 00:01:29.680 So this is a diagram showing how uh VLM supports both Nvidia and AMD. s16 00:01:29.680 --> 00:01:34.560 If you add extra hardware the the the diagram becomes even more difficult. s17 00:01:34.560 --> 00:01:48.880 Now today folks actually uh deploy not only using VLM but also SG lang TRTLM uh Llama CPP etc. So at Modra we thought well this is kind of a huge mess. s18 00:01:48.880 --> 00:02:00.560 Let's rearchitect the uh the stack from the ground up and let's build a unifying layer uh that unifies all the hardware so everything runs on our stack. s19 00:02:00.880 --> 00:02:12.800 To realize this, you have to actually go back to first principles and build a complete software stack with portability, performance, and programmability in mind. s20 00:02:12.800 --> 00:02:23.200 So at the bottommost level, we've developed Mojo, a heterogeneous programming language that's targeted towards AI as well as general purpose programming uh as well. s21 00:02:23.200 --> 00:02:25.520 It's kind of pythonic in style. s22 00:02:25.520 --> 00:02:31.200 So folks that actually know Python today pretty much know 90% of Mojo. s23 00:02:31.200 --> 00:02:38.160 Uh so we've built that language and that targets both CPUs, GPUs as well as all our accelerators. s24 00:02:38.160 --> 00:02:44.160 On top of that, we leverage Mojo to build max our inference engine. s25 00:02:44.160 --> 00:02:53.599 Uh so that provides you with uh AI inference for both for a bunch of different modalities like image, video, speech as well as text. s26 00:02:53.599 --> 00:03:00.080 We've built all our kernels underneath the hood that are built on top of Mojo. s27 00:03:00.080 --> 00:03:03.280 Finally, today's AI inference is distributed. s28 00:03:03.280 --> 00:03:06.080 So you actually need a cloud solution as well. s29 00:03:06.080 --> 00:03:18.400 uh you need to pick which hardware to deploy your model on, how to route uh different user requests to different uh machines, how to do KV cache offloading and and and distributed inference s30 00:03:18.400 --> 00:03:26.239 and disagreed inference uh to be able to serve the model at the speed and the cost that that folks expect. s31 00:03:26.239 --> 00:03:31.840 So putting it all together, we built a modular platform that's fully programmable. s32 00:03:31.840 --> 00:03:33.920 It's hardware abstracted. s33 00:03:33.920 --> 00:03:36.080 It's it's soda perf. s34 00:03:36.080 --> 00:03:41.440 Uh and then it actually can be deployed both on prem as well as on our cloud as well. s35 00:03:41.440 --> 00:03:49.519 So it can be if you have a cloud uh uh setup, you can actually download the modular cloud and and deployed there. s36 00:03:49.519 --> 00:03:54.319 So let me kind of dig into each one at a time, each part of the stack. s37 00:03:54.319 --> 00:03:56.640 So the first is Mojo. s38 00:03:56.640 --> 00:04:01.360 Uh so Mojo as I mentioned is a Pythonic system programming language. s39 00:04:01.360 --> 00:04:06.000 It's not unlike C++, Rust or CUDA C++. s40 00:04:06.000 --> 00:04:11.519 The thing is what it enables is uh programmability across different accelerators. s41 00:04:11.519 --> 00:04:13.040 It's blazing fast. s42 00:04:13.040 --> 00:04:18.479 It's again it actually competes with CUDA in terms of performance. s43 00:04:18.479 --> 00:04:26.880 uh and you can think of it as the best way to program CPUs and GPUs within a Python-like programming language. s44 00:04:27.280 --> 00:04:37.280 In terms of how we think about Mojo, uh you know, you can think about there's three different quadrants, usability, performance, and portability. s45 00:04:37.280 --> 00:04:44.560 Rockcom and KOD and CUDA gives you massive performance, but they sacrifice on both usability and programmability. s46 00:04:44.560 --> 00:04:48.960 A lot of folks use agents to actually program these system nowadays. s47 00:04:48.960 --> 00:04:52.880 Open CL back in you know a few years ago was popular. s48 00:04:52.880 --> 00:04:57.280 It gives you great portability but is sacrificed on performance and usability. s49 00:04:57.280 --> 00:05:10.320 Triton is has a good sweet spot that gives you 70% of the Perf uh but it gives you portability and usability and Mojo is is trying to kind of fill the niche of the sweet spot s50 00:05:10.320 --> 00:05:11.919 where it's the best of both worlds. s51 00:05:11.919 --> 00:05:23.600 It gives you the abstraction to allow you to write portable kernels and and uh and usable kernels while giving you also the low-level access uh to get the maximum performance. s52 00:05:23.840 --> 00:05:29.360 We've actually open sourced uh practically all of our kernels so you can actually test this stuff. s53 00:05:29.360 --> 00:05:31.680 You can benchmark it on your own machine. s54 00:05:31.680 --> 00:05:36.320 Uh we have state-of-the-art kernels on Nvidia, AMD and other hardware. s55 00:05:36.320 --> 00:05:45.280 So to tell you what state-of-the-art is, it means that our matrix multiplication is faster than the coupe losses, the hip losses, everything out there. s56 00:05:45.280 --> 00:05:48.639 And it's not by 1% in some cases by 2%. s57 00:05:48.639 --> 00:05:59.039 We have the fastest attention uh kernel out there, faster than flash attention for uh as well as like the AI it and things like that. s58 00:05:59.360 --> 00:06:06.960 Uh one thing that we've announced uh a month ago or so is Mojo 1.0, know the stabilization of our APIs. s59 00:06:06.960 --> 00:06:10.880 Uh, and we're going to be releasing Mojo 1.0 in the next few months. s60 00:06:10.880 --> 00:06:16.400 And we're also open sourcing Mojo uh, in the next few months as well. s61 00:06:17.520 --> 00:06:21.199 Sitting on top of Mojo is Max. s62 00:06:21.199 --> 00:06:29.039 So, Max is trying to kind of solve a few of the uh, assumptions that were made uh, back in the day for PyTorch. s63 00:06:29.039 --> 00:06:31.360 PyTorch is about 10 years old. s64 00:06:31.360 --> 00:06:35.520 It was not really designed for the genai inference of today. s65 00:06:35.520 --> 00:06:44.639 As a result, you have like VLM and SG lang that stepped into the void to give you things like KV cache, page detention and and things like that. s66 00:06:45.440 --> 00:06:50.240 And but all of that kind of kind of complicated the entire PyTorch ecosystem. s67 00:06:50.240 --> 00:06:54.639 A lot of folks don't really uh use PyTorch as is. s68 00:06:54.639 --> 00:06:58.639 They're resorting to these layers on top. s69 00:06:58.880 --> 00:07:02.560 So think of a new approach and we call that the max framework. s70 00:07:02.560 --> 00:07:12.080 So again it's built from the ground up uh with the best technologies with the best state-of-the-art graph compilers uh the eager mode execution. s71 00:07:12.080 --> 00:07:13.680 It's portable across devices. s72 00:07:13.680 --> 00:07:25.759 It understands your mojo programs to do novel fusion uh and it gives you the maximum performance and deployability that you're otherwise not able to do in in in regular PyTorch. s73 00:07:26.800 --> 00:07:39.919 So compared to other stacks uh this is a single stack that allows you to serve develop the model as well as optimize everything in one cohesive uh system. s74 00:07:40.560 --> 00:08:00.720 it within Max we support thousands and thousands of models in fact thousands of architectures uh everything from Gemma 4 to M uh Miniax 3 to GLM uh so any model that you think about is is practically supported in uh in in uh in Max s75 00:08:00.720 --> 00:08:06.400 and it has the same interface that you'd expect uh from what is standard today. s76 00:08:06.400 --> 00:08:12.800 So we also provide a open AI endpoints that you can use uh within the stack. s77 00:08:14.400 --> 00:08:25.919 If you're familiar with like PyTorch programming uh the figure on the right would or the left will not be like massive departure of your mental model. s78 00:08:25.919 --> 00:08:28.000 Uh you can build a model as you expect. s79 00:08:28.000 --> 00:08:30.639 You know you can import modules. s80 00:08:30.639 --> 00:08:34.640 The uh programming language for developing models is Python. s81 00:08:34.640 --> 00:08:42.719 So as familiar to folks uh and you can compose uh things as as you as you as you um as you build them. s82 00:08:42.719 --> 00:08:48.080 We also support like the eager execution to enable debugging as well. s83 00:08:49.440 --> 00:09:00.399 Now the other thing that one has to realize is modern LLMs uh are run on across different GPUs and in many cases multiple nodes. s84 00:09:00.399 --> 00:09:14.240 So also max the max APIs provide you with a way to uh program multiGPU system AC as well as multi-node systems from within the API so that the model is natively s85 00:09:14.240 --> 00:09:19.360 supported across uh different GPUs and different nodes as well. s86 00:09:19.360 --> 00:09:21.120 And this can also be heterogeneous. s87 00:09:21.120 --> 00:09:35.760 So there's nothing to stop you from saying the prefill is on hardware X and the decode is on hardware Y. Finally, layered on top of all of this is the modular cloud. s88 00:09:35.760 --> 00:09:40.160 So, as I mentioned, uh today's AI inference is distributed. s89 00:09:40.160 --> 00:09:42.720 It runs on many many different machines. s90 00:09:42.720 --> 00:09:44.880 Uh it's heterogeneous. s91 00:09:44.880 --> 00:09:55.279 So we provide the module cloud which is a performance native you know we think of it as a glass box because you can actually tinker with all the variables s92 00:09:55.279 --> 00:10:02.160 uh that are available for you in both max mojo and everything in between uh within your endpoints. s93 00:10:02.160 --> 00:10:09.360 So it provides provides endpoint deployment as well as bring your own cloud deployments as well. s94 00:10:09.360 --> 00:10:21.839 We also have a module cloud lab that allows you to experiment with all these different parameters and pick the best point in the perto frontier that you have for that model or based on your workload as well. s95 00:10:22.240 --> 00:10:31.920 Uh so in terms of like the UI uh if you're familiar with like any uh cloud provider I mean it's it should be pretty familiar. s96 00:10:31.920 --> 00:10:42.560 You just go to the website, you click on which model to you want to run, you get an AP an API key and then you plug that in uh to your HTTP request s97 00:10:42.560 --> 00:10:45.440 and uh you can run it. s98 00:10:45.440 --> 00:10:49.519 You can have dedicated endpoints as well as we can serve your own custom model. s99 00:10:49.519 --> 00:10:55.760 Uh you know in many cases folks have fine-tuned models or models with their own custom speculative decoders. s100 00:10:55.760 --> 00:11:00.240 Uh we could do all of that uh for you as well. s101 00:11:00.399 --> 00:11:03.120 Now, why do you care about all of this, right? s102 00:11:03.120 --> 00:11:04.880 It's a lot of tech. s103 00:11:04.880 --> 00:11:06.880 Uh why do folks care? s104 00:11:06.880 --> 00:11:14.880 Well, it's because you can actually uh accomplish more things uh through the system, right? s105 00:11:14.880 --> 00:11:17.040 So, you can actually decrease cost. s106 00:11:17.040 --> 00:11:21.120 You can get greater flexibility because you can actually use different hardware. s107 00:11:21.120 --> 00:11:23.839 You're not beholden to one hardware provider. s108 00:11:23.839 --> 00:11:25.360 Uh you get portability. s109 00:11:25.360 --> 00:11:27.519 You can use multiple different models. s110 00:11:27.519 --> 00:11:32.240 Uh etc. So how does Perf look like? s111 00:11:32.240 --> 00:11:40.203 Well, we benchmarked a lot of these uh and this is like against uh state-of-the-art, you know, the best combination of VLM or SG lang s112 00:11:40.203 --> 00:11:40.320 [snorts] s113 00:11:40.320 --> 00:11:41.440 in a bunch of cases. s114 00:11:41.440 --> 00:11:45.680 These are these benchmarks are independently verified as well. s115 00:11:45.680 --> 00:11:55.600 So in terms of Kimmy Kimmy K2.5, we're 5.5 faster on TTFT uh compared to VLM. s116 00:11:55.600 --> 00:11:59.279 We can also get 1.5x throughput improvements. s117 00:11:59.279 --> 00:12:09.120 So the token per minutes on per GPU is 1.5x faster uh on B200 than it is on VLM. s118 00:12:11.680 --> 00:12:22.959 In terms of Flux 2, and I encourage folks to actually visit the booth behind this curtain, uh there's a nice demo of Flux 2. uh we're about 7x faster s119 00:12:22.959 --> 00:12:26.320 uh compared to ters that compile with diffusers. s120 00:12:26.320 --> 00:12:30.560 So again, Max supports different modalities within the same framework. s121 00:12:30.560 --> 00:12:36.160 So you don't have to use an omni variant of a of a piece of software or anything like that. s122 00:12:36.160 --> 00:12:42.959 We support both video generation, audio generation, image generation as well as text generation. s123 00:12:42.959 --> 00:12:55.519 Uh we also support you know things on AMD where you can get a TCO benefit as well where we're 4x faster than the equivalent uh on um on on the MI355 s124 00:12:55.519 --> 00:12:56.880 systems. s125 00:12:56.880 --> 00:13:10.240 Actually you know some of this is not a fair comparison because you know some folks might say oh 7x is much larger than 4x but this is you cannot do that comparison very easily. s126 00:13:10.240 --> 00:13:21.839 uh that you can you have to factor in the cost uh per uh GPU and AMD is usually 30% cheaper than uh the Nvidia equivalent. s127 00:13:22.560 --> 00:13:30.639 Uh artificial analysis actually we've been partnering with them uh and we've been and they've independently been verifying our results. s128 00:13:30.639 --> 00:13:34.399 Uh this is our results like as of uh today. s129 00:13:34.399 --> 00:13:42.320 Uh we're about 3x faster than the next competitor uh on the flux 2 dev model as well. s130 00:13:42.320 --> 00:13:48.160 And this is not playing tricks of distillation and you know quantization or anything like that. s131 00:13:48.160 --> 00:13:59.519 This is just raw speed because we can leverage the hardware from the ground up and make sure that uh we're getting the max uh the max PF benefit from the model s132 00:14:00.160 --> 00:14:05.839 and you can actually go to artificial analysis today and see and see these results. s133 00:14:05.920 --> 00:14:08.480 Now what does that mean to folks again? s134 00:14:08.480 --> 00:14:16.240 It means you can open up applications that are otherwise kind of not possible because of slow inference. s135 00:14:16.240 --> 00:14:20.240 So this is actually uh a kind of a live demo. s136 00:14:20.240 --> 00:14:22.959 It's just somebody's pasting in the prompts. s137 00:14:22.959 --> 00:14:26.399 Uh but the generation is is accurate. s138 00:14:26.399 --> 00:14:34.399 So you can generate things in subsec generate images in subsecond uh per in many cases these are.3 seconds. s139 00:14:34.399 --> 00:14:35.920 Again you can go to the booth. s140 00:14:35.920 --> 00:14:46.160 There's a nice demo where uh you can give a prompt and then it reads the video feed and then renders it as well. s141 00:14:47.920 --> 00:14:57.680 Uh the other one data point is Gemma 4. So Gemma 4 was a a recent model that was just introduced about a month ago, a month and a half ago. s142 00:14:57.680 --> 00:15:04.000 Uh again we're quite faster than other any other competition out there. s143 00:15:04.000 --> 00:15:10.560 Uh so uh you know many cases actually some of these are uh hardware companies. s144 00:15:10.560 --> 00:15:13.440 you know, Samanova is a hardware accelerator. s145 00:15:13.440 --> 00:15:18.000 Uh we're, you know, let's say 6x faster than that. s146 00:15:18.000 --> 00:15:20.320 And in general, we're much much faster than others. s147 00:15:20.320 --> 00:15:23.839 And by the way, we're for all these cases, we're just starting. s148 00:15:23.839 --> 00:15:29.600 There's more optimizations that we're landing uh as as as time goes on. s149 00:15:29.600 --> 00:15:34.000 So, we're never never stopping to improving these things. s150 00:15:34.000 --> 00:15:38.000 So, we have a bunch of uh places where you can get involved. s151 00:15:38.000 --> 00:15:39.920 You can join our open source community. s152 00:15:39.920 --> 00:15:42.079 So, a lot of our code is open source. s153 00:15:42.079 --> 00:15:46.800 We have over 700,000 lines of kernel code that we've open sourced. s154 00:15:46.800 --> 00:15:52.399 We've over open open sourced all of our model definitions, all of our max serve. s155 00:15:52.399 --> 00:15:54.560 Um, we have AI skills. s156 00:15:54.560 --> 00:16:06.480 So, if you're not familiar with Mojo or Max, you can actually use the AI skills to port uh from languages that you're familiar with into Mojo or Max. s157 00:16:06.480 --> 00:16:07.600 You can also join us. s158 00:16:07.600 --> 00:16:08.880 We're hiring. s159 00:16:08.880 --> 00:16:14.079 So, there's pamphlets on the booth where we're hiring for a variety of different positions. s160 00:16:14.079 --> 00:16:24.480 So, as I you probably know, we we hire from low-level kernel engineers to high level folks that work on Kubernetes and everything in between. s161 00:16:24.480 --> 00:16:26.079 You can actually try all of this. s162 00:16:26.079 --> 00:16:29.759 So, this is not uh demoare or slideware. s163 00:16:29.759 --> 00:16:33.920 You can pip install modular right now if you want to and try all of this stuff. s164 00:16:33.920 --> 00:16:38.000 And I welcome everybody to validate these results. s165 00:16:38.160 --> 00:16:42.079 So, with that, I kind of that's it. s166 00:16:42.079 --> 00:16:44.480 And uh but I'm open to questions if folks have it. s167 00:16:44.480 --> 00:16:49.600 And I'll also be on the booth uh for the next few hours as well. s168 00:17:05.086 --> 00:17:07.086 [music]