WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=KB41dTlX1Uc NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/KB41dTlX1Uc.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.480 --> 00:00:14.160 Can you guys hear us? s3 00:00:14.160 --> 00:00:15.040 Sound check. s4 00:00:15.040 --> 00:00:19.119 All right, give it up for local AI, everyone. s5 00:00:20.305 --> 00:00:22.305 [applause] s6 00:00:23.279 --> 00:00:25.199 I hope you guys are excited as we are. s7 00:00:25.199 --> 00:00:26.720 Um, this is woo. s8 00:00:26.720 --> 00:00:28.240 This is the local AI summit. s9 00:00:28.240 --> 00:00:31.199 So, we're going to be here all day talking about local AI. s10 00:00:31.199 --> 00:00:35.200 And the reason why is we hit an inflection point this year. s11 00:00:35.200 --> 00:00:39.520 Not only did the models get really good, but the harnesses got really good. s12 00:00:39.520 --> 00:00:41.360 And this happened really fast. s13 00:00:41.360 --> 00:00:44.160 It's been, I think, a struggle for anyone here to keep up. s14 00:00:44.160 --> 00:00:47.600 I felt that most when I saw one of Andre Karpathy's tweets in November. s15 00:00:47.600 --> 00:00:51.039 He tweeted that you can't really trust these coding agents alone yet. s16 00:00:51.039 --> 00:00:53.920 You have to monitor them with an eye like a hawk. s17 00:00:53.920 --> 00:00:59.440 Three months later, he tweets that he's struggling to keep up with the capabilities of how good this all has gotten. s18 00:00:59.440 --> 00:01:01.440 And the thing is, both times he was right. s19 00:01:01.440 --> 00:01:03.680 This space is progressing really quickly. s20 00:01:03.680 --> 00:01:05.519 And it's wild how much you can do. s21 00:01:05.519 --> 00:01:10.080 And honestly, the only thing to do is just try to use it a little bit more today than you did yesterday. s22 00:01:10.080 --> 00:01:11.840 That's how to keep up with this space. s23 00:01:11.840 --> 00:01:13.920 And that's exactly what you guys are doing right here. s24 00:01:13.920 --> 00:01:16.240 And so I'm really excited about this panel. s25 00:01:16.240 --> 00:01:18.320 The way that we use AI has also changed. s26 00:01:18.320 --> 00:01:20.080 I'm not just using chat bots. s27 00:01:20.080 --> 00:01:22.560 I'm not just asking simple questions. s28 00:01:22.560 --> 00:01:28.640 When we got reasoning models, the the profile of how the AI model responded changed. s29 00:01:28.640 --> 00:01:33.439 Not only is it bursty and responding to me, but before the burst, it kind of plateaus for a bit. s30 00:01:33.439 --> 00:01:34.159 It's reasoning. s31 00:01:34.159 --> 00:01:38.079 It's turnurning on tokens that I'm not consuming. s32 00:01:38.079 --> 00:01:39.920 And then we got agents. s33 00:01:39.920 --> 00:01:42.960 And suddenly I don't even want these agents to turn off. s34 00:01:42.960 --> 00:01:44.320 I want these always on agents. s35 00:01:44.320 --> 00:01:47.280 They can always be productive if we set them up. s36 00:01:47.280 --> 00:01:53.680 And so we have enterprises that want to put a lot of their IP into this because it becomes more useful. s37 00:01:53.680 --> 00:01:55.360 We have consumers with the same thing. s38 00:01:55.360 --> 00:01:58.159 I want to give it my health data, my medical records. s39 00:01:58.159 --> 00:02:00.240 I want to give it footage from my home camera. s40 00:02:00.240 --> 00:02:04.640 And both enterprises and consumers, we don't want that stuff to leak. s41 00:02:04.640 --> 00:02:11.680 And as you also have a profile of uh tokens continuously generating, suddenly costs matter. s42 00:02:11.680 --> 00:02:14.400 And so local is amazing for both of those things. s43 00:02:14.400 --> 00:02:22.800 you get to make sure that you are plateaued on the costs uh for the tokens that you're generating and also uh everything sits in that room. s44 00:02:22.800 --> 00:02:28.800 So we have amazing demos here after these talks where the everything that's being run stays on those devices. s45 00:02:28.800 --> 00:02:31.599 It stays in this room and so that's a really nice guarantee. s46 00:02:31.599 --> 00:02:36.560 So as we turn over to the panels um first do you guys want to introduce yourselves? s47 00:02:36.560 --> 00:02:36.800 Yeah. s48 00:02:36.800 --> 00:02:38.400 Should I start please? s49 00:02:38.400 --> 00:02:38.959 You got the mic? s50 00:02:38.959 --> 00:02:40.879 Yeah, you're miked up. s51 00:02:40.879 --> 00:02:42.400 So yeah, I'm Alex. s52 00:02:42.400 --> 00:02:46.959 I'm the co-founder and CEO of ExoLabs and also the creator of local.ai. s53 00:02:46.959 --> 00:02:59.120 So we're we've been working on local AI for over two years which feels like a lot longer in this space and you know from the days of you know running Llama for AB on two MacBooks s54 00:02:59.120 --> 00:03:11.519 to now where we have you know demo there of Neimatron ultra running on four sparks you know our mission is to make AI more accessible and you know it's crazy to see what is possible now s55 00:03:11.519 --> 00:03:12.640 um s56 00:03:12.640 --> 00:03:17.920 cool hey everybody I'm Matt um I make content and uh videos. s57 00:03:17.920 --> 00:03:20.560 We have a newsletter all about artificial intelligence. s58 00:03:20.560 --> 00:03:22.720 I'm an AI enthusiast. s59 00:03:22.720 --> 00:03:25.680 Uh and yeah, thanks for having me. s60 00:03:25.680 --> 00:03:26.480 Hey everyone. s61 00:03:26.480 --> 00:03:28.480 Um my name is Ahmed Osman. s62 00:03:28.480 --> 00:03:31.040 Uh I am the founder and CEO of Osmantic. s63 00:03:31.040 --> 00:03:36.239 Uh somebody jumped into my DMs asked, "Hey, what does OS man stand for? s64 00:03:36.239 --> 00:03:37.120 Open source man." s65 00:03:37.120 --> 00:03:38.772 So that became a company. s66 00:03:38.772 --> 00:03:39.200 [laughter] s67 00:03:39.200 --> 00:03:42.239 And uh I also moderate local lamb subreddit. s68 00:03:42.239 --> 00:03:46.319 I have been in this local AI space since 2022. s69 00:03:46.319 --> 00:03:50.080 And uh yeah, let's uh let's make open source and local AI one. s70 00:03:50.080 --> 00:03:52.319 Love it. s71 00:03:52.527 --> 00:03:54.527 [applause] s72 00:03:55.360 --> 00:03:57.767 Open source man gets the round of applause. s73 00:03:57.767 --> 00:03:59.200 [laughter] s74 00:03:59.200 --> 00:04:01.519 That's appropriate. s75 00:04:01.519 --> 00:04:04.400 Today is uh I think a pretty momentous day for the panel. s76 00:04:04.400 --> 00:04:10.319 Um you know, Fable just came back, but it's a good reminder of why we need to have access to Frontier Intelligence to be able to build anything. s77 00:04:10.319 --> 00:04:11.120 Uh I'm Joseph. s78 00:04:11.120 --> 00:04:12.959 I'm co-founder CEO at Roboflow. s79 00:04:12.959 --> 00:04:14.319 We do all things vision. s80 00:04:14.319 --> 00:04:22.560 I kind of like to joke that vision is like the original local AI because like everything needs to run probably concurrent with like low latency alongside your video. s81 00:04:22.560 --> 00:04:25.199 Uh where the images and data is being captured. s82 00:04:25.199 --> 00:04:31.360 Uh looking forward to the discussion today and all things why local AI is the future of making AI useful for everyone. s83 00:04:31.360 --> 00:04:35.103 Everyone let's give Joseph and everyone a round of applause. s84 00:04:35.103 --> 00:04:37.103 [applause] s85 00:04:37.520 --> 00:04:43.600 So one thing that I'm really excited about for all the panelists is you guys were all early to the space in your own way. s86 00:04:43.600 --> 00:04:52.160 Uh and you know very early and I think as we all feel this inflection point now uh when do you guys feel like you felt the inflection point? s87 00:04:52.800 --> 00:04:54.000 Yeah I can go first. s88 00:04:54.000 --> 00:04:54.240 Sure. s89 00:04:54.240 --> 00:05:02.720 I mean look when I first saw Llama right I got really excited the idea that I can actually download this intelligence run it from my local computer. s90 00:05:02.720 --> 00:05:03.759 It was very exciting. s91 00:05:03.759 --> 00:05:05.199 I'm a tinkerer at heart. s92 00:05:05.199 --> 00:05:06.800 I'm a builder at heart. s93 00:05:06.800 --> 00:05:09.360 I I've overclocked PCs for years. s94 00:05:09.360 --> 00:05:11.759 So, this just felt like so good. s95 00:05:11.759 --> 00:05:19.680 So, felt so interesting to be able to have this crazy alien intelligence running on on my device in my office. s96 00:05:19.680 --> 00:05:25.919 Uh and that that was the first time where I didn't even think it was possible to be on till I saw Llama and I was like, "Oh, wow. s97 00:05:25.919 --> 00:05:27.120 This is incredible." s98 00:05:27.120 --> 00:05:30.639 Um so, that was really the point at which I was very turned on to it. s99 00:05:30.639 --> 00:05:33.440 I I think I um it's kind of the same for me. s100 00:05:33.440 --> 00:05:35.199 Uh yeah, it's Lamatu. s101 00:05:35.199 --> 00:05:49.840 I think that um finally on my 4090 RTX 4090, I'm like, "Wow, I can actually understand this black box and customize it and play with parameters and like you know, sampling parameters and all those uh configurations and see how inference engines s102 00:05:49.840 --> 00:05:50.240 work." s103 00:05:50.240 --> 00:05:52.880 Uh that that that made me feel like you know something clicked. s104 00:05:52.880 --> 00:05:53.919 It's like oh magic. s105 00:05:53.919 --> 00:05:55.600 I can be a wizard now with this thing. s106 00:05:55.600 --> 00:05:59.600 I can control how it how it plays to my ways of thinking. s107 00:05:59.600 --> 00:06:02.479 Um and from there it was you know the rest is history. s108 00:06:02.479 --> 00:06:15.840 Basically, I've been very vocal about why local and open source AI must coexist with cloud and how how we are supposed to like, you know, push for that and try to understand it as much as possible and teach people about it. s109 00:06:15.840 --> 00:06:16.319 Totally. s110 00:06:16.319 --> 00:06:17.600 I actually felt this as well. s111 00:06:17.600 --> 00:06:18.479 I was on a plane. s112 00:06:18.479 --> 00:06:23.840 I think it was in 2023 and I had uh a model running on my phone and it was awful. s113 00:06:23.840 --> 00:06:35.039 It took 20 minutes to complete a sentence, but I was I didn't have internet and so I was able to like ask uh essentially intelligence something there in the plane and that was my first feeling of like hm this is going to be something really cool. s114 00:06:35.039 --> 00:06:39.360 Do you know that you can run the equivalent of GT40 on your iPhone now? s115 00:06:39.360 --> 00:06:39.759 Really? s116 00:06:39.759 --> 00:06:40.000 Yeah. s117 00:06:40.000 --> 00:06:43.680 It's quen 3.5 the four parameter for billion parameters. s118 00:06:43.680 --> 00:06:52.720 It it's basically the same quality as something that used to be served in data centers and it's in a device in your pocket. s119 00:06:52.720 --> 00:06:54.240 Yeah, that's crazy. s120 00:06:54.240 --> 00:07:03.680 Yeah, I think if you just think about like how crazy that GPT40 moment was and now you can run it on the phone, it's like I think a lot of this stuff was just, s121 00:07:03.680 --> 00:07:08.240 you know, you had to see the vision of like where things were going because it was a toy, right? s122 00:07:08.240 --> 00:07:11.840 It was like, you know, the the first I think there were a few key points for me. s123 00:07:11.840 --> 00:07:15.039 One was like running Llama 45b locally, right? s124 00:07:15.039 --> 00:07:18.080 That was like the first really big open source model. s125 00:07:18.080 --> 00:07:32.000 And I remember like the gap it closed the gap quite significantly with the frontier closed models and open uh but it ran at two tokens a second so it wasn't useful right um and then you know another big moment was deepseek s126 00:07:32.000 --> 00:07:48.639 um both v3 and r1 deepseek v3 was like a massive so like you know llama 4 was dense so it's super slow seemed to unlock the performance so it was like oh wow you know with the devices I already have like with a you know a Mac uh Mac studio or s127 00:07:48.639 --> 00:07:56.960 Spark, I can run, you know, this massive model at actually like decent performance which is comparable with, you know, what you can run in the cloud. s128 00:07:56.960 --> 00:08:02.879 And then uh I think just recently to me GLM 5.2 is a big moment as well. s129 00:08:02.879 --> 00:08:13.120 Uh because again it's closing the gap and it's like you know Opus level um and you know we have it running over there on a device that like literally can fit on your desk s130 00:08:13.120 --> 00:08:14.560 uh the DJX station. s131 00:08:14.560 --> 00:08:24.879 So this is to me just like you know a trend and it's going to keep increasing right like there's going to be smaller and smaller devices with less memory better compression s132 00:08:24.879 --> 00:08:31.919 we'll be able to run more capable models locally and you know soon it'll be the default s133 00:08:31.919 --> 00:08:35.519 I'll keep with the uh the theme of of airplane stories. s134 00:08:35.519 --> 00:08:40.800 So yeah, I I agree that there's been like multiple moments over over time of what's been what's been going on. s135 00:08:40.800 --> 00:08:41.599 But one person s136 00:08:41.599 --> 00:08:43.360 I have a presentation for that after this. s137 00:08:43.360 --> 00:08:44.640 So stay around, please. s138 00:08:44.640 --> 00:08:45.600 All right. s139 00:08:45.600 --> 00:08:49.360 Uh one example moment where this like came up for me was uh I was on a plane. s140 00:08:49.360 --> 00:09:01.279 I was sitting next to someone man this is maybe two three years ago and they were hard of sight and they were using if you on Apple you know if you take a photo of something you can use the accessibility settings and it'll describe the photo to you s141 00:09:01.279 --> 00:09:12.880 and we were seated on the plane together and they were taking constant photos to understand where we seated and how to get buckled in and these sorts of things and I remember they took a photo of the seat back in front of them and the Apple accessibility s142 00:09:12.880 --> 00:09:20.640 described the photo described it as like a printer or something and it was like oh yeah like you're seated like by a printer and of course the individual knew from the context that couldn't be right. s143 00:09:20.640 --> 00:09:31.200 Uh, and I was like, you know, I wonder Lava had just come out, a multimodal model that like kind of just described things with like the pivot to text base rather than like natively multimodal, but still kind of a useful example. s144 00:09:31.200 --> 00:09:31.839 And I was curious. s145 00:09:31.839 --> 00:09:33.760 I was like, well, let's see how Lava would do on the same thing. s146 00:09:33.760 --> 00:09:35.600 So, I took a photo of like the seat tray in front of me. s147 00:09:35.600 --> 00:09:36.720 And it aptly described it. s148 00:09:36.720 --> 00:09:37.839 It's like, hey, you're on an airplane. s149 00:09:37.839 --> 00:09:38.720 That's the seat tray. s150 00:09:38.720 --> 00:09:44.320 And I showed the person sitting next to me who had never seen local models and never experienced local AI. s151 00:09:44.320 --> 00:09:52.240 And what that stood out to me is I was like, man, a company that's a trillion dollar plus market cap business, shipping the latest intelligence on their phones for describing s152 00:09:52.240 --> 00:09:56.959 visual settings is inferior to something that's broadly accessible and available to anyone. s153 00:09:56.959 --> 00:10:03.120 And that was such like a watershed clear moment that like even the largest companies don't have a monopoly on the frontier of intelligence. s154 00:10:03.120 --> 00:10:12.640 And so increasingly making that be accessible to others um is I think going to be critical for understanding all the massive impacts the tech will have. s155 00:10:12.640 --> 00:10:13.920 And that was lava. s156 00:10:13.920 --> 00:10:14.397 Yeah. s157 00:10:14.397 --> 00:10:15.120 [laughter] s158 00:10:15.120 --> 00:10:15.360 Yeah. s159 00:10:15.360 --> 00:10:16.560 It's interesting to hear s160 00:10:16.560 --> 00:10:18.399 that's like 2022 2023, right? s161 00:10:18.399 --> 00:10:18.880 Around there. s162 00:10:18.880 --> 00:10:19.200 Yeah. s163 00:10:19.200 --> 00:10:19.360 Yeah. s164 00:10:19.360 --> 00:10:20.240 That was a while ago. s165 00:10:20.240 --> 00:10:23.279 It's interesting to hear you say about the the accessibility of that, right? s166 00:10:23.279 --> 00:10:31.440 The uh taking this Frontier intelligence, what made it useful was giving it peripherals, in this case, a camera so that it could access the data in front of you. s167 00:10:31.440 --> 00:10:38.480 And I think that's where the inflection point this year was so much more than just models, but also these harnesses and what you can give it access to. s168 00:10:38.480 --> 00:10:43.200 It was CLI so that you can give business systems uh and plug those right into your agent. s169 00:10:43.200 --> 00:10:47.279 It was uh you know 40 was really exciting because I was coding with it, right? s170 00:10:47.279 --> 00:10:51.760 I was taking code snippets from my codebase and then I'd go to chatgpt and I'd paste it. s171 00:10:51.760 --> 00:10:57.360 I'd give it as much context as I thought I should and then I would take the result and I'd go back to my codebase and paste it. s172 00:10:57.360 --> 00:11:07.200 And what was really great about tools like cursor was it essentially was this harness that said what if I just had the full file system and I can let these agents reason on what files they needed and so on. s173 00:11:07.200 --> 00:11:17.839 And so the way that you know you get more use out of all of these local uh out of all of these local models and systems is how does it how can it interact with the real world. s174 00:11:17.839 --> 00:11:28.079 I'm curious um Joseph so you guys got the start in vision and it seems like vision had to learn the hard way what a lot of LLM are dealing uh discovering right now. s175 00:11:28.079 --> 00:11:34.720 What do you think is like a big lesson that the language world is discovering right now that uh we could look to vision for? s176 00:11:34.720 --> 00:11:40.560 I think um one of the things that language has an advantage of is it's inherently a human construct. s177 00:11:40.560 --> 00:11:51.279 So language generally exists where people exist and that also means usually you can use uh infinite amounts of of computer what is available in in a data center whereas a lot of vision s178 00:11:51.279 --> 00:11:52.640 uh you're computed. s179 00:11:52.640 --> 00:11:54.720 You're running maybe where there's low internet connectivity. s180 00:11:54.720 --> 00:11:55.920 You're running on a device. s181 00:11:55.920 --> 00:12:01.760 You're running on a robot and the amount of compute you have available to you is is what um is what you get. s182 00:12:01.760 --> 00:12:02.720 And so what did that do? s183 00:12:02.720 --> 00:12:06.320 So that created I think an emphasis on specialized learners faster s184 00:12:06.320 --> 00:12:16.399 because it's like okay I need this model to run in a limited context um and I'm not going to prioritize full world scale generalizability instead I'm going to have specialized s185 00:12:16.399 --> 00:12:24.720 in domain context work and what's interesting is I think you're actually seeing language follow a similar thing you're describing in the context of like harnesses for a given context s186 00:12:24.720 --> 00:12:37.120 um but you know tools like uh any sort of coding agent where you have a really good harness and it's specialized for doing those sorts of tasks but you're also seeing that even in like you know tax preparation or legal preparation of last mile fine-tuning s187 00:12:37.120 --> 00:12:48.000 adaptation specialized learners and so in some ways I feel like the pendulum swinging back to actually having specialized models even in language just as much as vision I think is one thing s188 00:12:48.000 --> 00:12:55.040 um and then you know there's a whole different thing around how you get the most out of compute optimization running on on a niche device s189 00:12:55.040 --> 00:12:57.920 um but I know that probably the exo folks can talk speak more s190 00:12:57.920 --> 00:12:59.279 absolutely s191 00:12:59.279 --> 00:13:07.360 um but yeah I think specialized models I mean, you remember there was like one model will rule them all world was kind of thought and I feel like the pendulum is swung back to people realizing special s192 00:13:07.360 --> 00:13:08.160 specialized models. s193 00:13:08.160 --> 00:13:12.160 I think we definitely at NVIDIA we see that it's going to be a multimodel world. s194 00:13:12.160 --> 00:13:13.600 We definitely agree with that. s195 00:13:13.600 --> 00:13:19.680 Um there's actually one of the panels that's coming on later I think at 3:20 p.m. is all about model routing and so that's that's really exciting. s196 00:13:19.680 --> 00:13:25.519 I guess as an enthusiast uh and as you use some of this is that uh how your usage pattern looks? s197 00:13:25.519 --> 00:13:26.560 Are you using many models? s198 00:13:26.560 --> 00:13:27.680 What does that kind of look like? s199 00:13:27.680 --> 00:13:28.399 Yeah, absolutely. s200 00:13:28.399 --> 00:13:40.880 everything from obviously like the top models Fable uh to kind of the more workhorse models with Sonnet local models for things that maybe don't depend as much on low latency. s201 00:13:40.880 --> 00:13:46.639 Um yeah, like the multimodel world seems like a no-brainer at this point. s202 00:13:46.639 --> 00:14:00.560 uh especially as you're seeing all these enterprise companies come out and look at their budgets and and think, okay, well, I I need to continue to increase the the total number of tokens that I'm consuming as a company, but I also don't want to just completely blow out my budget. s203 00:14:00.560 --> 00:14:11.360 I know Coinbase and Brian Armstrong just came out with that just great post the other day talking about uh how their tokens are are exploding yet their costs are staying flat. s204 00:14:11.360 --> 00:14:14.079 And and that is because they're using a mixture of different models. s205 00:14:14.079 --> 00:14:23.680 You don't need the top model for every single use case and in fact most use cases you don't uh I think the most obvious application is let the top model plan s206 00:14:23.680 --> 00:14:31.920 uh the the architecture whatever the kind of top level plan is and then the actual execution of the code can go to uh a more reasonably priced smaller model. s207 00:14:31.920 --> 00:14:43.120 Your most intelligent should provide you with the overall plan and then subtasks for your smaller executioner like executioner models and that's exactly the future. s208 00:14:43.120 --> 00:14:43.600 Yeah. s209 00:14:43.600 --> 00:14:55.279 And especially like you know we're talking about local um local models are great at writing code but maybe we offload the actual top level planning to one of the frontier models and you you save a bunch s210 00:14:55.279 --> 00:14:57.040 you control more of the workflow. s211 00:14:57.040 --> 00:14:59.760 It's it's a it's a nice pattern. s212 00:14:59.760 --> 00:15:00.000 Yeah. s213 00:15:00.000 --> 00:15:03.519 I I think the market wants this. s214 00:15:03.519 --> 00:15:10.959 So it wasn't clear like a few years ago when this stuff was just starting to play out, you know, would there just be one big model that everyone is using? s215 00:15:10.959 --> 00:15:12.959 But clearly that is not what people want. s216 00:15:12.959 --> 00:15:14.880 That is not what enterprises want. s217 00:15:14.880 --> 00:15:18.240 They don't want to be told what they can do by Dario. s218 00:15:18.240 --> 00:15:27.360 They don't want to be paying for the same model for all their workloads when some some workloads don't actually need, you know, a giant gigantic model that costs $50 s219 00:15:27.360 --> 00:15:28.880 per million tokens. s220 00:15:28.880 --> 00:15:38.880 Um, they want control, they want sovereignty, they want the ability to switch out models, they don't want to get rugpulled, you know, uh, from one day to the other, s221 00:15:38.880 --> 00:15:41.360 uh, because of some safety risk or whatever. s222 00:15:41.360 --> 00:15:45.360 Um, and you know, that's really what's driving a lot of this progress. s223 00:15:45.360 --> 00:15:51.040 So, you know, I'm more optimistic than ever, I think, about sort of, you know, where things are going with local. s224 00:15:51.040 --> 00:16:00.720 I think that, you know, the market is basically pulling a lot of this stuff um out of, you know, startups, out of enterprises are building solutions around this and, you know, s225 00:16:00.720 --> 00:16:03.279 it's it's yeah, it's it's happening, right? s226 00:16:03.279 --> 00:16:03.600 Totally. s227 00:16:03.600 --> 00:16:08.079 I feel like this is a new frontier where as we go multimodel, that becomes really difficult. s228 00:16:08.079 --> 00:16:14.880 Of course, you need to route between the models, but then you have to figure out how to provide necessary context to whichever model you're routing to. s229 00:16:14.880 --> 00:16:18.880 If you were making a plan and then breaking it off into sub aents, what is the framework to do? s230 00:16:18.880 --> 00:16:19.839 So, how should you do this? s231 00:16:19.839 --> 00:16:20.880 These are all open problems. s232 00:16:20.880 --> 00:16:25.360 And, you know, I think just being playful is uh kind of the best way to to approach this. s233 00:16:25.360 --> 00:16:28.800 But I like the way that you worded that of like uh the market's kind of pulling for these answers. s234 00:16:28.800 --> 00:16:30.079 It's reaching in this direction. s235 00:16:30.079 --> 00:16:32.720 We're looking for startups and companies to fill this gap. s236 00:16:32.720 --> 00:16:34.639 Um, makes a ton of sense. s237 00:16:34.639 --> 00:16:35.360 You know what's funny? s238 00:16:35.360 --> 00:16:40.560 Um, as any any enterprise consumes any any piece of software, it matters a lot. s239 00:16:40.560 --> 00:16:43.120 If you're going to build a foundation, you want to know what the versions are. s240 00:16:43.120 --> 00:16:48.000 You want to know what your model is, which whether that version changed in order to see change behavior later. s241 00:16:48.000 --> 00:16:54.079 So having uh also more control over all of this so that you know exactly what version it is, it can't be changed. s242 00:16:54.079 --> 00:17:04.000 Uh not even because of any sort of you know regulation, but just simply if there are updates, you choose when you're opting into whatever it is on your entire stack. s243 00:17:04.000 --> 00:17:04.480 Yeah. s244 00:17:04.480 --> 00:17:24.240 Uh so [clears throat] when you think about um this wave that we're seeing here of like this civilizational infrastructure that is called AI you have to consider the the potential of you know things being taken away from you and your sovereignty s245 00:17:24.240 --> 00:17:35.360 and uh how can you be in charge of you know this thing full stack end to end that's hardware software and everything in between That's the model weights. s246 00:17:35.360 --> 00:17:38.880 That's the specific version of the model as you were saying. s247 00:17:38.880 --> 00:17:41.120 That's how you fine-tune it. s248 00:17:41.120 --> 00:17:46.640 And picking up on Jos's you know thing um statement here [clears throat] about small and specialized models. s249 00:17:46.640 --> 00:17:51.760 I have been like, you know, I have been a proponent of that for quite some time. s250 00:17:51.760 --> 00:18:02.160 I I have tweets about that in 2024 saying small and specialized models are the future and it is going to be very user by per use case per workflow s251 00:18:02.160 --> 00:18:13.120 for businesses you know to to decide on their niche domains and you need to start on boarding them now because you want to collect the data points that races that's what you're asking about N you're asking about s252 00:18:13.120 --> 00:18:25.280 how do we get there how do we decide which model gets routed to how do we decide which use case gets goes to which model or which endpoint you You need to collect data and you need to be ready to basically s253 00:18:25.280 --> 00:18:38.559 break down um the traces that you've collected from your employees from everybody in your organization and decide how can we make the most optimal use case of these data points to which models and collect s254 00:18:38.559 --> 00:18:42.320 feedback as well and you can actually automate that with agents as well. s255 00:18:42.320 --> 00:18:55.840 So like this is like something that it's you know the the whole thing about RSI right now recursive self-improvement it also applies to even agents and harnesses and to to workflows and to use cases and to enterprises. s256 00:18:55.840 --> 00:18:59.520 You want to describe that for everyone too maybe for folks who don't know RSI s257 00:18:59.520 --> 00:19:17.120 uh recursive self-improvement it means that uh a model would basically rent its own compute and then start training its own checkpoints and then you know uh deploy it its next version so that it gets updated in certain ways change its behavior in certain ways so basically a model is training itself s258 00:19:17.120 --> 00:19:28.559 totally for our product brev which just makes it really easy to get a GPU we've been focusing more on agents as a first class audience because we're seeing this so we're seeing growing usage from agents that just want to go grab a GPU s259 00:19:28.559 --> 00:19:29.360 uh directly. s260 00:19:29.360 --> 00:19:30.000 Exactly. s261 00:19:30.000 --> 00:19:31.360 So yeah, you know, it's funny. s262 00:19:31.360 --> 00:19:39.440 So the second thing you talked about was uh not just multimodel but then all these optimizations to actually run it performant when you have less compute than in a say data center. s263 00:19:39.440 --> 00:19:41.840 I think Alex this is some fun work that we did. s264 00:19:41.840 --> 00:19:46.799 So um Alex has actually set up a second headquarters inside of Nvidia. s265 00:19:46.799 --> 00:19:59.520 we got a conference room and uh we were we were having dinner and he mentioned that he really was motivated to get to squeeze out every drop of performance we could on the DGX Spark and so we said hey let's get a conference room at Nvidia s266 00:19:59.520 --> 00:20:06.799 come bring your team and anytime you need an expert across any pillar let's just go and pull that person into the room and yeah s267 00:20:06.799 --> 00:20:16.720 I I actually have a follow-up here uh so since I have been part of local lama for quite some time um when I think about home labers I think about the individuals that are actually s268 00:20:16.720 --> 00:20:26.080 are within enterprises on boardrooms making decisions about you know the models that we're going to host or you know where we're going to get our AI from. s269 00:20:26.080 --> 00:20:30.000 Uh the things that these home labers focus on are optimizations. s270 00:20:30.000 --> 00:20:36.720 It's basically how can I extract the most economical value out of the hardware the constraints that I have the software that I have. s271 00:20:36.720 --> 00:20:39.440 You know quantizations came to be because of that. s272 00:20:39.440 --> 00:20:41.919 When we're thinking about enterprises it's the same thing. s273 00:20:41.919 --> 00:20:42.799 It's constraints. s274 00:20:42.799 --> 00:20:43.840 It's budget limitations. s275 00:20:43.840 --> 00:20:48.880 It's how can I make the most use of the number of tokens that I can get under the hardware that I have. s276 00:20:48.880 --> 00:21:02.400 Whether that's a DGX station, you know, um centrally in a small medalsized business or, you know, a DGXP300 cluster, it's all about how can I optimize the software for the latest and greatest frontier s277 00:21:02.400 --> 00:21:08.559 opensource model out there and get as much value as possible out of it under the economical constraints that we have. s278 00:21:08.559 --> 00:21:21.600 I think it's it's crucial to see home labers and that's why you know this is local AI it's local AI but it could it could be on premises it could be colloccated hardware it could be rented clusters it could be from brev s279 00:21:21.600 --> 00:21:36.159 basically it's just the idea about controlling this thing end to end from hardware to software to model endpoints to model weights and you know collecting all of that data to train your specialized and smaller models that will be more efficient as as you go. s280 00:21:36.159 --> 00:21:39.534 Um yeah, the future is great and the future is local. s281 00:21:39.534 --> 00:21:39.679 [laughter] s282 00:21:39.679 --> 00:21:41.440 Yeah, absolutely. s283 00:21:41.440 --> 00:21:45.200 Um you know, making sure it's your dates, your your weights, your compute. s284 00:21:45.200 --> 00:21:54.880 I think one thing that was really interesting, we did end up getting 10x performant improvements on the DGX Spark and we sent an update to Jensen, to the executive staff, to the teams that were helping. s285 00:21:54.880 --> 00:21:59.840 And one line I really liked in that email was that we didn't solve any new computer science to do this. s286 00:21:59.840 --> 00:22:04.480 We actually took things that the experts at NVIDIA had already solved and was out there. s287 00:22:04.480 --> 00:22:08.080 And I think what we worked together to do really nicely was assemble it in a bouquet. s288 00:22:08.080 --> 00:22:11.200 And I think that speaks to some of the the usability here, right? s289 00:22:11.200 --> 00:22:13.120 We're talking about what is capable. s290 00:22:13.120 --> 00:22:14.480 What what are the capabilities? s291 00:22:14.480 --> 00:22:18.799 But do you feel like it's just capabilities that are holding people back from adopting this? s292 00:22:18.799 --> 00:22:19.600 Yeah. s293 00:22:19.600 --> 00:22:22.480 So just to tell the story a little bit what happened here. s294 00:22:22.480 --> 00:22:29.760 So I think we had dinner on Thursday of the week and then you know an email got sent out on Friday. s295 00:22:29.760 --> 00:22:32.640 So this idea came up like why don't we do a lab, right? s296 00:22:32.640 --> 00:22:35.600 Why don't we like you know this to me sounded crazy at the time. s297 00:22:35.600 --> 00:22:45.600 I didn't think this would be possible but it was like you know let like Nata was like oh let's like get a bunch of people from Nvidia to help you guys and work with you guys to improve the performance on the spark s298 00:22:45.600 --> 00:22:51.120 right and I was like okay like let yeah let's I mean if we can that would be great. s299 00:22:51.120 --> 00:22:59.760 Email got sent out I think pretty much that night and then on Friday Nata told me be here on Monday um at NVIDIA HQ. s300 00:22:59.760 --> 00:23:11.760 uh we're going to have you know teams of people here that are going to work together with you on this and you know turned up on Monday and uh Jensen talks about this concept of swarming. s301 00:23:11.760 --> 00:23:18.400 Uh it's you know basically the idea that uh the the whole company will like mobilize around something. s302 00:23:18.400 --> 00:23:20.080 Have you ever seen little kids play soccer? s303 00:23:20.080 --> 00:23:21.840 There's a ball and everyone just attacks it. s304 00:23:21.840 --> 00:23:22.720 It's like a tackle. s305 00:23:22.720 --> 00:23:24.640 It's just the mit of the soccer ball. s306 00:23:24.640 --> 00:23:24.799 Yeah. s307 00:23:24.799 --> 00:23:28.240 And I had heard about this idea, but like I'd never experienced it. s308 00:23:28.240 --> 00:23:31.280 And I can tell you like we were just back toback that whole day. s309 00:23:31.280 --> 00:23:39.919 People coming in and out of the room, you know, from all various teams like the Neatron team, um, you know, people working in in on data center stuff, people working s310 00:23:39.919 --> 00:23:40.960 um on VLM. s311 00:23:40.960 --> 00:23:43.840 I I realized, you know, Nvidia has a team for everything. s312 00:23:43.840 --> 00:23:55.600 So there's like a VLM for Spark team which is oddly specific but like you know they have all these teams and you know we basically got to pull those resources in and you know s313 00:23:55.600 --> 00:24:09.280 you know in those three weeks or so as you mentioned like we basically got 10x performance versus you know what um Nvidia had running on the spark um in their existing playbook which was using Hermes agent. s314 00:24:09.280 --> 00:24:17.520 So we did a bunch of optimizations there um using VLM um as the the sort of inference back end there. s315 00:24:17.520 --> 00:24:25.840 Uh doing a lot of work with the like tuning the models like quantizing the models uh to you know be fit for local. s316 00:24:25.840 --> 00:24:36.320 So what you'll find is I think one of the great things Nvidia has done with the local hardware is it's the same architecture that's running in the data center as running on the spark. s317 00:24:36.320 --> 00:24:38.080 So it's Grace Blackwell. s318 00:24:38.080 --> 00:24:42.960 Um so the hardware is like fundamentally the same meaning you actually get a lot of things for free. s319 00:24:42.960 --> 00:24:45.840 So for example like the kernels are already really good. s320 00:24:45.840 --> 00:24:54.880 However there is like a lot of tuning and a lot of like configuration that is right now designed specifically for the data center. s321 00:24:54.880 --> 00:25:03.520 So a lot of the work that we did was not like inventing anything new but it was actually just tweaking things to work uh more performantly on the spark. s322 00:25:03.520 --> 00:25:06.320 And you know the hardware is extremely capable. s323 00:25:06.320 --> 00:25:10.559 It's like you know it is data center level hardware that's literally can sit on your desk. s324 00:25:10.559 --> 00:25:22.159 Um so it's just about like you know how do we activate that and I think this has been one of my learnings from the last few years is like we already have the hardware like the hardware is already really good s325 00:25:22.159 --> 00:25:24.320 and you know the models are getting better. s326 00:25:24.320 --> 00:25:28.559 Um they're getting a lot better at compression so you can fit more on a smaller device. s327 00:25:28.559 --> 00:25:44.240 So like you know I think it really is just a matter of more people looking at the space and working in it and you know we will be able to do amazing things like we have you know neatron 3 ultra it's a 550 billion parameter model running on four sparks over there 30 tokens per second s328 00:25:44.240 --> 00:25:45.200 so like s329 00:25:45.200 --> 00:25:58.240 that's that's huge I want to say something here like it's it's so great to see the community being proactive I think of Alex Gmail right here who's exo as somebody who's member of the community who's trying to push the frontier opensource s330 00:25:58.240 --> 00:25:59.919 intelligence to the next level. s331 00:25:59.919 --> 00:26:02.640 I love how Nvidia is working with them on that. s332 00:26:02.640 --> 00:26:09.840 When I think about, you know, this space, I think we're in the '9s of the Linux operating system and we are like just starting. s333 00:26:09.840 --> 00:26:11.440 The infrastructure is not there yet. s334 00:26:11.440 --> 00:26:13.120 We need so much more. s335 00:26:13.120 --> 00:26:16.000 Uh I was even telling Alex, I want to see how we can collaborate. s336 00:26:16.000 --> 00:26:24.159 We have like an open source uh uh deployment system called ODS that is a bunch of open source tools that we deploy in each hardware. s337 00:26:24.159 --> 00:26:33.919 It configures stuff, configures agents and basically uh gets you set like you know get gets you set up with the entire infrastructure end to end that you need for your agents locally. s338 00:26:33.919 --> 00:26:45.760 This kind of stuff we need that optimized for every piece of hardware out there so that we can get more and more people on boarded have the open source adoption s339 00:26:45.760 --> 00:26:47.760 you know just go to the moon. s340 00:26:47.760 --> 00:26:49.360 That's what I want right now. s341 00:26:49.360 --> 00:26:50.400 This is why we're here. s342 00:26:50.400 --> 00:27:05.520 We want everybody to know that local local and open source AI can run on anything starting from your phone to your DJX Spark as you're saying Alex to DJX stations to the next level in the data centers and it will deliver you frontier level intelligence. s343 00:27:05.520 --> 00:27:06.400 Yeah, absolutely. s344 00:27:06.400 --> 00:27:09.360 I think this is a really good question for you Matt. s345 00:27:09.360 --> 00:27:11.440 You know you test everything for a mainstream audience. s346 00:27:11.440 --> 00:27:13.120 You're hearing about all of this technology. s347 00:27:13.120 --> 00:27:14.240 You're using this aggressively. s348 00:27:14.240 --> 00:27:18.640 Where does it still fall short for the uh average user for AI? s349 00:27:18.640 --> 00:27:20.559 So I I think about [clears throat] two things. s350 00:27:20.559 --> 00:27:25.600 The average kind of personal user then the average enterprise using uh open source. s351 00:27:25.600 --> 00:27:31.919 I mean it's it it needs to basically be as simple as opening cursor. s352 00:27:31.919 --> 00:27:38.559 It needs to be maybe slightly more complicated than that or slightly more complicated than just installing codecs and opening it up. s353 00:27:38.559 --> 00:27:41.840 Right now it to be fair it is quite far from that. s354 00:27:41.840 --> 00:27:50.960 it like the the stuff these guys are doing is incredible, but it is more sophisticated than what most people including myself are going to be capable of. s355 00:27:50.960 --> 00:27:54.000 Um, you know, let alone a business. s356 00:27:54.000 --> 00:27:54.880 Uh, I think s357 00:27:54.880 --> 00:27:57.279 a full-time job to just do this kind of stuff. s358 00:27:57.279 --> 00:27:58.640 That's why we need to automate it. s359 00:27:58.640 --> 00:28:01.200 And it it really does need to be point-and-click. s360 00:28:01.200 --> 00:28:09.360 And once it gets there, and there's there's a lot of great open source projects, there's a lot of great projects in general that are getting there, but we're still not quite there. s361 00:28:09.360 --> 00:28:21.760 Uh the other thing to make it widely adopted is to allow people to better understand what use cases are appropriate for what what type of model for what type of harness what type of hardware s362 00:28:21.760 --> 00:28:34.880 knowing exactly the use case that I can get out of my home system or something that I'm I'm you know renting from uh service center uh that that is incredibly important as well s363 00:28:34.880 --> 00:28:37.360 and I I think it shouldn't be just in documentation. s364 00:28:37.360 --> 00:28:40.000 I think that's where open source becomes so difficult for people. s365 00:28:40.000 --> 00:28:44.240 I think it needs to be a point and click and it figures it out on its own. s366 00:28:44.240 --> 00:28:45.760 That's what ODS is about. s367 00:28:45.760 --> 00:28:47.279 That's what I think EXO is about. s368 00:28:47.279 --> 00:28:59.279 That's what you know as we grow more and more and building this infrastructure, we need to be thinking about the user experience for your average everyday user, not us as technical folks here because this is AI engineering s369 00:28:59.279 --> 00:29:04.559 um you know summit and I'm pretty sure that we all can manage our way around this stuff. s370 00:29:04.559 --> 00:29:15.520 But your chat GBT users, your cloud users, whatever out there, they want this to be an alternative that is just click play, send a message, use an agent, and done. s371 00:29:15.520 --> 00:29:15.760 Yeah. s372 00:29:15.760 --> 00:29:19.120 Most most people really don't want to know about the details. s373 00:29:19.120 --> 00:29:20.399 They they just want it to work. s374 00:29:20.399 --> 00:29:30.000 And even if it worked seamlessly within cursor or codeex or any of these other things and it just worked and they didn't have to think about it, that is number one uh that for for the vast majority of people. s375 00:29:30.000 --> 00:29:39.679 So [clears throat] interesting on that in a multimodel world where we have to pick a model do you think that that's something that is a product um as we're or is that is that plumbing is that something s376 00:29:39.679 --> 00:29:49.840 matter you talked about the harness it's exactly that it should be some like whatever the kind of the front entry point to somebody's AI experience is that should be what is choosing s377 00:29:49.840 --> 00:30:01.279 which model at the right time for the right use case this is a very difficult problem there are you know open source projects closed source uh companies building routing That's only one piece of it, right? s378 00:30:01.279 --> 00:30:02.080 So, go ahead. s379 00:30:02.080 --> 00:30:05.520 No, like Carter from Nvidia, he's he's going to be a moderator here. s380 00:30:05.520 --> 00:30:06.399 Hey, Carter. s381 00:30:06.399 --> 00:30:07.919 Uh, looking good, man. s382 00:30:07.919 --> 00:30:20.799 So, um, he I I I shared ODS with him and he told me, I love how like it immediately downloaded that two billion parameter models allowed me to start playing with it and then it started downloading the next model that would work perfectly on on my device. s383 00:30:20.799 --> 00:30:25.120 That's the kind of experience we need to be giving these users when they we're on boarding them. s384 00:30:25.120 --> 00:30:30.880 Don't make them sit down and have to think about all these quantizations and all these extensions and all these weird things. s385 00:30:30.880 --> 00:30:33.200 That is too much work for your average user. s386 00:30:33.200 --> 00:30:34.399 That's how we lose them. s387 00:30:34.399 --> 00:30:35.200 Mhm. s388 00:30:35.200 --> 00:30:35.440 Yeah. s389 00:30:35.440 --> 00:30:36.159 It's interesting, right? s390 00:30:36.159 --> 00:30:43.520 Like the when you're not having to deal with multimodel and you have just this generally good but big model, you can kind of ask anything. s391 00:30:43.520 --> 00:30:47.279 You don't have to be as sophisticated or you don't have to understand your use case too well. s392 00:30:47.279 --> 00:30:52.320 But as we talk about specialized models, if you're going to specialize in something, you need to know what that something is. s393 00:30:52.320 --> 00:31:00.240 And so the amount that I mean can you guys speak to maybe what it's like to move into a specialized model or to build one or have you have you s394 00:31:00.240 --> 00:31:01.200 I can start with that. s395 00:31:01.200 --> 00:31:04.320 Uh on the cloud uh you're basically getting the normal distribution. s396 00:31:04.320 --> 00:31:18.240 So every time that they are training something they are taking the average feedback from everybody when they are being happy or sad or like you know um they are okay with the answer or hey no this is not what I meant go back or change the model and answer with a different model. s397 00:31:18.240 --> 00:31:23.039 That's the kind of feedback that allows you, you know, cloud providers to fine-tune the next model. s398 00:31:23.039 --> 00:31:28.399 But if we're talking about small and specialized models for use cases, that's a lot of compute to train models. s399 00:31:28.399 --> 00:31:31.679 That's a lot of um storage for these weights. s400 00:31:31.679 --> 00:31:44.720 Uh that does not happen without each use case or each business entity etc. focusing on their own um patterns and use cases and like how they handle agents and how their employees s401 00:31:44.720 --> 00:31:47.519 uh message these agents and what workflows they're interested in. s402 00:31:47.519 --> 00:31:58.080 So it's really you know it's per use case it's per entity it it's not something that we can just generalize and that's why you know continual learning is not being talked about by the frontier model s403 00:31:58.080 --> 00:32:05.120 uh as much it's coming though it will happen and it needs to be running on local hardware for it to happen. s404 00:32:05.120 --> 00:32:19.200 That's how you can have something that is so optimized that is not just plotted markdown files sitting in one rebel and you know you think that that agent is not going to lose track of which skill to update or what to edit or which memories to update s405 00:32:19.200 --> 00:32:31.120 because at some point context lens becomes inefficient and that's one issue that's the current paradigm of agents is basically just saving to markdowns the next one will be updating the weights and that needs to happen locally. s406 00:32:31.120 --> 00:32:37.840 Yeah, there's it's funny too because you know uh there was a tweet I think by Swix uh that was like why has fine-tuning as a service not taken off. s407 00:32:37.840 --> 00:32:39.200 It was like a couple months ago. s408 00:32:39.200 --> 00:32:44.080 Um and I think a big reason for that is that model customization itself is also a very hard problem. s409 00:32:44.080 --> 00:32:46.080 And then what models can you hack on? s410 00:32:46.080 --> 00:32:48.480 Uh that's I mean that's why Nvidia releases Neotron. s411 00:32:48.480 --> 00:32:58.640 It's a open source model but everything from the dates sorry the data to the weights to the uh the recipes for how to do so and then of course you know the final model everything is open sourced s412 00:32:58.640 --> 00:33:03.200 um just so that it is a model that you know you can safely uh use and customize s413 00:33:03.200 --> 00:33:12.960 you know it's interesting I keep kind of flip-flopping on this point I think people look at how good the generalized models are and you give it the right context s414 00:33:12.960 --> 00:33:27.039 is that going to be better than than having a fine-tuned model and all the work that comes with that And I I keep going back and forth because there is a lot of value in being able to have kind of a smaller very specialized model and and maybe a bunch of them working in unison s415 00:33:27.039 --> 00:33:29.120 to accomplish whatever task you have. s416 00:33:29.120 --> 00:33:32.880 Um but I I yeah I'm not sure what what do you what do you guys think about that? s417 00:33:32.880 --> 00:33:40.880 Um, if I can chime in on this one, like I think the the beauty of like the open source ecosystem is that like all of these paths kind of get to be explored s418 00:33:40.880 --> 00:33:59.039 and then whatever wins like you know it's it it's like what happened with speculative decoding um you know there's all these like different ways of doing speculative decoding which is this idea that you can use a smaller model to basically approximate a larger model to speed it up. s419 00:33:59.039 --> 00:34:12.720 Um and you know you've seen like recently literally I think in the last week there's been like three different like quite you know seem to be breakthroughs in like speculative decoding that have just come from like different places. s420 00:34:12.720 --> 00:34:24.399 One from deepseek um there was some work that was done by like modal and uh the SG Lang team to like build uh Dlash draft models for like various Quen models that s421 00:34:24.399 --> 00:34:27.040 are like a big improvement on the previous ones. s422 00:34:27.040 --> 00:34:36.480 um there's like all this work being done and so like we kind kind of get to see like all of it and you know basically like whatever ends up being the best thing will just be the thing that kind of right. s423 00:34:36.480 --> 00:34:47.040 That's a great point where maybe it's not users and consumers actually customizing their own models and using the specialized models but someone who has a need does so and does so in an open source fashion so that someone else can just adopt it. s424 00:34:47.040 --> 00:34:47.440 Right? s425 00:34:47.440 --> 00:34:52.000 If I'm going to start to do image gen for a particular use case I can just go find a model. s426 00:34:52.000 --> 00:34:56.159 Actually we saw this a lot also with like early Lauras on like a lot of image gen models. s427 00:34:56.159 --> 00:34:59.760 I feel like that was really big as you know in the 40 era as well. s428 00:34:59.760 --> 00:35:07.440 I think ultimately what the end user cares about is does it solve my use case and can it do so within my budget s429 00:35:07.440 --> 00:35:17.760 and if there's another service that can manage the entire fine-tuning process and and I don't have to think I know I keep saying I don't have to think about it sometimes I do think about things but like in this specific use case um s430 00:35:17.760 --> 00:35:20.480 thinking about too many things already to add one more thing to s431 00:35:20.480 --> 00:35:28.320 like I I care about my business I don't want to think about the uh I want it to be abstracted away from what I'm worrying about dayto-day s432 00:35:28.320 --> 00:35:35.119 totally I could uh describe maybe a common flow that we see that's like distillation or uh general model specific model. s433 00:35:35.119 --> 00:35:45.520 So if you think about it um in a lot of cases it's like if the the challenge I like to pose that's like a thought experiment is like if you tell someone think of any object s434 00:35:45.520 --> 00:35:52.320 and it's like think of an object in your fridge the latter of those is actually easier and the latter of those is actually where a lot of models kind of get deployed into real world settings. s435 00:35:52.320 --> 00:35:58.160 So, for example, um you know, Monterey Bay and the Monterey Bay Aquarium Research Institute, Embari they're called. s436 00:35:58.160 --> 00:36:09.280 They discovered a new fish species recently and they build with RobFlow to process all of the underwater um footage that they capture from their deep sea um submarines. s437 00:36:09.280 --> 00:36:21.520 And they use large models like segmented anything 3 and LMS as judge to basically say hey let's take all this video footage and let's build a autolel pipeline to understand as many things as we can about all the video footage that we've collected s438 00:36:21.520 --> 00:36:31.920 but then ultimately you know Sam knows everything about uh you know things from fish to maybe architectural diagrams to um items in your fridge and everything in between. s439 00:36:31.920 --> 00:36:35.599 But they only care about things that are you know in this case underwater deep sea exploration. s440 00:36:35.599 --> 00:36:46.000 And so a very common flow that we see for like the fine-tune last mile distillation is okay let's take the large context of an array of models have those maybe all have a pass at understanding something s441 00:36:46.000 --> 00:36:56.560 use LMS as judge to say hey we agree with some amount of consensus and now we have a specialized data set then we can use that specific model and actually run that on the submarine in real time and also post-processing s442 00:36:56.560 --> 00:37:09.119 for faster video and if you think about it like a lot of problems are that shape where it's like yeah I want like general as much intelligence to understand the thing and then ultimately the problem I'm solving is specific enough even if not fully unbounded. s443 00:37:09.119 --> 00:37:14.720 And a mistake I see pretty frequently is people thinking like SAM 3 which is an awesome model and a great family of models. s444 00:37:14.720 --> 00:37:24.640 It's like okay well I should take SAM 3 and then maybe just fine-tune SAM 3. And in some ways that actually doesn't make as much sense because you lose the thing that makes SAM 3 awesome which is the open vocabulary capabilities. s445 00:37:24.640 --> 00:37:41.200 What might be better is like if you know you're distilling down to a specific fixed class list then you can actually drop the large expensive autoenccoder portion of SAM and use a specific maybe like DTOR or more specialized model depending on the task that you're solving and get all the benefits of you know speed up and accuracy s446 00:37:41.200 --> 00:37:47.200 uh while still having the general knowledge of preparing and curating your problem and I think a lot of problems are of that shape. s447 00:37:47.200 --> 00:37:51.599 Are you are you managing that pipeline for your customer? s448 00:37:51.599 --> 00:37:54.640 the yeah the tooling makes it so they can do that. s449 00:37:54.640 --> 00:37:55.040 Um s450 00:37:55.040 --> 00:37:56.880 do you do it on their behalf? s451 00:37:56.880 --> 00:37:57.280 Uh s452 00:37:57.280 --> 00:37:59.200 or do you give them the tooling to do it? s453 00:37:59.200 --> 00:38:10.640 Give them the platform to do that and then there's like recipes where someone can go and do that and then like any good AI company there's an FTE that if you want I can sell you get around back here and they can do it for you. s454 00:38:10.880 --> 00:38:24.400 That's a great example basically of the use cases and workflows that I was talking about when you're trying to find like you know collect the data as you go for your business for your enterprise and then decide how you're going to fine-tune s455 00:38:24.400 --> 00:38:30.800 a model a small specialized model on those use cases as you grow that's how you become more token efficient. s456 00:38:30.800 --> 00:38:31.119 Yeah. s457 00:38:31.119 --> 00:38:31.440 Yeah. s458 00:38:31.440 --> 00:38:42.720 I I I think this year and and next year you're going to see a lot of using these like monster frontier models to bootstrap, you know, like a more efficient setup that runs on open source. s459 00:38:42.720 --> 00:38:44.800 And I think, you know, that's great. s460 00:38:44.800 --> 00:38:49.040 Like I think this is how a lot of the open source models have been built right now. s461 00:38:49.040 --> 00:38:53.200 And it's it's it's proving to be like quite hard to, you know, stop that. s462 00:38:53.200 --> 00:39:05.520 And I would just encourage people to just you know move away potentially from like you know these uh frontier models but like you know use them use them for what for for bootstrapping that right so like um s463 00:39:05.520 --> 00:39:19.040 I love that you said that actually because that's how the word AI engineer got coined right when Swix released that blog post that coined it the way that we used to build AI products was we would start with the with the machine learning we would start by training a model then we would go try to discover a s464 00:39:19.040 --> 00:39:23.359 use case and what these big models allowed us to do is actually flip flip it. s465 00:39:23.359 --> 00:39:27.920 We said we could start with discovering a use case and then get into ML if it makes sense. s466 00:39:27.920 --> 00:39:37.200 And that was actually uh honestly amazing foresight from Swix because he kind of defined that pattern uh for us a few years ago and gave this amazing conference as well. s467 00:39:37.200 --> 00:39:40.320 Yeah, it's crazy how far this conference has come just three years. s468 00:39:40.320 --> 00:39:41.119 Yeah, totally. s469 00:39:41.119 --> 00:39:41.839 I have a question. s470 00:39:41.839 --> 00:39:43.839 Do you guys have questions in the in the crowd? s471 00:39:43.839 --> 00:39:49.839 We have a few more minutes and I know we were talking about opening this up if you guys want to ask the panelists directly. s472 00:39:49.839 --> 00:39:51.200 Just just shout. s473 00:39:51.200 --> 00:39:53.520 Yeah, please go ahead. s474 00:39:53.520 --> 00:39:56.000 What are the big open problems in local? s475 00:39:56.000 --> 00:40:00.000 I feel like you're the luminaries of the field, you know. s476 00:40:01.599 --> 00:40:02.640 You want to repeat the question? s477 00:40:02.640 --> 00:40:03.200 Repeat the one. s478 00:40:03.200 --> 00:40:05.760 You just asked what are the open problems in local AI? s479 00:40:05.760 --> 00:40:06.880 I I can repeat it. s480 00:40:06.880 --> 00:40:09.920 It's what are the biggest open problems in local AI? s481 00:40:09.920 --> 00:40:10.480 Uh s482 00:40:10.480 --> 00:40:14.960 uh it continues to be optimizations and u for inference. s483 00:40:14.960 --> 00:40:23.760 It continues to be getting things easily kickstarted which is you know what ODS is XOS for specific hardware. s484 00:40:23.760 --> 00:40:40.480 It continues to be how can I make the most out of my budget constraints and hardware constraints and how can I squeeze the most performance out of that uh you know the the models that you know I still have the same 3090s that I used to run lama 2 and like dolphin fine tunes on and now they're s485 00:40:40.480 --> 00:40:48.880 running coins 3.5 coins 3.6 6 27 billion parameters with excellent performance you know more on that in my next presentation. s486 00:40:48.880 --> 00:40:58.079 So you know uh it comes down to the optimization and we are still very early that there is space for so many players for so many contributors. s487 00:40:58.079 --> 00:41:04.880 We need all the help we can get to make this thing the success that it needs to be that we need to make local AI the default. s488 00:41:04.880 --> 00:41:07.280 This has been my stance for years now. s489 00:41:07.280 --> 00:41:10.720 I have been saying open source AI must win for like since forever. s490 00:41:10.720 --> 00:41:25.839 And the way we do that is by giving the people whether that's individuals at home, medalsized businesses or enterprises an easy way to use these models in a very efficient way that doesn't give them headaches more than solve their problems. s491 00:41:25.839 --> 00:41:26.160 Totally. s492 00:41:26.160 --> 00:41:32.880 And if you look at the panels that are that are happening today, those are what we believe to be the biggest open-ended questions, which is why we try to assemble this panel. s493 00:41:32.880 --> 00:41:40.480 And so the you know we have quantization so all about talking about the footprint so that the models do fit on these uh uh on these smaller hardware footprints. s494 00:41:40.480 --> 00:41:49.680 Uh there's model routing there's models generally uh so those are kind of the what we feel like are the big problems now and as we do future local AI summits s495 00:41:49.680 --> 00:41:57.520 um I you know every time the panels should change to be the topics dour that are kind of holding the or or going to be going to usher the next chapter in. s496 00:41:57.520 --> 00:41:59.119 Can I add one thing to that? s497 00:41:59.119 --> 00:42:07.839 One of the biggest challenge so I think there's two two big challenges One is what we've been talking about of basically the trade-off of simplicity versus customizability. s498 00:42:07.839 --> 00:42:12.319 It's local, it's yours, you can do different things, optimization, if it's hosted, it's built out of the box. s499 00:42:12.319 --> 00:42:14.319 The way that trade-off is is always difficult. s500 00:42:14.319 --> 00:42:22.319 The second, which I actually encourage people in this room to help solve is the importance of open models is becoming increasingly in question. s501 00:42:22.319 --> 00:42:26.640 And I actually think that like if you think local AI is important, then you think open source AI is important. s502 00:42:26.640 --> 00:42:32.640 And it's actually really important to be an advocate for being able to use, change, adapt, and toy with models. s503 00:42:32.640 --> 00:42:40.960 And so I think that that's a problem that um could increasingly be something that we feel less control over absent advocating. s504 00:42:40.960 --> 00:42:41.760 That's a great point. s505 00:42:41.760 --> 00:42:44.800 I think everyone here feels very passionate about open source. s506 00:42:44.800 --> 00:42:47.520 That is why we actually have access to the space at all. s507 00:42:47.520 --> 00:42:49.280 That's why the space is has progressed. s508 00:42:49.280 --> 00:42:52.079 It's a it's a necessary competitive environment. s509 00:42:52.079 --> 00:42:55.119 It allows for the best ideas to make their way to everybody. s510 00:42:55.119 --> 00:43:03.359 Um, so definitely when there's talks about that being a threat, you know, we need to invest and advocate for open source and open source is so much bigger than AI, right? s511 00:43:03.359 --> 00:43:13.119 Like there's a reason why computation like compute was invented in on the east coast, but Silicon Valley happened here and it's because hippies realize that they could share ideas for free in software. s512 00:43:13.119 --> 00:43:19.520 I think Silicon Valley is this kind of tension between the capitalists that want to make money and hippies that want to give these ideas away for free. s513 00:43:19.520 --> 00:43:22.319 And that tension is what creates such an incredible environment here. s514 00:43:22.319 --> 00:43:23.920 They can coexist, by the way. s515 00:43:23.920 --> 00:43:26.400 you can have consumers and you can sell to businesses s516 00:43:26.400 --> 00:43:28.640 and that's when this space works works best. s517 00:43:28.640 --> 00:43:36.720 Can I just say like if you care about this and you want to be more of an active participant but maybe you know you don't want to get involved in the technical side you're not technical then there are ways to get involved. s518 00:43:36.720 --> 00:43:47.200 So there's a website that just came out called right to intelligence.org and this is a way for you to like get involved and actually like advocate for opensource and to ensure that you know we s519 00:43:47.200 --> 00:43:49.200 maintain freedom of intelligence. s520 00:43:49.200 --> 00:43:49.760 Awesome. s521 00:43:49.760 --> 00:43:50.880 So we're out of time. s522 00:43:50.880 --> 00:43:52.240 I want to thank you guys so much. s523 00:43:52.240 --> 00:43:54.319 Locally Summit's going to be a ton of fun. s524 00:43:54.319 --> 00:43:56.720 Thank you guys for this incredible way to kick this off. s525 00:43:56.720 --> 00:43:58.079 We're gonna have amazing demos. s526 00:43:58.079 --> 00:44:01.680 We're running foundational intelligence models here inside of this room. s527 00:44:01.680 --> 00:44:04.720 Uh we're gonna have a couple of amazing uh a few more panels. s528 00:44:04.720 --> 00:44:06.000 It's going to be an exciting day. s529 00:44:06.000 --> 00:44:10.079 Um and we're all lingering here, so ask questions and please participate. s530 00:44:10.079 --> 00:44:11.470 Thank you. s531 00:44:11.470 --> 00:44:13.470 [applause] s532 00:44:27.190 --> 00:44:29.190 [music]