WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ZZcE0HeO-Hc NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ZZcE0HeO-Hc.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:13.599 --> 00:00:15.519 Yeah, I guess uh let's get started. s2 00:00:15.519 --> 00:00:16.480 I'm I'm Armen. s3 00:00:16.480 --> 00:00:20.000 I'm I'm the co-founder and CEO of Perceptron. s4 00:00:20.000 --> 00:00:32.239 Um, we'll get a little bit into into what we do, but uh, primarily what I want to talk about today is kind of our research stance that we want to move away from distinctions between VLMs, VAS, s5 00:00:32.239 --> 00:00:37.840 world models, whatever you want to call it, to something that we call embodied foundation models. s6 00:00:42.559 --> 00:00:52.480 So specifically what we what we do at Perceptron kind of our northstar really is we want to be able to build physical AI foundations that give us the ability to perceive, understand s7 00:00:52.480 --> 00:00:55.600 and interact with the physical world in real time. s8 00:00:55.600 --> 00:01:06.080 And so kind of the the northstar mission is really to bridge the the physical and digital world uh worlds fundamentally meaning that our our goal is kind of wherever there is a s9 00:01:06.080 --> 00:01:14.560 um a device, an instrument, a robot, a camera, a sensor, uh we're essentially there providing intelligence to it. s10 00:01:14.880 --> 00:01:23.040 And specifically when we talk about this paradigm of being able to perceive, being able to reason, being able to act, we view this as kind of a a unification s11 00:01:23.040 --> 00:01:25.920 of uh traditional multimodal modeling. s12 00:01:25.920 --> 00:01:34.960 So I came from fair, I was there for six years and and and my target there was really to try to figure out how to scale up recipes for multimodal models. s13 00:01:34.960 --> 00:01:40.479 And so one of the early things that we we started working on and we've published a lot in this domain was around early fusion. s14 00:01:40.479 --> 00:01:49.759 So this concept that you want to bring in all these modalities as early as you can and and the real complexity there is trying to figure out what is the correct way to actually properly represent s15 00:01:49.759 --> 00:02:00.880 all the different modalities uh both on the input and on on the output that you want to be able to represent holistically and so VLMs have kind of became the when I talk about multimodal models s16 00:02:00.880 --> 00:02:02.479 um you probably think of VLM. s17 00:02:02.479 --> 00:02:08.319 So this is the ability to take in some image video and some text and be able to output some some text essentially. s18 00:02:08.319 --> 00:02:10.160 And there's there's variations of this. s19 00:02:10.160 --> 00:02:20.400 There's there's models like the ER models, the embodied reasoning models that are able to output maybe some grounding points um that are able to do spatial understanding or reasoning a little bit better. s20 00:02:20.400 --> 00:02:26.000 Then we have things like VALAS that extend kind of the output domain apart from just text to now actions. s21 00:02:26.000 --> 00:02:30.959 And these are traditionally of course continue to be built on by standard VLM backbones. s22 00:02:30.959 --> 00:02:41.760 Although there's been some efforts to try to migrate away from uh from VLMs to things like world models or world action models, although nothing that's been super fruitful just yet. s23 00:02:41.760 --> 00:02:56.160 And then we get into kind of more interesting and complex uh variants of multimodal models like world models where you essentially are outputting video uh from some inputs and the inputs can be either image or video or honestly image video and and actions. s24 00:02:56.160 --> 00:03:06.159 Um and the last point that I'll talk about is something recent which we call semantic world models which is you don't really output anything but you learn some type of representation that you think is useful s25 00:03:06.159 --> 00:03:07.360 um in the future. s26 00:03:07.360 --> 00:03:20.080 What we kind of view as an embodied foundation model is actually uh a framing that allows you to both to do the standard perception the embodied reasoning and then the northstar target of control all within one model. s27 00:03:20.080 --> 00:03:31.120 Um, so being able to reason across different sense of modalities on the input and being able to do kind of most of what I mentioned on the output all within one single unified model. s28 00:03:31.599 --> 00:03:40.080 And so very quickly I'm going to talk about two challenges and these are kind of fundamental research challenges that we face and I'll talk about kind of how our company has approached this s29 00:03:40.080 --> 00:03:43.599 um and and and what other folks are doing in here as well. s30 00:03:43.599 --> 00:03:54.560 So the very first thing is think about purely if you're going to try to model something like video like one hour of video depending on the representation you might have something like 1 million visual tokens that are coming in. s31 00:03:54.560 --> 00:04:00.080 The truth is is that there's not actually any ground truth that you can use uh effectively, right? s32 00:04:00.080 --> 00:04:13.680 So you can do things like and people have done this of course like pull out the transcripts uh predict the transcripts from the video or or maybe synthetically label some frames, ask it some questions and it turns out that this is kind of a humongous waste, right? s33 00:04:13.680 --> 00:04:23.520 So if you think about what's going into your model, you have 1 million tokens going in and you're essentially calculating the loss on something like you know 2% of all the tokens that are going in. s34 00:04:23.520 --> 00:04:31.280 And so this is very fundamentally problematic and and and the truth is is any way you try to figure out how to fix this, you're essentially injecting a wrong training signal. s35 00:04:31.280 --> 00:04:36.240 Either the signal is too sparse, it's too synthetic, or it's too indiscriminate. s36 00:04:36.240 --> 00:04:41.600 And so approaches beyond just synthetic enrichment have been, well, let's predict every single pixel. s37 00:04:41.600 --> 00:04:46.479 I mean, true, this is a very dense signal, but it actually very poorly allocates attention. s38 00:04:46.479 --> 00:04:56.400 there's, you know, you're comparing a, you're treating a background pixel with the same degree of importance as you're treating a gripper tip or the contact points or the specific failures or the physics. s39 00:04:56.400 --> 00:05:04.720 And so what we do kind of at Perceptron is, and this is kind of some of our core IP is we think about what does a natural perceptive objective look like? s40 00:05:04.720 --> 00:05:11.600 So specifically, how can I predict the percepts that we think will matter in in the future in a very very automatic way. s41 00:05:11.600 --> 00:05:20.160 So, as an example, you might hardcode something like, well, if you have a robotic arm, well, the the the tip of the grippers turns out to be a very useful percept that you can predict into the future. s42 00:05:20.160 --> 00:05:21.520 And folks have started doing this. s43 00:05:21.520 --> 00:05:24.800 Um, like the Momo app folks from AI2 have done this. s44 00:05:24.800 --> 00:05:26.639 There are other VAS that have done this. s45 00:05:26.639 --> 00:05:28.479 But this is still a hard-coded percept. s46 00:05:28.479 --> 00:05:36.160 So, the question is, can you figure out an automatic way that the model semantically is able to learn this very unique objective? s47 00:05:36.160 --> 00:05:38.479 And the truth is, we have figured out a way. s48 00:05:38.479 --> 00:05:46.240 uh we're not going to share how we do it here, but this is just kind of hinting at how we approach the the problem of of of sparity. s49 00:05:46.400 --> 00:05:51.039 The the second core problem that we've spent a lot of time focusing on is is context bloat. s50 00:05:51.039 --> 00:06:01.360 So, if you have always on cameras, if you have robots that don't necessarily wait, they don't stop, you're essentially having to reason over a very very long um very long amount of tokens. s51 00:06:01.360 --> 00:06:05.440 And and so text is relatively dense and video is relatively sparse. s52 00:06:05.440 --> 00:06:15.039 And so the question is, are there architectural breakthroughs that are there in order for you to be able to deal with this problem problem natively rather than just trying to s53 00:06:15.039 --> 00:06:19.840 on some synthetic level figure out how to fix this imbalance? s54 00:06:19.919 --> 00:06:23.280 Um, and so there's a couple of things that you can do. s55 00:06:23.280 --> 00:06:29.280 One kind of core first principle is you need to start treating different uh modalities as completely different. s56 00:06:29.280 --> 00:06:34.960 So you can't treat text tokens as the same as image tokens as the same as audio or video tokens. s57 00:06:34.960 --> 00:06:40.080 So, one thing that you can start thinking about doing is focusing on spatial compression or token compression. s58 00:06:40.080 --> 00:06:43.759 And people do really dumb things and we started off doing the dumb things and it does work. s59 00:06:43.759 --> 00:06:50.160 You can start thinking about averaging, you know, patch-wise representations across a a video or an image. s60 00:06:50.160 --> 00:06:55.680 Um, and you can start getting some interesting compression rates, you know, up up up to 10x. s61 00:06:55.680 --> 00:07:02.880 But still, this is relatively of a hack and there's there's not, you know, wellused architectural methods to actually solve this. s62 00:07:02.880 --> 00:07:16.160 Um, so what we've done in in in the last couple of months, we've released what we think is our approach to dealing with um varying degrees of sparity, which is just let the model figure out what tokens it should look at and what tokens it shouldn't look like. s63 00:07:16.160 --> 00:07:21.199 And so we released our data sparse mixture of experts paper, which essentially allows you to do this. s64 00:07:21.199 --> 00:07:29.039 Um, it allows the model, well, there's a router in the model allows it to actually predict what token I should input, what token I should skip. s65 00:07:29.039 --> 00:07:34.560 Um, and it allows us to do it kind of for free through all the different layers. s66 00:07:34.560 --> 00:07:43.759 And it turns out that if you just let the model learn, if you're not actually hard coding any significant um architectural priors, um, the model actually does learn. s67 00:07:43.759 --> 00:07:55.919 So if you end up visualizing the data sparse compute uh that our models use, you'll actually see that our models innately learn to start focusing on very um high density information or task relevant information. s68 00:07:55.919 --> 00:08:04.960 Um so in in in the upper right you can kind of see that the model decides to focus in on the graph which actually you as a human would also kind of zoom into because this is likely s69 00:08:04.960 --> 00:08:07.039 uh what's interesting within the figure. s70 00:08:07.039 --> 00:08:14.319 Um and it turns out that even task dependent um uh task dependent allocation ends up happening as well. s71 00:08:14.319 --> 00:08:22.240 So if you look at the bottom left if you just ask you know a very very general question you're kind of going to see an attention graph that is throughout the whole image. s72 00:08:22.240 --> 00:08:25.599 So the model just doesn't know what the proper way to allocate compute is. s73 00:08:25.599 --> 00:08:33.200 At the same time, if you ask it to do something like, you know, segment out all the fruit, you can see that it's going to allocate more tokens to what it thinks are fruit tokens. s74 00:08:33.200 --> 00:08:43.519 And so this is a very nice and clever trick that that we use and we've published and and other folks are starting to use around embedding priors into the architecture that are s75 00:08:43.519 --> 00:08:53.200 useful to deal with the sparsity imbalances of your modalities but not too harsh to the point that the models aren't actually learning uh natively. s76 00:08:54.160 --> 00:09:03.600 And so kind of we we put all this together and you guys might have seen the release, but we essentially released our our our model which was uh what we considered to be the first embodied s77 00:09:03.600 --> 00:09:05.600 foundation model a couple weeks ago. s78 00:09:05.600 --> 00:09:10.480 And this model is essentially frontier um with respect to Gemini 3.1 Pro. s79 00:09:10.480 --> 00:09:12.880 It's actually better than Gemini embodied reasoning. s80 00:09:12.880 --> 00:09:15.200 Um and it's something like 15 times cheaper. s81 00:09:15.200 --> 00:09:21.200 And it's essentially trained on this one pedibyte data set that we've collected across uh literally everything. s82 00:09:21.200 --> 00:09:29.040 It's uh from from internet crawls to to our own custommade training recipes or synthetic data pipelines. s83 00:09:29.040 --> 00:09:33.600 We have this one one pabyte of data across text, images, videos, trajectories. s84 00:09:33.600 --> 00:09:35.839 And these trajectories can be very very general. s85 00:09:35.839 --> 00:09:37.920 They can be desktop use trajectories. s86 00:09:37.920 --> 00:09:41.360 It can be playing a video game trajectory. s87 00:09:41.839 --> 00:09:46.480 And it turns out that once you start doing these things, very interesting properties end up emerging. s88 00:09:46.480 --> 00:09:55.279 And so the the biggest property that we kind of saw which is kind of obvious in retrospect is that you can actually start thinking of doing classical CV tasks s89 00:09:55.279 --> 00:09:57.519 as being an agentic task. s90 00:09:57.519 --> 00:10:01.120 And so in this case we essentially reframe detection as an agentic task. s91 00:10:01.120 --> 00:10:06.640 So our model can you know write code it can it can ask to zoom in into specific portions. s92 00:10:06.640 --> 00:10:09.200 Uh it can change the contrast. s93 00:10:09.200 --> 00:10:12.320 Um and you can actually see here this is a very hard problem. s94 00:10:12.320 --> 00:10:17.519 I think there's a whole Reddit subreddit of these problems of trying to find very hard objects and images. s95 00:10:17.519 --> 00:10:21.120 And our models essentially do this very well, but they do this in an agentic sense. s96 00:10:21.120 --> 00:10:25.360 So this isn't the this isn't a classical, you know, detect this one box. s97 00:10:25.360 --> 00:10:28.160 This is the model actually deciding that it needs to tile things up. s98 00:10:28.160 --> 00:10:29.839 It needs to change the contrast. s99 00:10:29.839 --> 00:10:31.519 It proposes a box here. s100 00:10:31.519 --> 00:10:36.959 I think here it yeah uh increases contrast and it can find the bird. s101 00:10:36.959 --> 00:10:42.320 So again, very hard to do if if like even for a human and humans are very good at perceptive tasks. s102 00:10:42.320 --> 00:10:44.320 This is a relatively tough thing to do. s103 00:10:44.320 --> 00:10:51.600 And this all comes from just having natively embodied models that actually understand how to look at different modalities. s104 00:10:51.600 --> 00:10:57.760 And so following up kind of how does this relate to the general physical AI stance around robotics? s105 00:10:57.760 --> 00:11:00.959 So I kind of stole this slide from from GDM folks. s106 00:11:00.959 --> 00:11:12.560 And so one thing we've we're starting to see from kind of robotics um u agentic systems is this separation between what we call kind of embodied reasoning models or orchestrators s107 00:11:12.560 --> 00:11:14.640 and tactile policy models. s108 00:11:14.640 --> 00:11:27.360 You can think of problems as like you know if I have a if if I'm making coffee and that takes me three minutes to do I mean one thing I can do is try to force my whole you know VLA to try to figure out how to do this individual task or what I can do is I s109 00:11:27.360 --> 00:11:39.360 can have an orchestrator model that breaks up this tasks into subtasks and there's a tactile control policy that's running on top and there's kind of a full spectrum between you know full VA only all the way to this kind of agentic system. s110 00:11:39.360 --> 00:11:46.240 Uh but the main thing I'm trying to highlight is that embodied reasoning is actually very interesting and complex problem that is yet to be solved. s111 00:11:46.240 --> 00:11:50.959 Um that being said, our models continue to be frontier on embodied reasoning. s112 00:11:50.959 --> 00:11:55.600 And because they're frontier, we start seeing really cool things that we haven't seen before. s113 00:11:55.600 --> 00:12:02.320 Here's a concrete example of doing very complex egocentric or not egocentric, but this is robotic data annotation. s114 00:12:02.320 --> 00:12:11.360 And you can kind of see here the model is jumping around looking at different portions of the video, uh, you know, clipping it, figuring out whether or not the captions are correct, self-ver s115 00:12:11.360 --> 00:12:12.959 uh, self-verifying. s116 00:12:12.959 --> 00:12:15.519 And we can all do this because AR models are fast. s117 00:12:15.519 --> 00:12:18.720 They're significantly cheaper than anything else that's out there. s118 00:12:18.720 --> 00:12:24.560 So, if you try to do this with Gemini, this video would probably cost you a couple of dollars where for us, it's probably in the sense. s119 00:12:24.560 --> 00:12:34.000 Um, and these all kind of emerged from being able to have these frontier embodied reasoning capabilities that we just previously have not seen from other models. s120 00:12:35.600 --> 00:12:37.839 Um, okay. s121 00:12:37.839 --> 00:12:46.240 Probably going to share the the biggest research breakthrough that we've had and I think we'll we'll share more of this uh in the upcoming weeks probably on Twitter. s122 00:12:46.240 --> 00:12:52.000 Uh, but one thing that we found is we've we've we've discovered new scaling laws for embodied foundation models. s123 00:12:52.000 --> 00:13:01.519 So these are models again that you can jointly do control-based training, you can do trajectory training, you can do perceptive training, you can do embodied reasoning training. s124 00:13:01.519 --> 00:13:11.279 If you just figure out what the right way to mix this all together is and the right objectives to use, you actually start seeing very interesting levers that you maybe previously haven't been able to see before. s125 00:13:11.279 --> 00:13:22.399 So the concrete lever that I'll talk about is this ability to trade um this ability to trade general video uh pre-training data for teleop data. s126 00:13:22.399 --> 00:13:24.800 So kind of as we know teleop data is very expensive. s127 00:13:24.800 --> 00:13:33.360 It's on the orders of you know $100 per hour of data for a hundred you know dollars I can collect significantly more video pre-training data. s128 00:13:33.360 --> 00:13:40.560 And so what this graph is showing is that um if you're just training pure VA's pure policies there's this band that you have. s129 00:13:40.560 --> 00:13:42.560 So you do actually still have scaling loss. s130 00:13:42.560 --> 00:13:45.360 So you do get benefits from more and more teop data. s131 00:13:45.360 --> 00:13:51.519 That being said, the benefits are not as substantial as if you are really training these unified embodied foundation models. s132 00:13:51.519 --> 00:13:55.600 And so this is what kind of the bottom bottom half of the graph is. s133 00:13:55.600 --> 00:14:06.560 Uh and the really cool kind of uh lever that we get is you can essentially trade 10x uh less teleop data if you have 10x more video pre-training data. s134 00:14:06.560 --> 00:14:10.720 And so far this is kind of held for the amount of compute that our company has. s135 00:14:10.720 --> 00:14:19.519 Um and it will be continuing to to kind of push the fold on uh how far you can push these embodied foundation models. s136 00:14:22.800 --> 00:14:30.800 So here's a couple of videos of of a policy that hopefully will will um we'll open source one of the smaller models in a couple of weeks. s137 00:14:30.800 --> 00:14:40.959 Uh but this is all running natively within a single model that is capable of doing the embodied reasoning in order to figure out the task actually is outputting control tokens. s138 00:14:40.959 --> 00:14:43.680 Um you can see it's a little bit jittery but that's okay. s139 00:14:43.680 --> 00:14:51.440 Hopefully it'll be figured out at scale and you can actually see very complex tasks that previously I think would be really tough for pure VA to do. s140 00:14:51.440 --> 00:15:02.639 So I think if you look at the right hand video, this is requiring them all to actually read the title of the book, have the knowledge about what type of book this is and then properly allocate it within one of the bins. s141 00:15:02.639 --> 00:15:12.480 Um so this is actually a multi-step task between perception and control that is really really tough to do if you have a kind of a pure endto-end control model that is not aware of s142 00:15:12.480 --> 00:15:18.079 the different perceptive tasks that it needs to accomplish in order to do this task. s143 00:15:23.519 --> 00:15:26.000 Yeah, it's pretty cool. s144 00:15:26.000 --> 00:15:29.440 It also works zero shot relatively well out of the box. s145 00:15:29.440 --> 00:15:35.680 So, we're excited to get this in the hands of folks in in a couple of weeks sometime in July. s146 00:15:38.160 --> 00:15:46.399 Um, yeah, going to leave a couple of minutes for for for general questions, but if you guys are interested, let's uh let's connect. s147 00:15:46.399 --> 00:15:51.920 So, one cool thing that we do with our company is we actually for a limited set of partners give access to our Mark1 weights. s148 00:15:51.920 --> 00:15:55.279 We give access to our larger embodied foundation models weights. s149 00:15:55.279 --> 00:15:59.279 So, email me, DM me on Twitter, uh, whatever is easier. s150 00:15:59.279 --> 00:16:00.800 Um, and then yeah, I'll open up. s151 00:16:00.800 --> 00:16:04.399 There's a couple minutes left for for questions. s152 00:16:17.759 --> 00:16:41.199 these models and like when I saw a lot of contss So can you talk a little bit on that? s153 00:16:41.199 --> 00:16:43.040 Yeah, I think it's it's so yeah. s154 00:16:43.040 --> 00:16:47.839 So the question is how are we able to nail temporal and uh understanding to this degree? s155 00:16:47.839 --> 00:16:48.959 Um it's a good question. s156 00:16:48.959 --> 00:16:51.120 I mean to be honest it's not nailed. s157 00:16:51.120 --> 00:16:54.720 So there's still a lot of work to actually get it to a place where you can reliably deploy it. s158 00:16:54.720 --> 00:16:58.560 The the core thing is how do you think about context management? s159 00:16:58.560 --> 00:17:00.880 So you have relatively limited context. s160 00:17:00.880 --> 00:17:08.160 So I think the the the models here have 1 million context, but that's relatively easy to fit in with the, you know, high FPS video. s161 00:17:08.160 --> 00:17:12.079 And so you have to start thinking about, are there interesting things that you can do? s162 00:17:12.079 --> 00:17:13.199 I'll throw something out there. s163 00:17:13.199 --> 00:17:19.600 We we used to do this, but we got past this, but like how do I think about like uh uh key frames versus delta frames? s164 00:17:19.600 --> 00:17:23.120 How can I manage my context by training these two off? s165 00:17:23.120 --> 00:17:32.720 Um, and then you start thinking about during your pre-training objective, how can I start kind of natively ingesting things that I think will be useful for the robotics tasks. s166 00:17:32.720 --> 00:17:40.960 Um, so for example, a very basic thing that even kind of the the Gemini models used to struggle at, I think the new ones are pretty good, but being able to tell cardalities. s167 00:17:40.960 --> 00:17:51.760 So like left and right is very hard to tell if you do internet scale crawls because no one on the internet is necessarily labeling things as, you know, this object is to the left of this object, it's below this object. s168 00:17:51.760 --> 00:17:56.320 So really thinking about data distributions early on gives you this ability relatively quickly. s169 00:17:56.320 --> 00:18:05.200 And it's also like er this type of embodied reasoning was a very concrete focus with this which is why I think we were able to kind of surpass Gemini ER s170 00:18:05.200 --> 00:18:08.640 with relatively less compute. s171 00:18:12.480 --> 00:18:26.080 One issue with sometimes I was wondering whether you looked your models are more robust than s172 00:18:28.160 --> 00:18:28.480 Yeah. s173 00:18:28.480 --> 00:18:41.919 So, probably the coolest robustness that we've seen is that uh for VA models specifically, like if you go and you take one of the Chinese ones um and you try to fine-tune it for a specific policy, if you just change the background s174 00:18:41.919 --> 00:18:48.480 of the I don't know, you can like in the table uh if you change the the background of the table, the policy will actually fail. s175 00:18:48.480 --> 00:18:57.440 What's really interesting, if you do this type of joint perceptive and control modeling, you're much more robust to these types of errors or if the light is hitting it a slightly different way. s176 00:18:57.440 --> 00:19:04.240 And I think we primarily view this as robustness to background uh in a way that I think traditional models don't necessarily have. s177 00:19:04.240 --> 00:19:08.080 That being said, I'm not going to overclaim like uh I mean like it's still relatively hard. s178 00:19:08.080 --> 00:19:13.760 I think if I was going to go and shine a flashlight into one of the one of the arms, it's probably not going to work. s179 00:19:13.760 --> 00:19:17.760 Uh but being able to jointly model these things helps a significant amount. s180 00:19:17.760 --> 00:19:19.919 We also do a lot of online augmentations. s181 00:19:19.919 --> 00:19:28.000 So, so we do actually, you know, fake um I don't know um one of the one of the arm cameras being off, right? s182 00:19:28.000 --> 00:19:32.320 We fake uh you know, sunlight coming in from a certain direction, right? s183 00:19:32.320 --> 00:19:35.679 So, we do these things during training to improve robustness. s184 00:19:35.679 --> 00:19:42.880 Uh but the big gains come from taking this early fusion paradigm and then moving it into the robotics domain. s185 00:19:44.880 --> 00:19:46.880 Cool. s186 00:19:47.039 --> 00:19:51.600 uh create knowledge base using the K1 model. s187 00:19:51.600 --> 00:19:53.039 Uh oh, knowledge bases. s188 00:19:53.039 --> 00:19:57.840 Um it's useful for like I I mean if you want to caption images, videos, it's relatively well. s189 00:19:57.840 --> 00:20:02.559 We work with robotics partners for like very complex egocentric annotation like this. s190 00:20:02.559 --> 00:20:10.880 Um so it's not necessarily building out an ontology, but being able to do kind of very deep structured extraction, I think our models are are very good at. s191 00:20:10.880 --> 00:20:11.200 Yeah. s192 00:20:11.200 --> 00:20:15.919 By the way, everything that I kind of showed here is kind of public APIs, so you can go play around with it. s193 00:20:15.919 --> 00:20:17.840 The benchmarks are public. s194 00:20:17.840 --> 00:20:19.760 Um, I think I'm out of time. s195 00:20:19.760 --> 00:20:20.799 They're cutting me off. s196 00:20:20.799 --> 00:20:25.840 So, uh, I can talk with folks outside, but thank you guys.