From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI

AI Engineer · 20 min · 196 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:13Yeah, I guess uh let's get started.
00:15I'm I'm Armen.
00:16I'm I'm the co-founder and CEO of Perceptron.
00:20Um, we'll get a little bit into into what we do, but uh, primarily what I want to talk about today is kind of our research stance that we want to move away from distinctions between VLMs, VAS,
00:32world models, whatever you want to call it, to something that we call embodied foundation models.
00:42So specifically what we what we do at Perceptron kind of our northstar really is we want to be able to build physical AI foundations that give us the ability to perceive, understand
00:52and interact with the physical world in real time.
00:55And so kind of the the northstar mission is really to bridge the the physical and digital world uh worlds fundamentally meaning that our our goal is kind of wherever there is a
01:06um a device, an instrument, a robot, a camera, a sensor, uh we're essentially there providing intelligence to it.
01:14And specifically when we talk about this paradigm of being able to perceive, being able to reason, being able to act, we view this as kind of a a unification
01:23of uh traditional multimodal modeling.
01:25So I came from fair, I was there for six years and and and my target there was really to try to figure out how to scale up recipes for multimodal models.
01:34And so one of the early things that we we started working on and we've published a lot in this domain was around early fusion.
01:40So this concept that you want to bring in all these modalities as early as you can and and the real complexity there is trying to figure out what is the correct way to actually properly represent
01:49all the different modalities uh both on the input and on on the output that you want to be able to represent holistically and so VLMs have kind of became the when I talk about multimodal models
02:00um you probably think of VLM.
02:02So this is the ability to take in some image video and some text and be able to output some some text essentially.
02:08And there's there's variations of this.
02:10There's there's models like the ER models, the embodied reasoning models that are able to output maybe some grounding points um that are able to do spatial understanding or reasoning a little bit better.
02:20Then we have things like VALAS that extend kind of the output domain apart from just text to now actions.
02:26And these are traditionally of course continue to be built on by standard VLM backbones.
02:30Although there's been some efforts to try to migrate away from uh from VLMs to things like world models or world action models, although nothing that's been super fruitful just yet.
02:41And then we get into kind of more interesting and complex uh variants of multimodal models like world models where you essentially are outputting video uh from some inputs and the inputs can be either image or video or honestly image video and and actions.
02:56Um and the last point that I'll talk about is something recent which we call semantic world models which is you don't really output anything but you learn some type of representation that you think is useful
03:06um in the future.
03:07What we kind of view as an embodied foundation model is actually uh a framing that allows you to both to do the standard perception the embodied reasoning and then the northstar target of control all within one model.
03:20Um, so being able to reason across different sense of modalities on the input and being able to do kind of most of what I mentioned on the output all within one single unified model.
03:31And so very quickly I'm going to talk about two challenges and these are kind of fundamental research challenges that we face and I'll talk about kind of how our company has approached this
03:40um and and and what other folks are doing in here as well.
03:43So the very first thing is think about purely if you're going to try to model something like video like one hour of video depending on the representation you might have something like 1 million visual tokens that are coming in.
03:54The truth is is that there's not actually any ground truth that you can use uh effectively, right?
04:00So you can do things like and people have done this of course like pull out the transcripts uh predict the transcripts from the video or or maybe synthetically label some frames, ask it some questions and it turns out that this is kind of a humongous waste, right?
04:13So if you think about what's going into your model, you have 1 million tokens going in and you're essentially calculating the loss on something like you know 2% of all the tokens that are going in.
04:23And so this is very fundamentally problematic and and and the truth is is any way you try to figure out how to fix this, you're essentially injecting a wrong training signal.
04:31Either the signal is too sparse, it's too synthetic, or it's too indiscriminate.
04:36And so approaches beyond just synthetic enrichment have been, well, let's predict every single pixel.
04:41I mean, true, this is a very dense signal, but it actually very poorly allocates attention.
04:46there's, you know, you're comparing a, you're treating a background pixel with the same degree of importance as you're treating a gripper tip or the contact points or the specific failures or the physics.
04:56And so what we do kind of at Perceptron is, and this is kind of some of our core IP is we think about what does a natural perceptive objective look like?
05:04So specifically, how can I predict the percepts that we think will matter in in the future in a very very automatic way.
05:11So, as an example, you might hardcode something like, well, if you have a robotic arm, well, the the the tip of the grippers turns out to be a very useful percept that you can predict into the future.
05:20And folks have started doing this.
05:21Um, like the Momo app folks from AI2 have done this.
05:24There are other VAS that have done this.
05:26But this is still a hard-coded percept.
05:28So, the question is, can you figure out an automatic way that the model semantically is able to learn this very unique objective?
05:36And the truth is, we have figured out a way.
05:38uh we're not going to share how we do it here, but this is just kind of hinting at how we approach the the problem of of of sparity.
05:46The the second core problem that we've spent a lot of time focusing on is is context bloat.
05:51So, if you have always on cameras, if you have robots that don't necessarily wait, they don't stop, you're essentially having to reason over a very very long um very long amount of tokens.
06:01And and so text is relatively dense and video is relatively sparse.
06:05And so the question is, are there architectural breakthroughs that are there in order for you to be able to deal with this problem problem natively rather than just trying to
06:15on some synthetic level figure out how to fix this imbalance?
06:19Um, and so there's a couple of things that you can do.
06:23One kind of core first principle is you need to start treating different uh modalities as completely different.
06:29So you can't treat text tokens as the same as image tokens as the same as audio or video tokens.
06:34So, one thing that you can start thinking about doing is focusing on spatial compression or token compression.
06:40And people do really dumb things and we started off doing the dumb things and it does work.
06:43You can start thinking about averaging, you know, patch-wise representations across a a video or an image.
06:50Um, and you can start getting some interesting compression rates, you know, up up up to 10x.
06:55But still, this is relatively of a hack and there's there's not, you know, wellused architectural methods to actually solve this.
07:02Um, so what we've done in in in the last couple of months, we've released what we think is our approach to dealing with um varying degrees of sparity, which is just let the model figure out what tokens it should look at and what tokens it shouldn't look like.
07:16And so we released our data sparse mixture of experts paper, which essentially allows you to do this.
07:21Um, it allows the model, well, there's a router in the model allows it to actually predict what token I should input, what token I should skip.
07:29Um, and it allows us to do it kind of for free through all the different layers.
07:34And it turns out that if you just let the model learn, if you're not actually hard coding any significant um architectural priors, um, the model actually does learn.
07:43So if you end up visualizing the data sparse compute uh that our models use, you'll actually see that our models innately learn to start focusing on very um high density information or task relevant information.
07:55Um so in in in the upper right you can kind of see that the model decides to focus in on the graph which actually you as a human would also kind of zoom into because this is likely
08:04uh what's interesting within the figure.
08:07Um and it turns out that even task dependent um uh task dependent allocation ends up happening as well.
08:14So if you look at the bottom left if you just ask you know a very very general question you're kind of going to see an attention graph that is throughout the whole image.
08:22So the model just doesn't know what the proper way to allocate compute is.
08:25At the same time, if you ask it to do something like, you know, segment out all the fruit, you can see that it's going to allocate more tokens to what it thinks are fruit tokens.
08:33And so this is a very nice and clever trick that that we use and we've published and and other folks are starting to use around embedding priors into the architecture that are
08:43useful to deal with the sparsity imbalances of your modalities but not too harsh to the point that the models aren't actually learning uh natively.
08:54And so kind of we we put all this together and you guys might have seen the release, but we essentially released our our our model which was uh what we considered to be the first embodied
09:03foundation model a couple weeks ago.
09:05And this model is essentially frontier um with respect to Gemini 3.1 Pro.
09:10It's actually better than Gemini embodied reasoning.
09:12Um and it's something like 15 times cheaper.
09:15And it's essentially trained on this one pedibyte data set that we've collected across uh literally everything.
09:21It's uh from from internet crawls to to our own custommade training recipes or synthetic data pipelines.
09:29We have this one one pabyte of data across text, images, videos, trajectories.
09:33And these trajectories can be very very general.
09:35They can be desktop use trajectories.
09:37It can be playing a video game trajectory.
09:41And it turns out that once you start doing these things, very interesting properties end up emerging.
09:46And so the the biggest property that we kind of saw which is kind of obvious in retrospect is that you can actually start thinking of doing classical CV tasks
09:55as being an agentic task.
09:57And so in this case we essentially reframe detection as an agentic task.
10:01So our model can you know write code it can it can ask to zoom in into specific portions.
10:06Uh it can change the contrast.
10:09Um and you can actually see here this is a very hard problem.
10:12I think there's a whole Reddit subreddit of these problems of trying to find very hard objects and images.
10:17And our models essentially do this very well, but they do this in an agentic sense.
10:21So this isn't the this isn't a classical, you know, detect this one box.
10:25This is the model actually deciding that it needs to tile things up.
10:28It needs to change the contrast.
10:29It proposes a box here.
10:31I think here it yeah uh increases contrast and it can find the bird.
10:36So again, very hard to do if if like even for a human and humans are very good at perceptive tasks.
10:42This is a relatively tough thing to do.
10:44And this all comes from just having natively embodied models that actually understand how to look at different modalities.
10:51And so following up kind of how does this relate to the general physical AI stance around robotics?
10:57So I kind of stole this slide from from GDM folks.
11:00And so one thing we've we're starting to see from kind of robotics um u agentic systems is this separation between what we call kind of embodied reasoning models or orchestrators
11:12and tactile policy models.
11:14You can think of problems as like you know if I have a if if I'm making coffee and that takes me three minutes to do I mean one thing I can do is try to force my whole you know VLA to try to figure out how to do this individual task or what I can do is I
11:27can have an orchestrator model that breaks up this tasks into subtasks and there's a tactile control policy that's running on top and there's kind of a full spectrum between you know full VA only all the way to this kind of agentic system.
11:39Uh but the main thing I'm trying to highlight is that embodied reasoning is actually very interesting and complex problem that is yet to be solved.
11:46Um that being said, our models continue to be frontier on embodied reasoning.
11:50And because they're frontier, we start seeing really cool things that we haven't seen before.
11:55Here's a concrete example of doing very complex egocentric or not egocentric, but this is robotic data annotation.
12:02And you can kind of see here the model is jumping around looking at different portions of the video, uh, you know, clipping it, figuring out whether or not the captions are correct, self-ver
12:11uh, self-verifying.
12:12And we can all do this because AR models are fast.
12:15They're significantly cheaper than anything else that's out there.
12:18So, if you try to do this with Gemini, this video would probably cost you a couple of dollars where for us, it's probably in the sense.
12:24Um, and these all kind of emerged from being able to have these frontier embodied reasoning capabilities that we just previously have not seen from other models.
12:35Um, okay.
12:37Probably going to share the the biggest research breakthrough that we've had and I think we'll we'll share more of this uh in the upcoming weeks probably on Twitter.
12:46Uh, but one thing that we found is we've we've we've discovered new scaling laws for embodied foundation models.
12:52So these are models again that you can jointly do control-based training, you can do trajectory training, you can do perceptive training, you can do embodied reasoning training.
13:01If you just figure out what the right way to mix this all together is and the right objectives to use, you actually start seeing very interesting levers that you maybe previously haven't been able to see before.
13:11So the concrete lever that I'll talk about is this ability to trade um this ability to trade general video uh pre-training data for teleop data.
13:22So kind of as we know teleop data is very expensive.
13:24It's on the orders of you know $100 per hour of data for a hundred you know dollars I can collect significantly more video pre-training data.
13:33And so what this graph is showing is that um if you're just training pure VA's pure policies there's this band that you have.
13:40So you do actually still have scaling loss.
13:42So you do get benefits from more and more teop data.
13:45That being said, the benefits are not as substantial as if you are really training these unified embodied foundation models.
13:51And so this is what kind of the bottom bottom half of the graph is.
13:55Uh and the really cool kind of uh lever that we get is you can essentially trade 10x uh less teleop data if you have 10x more video pre-training data.
14:06And so far this is kind of held for the amount of compute that our company has.
14:10Um and it will be continuing to to kind of push the fold on uh how far you can push these embodied foundation models.
14:22So here's a couple of videos of of a policy that hopefully will will um we'll open source one of the smaller models in a couple of weeks.
14:30Uh but this is all running natively within a single model that is capable of doing the embodied reasoning in order to figure out the task actually is outputting control tokens.
14:40Um you can see it's a little bit jittery but that's okay.
14:43Hopefully it'll be figured out at scale and you can actually see very complex tasks that previously I think would be really tough for pure VA to do.
14:51So I think if you look at the right hand video, this is requiring them all to actually read the title of the book, have the knowledge about what type of book this is and then properly allocate it within one of the bins.
15:02Um so this is actually a multi-step task between perception and control that is really really tough to do if you have a kind of a pure endto-end control model that is not aware of
15:12the different perceptive tasks that it needs to accomplish in order to do this task.
15:23Yeah, it's pretty cool.
15:26It also works zero shot relatively well out of the box.
15:29So, we're excited to get this in the hands of folks in in a couple of weeks sometime in July.
15:38Um, yeah, going to leave a couple of minutes for for for general questions, but if you guys are interested, let's uh let's connect.
15:46So, one cool thing that we do with our company is we actually for a limited set of partners give access to our Mark1 weights.
15:51We give access to our larger embodied foundation models weights.
15:55So, email me, DM me on Twitter, uh, whatever is easier.
15:59Um, and then yeah, I'll open up.
16:00There's a couple minutes left for for questions.
16:17these models and like when I saw a lot of contss So can you talk a little bit on that?
16:41Yeah, I think it's it's so yeah.
16:43So the question is how are we able to nail temporal and uh understanding to this degree?
16:47Um it's a good question.
16:48I mean to be honest it's not nailed.
16:51So there's still a lot of work to actually get it to a place where you can reliably deploy it.
16:54The the core thing is how do you think about context management?
16:58So you have relatively limited context.
17:00So I think the the the models here have 1 million context, but that's relatively easy to fit in with the, you know, high FPS video.
17:08And so you have to start thinking about, are there interesting things that you can do?
17:12I'll throw something out there.
17:13We we used to do this, but we got past this, but like how do I think about like uh uh key frames versus delta frames?
17:19How can I manage my context by training these two off?
17:23Um, and then you start thinking about during your pre-training objective, how can I start kind of natively ingesting things that I think will be useful for the robotics tasks.
17:32Um, so for example, a very basic thing that even kind of the the Gemini models used to struggle at, I think the new ones are pretty good, but being able to tell cardalities.
17:40So like left and right is very hard to tell if you do internet scale crawls because no one on the internet is necessarily labeling things as, you know, this object is to the left of this object, it's below this object.
17:51So really thinking about data distributions early on gives you this ability relatively quickly.
17:56And it's also like er this type of embodied reasoning was a very concrete focus with this which is why I think we were able to kind of surpass Gemini ER
18:05with relatively less compute.
18:12One issue with sometimes I was wondering whether you looked your models are more robust than
18:28Yeah.
18:28So, probably the coolest robustness that we've seen is that uh for VA models specifically, like if you go and you take one of the Chinese ones um and you try to fine-tune it for a specific policy, if you just change the background
18:41of the I don't know, you can like in the table uh if you change the the background of the table, the policy will actually fail.
18:48What's really interesting, if you do this type of joint perceptive and control modeling, you're much more robust to these types of errors or if the light is hitting it a slightly different way.
18:57And I think we primarily view this as robustness to background uh in a way that I think traditional models don't necessarily have.
19:04That being said, I'm not going to overclaim like uh I mean like it's still relatively hard.
19:08I think if I was going to go and shine a flashlight into one of the one of the arms, it's probably not going to work.
19:13Uh but being able to jointly model these things helps a significant amount.
19:17We also do a lot of online augmentations.
19:19So, so we do actually, you know, fake um I don't know um one of the one of the arm cameras being off, right?
19:28We fake uh you know, sunlight coming in from a certain direction, right?
19:32So, we do these things during training to improve robustness.
19:35Uh but the big gains come from taking this early fusion paradigm and then moving it into the robotics domain.
19:44Cool.
19:47uh create knowledge base using the K1 model.
19:51Uh oh, knowledge bases.
19:53Um it's useful for like I I mean if you want to caption images, videos, it's relatively well.
19:57We work with robotics partners for like very complex egocentric annotation like this.
20:02Um so it's not necessarily building out an ontology, but being able to do kind of very deep structured extraction, I think our models are are very good at.
20:10Yeah.
20:11By the way, everything that I kind of showed here is kind of public APIs, so you can go play around with it.
20:15The benchmarks are public.
20:17Um, I think I'm out of time.
20:19They're cutting me off.
20:20So, uh, I can talk with folks outside, but thank you guys.