WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=MpkARD5pq-M

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/MpkARD5pq-M.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.760 --> 00:00:13.600
Hello.

s3
00:00:13.600 --> 00:00:25.600
Um I want you to uh zoom out for a second and think about if somebody were to tell you today that we would take all the information on the internet

s4
00:00:25.600 --> 00:00:31.680
and then we would train a model on it and we would use that in order to solve scientific problems that were not on the internet.

s5
00:00:31.680 --> 00:00:34.160
I think a lot of people would tell you that that's crazy.

s6
00:00:34.160 --> 00:00:41.640
Um and uh what we're doing um has a lot of parallels to that story.

s7
00:00:41.640 --> 00:00:51.560
Um and so if you think about how intelligence works imagine for a second that you're in the movie Frozen and you're Olaf, the the snowman.

s8
00:00:51.560 --> 00:01:03.280
Um if if Olaf was an LLM and it there there's a scene which is one of my favorites which is where he rolls down the mountain and there's a little stone that come that pops up and Olaf

s9
00:01:03.280 --> 00:01:14.880
spreads out temporarily becomes a disassembled snowman and then becomes an assembled snowman and Olaf knows how to do that because Olaf can observe as he is acting.

s10
00:01:14.880 --> 00:01:26.360
Olaf can, unlike LLMs which auto aggressively predict tokens, um constantly part of that behavior is that it's constantly observing as it is acting.

s11
00:01:26.360 --> 00:01:36.640
And so um this is the big difference between you know, when you see LLMs try to solve robotics problems or when you look at native architectures like what we're working on,

s12
00:01:36.640 --> 00:01:42.280
um it's very very easy to think of LLMs in the demos that you see on Twitter as the thing that will solve everything.

s13
00:01:42.280 --> 00:01:47.080
There's fundamental limitations that we see uh where purpose-built models are much better fit.

s14
00:01:47.080 --> 00:01:52.280
Now today I'm going to talk a little bit about what the other side of that coin looks like.

s15
00:01:52.280 --> 00:01:56.600
Um and so for us it all started with Medal.

s16
00:01:56.600 --> 00:02:03.172
We built the largest video game recorder in the world um and we think of that as the internet.

s17
00:02:03.172 --> 00:02:08.892
So, people use Medal to record themselves playing tens of thousands of different video games.

s18
00:02:08.892 --> 00:02:18.972
And if you think about it, uh, as a human, we can teleoperate a robot with the exact same brain that we played that video game.

s19
00:02:18.972 --> 00:02:27.132
And in fact, um, in many cases, we also can use the same controller.

s20
00:02:27.132 --> 00:02:35.732
So, what this means is that the action space of video games, often because of game controllers, and the action space of teleoperations or the real world, in many cases, is the same.

s21
00:02:35.732 --> 00:02:41.972
Um, and so, every single robot built below, um, certain degrees of freedom ships with the game controller.

s22
00:02:41.972 --> 00:02:44.452
A majority of the supply chain is already worked this way.

s23
00:02:44.452 --> 00:02:54.652
And so, we thought, what if you just take all of the action data that we get through the video recorder, um, which associates the inputs and the actions with the actual frames.

s24
00:02:54.652 --> 00:03:02.932
We label the actions, so we under- understand semantically meaning behind every action, and we use it to give it to a vision-based model.

s25
00:03:02.932 --> 00:03:07.572
And so, what you see here on the left is just an intuitive way of how the model works.

s26
00:03:07.572 --> 00:03:12.572
So, you give the model a set of keys to choose from, essentially a computer use model.

s27
00:03:12.572 --> 00:03:16.852
It It It gives frames as the input, so you can give it a text prompt as well.

s28
00:03:16.852 --> 00:03:21.572
And you send it through API, and you say, "Somebody waves at you, wave back."

s29
00:03:21.572 --> 00:03:28.092
In the input space, model understands that you have a greet action available, it will output the action associated with that.

s30
00:03:28.092 --> 00:03:31.772
Um, one way of intu- of intuitively understanding how this works.

s31
00:03:31.772 --> 00:03:38.772
So, um, I've obviously simplified here because it's predicting next token, not the next word, but bear with me here.

s32
00:03:38.772 --> 00:03:47.292
Um, LLMs took large-scale internet data, then trained it to predict the next text token, and now use it for writing software.

s33
00:03:47.292 --> 00:03:55.692
Um, and these are all workloads that are really, really well fit for futures that are not simultaneously unfolding as you are predicting, right?

s34
00:03:55.692 --> 00:03:57.132
Code is turn-based.

s35
00:03:57.132 --> 00:04:02.654
You can say something, the model will do something, you can say something again, the model will do something me again.

s36
00:04:02.654 --> 00:04:12.414
And um you can use it for lots of valuable work, but at the end of the day, because it's turn-based, um it's very, very difficult for it to generalize

s37
00:04:12.414 --> 00:04:15.974
to environments, especially that include um space and time.

s38
00:04:15.974 --> 00:04:29.534
And so, our models, the way that they work is they're continuously receiving frames or receiving batches of essentially um information from the camera of the robot or it could be a a screen.

s39
00:04:29.534 --> 00:04:34.774
And it predicts everything as as if it were just a video game.

s40
00:04:34.774 --> 00:04:39.894
So, you can think of the way that Anthropic focused on just nailing coding.

s41
00:04:39.894 --> 00:04:43.614
So, you know, they were never frontier in just base text generation.

s42
00:04:43.614 --> 00:04:46.694
They always focused on being the best at generating code.

s43
00:04:46.694 --> 00:04:51.534
And then what the debate that they made is that more people would start bringing more problems into coding space.

s44
00:04:51.534 --> 00:05:03.454
If you think of general intuition, our bet is that you can as humans, we are generally pixel-to-action converters plus sensors, which in many cases you can convert into pixels as well.

s45
00:05:03.454 --> 00:05:06.374
And you can output actions at the same time.

s46
00:05:06.374 --> 00:05:09.494
And those actions are in this case either keyboard and mouse or a game controller.

s47
00:05:09.494 --> 00:05:18.134
So, the output tokens of the model are um essentially equivalent to that of uh enhanced computer use model with more more more actions on the output space to internalize it.

s48
00:05:18.134 --> 00:05:21.054
So, you send frames and you get actions.

s49
00:05:21.054 --> 00:05:28.334
This works again because every single thing that you see here can control um all of these different embodiments, right?

s50
00:05:28.334 --> 00:05:37.854
So, um as humans, we do not know how to predict motor torque.

s51
00:05:37.854 --> 00:05:41.614
We do not We do not know what value to send to a machine.

s52
00:05:41.614 --> 00:05:49.294
Um there's this beautiful device that has been made for many years, uh which actually does that work for us, so we can control them.

s53
00:05:49.294 --> 00:05:59.254
Most robotics companies today are trying to solve the problem at the end-to-end level, trying to predict exact poses or exact um values uh in the motor.

s54
00:05:59.254 --> 00:06:16.295
I instead of just looking and picking up the thing that every robotics company has built to generalize our very limited abilities to have physics understanding and predictive text values

s55
00:06:16.295 --> 00:06:17.175
and solve for it.

s56
00:06:17.175 --> 00:06:20.655
And so, our approach is let's just use that.

s57
00:06:20.655 --> 00:06:28.935
Um and so Metal at the moment sees about a billion videos uploaded per year.

s58
00:06:28.935 --> 00:06:31.495
This year on we're on track for 3 billion.

s59
00:06:31.495 --> 00:06:36.815
Um which lead to over a trillion and a half action tokens per month.

s60
00:06:36.815 --> 00:06:39.535
So, just for comparison, think about the bet.

s61
00:06:39.535 --> 00:06:46.015
The total amount of data that Frontier Labs used today to train robotics models, we collect in at the moment half a day.

s62
00:06:46.015 --> 00:06:59.615
Um and so I the the type of bet and again, think of the fact that humans can use the exact same controllers and their eyes to play video games that they can teleoperate robots.

s63
00:06:59.615 --> 00:07:00.575
It is the same thing.

s64
00:07:00.575 --> 00:07:12.959
We just bet on scaling pixel space the same way that Anthropic scaled coding space and betting that people will just as we scale data compute into pixel space that people will bring more problems into pixel space.

s65
00:07:12.959 --> 00:07:14.175
[snorts]

s66
00:07:14.175 --> 00:07:16.255
And so, what happens?

s67
00:07:16.255 --> 00:07:36.695
Um All of the sudden, you get an incredibly general agent that can go and solve problems on computers, in the real world, um and in addition, similarly to how Anthropic focuses on doing coding environments,

s68
00:07:36.695 --> 00:07:44.215
we focus on world models, which most of you know as heard of Mira release world models, to then generate additional environments for these models to learn in.

s69
00:07:44.215 --> 00:07:58.015
One way of internalizing a big advantage of this approach against LLMs is if you're an if you're an LLM company and you have to bet on the ability to generate environments for your models to learn, you have you have two problems.

s70
00:07:58.015 --> 00:08:04.615
One, you actually need to spend the compute to simulate environments and the more realistic you try to simulate, the more expensive that gets.

s71
00:08:04.615 --> 00:08:08.057
Um and so, imagine you have you want to learn a new thing.

s72
00:08:08.057 --> 00:08:14.777
You want to learn that new thing in the exact input space of the human, right?

s73
00:08:14.777 --> 00:08:18.297
So, you if if we are learning from pixels, we're we're learning by looking around.

s74
00:08:18.297 --> 00:08:23.737
If I'm trying to communicate something, I want my robot partner to observe that exact same thing.

s75
00:08:23.737 --> 00:08:33.097
And so, um what what I don't want is for data to go through an LLM, for a digital twin to have to be created, for the model to have to get deployed,

s76
00:08:33.097 --> 00:08:39.137
for there to be a bunch of verifier checks whether or not things are equivalent, and then for the robot to to learn much later, right?

s77
00:08:39.137 --> 00:08:48.497
And so, one way of thinking about it is when you can learn in the native modality of the human, you can all of the sudden start solving a lot more problems a lot faster.

s78
00:08:48.497 --> 00:08:57.257
You what you'll see when these models come out is they're significantly faster than any LLM um at doing these things.

s79
00:08:57.257 --> 00:09:01.017
We because games are so action-dense.

s80
00:09:01.017 --> 00:09:08.097
So, what you see, right, is um one thing that's interesting is uh you saw, for instance, the model predicting actions at the same time.

s81
00:09:08.097 --> 00:09:15.537
Um we're able to uh take that action density from gamers and convert them into really complex robotics workloads.

s82
00:09:15.537 --> 00:09:23.017
So, for example, if you are operating a drone and you need to make lots of little adjustments, you really really really want to start with games.

s83
00:09:23.017 --> 00:09:27.937
You don't want to start with uh teleoperations because the human will just take a lot less actions.

s84
00:09:27.937 --> 00:09:28.937
It has a lot less information.

s85
00:09:28.937 --> 00:09:31.257
So, you want a model that can predict really high APM.

s86
00:09:31.257 --> 00:09:40.457
Um And so, the APM of the data set puts a lot of these problems all of the sudden in reach so that it can do a lot of these actions simultaneously.

s87
00:09:40.457 --> 00:09:44.337
And as I mentioned, the model doesn't just understand the key it's predicting.

s88
00:09:44.337 --> 00:09:50.977
It also understands the semantic meaning associated with that key it's predicting, which is really important for things like um interpretability.

s89
00:09:50.977 --> 00:09:56.017
I So, I we will be releasing the models this year.

s90
00:09:56.017 --> 00:09:59.737
Um I we are very happy.

s91
00:09:59.737 --> 00:10:06.177
I think the correct word would be that we're we're stunned by the fact of what these models can do.

s92
00:10:06.177 --> 00:10:09.377
Um it is I Hi.

s93
00:10:09.377 --> 00:10:12.431
This is a live stream, so I unfortunately couldn't share the

s94
00:10:12.431 --> 00:10:12.658
[laughter]

s95
00:10:12.658 --> 00:10:17.138
release videos without breaking the surprise, but if you're in New York, we are also in New York.

s96
00:10:17.138 --> 00:10:20.738
And so, we have started doing a few demos of the models.

s97
00:10:20.738 --> 00:10:26.938
So, if you can know somebody on the team, reach out and we can start showing you some of the some of the capabilities live.

s98
00:10:26.938 --> 00:10:36.778
Um, but we have essentially one model, which can play video games to highest level and control over 10 types of robots using the exact same primitives.

s99
00:10:36.778 --> 00:10:39.698
So, pixels in, actions out completely real time.

s100
00:10:39.698 --> 00:10:45.098
Um, and we'll be rolling out early access to these models over next few months.

s101
00:10:45.098 --> 00:10:50.578
We already have some customers that are on them, but it's still private, so you're not hearing too much about it.

s102
00:10:50.578 --> 00:11:00.218
But, if you scan this QR code, you'll be able to get the models because we go by who signs up on the wait list at which times.

s103
00:11:00.218 --> 00:11:03.978
So, um, this is a great this is a great moment to get in touch.

s104
00:11:03.978 --> 00:11:05.098
As I mentioned, we are in New York.

s105
00:11:05.098 --> 00:11:15.698
I think what you'll find is similar to how uh, Jeff and a lot of these decision-making models have kind of opened people's eyes on what you can do when you have a new architectural primitive.

s106
00:11:15.698 --> 00:11:26.618
People are ready to kind of do away with the unpredictability of an LLM and put something in that is more reliable and faster and you just understand how it's going to react as opposed to having to work around it.

s107
00:11:26.618 --> 00:11:30.658
If that resonates, these models are exactly what you're looking for.

s108
00:11:30.658 --> 00:11:37.658
And all of the sudden, um, you can start exploring the environments that you generate with LLMs.

s109
00:11:37.658 --> 00:11:47.218
All of the sudden, one of the use cases that I'm really really excited about is well, these models, they can take actions as if a human was on the interface.

s110
00:11:47.218 --> 00:11:52.298
And so, if you're doing UX work and you're trying to understand, is this an intuitive interface?

s111
00:11:52.298 --> 00:11:54.738
Well, this is a very interesting way of solving the problem.

s112
00:11:54.738 --> 00:12:03.258
So, all these problems that LLMs can't solve because they can't actually explore like a human, you can use these problems to start you can start using these models for.

s113
00:12:03.258 --> 00:12:10.978
So, um, the way that they're going to be released is we still expect you to use an orchestrator model.

s114
00:12:10.978 --> 00:12:18.739
So, you may use something like Gemini or Astra in order to do long-term planning and and everything, and you'll be able to plug these into your workloads.

s115
00:12:18.739 --> 00:12:32.459
Um and we're actually the the models are so general, it it it is genuinely going to, I think, feel like the early LLM days when you have a new toy and you just want to figure out what it does and all the things you can use it for.

s116
00:12:32.459 --> 00:12:40.619
I my reaction was like that when when I when I first started using the checkpoints, and I think I think you'll have the same reaction.

s117
00:12:40.619 --> 00:12:53.299
Um and I think a lot of you will, just like with LLMs, if you're a bit early and you start understanding these primitives, I think you'll understand uh that you can even build lots of businesses on top of this, right?

s118
00:12:53.299 --> 00:12:58.659
You know, think about for instance, um a a sci- a science lab, right?

s119
00:12:58.659 --> 00:13:08.899
If you all of the sudden, if you can reliably orchestrate robots in space and time, and they can take text instructions and do things, all of the sudden, um

s120
00:13:08.899 --> 00:13:11.339
I this is a robotics problem, right?

s121
00:13:11.339 --> 00:13:15.859
Robotics is interesting because one of the big problems with solving robotics is making more robots.

s122
00:13:15.859 --> 00:13:26.259
And so, there are all these businesses that because it's so bottlenecked by intelligence, are going to emerge just from solving this one simple problem, which is pixels to actions.

s123
00:13:26.259 --> 00:13:35.259
And I think if you're early on getting your hands on on these, it will have a similar effect on getting your your hands on early LLMs.

s124
00:13:35.259 --> 00:13:46.739
Um I am most excited also uh one of the things that that we're working on, which is interesting, is if you have any friends or family who uh are

s125
00:13:46.739 --> 00:13:55.619
um in some way um uh need accessibility help for using their computer, because these models can use computers like a human, all of the sudden, for instance, if you're if you're

s126
00:13:55.619 --> 00:14:07.419
uh if you only can use one arm, um all of the sudden, the model can allow you to play video games because you can you can use you can just control the mouse, and the model is predicting the actions for your keyboard.

s127
00:14:07.419 --> 00:14:12.059
And because the model also understands which actions you're predicting, because that's the input, right?

s128
00:14:12.059 --> 00:14:18.781
It's it's observing as it's acting, it will actually just not predict the things that you're doing and it will just predict the keys.

s129
00:14:18.781 --> 00:14:24.741
And so, it has all these downstream effects of things that you can build on top that I'm an engineer.

s130
00:14:24.741 --> 00:14:30.541
If you're interested in building in this space, uh this is going to be the most interesting toy that comes out this year.

s131
00:14:30.541 --> 00:14:34.061
I I I'm fairly certain.

s132
00:14:34.061 --> 00:14:37.421
Um and I also think a lot of the initial usage will sit in simulation.

s133
00:14:37.421 --> 00:14:45.341
So, if if a lot of you are into generating environments with Astra or if Gemini is perfect, now all of a sudden you can have something that can explore it and you don't have to do it yourself.

s134
00:14:45.341 --> 00:14:53.381
Um and I it's perfect timing because of all the environment generation capabilities going going into these models.

s135
00:14:53.381 --> 00:14:54.901
Um Okay.

s136
00:14:54.901 --> 00:14:58.741
So, really quickly before I wrap up, um I want to talk about the team.

s137
00:14:58.741 --> 00:15:01.221
Um so, we are based in New York City.

s138
00:15:01.221 --> 00:15:04.861
I the metal team and the general intuition team is here in New York City.

s139
00:15:04.861 --> 00:15:12.181
So, if you're here, you should reach out to us and I I we can both show you the models, uh especially if you want to join the team.

s140
00:15:12.181 --> 00:15:14.581
I think we'll we'll prioritize even higher.

s141
00:15:14.581 --> 00:15:15.045
Uh

s142
00:15:15.045 --> 00:15:15.421
[laughter]

s143
00:15:15.421 --> 00:15:24.181
but, um I we really are looking for um people who A want to understand these models as much as we do.

s144
00:15:24.181 --> 00:15:32.261
So, people if you're really really good at generating novel environments, uh typically if you have some sort of research uh background, but I don't have a research background, so that's okay.

s145
00:15:32.261 --> 00:15:38.181
If you you know, if you um if you come from an engineering background and you're very very happy to learn, this is this is fine, too.

s146
00:15:38.181 --> 00:15:49.501
Um and ideally what we want is somebody who um uh has experienced shipping uh end-to-end uh AI uh products, but it doesn't have to be on the research side.

s147
00:15:49.501 --> 00:15:53.021
So, we just want to kind of hire people that understand the entire thing.

s148
00:15:53.021 --> 00:15:55.861
Um Let's see.

s149
00:15:55.861 --> 00:16:05.581
I We have three teams, so one is world modeling, the other is the policy, so the agent, and the other one is working on um essentially the the data generation

s150
00:16:05.581 --> 00:16:06.621
and the environment generation.

s151
00:16:06.621 --> 00:16:07.381
So, these are the main teams.

s152
00:16:07.381 --> 00:16:11.061
If you're an engineer, the best one to join is probably the environment generation team.

s153
00:16:11.061 --> 00:16:14.821
Um and then you sort of grow into uh getting access later on.

s154
00:16:14.821 --> 00:16:18.702
Um yeah, we I I'm very very excited for November.

s155
00:16:18.702 --> 00:16:22.902
I really hope that I you'll be on the lookout.

s156
00:16:22.902 --> 00:16:26.822
Um as I mentioned, it will be either late November early December is the current planning.

s157
00:16:26.822 --> 00:16:32.062
Um one more thing, uh we are going to organize three hackathons.

s158
00:16:32.062 --> 00:16:35.382
Um so the first is going to be in New York City, it will be mid-November.

s159
00:16:35.382 --> 00:16:43.902
Um and uh we were we will send out um if you sign up for the partner waitlist, we'll send out some emails so that everybody can get access.

s160
00:16:43.902 --> 00:16:46.182
Uh we hope to see everybody at the hackathons.

s161
00:16:46.182 --> 00:16:50.862
I think it will be really the first time to get your hands on them the way that we're playing with them internally now.

s162
00:16:50.862 --> 00:16:53.982
And uh there will be one in New York City and one in London.

s163
00:16:53.982 --> 00:17:00.542
So if you're in either of the cities or want to travel to either of those cities, uh those would be a good uh times to meet the team.

s164
00:17:00.542 --> 00:17:03.102
Um yeah, let's see.

s165
00:17:03.102 --> 00:17:08.422
Um I have time for one or two questions and then uh I think we can we can wrap.

s166
00:17:08.422 --> 00:17:12.862
Uh let's see.

s167
00:17:16.742 --> 00:17:18.822
How is it different from Jeppa?

s168
00:17:18.822 --> 00:17:20.422
Uh yeah, how's it different from Jeppa?

s169
00:17:20.422 --> 00:17:32.422
So Jeppa is uh doesn't train alongside the action labels, so they bet on infrared action labels for the most part, which I think like one of the ways of thinking about it is Jeppa I think in the limit is the more correct approach, but it's just a lot less practical in the short term because you need to

s170
00:17:32.422 --> 00:17:36.102
bet on a separate scaling law on the action inference itself.

s171
00:17:36.102 --> 00:17:40.782
Um and I we really like Jeppa as a concept.

s172
00:17:40.782 --> 00:17:44.542
We think it's a uh uh you know, we we use some of the things in the models.

s173
00:17:44.542 --> 00:17:50.142
Um and uh but at the end of day it's it's it's an approach, not a product.

s174
00:17:50.142 --> 00:17:55.662
Um and I think we'll, you know, borrow concepts both ways.

s175
00:17:55.662 --> 00:18:06.182
Um the one way of thinking about it is um because we have so much action data, let's say you're flying a plane and you're moving the rudder in the back.

s176
00:18:06.182 --> 00:18:10.062
There's no way of being able to tell that from the video frames.

s177
00:18:10.062 --> 00:18:11.342
It's it's very very difficult.

s178
00:18:11.342 --> 00:18:20.582
So our mode on having the action data really transfers better to action sequences than it does to world models and to to generating the pixels.

s179
00:18:20.582 --> 00:18:24.823
And that's why we just went all in on training these general foundation models for action predictions.

s180
00:18:24.823 --> 00:18:27.343
And we use we use all sorts of techniques to do world models.

s181
00:18:27.343 --> 00:18:31.903
You know, we our own, we we really like other companies building them as well.

s182
00:18:31.903 --> 00:18:35.183
Um So.

s183
00:18:35.183 --> 00:18:35.703
Great.

s184
00:18:35.703 --> 00:18:37.943
Then one more and then I got to run, I think.

s185
00:18:37.943 --> 00:18:38.863
Great.

s186
00:18:38.863 --> 00:18:41.983
Um Great.

s187
00:18:41.983 --> 00:18:42.543
All right.

s188
00:18:42.543 --> 00:18:43.743
Thank you everyone.

s189
00:18:43.743 --> 00:18:44.647
Have a good one.

s190
00:18:44.647 --> 00:18:46.647
[applause]