Robots Will Be Trained on Video Games - Pim de Witte, General Intuition

AI Engineer · 18 min · 190 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hello.
00:13Um I want you to uh zoom out for a second and think about if somebody were to tell you today that we would take all the information on the internet
00:25and then we would train a model on it and we would use that in order to solve scientific problems that were not on the internet.
00:31I think a lot of people would tell you that that's crazy.
00:34Um and uh what we're doing um has a lot of parallels to that story.
00:41Um and so if you think about how intelligence works imagine for a second that you're in the movie Frozen and you're Olaf, the the snowman.
00:51Um if if Olaf was an LLM and it there there's a scene which is one of my favorites which is where he rolls down the mountain and there's a little stone that come that pops up and Olaf
01:03spreads out temporarily becomes a disassembled snowman and then becomes an assembled snowman and Olaf knows how to do that because Olaf can observe as he is acting.
01:14Olaf can, unlike LLMs which auto aggressively predict tokens, um constantly part of that behavior is that it's constantly observing as it is acting.
01:26And so um this is the big difference between you know, when you see LLMs try to solve robotics problems or when you look at native architectures like what we're working on,
01:36um it's very very easy to think of LLMs in the demos that you see on Twitter as the thing that will solve everything.
01:42There's fundamental limitations that we see uh where purpose-built models are much better fit.
01:47Now today I'm going to talk a little bit about what the other side of that coin looks like.
01:52Um and so for us it all started with Medal.
01:56We built the largest video game recorder in the world um and we think of that as the internet.
02:03So, people use Medal to record themselves playing tens of thousands of different video games.
02:08And if you think about it, uh, as a human, we can teleoperate a robot with the exact same brain that we played that video game.
02:18And in fact, um, in many cases, we also can use the same controller.
02:27So, what this means is that the action space of video games, often because of game controllers, and the action space of teleoperations or the real world, in many cases, is the same.
02:35Um, and so, every single robot built below, um, certain degrees of freedom ships with the game controller.
02:41A majority of the supply chain is already worked this way.
02:44And so, we thought, what if you just take all of the action data that we get through the video recorder, um, which associates the inputs and the actions with the actual frames.
02:54We label the actions, so we under- understand semantically meaning behind every action, and we use it to give it to a vision-based model.
03:02And so, what you see here on the left is just an intuitive way of how the model works.
03:07So, you give the model a set of keys to choose from, essentially a computer use model.
03:12It It It gives frames as the input, so you can give it a text prompt as well.
03:16And you send it through API, and you say, "Somebody waves at you, wave back."
03:21In the input space, model understands that you have a greet action available, it will output the action associated with that.
03:28Um, one way of intu- of intuitively understanding how this works.
03:31So, um, I've obviously simplified here because it's predicting next token, not the next word, but bear with me here.
03:38Um, LLMs took large-scale internet data, then trained it to predict the next text token, and now use it for writing software.
03:47Um, and these are all workloads that are really, really well fit for futures that are not simultaneously unfolding as you are predicting, right?
03:55Code is turn-based.
03:57You can say something, the model will do something, you can say something again, the model will do something me again.
04:02And um you can use it for lots of valuable work, but at the end of the day, because it's turn-based, um it's very, very difficult for it to generalize
04:12to environments, especially that include um space and time.
04:15And so, our models, the way that they work is they're continuously receiving frames or receiving batches of essentially um information from the camera of the robot or it could be a a screen.
04:29And it predicts everything as as if it were just a video game.
04:34So, you can think of the way that Anthropic focused on just nailing coding.
04:39So, you know, they were never frontier in just base text generation.
04:43They always focused on being the best at generating code.
04:46And then what the debate that they made is that more people would start bringing more problems into coding space.
04:51If you think of general intuition, our bet is that you can as humans, we are generally pixel-to-action converters plus sensors, which in many cases you can convert into pixels as well.
05:03And you can output actions at the same time.
05:06And those actions are in this case either keyboard and mouse or a game controller.
05:09So, the output tokens of the model are um essentially equivalent to that of uh enhanced computer use model with more more more actions on the output space to internalize it.
05:18So, you send frames and you get actions.
05:21This works again because every single thing that you see here can control um all of these different embodiments, right?
05:28So, um as humans, we do not know how to predict motor torque.
05:37We do not We do not know what value to send to a machine.
05:41Um there's this beautiful device that has been made for many years, uh which actually does that work for us, so we can control them.
05:49Most robotics companies today are trying to solve the problem at the end-to-end level, trying to predict exact poses or exact um values uh in the motor.
05:59I instead of just looking and picking up the thing that every robotics company has built to generalize our very limited abilities to have physics understanding and predictive text values
06:16and solve for it.
06:17And so, our approach is let's just use that.
06:20Um and so Metal at the moment sees about a billion videos uploaded per year.
06:28This year on we're on track for 3 billion.
06:31Um which lead to over a trillion and a half action tokens per month.
06:36So, just for comparison, think about the bet.
06:39The total amount of data that Frontier Labs used today to train robotics models, we collect in at the moment half a day.
06:46Um and so I the the type of bet and again, think of the fact that humans can use the exact same controllers and their eyes to play video games that they can teleoperate robots.
06:59It is the same thing.
07:00We just bet on scaling pixel space the same way that Anthropic scaled coding space and betting that people will just as we scale data compute into pixel space that people will bring more problems into pixel space.
07:12[snorts]
07:14And so, what happens?
07:16Um All of the sudden, you get an incredibly general agent that can go and solve problems on computers, in the real world, um and in addition, similarly to how Anthropic focuses on doing coding environments,
07:36we focus on world models, which most of you know as heard of Mira release world models, to then generate additional environments for these models to learn in.
07:44One way of internalizing a big advantage of this approach against LLMs is if you're an if you're an LLM company and you have to bet on the ability to generate environments for your models to learn, you have you have two problems.
07:58One, you actually need to spend the compute to simulate environments and the more realistic you try to simulate, the more expensive that gets.
08:04Um and so, imagine you have you want to learn a new thing.
08:08You want to learn that new thing in the exact input space of the human, right?
08:14So, you if if we are learning from pixels, we're we're learning by looking around.
08:18If I'm trying to communicate something, I want my robot partner to observe that exact same thing.
08:23And so, um what what I don't want is for data to go through an LLM, for a digital twin to have to be created, for the model to have to get deployed,
08:33for there to be a bunch of verifier checks whether or not things are equivalent, and then for the robot to to learn much later, right?
08:39And so, one way of thinking about it is when you can learn in the native modality of the human, you can all of the sudden start solving a lot more problems a lot faster.
08:48You what you'll see when these models come out is they're significantly faster than any LLM um at doing these things.
08:57We because games are so action-dense.
09:01So, what you see, right, is um one thing that's interesting is uh you saw, for instance, the model predicting actions at the same time.
09:08Um we're able to uh take that action density from gamers and convert them into really complex robotics workloads.
09:15So, for example, if you are operating a drone and you need to make lots of little adjustments, you really really really want to start with games.
09:23You don't want to start with uh teleoperations because the human will just take a lot less actions.
09:27It has a lot less information.
09:28So, you want a model that can predict really high APM.
09:31Um And so, the APM of the data set puts a lot of these problems all of the sudden in reach so that it can do a lot of these actions simultaneously.
09:40And as I mentioned, the model doesn't just understand the key it's predicting.
09:44It also understands the semantic meaning associated with that key it's predicting, which is really important for things like um interpretability.
09:50I So, I we will be releasing the models this year.
09:56Um I we are very happy.
09:59I think the correct word would be that we're we're stunned by the fact of what these models can do.
10:06Um it is I Hi.
10:09This is a live stream, so I unfortunately couldn't share the
10:12[laughter]
10:12release videos without breaking the surprise, but if you're in New York, we are also in New York.
10:17And so, we have started doing a few demos of the models.
10:20So, if you can know somebody on the team, reach out and we can start showing you some of the some of the capabilities live.
10:26Um, but we have essentially one model, which can play video games to highest level and control over 10 types of robots using the exact same primitives.
10:36So, pixels in, actions out completely real time.
10:39Um, and we'll be rolling out early access to these models over next few months.
10:45We already have some customers that are on them, but it's still private, so you're not hearing too much about it.
10:50But, if you scan this QR code, you'll be able to get the models because we go by who signs up on the wait list at which times.
11:00So, um, this is a great this is a great moment to get in touch.
11:03As I mentioned, we are in New York.
11:05I think what you'll find is similar to how uh, Jeff and a lot of these decision-making models have kind of opened people's eyes on what you can do when you have a new architectural primitive.
11:15People are ready to kind of do away with the unpredictability of an LLM and put something in that is more reliable and faster and you just understand how it's going to react as opposed to having to work around it.
11:26If that resonates, these models are exactly what you're looking for.
11:30And all of the sudden, um, you can start exploring the environments that you generate with LLMs.
11:37All of the sudden, one of the use cases that I'm really really excited about is well, these models, they can take actions as if a human was on the interface.
11:47And so, if you're doing UX work and you're trying to understand, is this an intuitive interface?
11:52Well, this is a very interesting way of solving the problem.
11:54So, all these problems that LLMs can't solve because they can't actually explore like a human, you can use these problems to start you can start using these models for.
12:03So, um, the way that they're going to be released is we still expect you to use an orchestrator model.
12:10So, you may use something like Gemini or Astra in order to do long-term planning and and everything, and you'll be able to plug these into your workloads.
12:18Um and we're actually the the models are so general, it it it is genuinely going to, I think, feel like the early LLM days when you have a new toy and you just want to figure out what it does and all the things you can use it for.
12:32I my reaction was like that when when I when I first started using the checkpoints, and I think I think you'll have the same reaction.
12:40Um and I think a lot of you will, just like with LLMs, if you're a bit early and you start understanding these primitives, I think you'll understand uh that you can even build lots of businesses on top of this, right?
12:53You know, think about for instance, um a a sci- a science lab, right?
12:58If you all of the sudden, if you can reliably orchestrate robots in space and time, and they can take text instructions and do things, all of the sudden, um
13:08I this is a robotics problem, right?
13:11Robotics is interesting because one of the big problems with solving robotics is making more robots.
13:15And so, there are all these businesses that because it's so bottlenecked by intelligence, are going to emerge just from solving this one simple problem, which is pixels to actions.
13:26And I think if you're early on getting your hands on on these, it will have a similar effect on getting your your hands on early LLMs.
13:35Um I am most excited also uh one of the things that that we're working on, which is interesting, is if you have any friends or family who uh are
13:46um in some way um uh need accessibility help for using their computer, because these models can use computers like a human, all of the sudden, for instance, if you're if you're
13:55uh if you only can use one arm, um all of the sudden, the model can allow you to play video games because you can you can use you can just control the mouse, and the model is predicting the actions for your keyboard.
14:07And because the model also understands which actions you're predicting, because that's the input, right?
14:12It's it's observing as it's acting, it will actually just not predict the things that you're doing and it will just predict the keys.
14:18And so, it has all these downstream effects of things that you can build on top that I'm an engineer.
14:24If you're interested in building in this space, uh this is going to be the most interesting toy that comes out this year.
14:30I I I'm fairly certain.
14:34Um and I also think a lot of the initial usage will sit in simulation.
14:37So, if if a lot of you are into generating environments with Astra or if Gemini is perfect, now all of a sudden you can have something that can explore it and you don't have to do it yourself.
14:45Um and I it's perfect timing because of all the environment generation capabilities going going into these models.
14:53Um Okay.
14:54So, really quickly before I wrap up, um I want to talk about the team.
14:58Um so, we are based in New York City.
15:01I the metal team and the general intuition team is here in New York City.
15:04So, if you're here, you should reach out to us and I I we can both show you the models, uh especially if you want to join the team.
15:12I think we'll we'll prioritize even higher.
15:14Uh
15:15[laughter]
15:15but, um I we really are looking for um people who A want to understand these models as much as we do.
15:24So, people if you're really really good at generating novel environments, uh typically if you have some sort of research uh background, but I don't have a research background, so that's okay.
15:32If you you know, if you um if you come from an engineering background and you're very very happy to learn, this is this is fine, too.
15:38Um and ideally what we want is somebody who um uh has experienced shipping uh end-to-end uh AI uh products, but it doesn't have to be on the research side.
15:49So, we just want to kind of hire people that understand the entire thing.
15:53Um Let's see.
15:55I We have three teams, so one is world modeling, the other is the policy, so the agent, and the other one is working on um essentially the the data generation
16:05and the environment generation.
16:06So, these are the main teams.
16:07If you're an engineer, the best one to join is probably the environment generation team.
16:11Um and then you sort of grow into uh getting access later on.
16:14Um yeah, we I I'm very very excited for November.
16:18I really hope that I you'll be on the lookout.
16:22Um as I mentioned, it will be either late November early December is the current planning.
16:26Um one more thing, uh we are going to organize three hackathons.
16:32Um so the first is going to be in New York City, it will be mid-November.
16:35Um and uh we were we will send out um if you sign up for the partner waitlist, we'll send out some emails so that everybody can get access.
16:43Uh we hope to see everybody at the hackathons.
16:46I think it will be really the first time to get your hands on them the way that we're playing with them internally now.
16:50And uh there will be one in New York City and one in London.
16:53So if you're in either of the cities or want to travel to either of those cities, uh those would be a good uh times to meet the team.
17:00Um yeah, let's see.
17:03Um I have time for one or two questions and then uh I think we can we can wrap.
17:08Uh let's see.
17:16How is it different from Jeppa?
17:18Uh yeah, how's it different from Jeppa?
17:20So Jeppa is uh doesn't train alongside the action labels, so they bet on infrared action labels for the most part, which I think like one of the ways of thinking about it is Jeppa I think in the limit is the more correct approach, but it's just a lot less practical in the short term because you need to
17:32bet on a separate scaling law on the action inference itself.
17:36Um and I we really like Jeppa as a concept.
17:40We think it's a uh uh you know, we we use some of the things in the models.
17:44Um and uh but at the end of day it's it's it's an approach, not a product.
17:50Um and I think we'll, you know, borrow concepts both ways.
17:55Um the one way of thinking about it is um because we have so much action data, let's say you're flying a plane and you're moving the rudder in the back.
18:06There's no way of being able to tell that from the video frames.
18:10It's it's very very difficult.
18:11So our mode on having the action data really transfers better to action sequences than it does to world models and to to generating the pixels.
18:20And that's why we just went all in on training these general foundation models for action predictions.
18:24And we use we use all sorts of techniques to do world models.
18:27You know, we our own, we we really like other companies building them as well.
18:31Um So.
18:35Great.
18:35Then one more and then I got to run, I think.
18:37Great.
18:38Um Great.
18:41All right.
18:42Thank you everyone.
18:43Have a good one.
18:44[applause]