WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=jFHteJjRl8A NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/jFHteJjRl8A.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.559 --> 00:00:17.600 I think uh all of you know here that how much progress has AI made in the last five years. s3 00:00:17.600 --> 00:00:22.080 Uh seems like it's more than uh last 100 years of progress. s4 00:00:22.080 --> 00:00:36.800 And what is happening especially in AI is AI came for language then speech audio video and everybody's excitement is next is robotics okay and this excitement has is going on peak uh this year uh s5 00:00:36.800 --> 00:00:46.559 for some reason which is still beyond my uh understanding and you can also see uh leaders like Jensen talking about physical AI as the next frontier of robotics. s6 00:00:46.559 --> 00:00:58.719 So it seems like if you open Twitter or LinkedIn, it seems like robots robotics is already here and we'll have home robots in our homes sometime by December uh as uh as it's being said by multiple people. s7 00:00:58.719 --> 00:01:06.155 But I'm here to highlight that robotics has been almost here for the last 70 years s8 00:01:06.155 --> 00:01:06.799 [sighs] s9 00:01:06.799 --> 00:01:08.000 now. s10 00:01:08.000 --> 00:01:20.640 Those of you who got into robotics in the last four or five years, go and take any any any course in robotics and if you can distinguish the videos in robotics from 30 years ago versus today, s11 00:01:20.640 --> 00:01:23.360 uh I owe you a dinner. s12 00:01:23.360 --> 00:01:26.000 So to give you an example, let's look at this robot. s13 00:01:26.000 --> 00:01:37.600 So this robot here is an image and the goal of this robot is to look at this image of blocks and arrange these blocks in front of it so that it looks the same pattern as in the image. s14 00:01:37.600 --> 00:01:40.079 Okay, this is an extremely simple task. s15 00:01:40.079 --> 00:01:41.920 But if you think about this, it's a 3D block. s16 00:01:41.920 --> 00:01:44.159 You have to look at understand 3D from each side. s17 00:01:44.159 --> 00:01:47.360 And the robot can do this pretty precisely. s18 00:01:47.360 --> 00:01:51.040 Any guesses how old this result is? s19 00:01:51.280 --> 00:01:51.680 Anyone? s20 00:01:51.680 --> 00:01:52.159 It's a guess. s21 00:01:52.159 --> 00:01:53.520 You can make any guess. s22 00:01:53.520 --> 00:01:54.560 Huh? s23 00:01:54.560 --> 00:01:55.280 40 years. s24 00:01:55.280 --> 00:01:57.901 Okay, that's the highest. s25 00:01:57.901 --> 00:01:58.960 [panting] s26 00:01:58.960 --> 00:02:01.040 This is from 1960s. s27 00:02:01.040 --> 00:02:03.200 This is way before there were computers. s28 00:02:03.200 --> 00:02:04.399 Uh, okay. s29 00:02:04.399 --> 00:02:06.640 So, this is this is called MIT copy demo. s30 00:02:06.640 --> 00:02:10.879 Uh, this was the start of uh this is this predates AI. s31 00:02:10.879 --> 00:02:15.040 Uh, as you know, today look at this one. s32 00:02:17.599 --> 00:02:22.000 One exhibit at the Nuclear Congress in Philadelphia has the perfect formula for s33 00:02:22.000 --> 00:02:23.120 what's happening here. s34 00:02:23.120 --> 00:02:27.920 The guy behind the scene is controlling these robots, a leader follower system. s35 00:02:27.920 --> 00:02:38.480 And if you look at any big lab, any big company, any big academic lab, how they get data, they use teley operation system, which follows the exact same principles. s36 00:02:38.480 --> 00:02:41.920 Any guesses for the year for this one? s37 00:02:42.480 --> 00:02:49.920 Now, everybody would correct aggressively on the year, but this is from 1957, 68 years ago. s38 00:02:49.920 --> 00:02:51.120 Okay? s39 00:02:51.120 --> 00:03:03.599 Ask anybody in your family who is older than actually they may not be around like so because this is for 60 you have to ask somebody who is 80 year old what was the state of technology at that time like RAM s40 00:03:03.599 --> 00:03:11.440 for few KB of RAM you would have a size of a room uh like this big of a setup and this is from that time when people could make it work s41 00:03:11.440 --> 00:03:29.680 on not just this you keep coming back every decade the videos look as cool like here is robot a humanoid juggling playing foosball uh with human and this is also from you can see this is all they were all predate deep learning this from 8 years ago so this robot is a result from Berkeley can clean up s42 00:03:29.680 --> 00:03:38.560 the whole table can arrange items in a in a in a box this was done by one grad student with a single GPU machine nothing more than that okay s43 00:03:38.560 --> 00:03:47.287 now this picture of and if I colorize all these videos from past and apply genai filter for the modern voice you cannot tell that this s44 00:03:47.287 --> 00:03:47.519 [laughter] s45 00:03:47.519 --> 00:03:54.319 video is from 19657 or this is from today And these are not even the oldest results the oldest ones go to 1940s. s46 00:03:54.319 --> 00:03:55.120 Okay. s47 00:03:55.120 --> 00:04:08.000 So what is this that if you look at last 70 years tech every other technology every other technology has come way far language understanding computer vision mobile phone chips s48 00:04:08.000 --> 00:04:14.879 why is robotics is stuck in this primitive age uh for for this long and the reason is uh a general brain. s49 00:04:14.879 --> 00:04:25.520 Robotics has always been approached as a hardware problem from ground up and this the reason that everything around robotics has progressed but robotics is still stuck in the same land from the last 70 years. s50 00:04:25.520 --> 00:04:28.320 So there's a very famous paradox called Morave paradox. s51 00:04:28.320 --> 00:04:29.600 I don't know if you know about Moravec. s52 00:04:29.600 --> 00:04:34.240 He was also a CMU professor and I'm also CMU professor so have to quote him for sure. s53 00:04:34.240 --> 00:04:36.880 He was one of the founding figures in in AI. s54 00:04:36.880 --> 00:04:41.360 After 30 years of working in robotics he arrived at a very simple conclusion. s55 00:04:41.360 --> 00:04:43.600 Hard is easy, easy is hard. s56 00:04:43.600 --> 00:04:44.080 Okay. s57 00:04:44.080 --> 00:04:48.960 Whatever human believe to be hard is extremely easy for computers and vice versa. s58 00:04:48.960 --> 00:04:51.520 And you can see it happening right in front of your eyes. s59 00:04:51.520 --> 00:04:53.759 You would say doing math is hard. s60 00:04:53.759 --> 00:04:56.560 Math Olympiad gold medal is really really hard. s61 00:04:56.560 --> 00:04:59.040 Climbing a stair oh super easy. s62 00:04:59.040 --> 00:05:04.080 Now look around you in technology where we are in terms of what has been solved and what's not being solved. s63 00:05:04.080 --> 00:05:05.520 So this is robotics for you. s64 00:05:05.520 --> 00:05:05.759 Okay. s65 00:05:05.759 --> 00:05:09.440 It's very it is not yet another application of AI. s66 00:05:09.440 --> 00:05:14.080 It is what AI was founded for in the very beginning and has made very little progress towards. s67 00:05:14.080 --> 00:05:18.320 This is not yet another application where deep networks can come in and attack the field. s68 00:05:18.320 --> 00:05:23.600 This is has to be thought of with fundamentally first principles from the ground up. s69 00:05:23.600 --> 00:05:38.080 This is a very famous example of where you can have a you know a computer beat uh Gary Casper on chess in '90s but you can you still don't have a a a computer or robot that can pick up the chess pieces and arrange them on any chess board. s70 00:05:38.080 --> 00:05:41.840 Now, so how do we go about solving this? s71 00:05:41.840 --> 00:05:49.440 Okay, so on a more positive note, uh we have a magic sauce for AI success, right? s72 00:05:49.440 --> 00:05:53.520 You get big data set, you train big models and magic happens. s73 00:05:53.520 --> 00:05:59.199 Okay, now we know this template is working well very well in many topics. s74 00:05:59.199 --> 00:06:02.720 Okay, now can we apply the same recipe to robotics? s75 00:06:02.720 --> 00:06:07.039 Well, it's not uh directly applicable because we have no data. s76 00:06:07.039 --> 00:06:22.639 There is no internet of robotics data and as I said earlier you can go and collect data uh manually on the robot called telly operation and as you notice teley operation is not 5 year old thing this is a 68 or 70 year old thing so s77 00:06:22.639 --> 00:06:33.280 the the funny part is if you go back and you look at the whole evolution of GPD3 GPD4 models go back to GPD3 uh three years ago seems like a like a lifetime ago s78 00:06:33.280 --> 00:06:40.160 GPD3 only began working and caught people's attention when you could train those models on trillions of tokens. s79 00:06:40.160 --> 00:06:49.759 So GPD3 was already trained 30 trillion plus token and today hundreds of trillions of tokens and if you collect data by manually by tell operation it takes you about 1 minute to get one example. s80 00:06:49.759 --> 00:07:00.720 You can do the math if you hire all of US population it will take you more than a century to reach the same scale as of GPD3 uh which is like uh and nobody uses GP3 today. s81 00:07:00.720 --> 00:07:01.039 Okay. s82 00:07:01.039 --> 00:07:02.639 So this is very slow and expensive. s83 00:07:02.639 --> 00:07:14.080 So in robotics I would argue nobody has really scaled robotics yet and we are very far from talking about scale in robotics with the way other uh other areas have seen scale. s84 00:07:14.080 --> 00:07:22.800 Okay so this is where uh we have been focusing on uh our scaled is about uh 3 years old but uh I have been working on the problem for more than a decade s85 00:07:22.800 --> 00:07:25.520 uh that's all I've done in my career uh nothing else. s86 00:07:25.520 --> 00:07:42.479 Uh so at scale what our thesis is is to build what we call an omniodied intelligence any robot any task one brain okay any robot it can be a humanoid it can be a quadriped it can be a robotic arm on a conveyor belt or a dextrous hand doesn't really matter s87 00:07:42.479 --> 00:07:54.800 this even this hypothesis is way more general than uh one would argue humans are because we control our own body and why do we have to go so general well the argument is in robotics s88 00:07:54.800 --> 00:07:56.080 there there is no data anyway. s89 00:07:56.080 --> 00:07:59.840 So I can't pick and choose which hardware do I use data from. s90 00:07:59.840 --> 00:08:04.720 And we should be able to use data from any kind of hardware, any kind of task, any kind of scenario. s91 00:08:04.720 --> 00:08:10.560 And that's the only way to truly achieve the scale of uh of what language models achieved uh three years ago. s92 00:08:10.560 --> 00:08:10.800 Okay. s93 00:08:10.800 --> 00:08:15.120 So the goal here is this uh this whole idea of any robot any task one brain. s94 00:08:15.120 --> 00:08:20.160 And through this talk I'll hopefully convince you why this is the this is the way to go towards robotics. s95 00:08:20.160 --> 00:08:20.720 Okay. s96 00:08:20.720 --> 00:08:28.240 So this is the uh rough intro but before I go into the more details let me show you just a teaser result. s97 00:08:28.800 --> 00:08:42.240 So in this result, every single robot uh it's from a different company, different hardware and they're all controlled by uh skilled brain whether it's humanoids going up and down stairs, any kind of scenarios. s98 00:08:42.240 --> 00:08:43.760 And these are not new results. s99 00:08:43.760 --> 00:08:47.920 They're like a couple of year old uh results in here. s100 00:08:47.920 --> 00:08:50.720 Robust to dust disturbances. s101 00:08:51.200 --> 00:08:54.720 You can put them zero shot in new scenarios. s102 00:09:02.959 --> 00:09:12.180 So the idea here any every single robot in this video takes any action anywhere in the world the brain behind the scene s103 00:09:12.180 --> 00:09:12.480 [music] s104 00:09:12.480 --> 00:09:15.360 improves because it's an omni-body brain. s105 00:09:15.360 --> 00:09:19.760 Yeah, this is really hard right because like eggs are okay. s106 00:09:32.080 --> 00:09:34.560 So that's a teaser of what we what we work on. s107 00:09:34.560 --> 00:09:39.600 So I want to make this talk more scientific and more uh more informative than a company ad. s108 00:09:39.600 --> 00:09:42.480 So I'll talk about uh how do we scale data in robotics. s109 00:09:42.480 --> 00:09:46.320 Okay, let's take a tour back as to how have people addressed this. s110 00:09:46.320 --> 00:09:52.080 uh I'll take a look back at my own career and my hypothesis for data has been changing over the years. s111 00:09:52.080 --> 00:09:52.480 Okay. s112 00:09:52.480 --> 00:10:03.600 When I began uh uh working scaling things the idea was you can have robots you can collect data for robots manually but it is too slow and robots should be allowed to collect data by themselves. s113 00:10:03.600 --> 00:10:06.560 So we had this whole idea of curiositydriven exploration. s114 00:10:06.560 --> 00:10:13.600 Uh allow robots to explore in a curious way in the environment and it has a lot of good parts like no human required. s115 00:10:13.600 --> 00:10:15.839 Robots can go and keep playing collect more data. s116 00:10:15.839 --> 00:10:17.839 We called it play data at the time. s117 00:10:17.839 --> 00:10:21.519 It's very rich because it has force sensors and joint angles. s118 00:10:21.519 --> 00:10:24.880 But the difficulty is it is only in the physical world. s119 00:10:24.880 --> 00:10:26.800 So it's very difficult to scale. s120 00:10:26.800 --> 00:10:28.880 E Google was at it at the time. s121 00:10:28.880 --> 00:10:31.440 Uh h having hundreds of robots. s122 00:10:31.440 --> 00:10:34.880 Even for Google it's too expensive and too slow to scale. s123 00:10:34.880 --> 00:10:36.560 The other idea is again telly operation. s124 00:10:36.560 --> 00:10:43.200 As I said it's an old idea but again the same issues impossible to scale because now you don't need robot you also need humans. s125 00:10:43.200 --> 00:10:53.839 Uh so it's even more expensive and diversity is very limited because even if you put a robot in one setup with a human you can get thousand examples but they'll all be in the same setup. s126 00:10:53.839 --> 00:11:01.680 So what you ideally want carry the robot to new home every day in new scenario and that's just impossibly hard to scale. s127 00:11:01.680 --> 00:11:04.720 Then we had major breakthrough in learning from simulation. s128 00:11:04.720 --> 00:11:13.360 This was one of the first result where uh one could show deep reinforcement learning uh trained in simulation transfer to real robot. s129 00:11:13.360 --> 00:11:20.640 This is one of one of the award-winning paper uh at the time and now it's used in every humanoid uh every company out there. s130 00:11:20.640 --> 00:11:23.200 uh this was from our lab at Berkeley and CMU. s131 00:11:23.200 --> 00:11:25.519 But again they there are pros and cons. s132 00:11:25.519 --> 00:11:31.760 It's very easy to scale but diversity is hard to get because you have to engineer every scene in simulation manually. s133 00:11:31.760 --> 00:11:35.680 So if you're listening to this like there is no single answer I'm coming to. s134 00:11:35.680 --> 00:11:39.200 Uh there is no uh no no uh no single solution. s135 00:11:39.200 --> 00:11:41.440 This is uh another work we did earlier. s136 00:11:41.440 --> 00:11:44.560 This is all before skilled learning from human videos. s137 00:11:44.560 --> 00:11:47.279 You watch the human do things and then robot copies this. s138 00:11:47.279 --> 00:11:48.640 Again high diversity. s139 00:11:48.640 --> 00:11:56.320 You can use videos from YouTube etc. Highly scalable but it's a very poor form of data because it's very far from robot. s140 00:11:56.320 --> 00:11:57.279 Okay. s141 00:11:57.279 --> 00:11:58.959 So what is the solution here? s142 00:11:58.959 --> 00:12:01.680 The answer is there is no there is no golden path. s143 00:12:01.680 --> 00:12:06.800 You have to think about data in the context of different features. s144 00:12:06.800 --> 00:12:12.240 And in my opinion there are only three features that matter in in robotics. s145 00:12:12.240 --> 00:12:18.320 scalability, diversity and how close you are to your robot uh uh robot joint angles. s146 00:12:18.320 --> 00:12:22.399 Okay, scalability means can I quickly scale it across scenarios. s147 00:12:22.399 --> 00:12:31.839 Diversity means can I get diverse data because just having 100 trillion tokens is completely useless if they're all in the same environment and same scenarios. s148 00:12:31.839 --> 00:12:32.639 Okay. s149 00:12:32.639 --> 00:12:38.079 And closeness to robot mean how far are you from the ground truth of robot own joint angles. s150 00:12:38.079 --> 00:12:49.839 So if you look at simulation very scalable low diverse uh uh but moderately close to robot human videos are very scalable and diverse but very far from robot data you have to learn to map the human to robot s151 00:12:49.839 --> 00:13:01.920 this other two teleop and the and the manipulation interfaces they are kind of in between and you can see here uh right now around the world if you look at different companies they're all focusing on one of these approaches s152 00:13:01.920 --> 00:13:11.519 most of them are on teleop or or yumi setups very few on everything else and but there There is no golden answer like they every each one of them have a downside s153 00:13:11.519 --> 00:13:20.240 but the golden light here is that they are all complemented to each other like they are not their cons are not exactly matching from each other. s154 00:13:20.240 --> 00:13:37.920 So this is where uh the recipe that we have converged to over the years is to realize separate the training into two parts pre-training and post- training but that's not surprise right but how do we pre-train you want to pre-train using data which is highly scalable and diverse but maybe low quality so simulation human videos etc. So s155 00:13:37.920 --> 00:13:42.560 that's how you pre-train and then for post training you use data from telly operation. s156 00:13:42.560 --> 00:13:44.000 Now what is teleoperation data? s157 00:13:44.000 --> 00:13:46.399 It's very high quality but low in amount. s158 00:13:46.399 --> 00:13:48.560 Okay, high quality low in amount. s159 00:13:48.560 --> 00:13:51.760 It exactly reminds us of the recipe in language models. s160 00:13:51.760 --> 00:14:02.560 You pre-train data on the internet then you post train for coding for uh uh for your own company etc. But in robotics there is one more bucket which is deployment data s161 00:14:02.560 --> 00:14:07.519 and deployment data is highly scalable once deployment scaled. s162 00:14:07.519 --> 00:14:11.279 So over time this data will take over everything else. s163 00:14:11.279 --> 00:14:20.000 And what we are trying to build is what we call this data flywheel which goes which takes this data from post training time and puts back in pre-training. s164 00:14:20.000 --> 00:14:30.800 And now you can see why this idea of omniodied brain is extremely important because this is very unlikely that you have only one robot only one version deployed forever s165 00:14:30.800 --> 00:14:32.480 in every task around the world. s166 00:14:32.480 --> 00:14:38.000 This never happens in any area of hardware except chips because they are very hard to manufacture. s167 00:14:38.000 --> 00:14:43.760 Uh and you can still see even in the chips uh inference chips are coming left and right for many companies these days. s168 00:14:43.760 --> 00:14:45.680 So in hardware this is never the case. s169 00:14:45.680 --> 00:14:54.720 you have only one version of the robot, one shape, which is why omniodied intelligence is the enabler of what we call a deployment data flywheel. s170 00:14:54.720 --> 00:14:59.040 Okay, so let's now look at a few results uh very quick. s171 00:14:59.040 --> 00:15:00.959 Now this brain is extremely general. s172 00:15:00.959 --> 00:15:03.680 So we can do variety of tasks very quickly. s173 00:15:03.680 --> 00:15:10.160 You may have you have se you may have seen many tasks like laundry folding etc. It's very popular task in Silicon Valley for some reason. s174 00:15:10.160 --> 00:15:22.720 uh and uh and the and the argument here is uh when have you ever thought while folding a t-shirt that if I miss my hand by 1 cm my t-shirt fold will be a blunder s175 00:15:22.720 --> 00:15:34.880 you don't think like this people don't even think while folding they just hold anywhere you just do something so that the tolerance for error is extremely extremely high then why is this task hard s176 00:15:34.959 --> 00:15:42.399 anybody why why do people get why do people get fooled into believing this task is Let me put it that way. s177 00:15:42.399 --> 00:15:43.360 Fabric, right? s178 00:15:43.360 --> 00:15:46.079 Why is fabric hard? s179 00:15:46.399 --> 00:15:46.959 Simulation. s180 00:15:46.959 --> 00:15:48.000 But nobody's using simulation anyway. s181 00:15:48.000 --> 00:15:49.440 This is all from telly operation. s182 00:15:49.440 --> 00:15:51.040 Why is this hard? s183 00:15:51.040 --> 00:15:52.480 You know why is it hard? s184 00:15:52.480 --> 00:15:56.480 Because it is hard for classical version of robotics. s185 00:15:56.480 --> 00:16:01.360 Classically in robotics, people would model the whole physics, create models by hand and then do this. s186 00:16:01.360 --> 00:16:02.560 It's very hard for that. s187 00:16:02.560 --> 00:16:08.240 But for deep learning based robotics, this is the easiest task possible because it has high tolerance for error. s188 00:16:08.240 --> 00:16:12.320 So it is completely uh now so many companies focusing on this task. s189 00:16:12.320 --> 00:16:16.800 Now what is hard is I would say something about like let's say this task. s190 00:16:17.040 --> 00:16:28.240 If I ask you before seeing this video is this task doable without having hands very likely half the people will say no because it requires putting like how many of you have lost airports? s191 00:16:28.240 --> 00:16:34.079 Uh like and in our company there's a channel called right airpod because people keep losing their right. s192 00:16:34.079 --> 00:16:37.600 Now in this case the robot does not have hand it has gripper. s193 00:16:37.600 --> 00:16:41.279 So here the task is very hard for the gripper. s194 00:16:41.279 --> 00:16:54.399 So it requires a higher level of intelligence because the arm has to go and orient itself to pick up the airpod in the right manner such that it can be inserted because the grippers can only close up and like like this. s195 00:16:54.399 --> 00:16:55.759 They're parallejo grippers. s196 00:16:55.759 --> 00:17:00.000 So you cannot turn the uh the the the airpod at the very end. s197 00:17:00.000 --> 00:17:02.399 Now what you are seeing here these are not real airpods. s198 00:17:02.399 --> 00:17:06.000 They are fake ones from teu uh like $5 $10 each. s199 00:17:06.000 --> 00:17:07.679 So they don't have a magnet inside. s200 00:17:07.679 --> 00:17:13.600 So the robot has to really work hard to put this inside properly because there is no magnet to pull it uh easily. s201 00:17:13.600 --> 00:17:15.360 So these are all fake uh fake ones. s202 00:17:15.360 --> 00:17:18.799 This is even harder than it appears uh in the video. s203 00:17:18.799 --> 00:17:29.280 You can also uh like once you build the general brain behind the scene, you can also go and learn it from variety of just human videos without actually having any finetuning data at tell operation time. s204 00:17:29.280 --> 00:17:34.480 So in this scenario what we did, we trained on human videos like this like egocentric videos. s205 00:17:34.480 --> 00:17:43.200 we are now trying to transfer it to more third person uh uh videos and if third person works you can learn from YouTube any any kind of open source data. s206 00:17:43.200 --> 00:17:47.760 So in this case we see the video and then we add less than 1 hour of robot data. s207 00:17:47.760 --> 00:17:52.960 So very quick uh transformation and then the it can transfer to humanoids. s208 00:17:52.960 --> 00:17:54.960 Now these ones have hands. s209 00:17:54.960 --> 00:17:57.120 Hands or no hands is a big debate. s210 00:17:57.120 --> 00:18:02.480 People often use hands as an excuse as to why robotics is not here but that's not the case. s211 00:18:02.480 --> 00:18:06.640 It's always the intelligence uh behind the scene. s212 00:18:07.440 --> 00:18:12.080 We don't deploy hands right now because there are none available which can be deployed in factories. s213 00:18:12.080 --> 00:18:13.760 They all break within 100 yards. s214 00:18:13.760 --> 00:18:16.640 Uh pick anyone. s215 00:18:17.679 --> 00:18:22.720 If you have better hand uh I would love to buy 10 units ASAP to test. s216 00:18:22.720 --> 00:18:25.280 So it's robust to scenarios. s217 00:18:25.280 --> 00:18:28.400 Now the idea is you can do many tasks by watching humans. s218 00:18:28.400 --> 00:18:39.200 And the reason we can work with very less data which is less than one hour of robot data is because robot imagines things in its own head and tries to multiply the learning from for many scenarios. s219 00:18:39.200 --> 00:18:43.919 So what you see here is a completely fake video uh you can call it robot's dream. s220 00:18:43.919 --> 00:18:49.280 So it's inside the robot's own model where it can imagine scenarios uh in so this is not real data. s221 00:18:49.280 --> 00:18:52.880 This is all fake uh from own robot's own model. s222 00:18:52.880 --> 00:18:57.600 You can transfer it to you know more complex task and even lower cost hardware. s223 00:18:57.600 --> 00:19:02.080 So this is a egg uh uh egg making like omelette making task. s224 00:19:02.080 --> 00:19:06.640 Uh now here the these this whole setup costs about $4,000. s225 00:19:06.640 --> 00:19:11.280 So extremely cheap arms uh compared to the uh what you see out there. s226 00:19:11.280 --> 00:19:16.960 And if you notice here this setup is too cheap to even have any sensors. s227 00:19:16.960 --> 00:19:19.039 So the only sensor here is just a camera. s228 00:19:19.039 --> 00:19:20.559 No force nothing else. s229 00:19:20.559 --> 00:19:25.039 And the robot can do task which require forces uh from vision. s230 00:19:25.039 --> 00:19:27.919 Now I'm not saying that's a future like you should not be doing this. s231 00:19:27.919 --> 00:19:31.200 I'm sure the sensors will improve but s232 00:19:31.200 --> 00:19:37.679 from the existing sensors we are way behind than where we can be from just intelligence perspective. s233 00:19:37.679 --> 00:19:38.480 Okay. s234 00:19:38.480 --> 00:19:40.000 So this can keep on going. s235 00:19:40.000 --> 00:19:42.080 This is my adviser from Berkeley. s236 00:19:42.080 --> 00:19:43.679 He did not believe the robots can do it. s237 00:19:43.679 --> 00:19:47.440 He just kept standing for like uh 1 hour and the robot kept making omelette. s238 00:19:47.440 --> 00:19:51.360 Uh and and the interesting part here is this is fully end to end system. s239 00:19:51.360 --> 00:19:53.360 There is no state machine nothing. s240 00:19:53.360 --> 00:19:56.799 So the reason robot is looking egg because there is an empty plate in the front. s241 00:19:56.799 --> 00:19:57.919 I don't know why it's fluctuating. s242 00:19:57.919 --> 00:20:00.080 It's not in the there's no cut in the video. s243 00:20:00.080 --> 00:20:01.520 I I assure you of that. s244 00:20:01.520 --> 00:20:02.960 It's HDMI problem. s245 00:20:02.960 --> 00:20:08.880 So as soon as you remove the plate uh sorry I don't know why this is happening. s246 00:20:08.880 --> 00:20:09.120 Yeah. s247 00:20:09.120 --> 00:20:11.840 So there's no state machine when the robot starts when the robot ends. s248 00:20:11.840 --> 00:20:15.039 It's all automated and it's very robust to even disturbances. s249 00:20:15.039 --> 00:20:16.720 You can change objects around. s250 00:20:16.720 --> 00:20:20.000 Uh you can add these are all unseen and unseen objects. s251 00:20:20.000 --> 00:20:22.400 Uh only the pan and the gas stove is same. s252 00:20:22.400 --> 00:20:25.360 Everything else is different and the robot can keep on going. s253 00:20:25.360 --> 00:20:27.360 So you can get basic robustness. s254 00:20:27.360 --> 00:20:30.640 So this was this is all trained with less than 10 hours of data. s255 00:20:30.640 --> 00:20:37.360 Uh so it's very low data to be learning this robustness and it's coming from the base model behind the scene. s256 00:20:37.360 --> 00:20:39.840 I'm skipping uh this in the interest of time. s257 00:20:39.840 --> 00:20:46.000 So unlike traditional robotics pipelines where you have planning, mapping etc. This is an end toend brain. s258 00:20:46.000 --> 00:20:50.480 Now end to end is heavily used term in in many areas of AI. s259 00:20:50.480 --> 00:20:53.679 But when I mean end to end, I really mean end to end. s260 00:20:53.679 --> 00:20:58.960 It reads directly from the cameras and it applies power directly to motors. s261 00:20:58.960 --> 00:21:04.880 So we use nothing uh in between except just a P at the very end to go from 100 to 500 Hz. s262 00:21:04.880 --> 00:21:06.799 But P is not robotics contribution. s263 00:21:06.799 --> 00:21:09.679 This is before it predates to like World War II or something. s264 00:21:09.679 --> 00:21:10.240 Okay. s265 00:21:10.240 --> 00:21:15.440 Now the the the interesting part is vision for us is yet another input. s266 00:21:15.440 --> 00:21:17.520 So nothing that crazy about it. s267 00:21:17.520 --> 00:21:24.960 So if you have seen, you may have seen a lot of lot of results of robots dancing, doing karate, kung fu, backflip. s268 00:21:24.960 --> 00:21:30.480 Go back and think how many results have you seen of robots going up and down stairs. s269 00:21:30.480 --> 00:21:31.120 Very few. s270 00:21:31.120 --> 00:21:39.200 Like even the top companies out there in humanoid, they have not shown beyond one sample stair just to check the tick tick box. s271 00:21:39.200 --> 00:21:39.840 You do. s272 00:21:39.840 --> 00:21:45.200 And people talk about humanoids are coming, China versus US, all of this is sort of BS. s273 00:21:45.200 --> 00:21:49.760 If humanoids cannot climb stairs, what's the point of having legs? s274 00:21:49.760 --> 00:21:51.760 All right, that's the only reason why you have legs. s275 00:21:51.760 --> 00:21:55.679 Now, this is actually a Moex paradox add action. s276 00:21:55.679 --> 00:21:58.159 Left one actually is much easier for a robot. s277 00:21:58.159 --> 00:22:01.200 A backflip, dancing, etc. is much easier for the robot. s278 00:22:01.200 --> 00:22:04.159 And you may think, okay, why is this paradox exist? s279 00:22:04.159 --> 00:22:05.760 Think one level deeper. s280 00:22:05.760 --> 00:22:13.360 When a robot is doing a back flip, it has to only know about its own body, nothing else. s281 00:22:13.360 --> 00:22:13.760 Right? s282 00:22:13.760 --> 00:22:32.640 So it's and when everything is known or fully observed that's what computers are good at because they can do they can simulate every possible uh uh setup but the right one even simply climbing on stairs requires seeing the stair at the first at the first place or what's the height what's the width s283 00:22:32.640 --> 00:22:46.000 you don't measure height and width exactly but you adjust to all the disturbances and that requires vision so right one requires understanding and that's why it is hard and you don't see any of this but for us it's just get another result it doesn't really matter so we And uh we put this result out one and a s284 00:22:46.000 --> 00:22:46.480 half year ago. s285 00:22:46.480 --> 00:22:48.480 We shot this like two and a half years ago. s286 00:22:48.480 --> 00:22:52.559 Uh but uh like this robot can go any scenario. s287 00:22:52.559 --> 00:22:53.919 It is not stumbling on things. s288 00:22:53.919 --> 00:22:57.039 It intentionally steps over things and there is no mapping or planning. s289 00:22:57.039 --> 00:22:59.600 It doesn't make any 3D map of the surroundings. s290 00:22:59.600 --> 00:23:02.080 It's all operating from camera on the torso. s291 00:23:02.080 --> 00:23:05.440 And you can put this in any kind of setup, any kind of stairs. s292 00:23:05.440 --> 00:23:07.440 Uh I'm going faster here. s293 00:23:07.440 --> 00:23:11.120 These are, you know, uh fire escape stairs. s294 00:23:11.120 --> 00:23:12.240 It's a bit weird. s295 00:23:12.240 --> 00:23:17.760 Fire escape stairs are to be used when there is fire in urgency and they are the hardest in every building. s296 00:23:17.760 --> 00:23:23.520 Uh so we are testing in all in all those setups but you can put this in anywhere you can disturb it and stay doesn't really matter. s297 00:23:23.520 --> 00:23:27.679 This is superhuman capability because robot cannot see it's being pulled. s298 00:23:27.679 --> 00:23:31.360 So it's a surprise factor for the robot that you're pulling the violet while climbing. s299 00:23:31.360 --> 00:23:35.120 And again it's the same model everywhere in all these scenarios. s300 00:23:35.120 --> 00:23:48.320 So very easy although easy parkour does look fun uh uh does look fun and people often find like uh this is uh you know the robots can do all this but this is s301 00:23:48.320 --> 00:24:03.440 so much easier than what I showed earlier and even if you see these videos right now many of you will find this more impressive even though I'm telling you this is easy it takes half an hour to train this while the previous one it takes much longer uh and much more difficult to train s302 00:24:03.440 --> 00:24:08.000 so you can take this Based model and you can transfer to variety of tasks very quickly. s303 00:24:08.000 --> 00:24:18.640 So here this is a very old result from one and a half year ago uh for inserting LAN cables etc. We have very advanced systems now but we are deploying these models already across variety of application. s304 00:24:18.640 --> 00:24:24.080 So for instance because to to build the data flywhe the hardest part is the deployment itself. s305 00:24:24.080 --> 00:24:29.039 There are so many other issues beside intelligence and hardware that come up when you deploy robotics. s306 00:24:29.039 --> 00:24:30.480 It is not the same. s307 00:24:30.480 --> 00:24:42.159 This is not same same as deploying chat GP over an app because you have to work around many of the and I think Skyio gave a talk before this and they can talk to you about all day about the hardness of deployment. s308 00:24:42.159 --> 00:24:49.600 So we have to start deployment now so that we can have this data flywheel in a foreseeable future and I'll give you a few examples of deployment we are doing. s309 00:24:49.600 --> 00:24:54.320 So one of the examples with Nvidia like Nvidia is opening their first factory in Houston. s310 00:24:54.320 --> 00:25:05.840 uh and their GPUs what you use right now they're all built outside US in in Taiwan mainly and it's all done manually over there and uh humos are really efficient and really uh really good at making these things s311 00:25:05.840 --> 00:25:18.159 but sustaining cost and scalability and throughput here in US it's very hard to maintain without this labor force so here we are automating uh this GPU assembly for them this was we did a live demo in Nvidia s312 00:25:18.159 --> 00:25:32.960 GTC this is deployed in factory already uh last week so it's already live but the interesting part I want to highlight this is a very traditional factory task but if you look at the the whole setup s313 00:25:32.960 --> 00:25:48.559 this is extremely randomized extremely noisy and you go to any factory traditional setup they're extremely clean so here the robot the same brain can not only do very precise task it can be very robust to any kind of disturbance which means now these robots s314 00:25:48.559 --> 00:26:00.000 do not need to be in a cage uh and they can be uh just deployed as is on on these factory lines without any change to any factory line alongside humans and wherever there is humans there is mess. s315 00:26:00.000 --> 00:26:02.799 Okay, you can put the same brain to delivery applications. s316 00:26:02.799 --> 00:26:06.640 So here the robot delivers from the from the truck to the to the door. s317 00:26:06.640 --> 00:26:08.240 It has to find where the front door is. s318 00:26:08.240 --> 00:26:09.279 So it's a common sense problem. s319 00:26:09.279 --> 00:26:10.480 It's not mobility problem. s320 00:26:10.480 --> 00:26:19.279 And we already have been delivering uh packages to with working behind the scene with many partners to delivering packages to front door of the houses uh in these areas. s321 00:26:19.279 --> 00:26:22.559 same setup works across warehouses and all this. s322 00:26:22.559 --> 00:26:34.320 Um now just to close the talk uh close the topic I mentioned this is an omnibody brain and I hope it's very clear why it's extremely essential to have an omnibody brain to build data flywheel s323 00:26:34.320 --> 00:26:49.360 but there's one extra benefit and the benefit is your robots may break over time and uh or the safety uh scenarios and safety is a byproduct of this omnibody brain because if you have a we can put the I'm just skipping very fast here so you can see the online this is all online s324 00:26:49.360 --> 00:26:58.559 uh but we can put the same brain across each of these systems each of these robots and in this case we did not even train on these robots. s325 00:26:58.559 --> 00:27:09.440 This is completely zero shot transfer to all these humanoids and all these things and even if the robot breaks like here the leg breaks it can recover within milliseconds because when your leg breaks s326 00:27:09.440 --> 00:27:12.080 your three-legged robot is a new robot. s327 00:27:12.080 --> 00:27:21.120 So it doesn't really matter what's the shape of the robot is anything changing here we disabled the legs of the robot it learns to learn it learns to walk on two legs in just three trials. s328 00:27:21.120 --> 00:27:27.360 So what you are seeing on screen is all the training for the robot that they are that that is happening in these systems. s329 00:27:27.360 --> 00:27:32.080 So it's it adapts in few milliseconds to uh uh 30 seconds or so. s330 00:27:32.080 --> 00:27:34.799 And like one example here it's it's going on wheels. s331 00:27:34.799 --> 00:27:37.760 You jam the wheel it starts walking. s332 00:27:37.760 --> 00:27:41.039 So this is a another byproduct of omnibody brain. s333 00:27:41.039 --> 00:27:45.440 Safety takes a very different meaning uh with these kind of models. s334 00:27:45.440 --> 00:27:50.880 when the robot can fly like or sorry when the robots can walk or or operate with only half the body. s335 00:27:50.880 --> 00:28:00.159 I removed some of the gory scenes from this but we also cut the robot in half and it can still work uh with both the two halves separately but for that you can go to YouTube uh and that's all I have. s336 00:28:00.159 --> 00:28:02.320 Thank you.