{ "video": { "id": "jFHteJjRl8A", "title": "Robotics Has Been Stuck for 70 Years — Deepak Pathak, Skild AI", "duration": 1698, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "jFHteJjRl8A", "title": "Robotics Has Been Stuck for 70 Years — Deepak Pathak, Skild AI", "one_liner": "Robotics looks the same as it did 70 years ago because it was treated as a hardware problem. Skild AI's answer is one 'omni-bodied' brain for any robot and any task. It is pre-trained on scalable but low-quality data (simulation, human videos), post-trained on teleoperation, and then improved through a deployment-data flywheel that only works if the brain spans many body types.", "summary": "Pathak says robotics demos from the 1950s–60s (the MIT copy demo, a 1957 leader-follower teleoperation rig) look just like today's, and that teleoperation can never reach LLM-scale data: at about one minute per example, even the whole US population would need more than a century to match GPT-3. He argues no single data source is enough. Each one trades off scalability, diversity and closeness to the robot's own joint angles, and their weaknesses complement each other. So Skild pre-trains on simulation and human video, post-trains on teleop, and builds a flywheel from deployment data. He shows end-to-end results: AirPod insertion with a parallel-jaw gripper, omelette-making on a $4,000 camera-only arm, stair-climbing humanoids with no mapping, NVIDIA GPU assembly in a factory, package delivery, and zero-shot transfer to new or damaged robot bodies.", "key_points": [ "Moravec's paradox ('hard is easy, easy is hard') explains why robotics has lagged: a computer beat Kasparov at chess in the '90s, but no robot can yet pick up chess pieces and arrange them on any board.", "Teleoperation, which all the big labs use to collect data, follows the same principle as a 1957 leader-follower system. At about 1 minute per example, getting to GPT-3 scale (which he puts at 30T+ tokens) would take all of the US population more than a century.", "Three data properties matter: scalability, diversity, and closeness to the robot's joint angles. Simulation is scalable but low in diversity. Human videos are scalable and diverse but far from the robot. Teleop and UMI-style interfaces sit in between. Most companies focus only on teleop or UMI setups.", "The recipe: pre-train on scalable, diverse, low-quality data (simulation, human videos) and post-train on teleop data (high quality, small amount). Deployment data forms a flywheel back into pre-training, and he expects it to eventually outweigh everything else.", "An omni-bodied brain (any robot, any task, one brain) is what makes the deployment flywheel possible, because hardware never settles on a single robot shape or version.", "Laundry folding is hard for classical physics-modelling robotics but is 'the easiest task possible' for deep learning because it tolerates error. Harder: inserting fake $5–10 AirPods (no magnet) into a case with a parallel-jaw gripper, which forces the arm to orient itself before grasping.", "Egocentric human video plus less than 1 hour of robot data transfers skills to humanoids, helped by the robot 'dreaming' imagined scenarios inside its own model. Omelette-making on a ~$4,000 camera-only setup (no force sensors) needed under 10 hours of data. It is fully end-to-end with no state machine and handles unseen objects.", "'Really end-to-end' means going straight from cameras to motor power, with only a PD controller in between to go from 100 to 500 Hz. Blind stair-climbing from a torso camera, with no 3D map or planner, is much harder than backflips or parkour (about half an hour to train) because stairs require perception. Deployments: NVIDIA GPU assembly at its Houston factory (demoed at GTC, live in the factory as of the week before the talk), uncaged in noisy settings, and truck-to-door package delivery. Zero-shot transfer across humanoids; recovery from a broken leg in milliseconds; learns to walk on two legs in three trials; switches to walking when its wheels are jammed." ], "takeaways": [ "Don't bet on a single data source. Combine scalable, diverse data (simulation, human video) for pre-training with a small amount of high-quality teleop data for post-training.", "Judge each data source on three axes: scalability, diversity, and closeness to the robot's own joint angles. 100 trillion tokens from the same environment are useless.", "Start deploying early. Deployment is the hardest part and the source of the data flywheel, and it raises many problems beyond intelligence and hardware.", "Train models across different body types so that deployment data from any hardware helps, and so the system tolerates damage and hardware changes.", "Don't read flashy demos (backflips, dancing, laundry folding) as progress. Perception-heavy tasks like climbing varied stairs are the real test." ], "topics": [ "robotics", "foundation models", "embodied ai", "robot learning data", "sim-to-real", "teleoperation", "data flywheel", "humanoids" ], "tools": [ "Skild AI", "NVIDIA", "NVIDIA GTC", "GPT-3", "GPT-4", "ChatGPT", "Google", "MIT copy demo", "UMI", "Temu", "AirPods", "Berkeley", "CMU" ], "quotes": [ { "text": "Hard is easy, easy is hard.", "at": "04:41", "url": "https://www.youtube.com/watch?v=jFHteJjRl8A&t=281s" }, { "text": "if you hire all of US population it will take you more than a century to reach the same scale as of GPD3", "at": "06:51", "url": "https://www.youtube.com/watch?v=jFHteJjRl8A&t=411s" }, { "text": "If humanoids cannot climb stairs, what's the point of having legs?", "at": "21:46", "url": "https://www.youtube.com/watch?v=jFHteJjRl8A&t=1306s" }, { "text": "when your leg breaks your three-legged robot is a new robot.", "at": "27:08", "url": "https://www.youtube.com/watch?v=jFHteJjRl8A&t=1628s" } ], "words": 6006 }, "summary_url": "/#jFHteJjRl8A", "transcript": { "html": "/transcripts/jFHteJjRl8A.html", "txt": "/transcripts/jFHteJjRl8A.txt", "vtt": "/transcripts/jFHteJjRl8A.vtt" } }