Physical AI's Next Bottleneck Is Finding the Right Video — Rafael Levi, Bright Data

AI Engineer · 16 min · 203 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:13Hi everybody.
3. 00:15Welcome.
4. 00:16For those that came to actually see me, thank you.
5. 00:20For those that are just hanging out here, also thank you.
6. 00:23I'm going to try to entertain you guys, teach you something new.
7. 00:28Um again my name is Rafael.
8. 00:30I work at Bright Data.
9. 00:31I've been with Brighta for over eight years.
10. 00:34And in Bright Data we are collecting data and uh we're trying to innovate.
11. 00:38So today I want to talk about video discovery for agentic world models training.
12. 00:44And uh we're going to explain what that is in a little bit.
13. 00:48So what does it take to take to build a system that can see understand and act right?
14. 00:57I mean, we figured out what an AI is.
15. 01:00We're constantly improving it.
16. 01:02Like, you know, that's kind of handled.
17. 01:04But now, the hard part is what data do you provide to it?
18. 01:08Because everybody knows AI without data is just a box, right?
19. 01:15So, let me take you a little bit back in history, right?
20. 01:18In 2022, Google taught a robot by images, real world actions, right?
21. 01:27Then going forward, there was a leap uh in 2023.
22. 01:32Uh we had uh an AI actually control multiple robots.
23. 01:37In 2024, the open source came out.
24. 01:40Now, this where the game changed.
25. 01:42Once we had an open source, all the labs had access to AI and they started already putting AI into robots, right?
26. 01:51And so now these models are already driving humanoid robots.
27. 01:56I'm you seeing some of course most here are remote controlled, but I mean in San Francisco you got weo driving without any drivers.
28. 02:05How cool is that?
29. 02:07First time I got in the car, I thought I was going to die, but it was actually very nice.
30. 02:12Right now, how do you think it learned how to drive?
31. 02:17By actually teaching itself.
32. 02:19By by learning on dashboard cameras, right?
33. 02:22Every car has a dashboard.
34. 02:24So, if you provide that to an AI, it can figure out how to actually operate a machine.
35. 02:31So, as I said, the AI is no longer the hard part.
36. 02:34The data is.
37. 02:35And this is what I want to talk about.
38. 02:38Right?
39. 02:39So for chat LLMs, there's trillions of words and texts, right?
40. 02:46So we have so much text data to train the LLMs.
41. 02:49For image generation, we got billions of images, labeled images.
42. 02:54So that's also a huge database.
43. 02:57But for robotics, there's only about a million videos.
44. 03:01It's a very small limited data sets of robots doing things, right?
45. 03:09And so the idea here is that of course you a lot of companies out there what they do is they pay people to record actions, right?
46. 03:20So for example, hey record me how you open a door or record to me how you're sitting on the chair.
47. 03:28But then the problem becomes is that when somebody is told to do something, they do not do it naturally.
48. 03:34So I call it instructed, right?
49. 03:37If you're told to record how you open the door and you're doing it for somebody, it's not going to be the same as if you just walk into the house.
50. 03:43The movement is totally different.
51. 03:45So is that data good for robotic training?
52. 03:49In my assumption, no. It's biased data and it doesn't deliver the same results.
53. 03:53So obviously the robot is not going to be as effective as if it's intuitive.
54. 03:59Right?
55. 04:00So I just threw in some examples on the slide, right?
56. 04:02So uh the built by hand data is not the same as the data that is kind of on the site, right?
57. 04:12So and also the fact is that people behave very differently when they on the camera.
58. 04:16Some of you know some people are camera shy.
59. 04:19So you know as soon as you put a camera on them you they change but in the natural world it's a totally different thing.
60. 04:27So this is what we are trying to solve in bright data right um again so why is the usual data sources are not enough right simulations uh virtual it's it's cheap but it's um you know everybody's playing video games the physics in video games are almost there
61. 04:47but they're not good enough where to train a robot on it right a hand control the person controlling a robot it's doable but how many hours a day can you record the robot, how many people you need to actually get huge scale data, right?
62. 05:01I mean, you can record maybe eight hours a day every day.
63. 05:04How many hours are you going to get in a year?
64. 05:07It's not really scalable.
65. 05:09And of course, there's already pre-built data sets, but as I said before, they're small.
66. 05:13There's only about a million videos at this point.
67. 05:15So, what is the alternative?
68. 05:17The alternative is the web.
69. 05:20Let's just think about YouTube.
70. 05:22How many videos are there on YouTube?
71. 05:25I don't know, five billion videos.
72. 05:28How many actions are in those videos that a robot can replicate?
73. 05:34I mean, let's think about it.
74. 05:35How many do you think videos are there of a person opening the door firsterson view?
75. 05:42Millions of hours.
76. 05:43You'll be surprised.
77. 05:46So, every day the web video shows gravity motions, right?
78. 05:54So um cause and effect obviously accidents big the biggest cause and effect right how handle how people handle objects and billions of hours great training material for robots but what is the problem the problem is that there's a lot of noise
79. 06:11right oh and for example right so one of the AI models that meta trained they gave it about a million hours of real world videos and then all it took is 62 hours of real robotics data to actually
80. 06:26control to control a real robot.
81. 06:29So if you think about it, the data was collected publicly, right?
82. 06:33So it trained the the the robot on just random things and then in 62 robotic hours, it was already moving.
83. 06:42So there no simulations needed, autonomous robotics delivered fast.
84. 06:52But the videos don't show how the robots move, right?
85. 06:55So you might think, well, listen, how useful is it a person pouring water into a cup for a robot.
86. 07:01So there is methods actually out there how an AI can distinguish and actually learn from the actions that are in a video.
87. 07:09If you take two frames frame by frame and the AI measures the movement this difference, right?
88. 07:15So you have an image A and image two and there's a small movement that is changing between the two images and AI can actually measure that change.
89. 07:24So if you keep doing that through the whole video and AI can actually figure out angles, distance and everything that it needs to train and robot right.
90. 07:34So we don't necessarily need the sensor's data but of course the video itself that you download from video from YouTube or the video that uh you extract the data from doesn't necessarily get you 100% there.
91. 07:48You do need to have some processing on top of that but it's available.
92. 07:53It's out there and all you need to do is process it.
93. 07:58So a few more examples right?
94. 08:00So for example, Nvidia when they train in the Cosmos robot, they're throwing out about 96% of the video.
95. 08:07So what does that mean?
96. 08:09They download a million hours of video and the only part is 4% is actually useful.
97. 08:15Everything else gets thrown out.
98. 08:17That's wasted compute.
99. 08:18That's wasted bandwidth.
100. 08:19That's wasted storage.
101. 08:21That's just a lot of wasted money.
102. 08:23Okay.
103. 08:23Stable video diffusion is throwing out 74% of the videos that they're downloading.
104. 08:30So in bright data we are trying to solve that problem and we're trying to take it a step forward.
105. 08:37So what we propose is search first collect second.
106. 08:41What we do is we are actually indexing videos and we are allowing you to search for specific actions.
107. 08:50Since we're talking about a person opening a door, let's keep using that uh example.
108. 08:55What you can do on our platform is actually input detailed information, right?
109. 09:01A query, person washing dishes, person folding a t-shirt and so on.
110. 09:06And we will provide the snippets from the videos of these actions.
111. 09:14So what does that mean?
112. 09:16That means that there's less waste of data collection.
113. 09:22there's less waste of storage and bandwidth, right?
114. 09:25So, a little more on how it works.
115. 09:27Obviously, you define what you're looking for.
116. 09:32We search through billions and billions of pre-indexed videos, not by keywords, but by actions, and we provide you with ready to use clips.
117. 09:44And obviously, they're already trimmed, prepared for your training.
118. 09:47So all you got to do is process them a little bit for maybe motion distance sensors and so on and it's ready to be ingested into a robot.
119. 09:57I have a small example of a video of how it actually works on our platform.
120. 10:02So let me just hit play here if I can find it.
121. 10:07So here we see person washing dishes by hand in sink removing grease from the plate using sponge and soap including rinse and placing dishes into drying rack.
122. 10:16close-up hand interactions.
123. 10:19So, right now what is happening is it's searching through billions of videos.
124. 10:24Now, on this video, it's a bit old.
125. 10:26We only got about 100 million index there, but now we're up to 1.1 billion videos.
126. 10:31And it will literally like this is a little sped up, so I didn't want to waste your guys time, but it will provide to you clips from videos of people washing dishes.
127. 10:41And you can see right here all the videos in That would be fine.
128. 10:47Now, some of these videos might not have to do anything with washing dishes.
129. 10:50They might have cleaning kitchen.
130. 10:52They might have to do something else.
131. 10:54And here's another example.
132. 10:55For example, human folding different types of clothes, right?
133. 11:00So, a little more description.
134. 11:01The more description you give, the more you actually get back.
135. 11:06So, again, a little speed up.
136. 11:08Let's see.
137. 11:14I want you to understand it could be anything.
138. 11:16A person putting on makeup.
139. 11:17Let's say you have a brand and you want to find videos where your brand is being found.
140. 11:21All that can be also provided.
141. 11:24So you see people folding clothes.
142. 11:26So now you can train a robot on how to fold clothes.
143. 11:34Of course everything is available via API.
144. 11:36We don't expect people to actually do anything manually, right?
145. 11:39So you can trigger it by API.
146. 11:41you get a returned snippet of the video, the URL.
147. 11:46We give you the link to the original video if you want to watch it.
148. 11:49But the main thing here is that we can give you the snippets of the video to train your robotics.
149. 11:57Now, I'm not sure if any of you training any robots, so I'm going to give you another example of how useful it could be.
150. 12:05Let's say you have a brand and you want to know where your brand is being demoed on videos.
151. 12:13The video titles doesn't matter because it doesn't actually consume.
152. 12:17It's not about your product.
153. 12:19It's just a person doing some podcast recording something.
154. 12:23But you see a woman putting on makeup and she you see on the table that this is your brand of a makeup.
155. 12:30So now all of a sudden you can find all videos for your brand where it's being used whatever it is right by searching name of the video you won't find it by by
156. 12:40video indexing you can actually find specific things that you are interested in apple falling from the tree and so on.
157. 12:48So what comes back?
158. 12:50We provide to you this time stamp.
159. 12:52We provide to you the score matching, how close it is to your query, and of course the frame count.
160. 12:58How many frames are about what you're looking for?
161. 13:04So what does this change?
162. 13:06This changes a lot.
163. 13:07Okay, less waste.
164. 13:09You don't need to download million of hours of videos.
165. 13:12you can actually get exactly the the specific actions that you are interested in.
166. 13:21Again, it's very crucial for robotic training because noise creates problems, hallucinations, right?
167. 13:29This removes the noise.
168. 13:34What is it useful for?
169. 13:35Not only robots, but obviously self-driving for example, Wimo, right?
170. 13:40trained on dash cam video cameras and there's millions, hundreds of millions of hours on YouTube of dash cams.
171. 13:47How many of you like to watch accidents?
172. 13:50Dash cam c accidents.
173. 13:52I'm sure everybody watched them.
174. 13:54Crazy driving in Russia, right?
175. 13:57So, this is what we're talking about, right?
176. 13:58The videos that are out there, okay?
177. 14:01A person running a red light, person stopping on red light, making a left turn, making a right turn, and so on.
178. 14:07All of that can be a very useful data for training and any other world models.
179. 14:13Physics, right?
180. 14:14Robots needs to understand physics.
181. 14:17What is the gravity?
182. 14:18How to sit?
183. 14:19How to walk?
184. 14:19How to move?
185. 14:21Again, of course, you can pre-record it.
186. 14:24Um, a lot of times right now I speak to some of the people and they have millions of people recording videos for them.
187. 14:30Hey, can you guys record me how you open doors?
188. 14:33That's crazy.
189. 14:34Why do you need a million people doing it when everything is available online?
190. 14:40Online is one of the hugest databases out there.
191. 14:43It's just people don't really like it's it's not that easy to use it, right?
192. 14:46I mean, right now the only search available to us is by the keyword of the name of the video.
193. 14:53And of course, one source, the whole public video web in one place.
194. 14:59I mean, we got YouTube, we got Vimeo, we got so many different uh video providers out there, billions and billions of hours of training data just there waiting for you to grab it, collect it and process it.
195. 15:14So basically this is a new product that we're doing a bright data right indexing videos so that you guys can find specific actions anything you can think about you know again it could be useful for brands
196. 15:28uh as well it's not only for training data anything you can think of could be useful maybe you are a gamer and you want to know how do you beat this level but there's no exactly video like that so you can search okay people beating the level on the video game, right?
197. 15:47Anything.
198. 15:49The world is yours.
199. 15:52So, I'm going to wrap it up.
200. 15:54I don't know if you have any questions, but uh if you want to connect with me, talk about it more, feel free to to add me on LinkedIn.
201. 16:02And um yeah, I'm going to thank you guys for your attention.
202. 16:06And uh I'm at a bright data booth if you want to talk more about it.
203. 16:11And thanks for coming.