WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=I_VEh7XSwyc NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/I_VEh7XSwyc.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:13.040 --> 00:00:15.200 Hi everybody. s3 00:00:15.200 --> 00:00:16.960 Welcome. s4 00:00:16.960 --> 00:00:20.720 For those that came to actually see me, thank you. s5 00:00:20.720 --> 00:00:23.519 For those that are just hanging out here, also thank you. s6 00:00:23.519 --> 00:00:28.000 I'm going to try to entertain you guys, teach you something new. s7 00:00:28.000 --> 00:00:30.560 Um again my name is Rafael. s8 00:00:30.560 --> 00:00:31.840 I work at Bright Data. s9 00:00:31.840 --> 00:00:34.160 I've been with Brighta for over eight years. s10 00:00:34.160 --> 00:00:38.800 And in Bright Data we are collecting data and uh we're trying to innovate. s11 00:00:38.800 --> 00:00:44.480 So today I want to talk about video discovery for agentic world models training. s12 00:00:44.480 --> 00:00:48.160 And uh we're going to explain what that is in a little bit. s13 00:00:48.160 --> 00:00:57.360 So what does it take to take to build a system that can see understand and act right? s14 00:00:57.360 --> 00:01:00.800 I mean, we figured out what an AI is. s15 00:01:00.800 --> 00:01:02.239 We're constantly improving it. s16 00:01:02.239 --> 00:01:04.400 Like, you know, that's kind of handled. s17 00:01:04.400 --> 00:01:08.240 But now, the hard part is what data do you provide to it? s18 00:01:08.240 --> 00:01:15.680 Because everybody knows AI without data is just a box, right? s19 00:01:15.680 --> 00:01:18.720 So, let me take you a little bit back in history, right? s20 00:01:18.720 --> 00:01:27.680 In 2022, Google taught a robot by images, real world actions, right? s21 00:01:27.680 --> 00:01:32.000 Then going forward, there was a leap uh in 2023. s22 00:01:32.000 --> 00:01:37.360 Uh we had uh an AI actually control multiple robots. s23 00:01:37.360 --> 00:01:40.000 In 2024, the open source came out. s24 00:01:40.000 --> 00:01:42.000 Now, this where the game changed. s25 00:01:42.000 --> 00:01:51.680 Once we had an open source, all the labs had access to AI and they started already putting AI into robots, right? s26 00:01:51.680 --> 00:01:56.720 And so now these models are already driving humanoid robots. s27 00:01:56.720 --> 00:02:05.920 I'm you seeing some of course most here are remote controlled, but I mean in San Francisco you got weo driving without any drivers. s28 00:02:05.920 --> 00:02:07.200 How cool is that? s29 00:02:07.200 --> 00:02:12.720 First time I got in the car, I thought I was going to die, but it was actually very nice. s30 00:02:12.720 --> 00:02:17.280 Right now, how do you think it learned how to drive? s31 00:02:17.280 --> 00:02:19.120 By actually teaching itself. s32 00:02:19.120 --> 00:02:22.560 By by learning on dashboard cameras, right? s33 00:02:22.560 --> 00:02:24.319 Every car has a dashboard. s34 00:02:24.319 --> 00:02:31.840 So, if you provide that to an AI, it can figure out how to actually operate a machine. s35 00:02:31.840 --> 00:02:34.160 So, as I said, the AI is no longer the hard part. s36 00:02:34.160 --> 00:02:35.840 The data is. s37 00:02:35.840 --> 00:02:38.640 And this is what I want to talk about. s38 00:02:38.640 --> 00:02:39.840 Right? s39 00:02:39.840 --> 00:02:46.000 So for chat LLMs, there's trillions of words and texts, right? s40 00:02:46.000 --> 00:02:49.840 So we have so much text data to train the LLMs. s41 00:02:49.840 --> 00:02:54.160 For image generation, we got billions of images, labeled images. s42 00:02:54.160 --> 00:02:57.120 So that's also a huge database. s43 00:02:57.120 --> 00:03:01.040 But for robotics, there's only about a million videos. s44 00:03:01.040 --> 00:03:09.200 It's a very small limited data sets of robots doing things, right? s45 00:03:09.760 --> 00:03:20.640 And so the idea here is that of course you a lot of companies out there what they do is they pay people to record actions, right? s46 00:03:20.640 --> 00:03:28.400 So for example, hey record me how you open a door or record to me how you're sitting on the chair. s47 00:03:28.400 --> 00:03:34.959 But then the problem becomes is that when somebody is told to do something, they do not do it naturally. s48 00:03:34.959 --> 00:03:37.200 So I call it instructed, right? s49 00:03:37.200 --> 00:03:43.840 If you're told to record how you open the door and you're doing it for somebody, it's not going to be the same as if you just walk into the house. s50 00:03:43.840 --> 00:03:45.519 The movement is totally different. s51 00:03:45.519 --> 00:03:49.760 So is that data good for robotic training? s52 00:03:49.760 --> 00:03:53.840 In my assumption, no. It's biased data and it doesn't deliver the same results. s53 00:03:53.840 --> 00:03:59.040 So obviously the robot is not going to be as effective as if it's intuitive. s54 00:03:59.040 --> 00:04:00.000 Right? s55 00:04:00.000 --> 00:04:02.720 So I just threw in some examples on the slide, right? s56 00:04:02.720 --> 00:04:12.319 So uh the built by hand data is not the same as the data that is kind of on the site, right? s57 00:04:12.319 --> 00:04:16.880 So and also the fact is that people behave very differently when they on the camera. s58 00:04:16.880 --> 00:04:19.680 Some of you know some people are camera shy. s59 00:04:19.680 --> 00:04:27.199 So you know as soon as you put a camera on them you they change but in the natural world it's a totally different thing. s60 00:04:27.199 --> 00:04:47.280 So this is what we are trying to solve in bright data right um again so why is the usual data sources are not enough right simulations uh virtual it's it's cheap but it's um you know everybody's playing video games the physics in video games are almost there s61 00:04:47.280 --> 00:05:01.520 but they're not good enough where to train a robot on it right a hand control the person controlling a robot it's doable but how many hours a day can you record the robot, how many people you need to actually get huge scale data, right? s62 00:05:01.520 --> 00:05:04.800 I mean, you can record maybe eight hours a day every day. s63 00:05:04.800 --> 00:05:07.199 How many hours are you going to get in a year? s64 00:05:07.199 --> 00:05:09.840 It's not really scalable. s65 00:05:09.840 --> 00:05:13.840 And of course, there's already pre-built data sets, but as I said before, they're small. s66 00:05:13.840 --> 00:05:15.919 There's only about a million videos at this point. s67 00:05:15.919 --> 00:05:17.600 So, what is the alternative? s68 00:05:17.600 --> 00:05:20.560 The alternative is the web. s69 00:05:20.560 --> 00:05:22.240 Let's just think about YouTube. s70 00:05:22.240 --> 00:05:25.280 How many videos are there on YouTube? s71 00:05:25.280 --> 00:05:28.320 I don't know, five billion videos. s72 00:05:28.320 --> 00:05:34.720 How many actions are in those videos that a robot can replicate? s73 00:05:34.720 --> 00:05:35.840 I mean, let's think about it. s74 00:05:35.840 --> 00:05:41.919 How many do you think videos are there of a person opening the door firsterson view? s75 00:05:42.560 --> 00:05:43.919 Millions of hours. s76 00:05:43.919 --> 00:05:46.160 You'll be surprised. s77 00:05:46.160 --> 00:05:54.000 So, every day the web video shows gravity motions, right? s78 00:05:54.000 --> 00:06:11.840 So um cause and effect obviously accidents big the biggest cause and effect right how handle how people handle objects and billions of hours great training material for robots but what is the problem the problem is that there's a lot of noise s79 00:06:11.840 --> 00:06:26.479 right oh and for example right so one of the AI models that meta trained they gave it about a million hours of real world videos and then all it took is 62 hours of real robotics data to actually s80 00:06:26.479 --> 00:06:29.360 control to control a real robot. s81 00:06:29.360 --> 00:06:33.840 So if you think about it, the data was collected publicly, right? s82 00:06:33.840 --> 00:06:42.080 So it trained the the the robot on just random things and then in 62 robotic hours, it was already moving. s83 00:06:42.080 --> 00:06:49.199 So there no simulations needed, autonomous robotics delivered fast. s84 00:06:52.880 --> 00:06:55.199 But the videos don't show how the robots move, right? s85 00:06:55.199 --> 00:07:01.840 So you might think, well, listen, how useful is it a person pouring water into a cup for a robot. s86 00:07:01.840 --> 00:07:09.680 So there is methods actually out there how an AI can distinguish and actually learn from the actions that are in a video. s87 00:07:09.680 --> 00:07:15.599 If you take two frames frame by frame and the AI measures the movement this difference, right? s88 00:07:15.599 --> 00:07:24.880 So you have an image A and image two and there's a small movement that is changing between the two images and AI can actually measure that change. s89 00:07:24.880 --> 00:07:34.880 So if you keep doing that through the whole video and AI can actually figure out angles, distance and everything that it needs to train and robot right. s90 00:07:34.880 --> 00:07:48.240 So we don't necessarily need the sensor's data but of course the video itself that you download from video from YouTube or the video that uh you extract the data from doesn't necessarily get you 100% there. s91 00:07:48.240 --> 00:07:53.199 You do need to have some processing on top of that but it's available. s92 00:07:53.199 --> 00:07:58.400 It's out there and all you need to do is process it. s93 00:07:58.560 --> 00:08:00.080 So a few more examples right? s94 00:08:00.080 --> 00:08:07.919 So for example, Nvidia when they train in the Cosmos robot, they're throwing out about 96% of the video. s95 00:08:07.919 --> 00:08:09.440 So what does that mean? s96 00:08:09.440 --> 00:08:15.520 They download a million hours of video and the only part is 4% is actually useful. s97 00:08:15.520 --> 00:08:17.039 Everything else gets thrown out. s98 00:08:17.039 --> 00:08:18.400 That's wasted compute. s99 00:08:18.400 --> 00:08:19.599 That's wasted bandwidth. s100 00:08:19.599 --> 00:08:21.360 That's wasted storage. s101 00:08:21.360 --> 00:08:23.360 That's just a lot of wasted money. s102 00:08:23.360 --> 00:08:23.759 Okay. s103 00:08:23.759 --> 00:08:30.240 Stable video diffusion is throwing out 74% of the videos that they're downloading. s104 00:08:30.240 --> 00:08:37.760 So in bright data we are trying to solve that problem and we're trying to take it a step forward. s105 00:08:37.760 --> 00:08:41.599 So what we propose is search first collect second. s106 00:08:41.599 --> 00:08:50.320 What we do is we are actually indexing videos and we are allowing you to search for specific actions. s107 00:08:50.320 --> 00:08:55.680 Since we're talking about a person opening a door, let's keep using that uh example. s108 00:08:55.680 --> 00:09:01.440 What you can do on our platform is actually input detailed information, right? s109 00:09:01.440 --> 00:09:06.959 A query, person washing dishes, person folding a t-shirt and so on. s110 00:09:06.959 --> 00:09:13.920 And we will provide the snippets from the videos of these actions. s111 00:09:14.000 --> 00:09:16.640 So what does that mean? s112 00:09:16.640 --> 00:09:22.720 That means that there's less waste of data collection. s113 00:09:22.720 --> 00:09:25.519 there's less waste of storage and bandwidth, right? s114 00:09:25.519 --> 00:09:27.600 So, a little more on how it works. s115 00:09:27.600 --> 00:09:31.279 Obviously, you define what you're looking for. s116 00:09:32.320 --> 00:09:44.080 We search through billions and billions of pre-indexed videos, not by keywords, but by actions, and we provide you with ready to use clips. s117 00:09:44.080 --> 00:09:47.200 And obviously, they're already trimmed, prepared for your training. s118 00:09:47.200 --> 00:09:57.360 So all you got to do is process them a little bit for maybe motion distance sensors and so on and it's ready to be ingested into a robot. s119 00:09:57.839 --> 00:10:02.640 I have a small example of a video of how it actually works on our platform. s120 00:10:02.640 --> 00:10:07.120 So let me just hit play here if I can find it. s121 00:10:07.360 --> 00:10:16.320 So here we see person washing dishes by hand in sink removing grease from the plate using sponge and soap including rinse and placing dishes into drying rack. s122 00:10:16.320 --> 00:10:19.120 close-up hand interactions. s123 00:10:19.519 --> 00:10:24.720 So, right now what is happening is it's searching through billions of videos. s124 00:10:24.720 --> 00:10:26.480 Now, on this video, it's a bit old. s125 00:10:26.480 --> 00:10:31.279 We only got about 100 million index there, but now we're up to 1.1 billion videos. s126 00:10:31.279 --> 00:10:41.440 And it will literally like this is a little sped up, so I didn't want to waste your guys time, but it will provide to you clips from videos of people washing dishes. s127 00:10:41.440 --> 00:10:47.279 And you can see right here all the videos in That would be fine. s128 00:10:47.279 --> 00:10:50.560 Now, some of these videos might not have to do anything with washing dishes. s129 00:10:50.560 --> 00:10:52.399 They might have cleaning kitchen. s130 00:10:52.399 --> 00:10:54.480 They might have to do something else. s131 00:10:54.480 --> 00:10:55.839 And here's another example. s132 00:10:55.839 --> 00:11:00.000 For example, human folding different types of clothes, right? s133 00:11:00.000 --> 00:11:01.839 So, a little more description. s134 00:11:01.839 --> 00:11:06.640 The more description you give, the more you actually get back. s135 00:11:06.959 --> 00:11:08.959 So, again, a little speed up. s136 00:11:08.959 --> 00:11:11.279 Let's see. s137 00:11:14.480 --> 00:11:16.240 I want you to understand it could be anything. s138 00:11:16.240 --> 00:11:17.680 A person putting on makeup. s139 00:11:17.680 --> 00:11:21.600 Let's say you have a brand and you want to find videos where your brand is being found. s140 00:11:21.600 --> 00:11:24.000 All that can be also provided. s141 00:11:24.000 --> 00:11:26.880 So you see people folding clothes. s142 00:11:26.880 --> 00:11:31.200 So now you can train a robot on how to fold clothes. s143 00:11:34.560 --> 00:11:36.640 Of course everything is available via API. s144 00:11:36.640 --> 00:11:39.600 We don't expect people to actually do anything manually, right? s145 00:11:39.600 --> 00:11:41.200 So you can trigger it by API. s146 00:11:41.200 --> 00:11:46.480 you get a returned snippet of the video, the URL. s147 00:11:46.480 --> 00:11:49.600 We give you the link to the original video if you want to watch it. s148 00:11:49.600 --> 00:11:57.440 But the main thing here is that we can give you the snippets of the video to train your robotics. s149 00:11:57.440 --> 00:12:05.040 Now, I'm not sure if any of you training any robots, so I'm going to give you another example of how useful it could be. s150 00:12:05.040 --> 00:12:13.680 Let's say you have a brand and you want to know where your brand is being demoed on videos. s151 00:12:13.839 --> 00:12:17.440 The video titles doesn't matter because it doesn't actually consume. s152 00:12:17.440 --> 00:12:19.120 It's not about your product. s153 00:12:19.120 --> 00:12:23.040 It's just a person doing some podcast recording something. s154 00:12:23.040 --> 00:12:30.079 But you see a woman putting on makeup and she you see on the table that this is your brand of a makeup. s155 00:12:30.079 --> 00:12:40.240 So now all of a sudden you can find all videos for your brand where it's being used whatever it is right by searching name of the video you won't find it by by s156 00:12:40.240 --> 00:12:48.959 video indexing you can actually find specific things that you are interested in apple falling from the tree and so on. s157 00:12:48.959 --> 00:12:50.639 So what comes back? s158 00:12:50.639 --> 00:12:52.399 We provide to you this time stamp. s159 00:12:52.399 --> 00:12:58.079 We provide to you the score matching, how close it is to your query, and of course the frame count. s160 00:12:58.079 --> 00:13:03.360 How many frames are about what you're looking for? s161 00:13:04.079 --> 00:13:06.399 So what does this change? s162 00:13:06.399 --> 00:13:07.600 This changes a lot. s163 00:13:07.600 --> 00:13:09.760 Okay, less waste. s164 00:13:09.760 --> 00:13:12.880 You don't need to download million of hours of videos. s165 00:13:12.880 --> 00:13:20.560 you can actually get exactly the the specific actions that you are interested in. s166 00:13:21.040 --> 00:13:29.760 Again, it's very crucial for robotic training because noise creates problems, hallucinations, right? s167 00:13:29.760 --> 00:13:32.880 This removes the noise. s168 00:13:34.160 --> 00:13:35.839 What is it useful for? s169 00:13:35.839 --> 00:13:40.560 Not only robots, but obviously self-driving for example, Wimo, right? s170 00:13:40.560 --> 00:13:47.920 trained on dash cam video cameras and there's millions, hundreds of millions of hours on YouTube of dash cams. s171 00:13:47.920 --> 00:13:50.399 How many of you like to watch accidents? s172 00:13:50.399 --> 00:13:52.160 Dash cam c accidents. s173 00:13:52.160 --> 00:13:54.000 I'm sure everybody watched them. s174 00:13:54.000 --> 00:13:57.120 Crazy driving in Russia, right? s175 00:13:57.120 --> 00:13:58.880 So, this is what we're talking about, right? s176 00:13:58.880 --> 00:14:01.680 The videos that are out there, okay? s177 00:14:01.680 --> 00:14:07.120 A person running a red light, person stopping on red light, making a left turn, making a right turn, and so on. s178 00:14:07.120 --> 00:14:13.760 All of that can be a very useful data for training and any other world models. s179 00:14:13.760 --> 00:14:14.880 Physics, right? s180 00:14:14.880 --> 00:14:17.120 Robots needs to understand physics. s181 00:14:17.120 --> 00:14:18.480 What is the gravity? s182 00:14:18.480 --> 00:14:19.199 How to sit? s183 00:14:19.199 --> 00:14:19.839 How to walk? s184 00:14:19.839 --> 00:14:21.839 How to move? s185 00:14:21.839 --> 00:14:24.880 Again, of course, you can pre-record it. s186 00:14:24.880 --> 00:14:30.639 Um, a lot of times right now I speak to some of the people and they have millions of people recording videos for them. s187 00:14:30.639 --> 00:14:33.360 Hey, can you guys record me how you open doors? s188 00:14:33.360 --> 00:14:34.800 That's crazy. s189 00:14:34.800 --> 00:14:40.000 Why do you need a million people doing it when everything is available online? s190 00:14:40.000 --> 00:14:43.360 Online is one of the hugest databases out there. s191 00:14:43.360 --> 00:14:46.720 It's just people don't really like it's it's not that easy to use it, right? s192 00:14:46.720 --> 00:14:53.920 I mean, right now the only search available to us is by the keyword of the name of the video. s193 00:14:53.920 --> 00:14:59.519 And of course, one source, the whole public video web in one place. s194 00:14:59.519 --> 00:15:14.959 I mean, we got YouTube, we got Vimeo, we got so many different uh video providers out there, billions and billions of hours of training data just there waiting for you to grab it, collect it and process it. s195 00:15:14.959 --> 00:15:28.560 So basically this is a new product that we're doing a bright data right indexing videos so that you guys can find specific actions anything you can think about you know again it could be useful for brands s196 00:15:28.560 --> 00:15:47.760 uh as well it's not only for training data anything you can think of could be useful maybe you are a gamer and you want to know how do you beat this level but there's no exactly video like that so you can search okay people beating the level on the video game, right? s197 00:15:47.760 --> 00:15:49.279 Anything. s198 00:15:49.279 --> 00:15:51.920 The world is yours. s199 00:15:52.240 --> 00:15:54.720 So, I'm going to wrap it up. s200 00:15:54.720 --> 00:16:02.160 I don't know if you have any questions, but uh if you want to connect with me, talk about it more, feel free to to add me on LinkedIn. s201 00:16:02.160 --> 00:16:06.800 And um yeah, I'm going to thank you guys for your attention. s202 00:16:06.800 --> 00:16:11.199 And uh I'm at a bright data booth if you want to talk more about it. s203 00:16:11.199 --> 00:16:14.880 And thanks for coming.