WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=bUJgirn4_yc NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/bUJgirn4_yc.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:02.080 --> 00:00:06.120 How could intelligent work with physical data? s2 00:00:06.120 --> 00:00:14.960 Video recordings, sensor data, robots telemetry, and sometimes all of those combined in a single multi-model project. s3 00:00:14.960 --> 00:00:18.440 If you try this, you probably have seen how badly it fails. s4 00:00:18.440 --> 00:00:28.840 And today we'll discuss the reasons and all the physics laws behind the problems and how to fix it. s5 00:00:30.080 --> 00:00:43.200 This year two frontier labs published very interesting results and surprising results that agents in general are not good at data. s6 00:00:43.200 --> 00:00:57.280 Uh, Anthropic published that accuracy for data projects on their agents is only 21% until you add specific data harnesses to them and provide context. s7 00:00:57.280 --> 00:01:07.640 OpenAI published a whole layers of context, six layers of context in order to make the data agent to work. s8 00:01:07.640 --> 00:01:20.200 And all of those are unstructured data which lives in a very houses with tables, execution engine, and all this like luxury. s9 00:01:20.200 --> 00:01:31.760 In my life, I don't have this luxury unfortunately because I live in a very extreme side of the data universe, messy unstructured data. s10 00:01:32.080 --> 00:01:35.640 Uh, I work with data for about 10 years. s11 00:01:35.640 --> 00:01:43.360 I built data version control project, Git for data, and now work on data chain. s12 00:01:45.400 --> 00:01:58.680 In order to make agent to work for unstructured data, for physical data, we need to build not only the brain which we already have, right? s13 00:01:58.680 --> 00:02:00.040 It's LLM. s14 00:02:00.040 --> 00:02:08.320 Uh but we need to make harness data harness to for agents to understand this physical world. s15 00:02:08.320 --> 00:02:10.880 It should see the data properly. s16 00:02:10.880 --> 00:02:14.520 It should be able to run kind of giving him a leg. s17 00:02:14.520 --> 00:02:25.640 Uh it should be able to touch data, verify the result, run tests, and also remember the crucial important data sets, important result. s18 00:02:25.640 --> 00:02:31.240 So, this information could be reused uh in the future project. s19 00:02:32.640 --> 00:02:37.400 And the first we start from the how you see data. s20 00:02:37.400 --> 00:02:46.720 Uh what does it take to understand all these complicated binary files that you have in your object storages? s21 00:02:47.400 --> 00:02:55.400 Uh in reality, how it usually looks, uh they are just a several files, sometimes several thousand of files, right? s22 00:02:55.400 --> 00:02:57.800 It doesn't seems like a big deal, right? s23 00:02:57.800 --> 00:03:04.440 Uh but what is usually happens, those files are very complex inside. s24 00:03:04.440 --> 00:03:09.120 That what makes unstructured multimodal data complicated. s25 00:03:09.120 --> 00:03:14.880 And from 2000 because video recordings might have clips inside. s26 00:03:14.880 --> 00:03:16.600 Clips might have frames. s27 00:03:16.600 --> 00:03:18.280 Frames objects. s28 00:03:18.280 --> 00:03:23.239 Objects confident, type, class, label. s29 00:03:23.239 --> 00:03:27.160 Uh there's connection between those pieces together. s30 00:03:27.160 --> 00:03:29.040 It makes kind of like explosion. s31 00:03:29.040 --> 00:03:33.959 It's it looks like uh neutron star. s32 00:03:33.959 --> 00:03:36.920 It small size on the surface, right? s33 00:03:36.920 --> 00:03:40.000 It's just a size of a one city. s34 00:03:40.000 --> 00:03:42.920 But the mass of this object is tremendous. s35 00:03:42.920 --> 00:03:45.600 It's more than the mass of our sun. s36 00:03:45.600 --> 00:03:56.080 Uh and what 2000 objects, 2000 files of videos could easily generate you millions of objects inside the videos. s37 00:03:56.280 --> 00:03:58.880 And how people usually do deal with these problems? s38 00:03:58.880 --> 00:04:02.000 They usually go through like a two major steps. s39 00:04:02.000 --> 00:04:09.600 First step, let's put this meta information to JSON files and put it on S3 next to the images, right? s40 00:04:09.600 --> 00:04:13.040 And they end up with a millions of JSONs. s41 00:04:13.040 --> 00:04:18.000 Crazy latency, not efficiency, not consistency. s42 00:04:18.000 --> 00:04:22.840 And the next idea, why don't we use database? s43 00:04:22.840 --> 00:04:23.520 Brilliant. s44 00:04:23.520 --> 00:04:26.480 And the most advanced team do exactly this. s45 00:04:26.480 --> 00:04:33.000 Let's put a centralized database when all the meta metadata is uh in there. s46 00:04:33.000 --> 00:04:34.480 Great. s47 00:04:34.480 --> 00:04:42.480 But this way you end up with a two system with a two programming languages and all the mess around uh two different stacks. s48 00:04:42.480 --> 00:04:49.320 And this stacks useless for most of the researchers because they don't want to deal with this complexity. s49 00:04:51.040 --> 00:04:58.120 We found that the easiest way for researchers and developers to deal with the schema is Pydantic. s50 00:04:58.120 --> 00:05:03.840 Uh so you use the same language for the data, for the schemas, as well as code. s51 00:05:03.840 --> 00:05:07.160 Uh there are no SQL island in your code base. s52 00:05:07.160 --> 00:05:15.080 So in a that way you kind of transition from the messy world of structured data to the structure. s53 00:05:15.080 --> 00:05:23.200 The only thing you need to just like transpliers to the SQL uh from from Python and I'll show you uh how it works. s54 00:05:23.200 --> 00:05:29.120 But that's the way when you can create schemas uh and work with the schemas efficiently. s55 00:05:29.120 --> 00:05:36.280 I want to emphasize that the problem we are solving here it's very specific for structured data, physical data. s56 00:05:36.280 --> 00:05:39.280 It doesn't exist in the structured data world, right? s57 00:05:39.280 --> 00:05:47.280 For example, in those uh OpenAI and uh Anthropic blog posts because they work with the structured business logic, right? s58 00:05:47.280 --> 00:05:52.200 We work with the physical data, binaries and such. s59 00:05:52.200 --> 00:05:54.880 Let's see how it works. s60 00:05:55.440 --> 00:06:04.280 I will be showing you our open source project data chain with data harness to coding agents. s61 00:06:04.280 --> 00:06:07.000 So, first you need to install the tool, right? s62 00:06:07.000 --> 00:06:08.360 Using pip install. s63 00:06:08.360 --> 00:06:11.560 So, we have already done this. s64 00:06:11.560 --> 00:06:14.520 Then, skill needs to be installed. s65 00:06:14.520 --> 00:06:20.320 So, you choose the coding tool that you use. s66 00:06:20.320 --> 00:06:22.200 We will be using cloud code. s67 00:06:22.200 --> 00:06:29.680 But, we support three more different uh coding agents. s68 00:06:29.680 --> 00:06:35.760 And then, you run your favorite coding coding agents. s69 00:06:35.760 --> 00:06:38.400 Uh of course, we are skipping permissions. s70 00:06:38.400 --> 00:06:41.680 Uh just to make it faster, right? s71 00:06:41.680 --> 00:06:45.919 And then, you define the prompt and just solve your problems. s72 00:06:45.919 --> 00:06:54.080 So, in this case, we will be analyzing uh motions uh in video cameras. s73 00:06:54.080 --> 00:07:00.640 Uh that's open data set from um uh dashcam recordings. s74 00:07:00.640 --> 00:07:08.120 Uh video it's a very usual use case for uh physical AI projects. s75 00:07:08.120 --> 00:07:14.120 Uh majority of the project do include uh video recordings uh as a one of the modalities. s76 00:07:14.120 --> 00:07:16.280 Sometimes, the major modalities. s77 00:07:16.280 --> 00:07:24.840 Uh and that's one of the I would say more interesting and more challenging problems to solve uh in this uh physical AI world. s78 00:07:24.840 --> 00:07:28.919 Let's use uh this modality as an example. s79 00:07:28.919 --> 00:07:36.680 This data harness asks uh some questions from users to understand the scope better. s80 00:07:36.680 --> 00:07:39.840 So, in this case, it uh we need to choose the model. s81 00:07:39.840 --> 00:07:43.600 So, it decided to use a YOLO model and we need to choose the size. s82 00:07:43.600 --> 00:07:46.440 So, let's use the smallest one. s83 00:07:46.440 --> 00:07:51.480 Uh we need to choose the velocity. s84 00:07:51.480 --> 00:07:55.440 Uh so, there's a few ways how to track like speed. s85 00:07:55.440 --> 00:07:58.320 Uh but, let's use the simplest one. s86 00:07:58.320 --> 00:08:07.640 So, the second one uh adds more uh will require more time for the compute. s87 00:08:07.640 --> 00:08:13.480 And, granularity, so per detection frame, per track. s88 00:08:13.480 --> 00:08:18.880 Uh let's do just per frame in this case. s89 00:08:18.880 --> 00:08:20.880 And, the data set itself. s90 00:08:20.880 --> 00:08:23.200 So, we ask for January data, right? s91 00:08:23.200 --> 00:08:32.240 91 clip, but it also ask for uh if we need to extend scope and uh analyze more images. s92 00:08:32.240 --> 00:08:35.840 So, let's let's stay here. s93 00:08:39.080 --> 00:08:46.960 So, it took us 24 minutes to analyze those uh 90 videos. s94 00:08:46.960 --> 00:08:51.640 And, now we have all the information, we can ask uh questions about it. s95 00:08:51.640 --> 00:09:07.480 For example, how many um videos have people in it? s96 00:09:15.760 --> 00:09:20.480 This information lives uh in the database, right? s97 00:09:20.480 --> 00:09:27.880 As we discussed, and it can very quickly query this database and return the result, right? s98 00:09:27.880 --> 00:09:29.080 So, okay. s99 00:09:29.080 --> 00:09:38.840 92 uh oh, 82 out of 91 clips have some people uh detected, right? s100 00:09:38.840 --> 00:09:48.000 And uh you don't need to kind of go through all the JSON files download and analyze and parse, right? s101 00:09:48.000 --> 00:09:50.480 Information is here in the local database. s102 00:09:50.480 --> 00:09:54.920 It can easily quickly answer the question. s103 00:09:54.920 --> 00:10:00.840 We can even see the source code. s104 00:10:08.501 --> 00:10:10.501 [snorts] s105 00:10:11.920 --> 00:10:23.440 Okay, just that easy and nice Python code to get get data get data data set, right? s106 00:10:23.440 --> 00:10:31.200 Analyze apply some simple filters when label is equal to person. s107 00:10:31.200 --> 00:10:34.440 Count the number, right? s108 00:10:34.440 --> 00:10:37.680 And then see the total total number. s109 00:10:37.680 --> 00:10:41.960 And it goes like it runs against database so super fast. s110 00:10:41.960 --> 00:10:50.160 So, how big is how many records do we have here? s111 00:10:50.160 --> 00:10:54.720 So, that's about the size of our neutron star, right? s112 00:10:54.720 --> 00:11:06.680 It's not supposed to be big because we analyze only like 90 videos and 90 videos generated 100,000 of records, right? s113 00:11:06.680 --> 00:11:09.480 So, that's how explosion is happening. s114 00:11:09.480 --> 00:11:13.640 If you analyze thousands of videos, right? s115 00:11:13.640 --> 00:11:27.680 It goes to like a million million scale and if you go to deeper to the objects, it can easily it can easily be multiplied by 10 20 even hundreds. s116 00:11:32.760 --> 00:11:36.440 Let's take a look at the source code. s117 00:11:36.960 --> 00:11:41.560 This code was generated by the agents. s118 00:11:41.560 --> 00:11:44.720 Uh power agents using the harness. s119 00:11:44.720 --> 00:11:50.080 Uh The magic part here is the data model. s120 00:11:50.080 --> 00:12:00.600 So, this data class uh that was generated based on our requirements and it's a just a by dentic uh usual by dentic data model. s121 00:12:00.600 --> 00:12:03.560 Uh with the file. s122 00:12:03.560 --> 00:12:05.840 Uh right, the video file. s123 00:12:05.840 --> 00:12:18.720 Uh frame ID, timestamp, uh class ID, that's how you know if it's a personal car or some object, confidence score, bounding box, there a few more items here. s124 00:12:18.720 --> 00:12:22.360 And the object is nested. s125 00:12:22.360 --> 00:12:25.320 If you look at the If you take a look at the bounding box, right? s126 00:12:25.320 --> 00:12:30.040 There's a several columns, but files is more interesting. s127 00:12:30.040 --> 00:12:31.120 One. s128 00:12:31.120 --> 00:12:40.600 Uh file has uh path and the checksums like a version of the file, e-tag, size of the file. s129 00:12:40.600 --> 00:12:45.280 So, everything that's your cloud storage providers provides for you. s130 00:12:45.280 --> 00:12:55.040 Uh and this information is just become a row in a database uh instead of a JSON file in S3. s131 00:12:55.040 --> 00:13:01.800 Uh that's why you can easily uh answer uh all the uh questions. s132 00:13:01.800 --> 00:13:04.320 Uh all the analytical questions, right? s133 00:13:04.320 --> 00:13:07.800 Like before we asked how many people are there. s134 00:13:07.800 --> 00:13:18.400 Uh and it's just a matter of like a simple Python code that runs against the database uh to return like results super quickly. s135 00:13:18.400 --> 00:13:25.480 Uh that's how you can make sense of all the messy data you have on your storage, right? s136 00:13:25.480 --> 00:13:37.400 In a file, uh but you analyze uh you analyze this a using using regular analytical tools and two tricks. s137 00:13:39.040 --> 00:13:42.720 With the schemas, you can see what is inside the object. s138 00:13:42.720 --> 00:13:45.040 You can see the scale of the problem. s139 00:13:45.040 --> 00:13:53.880 But, how to do the actual heavy lifting when we when we deal with the terabytes of this messy data? s140 00:13:53.880 --> 00:14:05.080 You do know that the data harness have to have execution engine in order to deal with physical data. s141 00:14:05.080 --> 00:14:07.240 It has to work efficiently. s142 00:14:07.240 --> 00:14:13.640 Sometimes it spend a lot of resources and tokens if you sell them. s143 00:14:13.640 --> 00:14:23.480 And if it breaks somewhere in the middle, so you should be able to recover and catch up with all the data that it's already processed. s144 00:14:23.480 --> 00:14:25.920 So, you don't want to waste your resources. s145 00:14:25.920 --> 00:14:31.240 In the sequel or structured data world, execution engine is your data warehouse, right? s146 00:14:31.240 --> 00:14:33.480 That's obvious and easy. s147 00:14:33.480 --> 00:14:37.839 In an unstructured data world, you have to reinvent one. s148 00:14:37.839 --> 00:14:46.200 And in some teams, they use different types of orchestration tools or distributed computer on Ray or Spark. s149 00:14:46.200 --> 00:14:53.880 I like model of Dask which connects computer with the data structured data. s150 00:14:53.880 --> 00:15:06.640 But, unstructured data needs its kind of like own approach when you connect the Python and schemas with the execution engine. s151 00:15:07.480 --> 00:15:24.200 To simplify it, you can connect those Python functions, Pydantic schemas, input input parameters, output parameters for the function, as well as the data warehouse with all the meta information s152 00:15:24.200 --> 00:15:29.880 and files on the storage all together in a in a holistic experience. s153 00:15:29.880 --> 00:15:43.040 Uh that's what this way it will be more simple for developers to code and it might be simple for the engine to distribute jobs to different machines or different threads s154 00:15:43.040 --> 00:15:46.480 based on the files they process, the schema they produce. s155 00:15:46.480 --> 00:15:47.440 Uh so forth and so on. s156 00:15:47.440 --> 00:15:53.960 It's kind of like a Dask approach if you wish but for unstructured data world. s157 00:15:54.080 --> 00:16:03.560 To make the execution efficient, uh you need to run this uh data processing data crunching function very efficiently, right? s158 00:16:03.560 --> 00:16:07.720 In a parallel way or distributed way. s159 00:16:07.720 --> 00:16:14.160 And we use data models and the data storage models a lot. s160 00:16:14.160 --> 00:16:30.000 So, it kind of connects the file which is a file in your S3 bucket uh to the result uh that's a set of detection object which goes to the data warehouse, right? s161 00:16:30.000 --> 00:16:36.960 So, this function connects the storage and the data warehouse in a regular Python way. s162 00:16:36.960 --> 00:16:40.520 Uh so, that's a very typical Python code. s163 00:16:40.520 --> 00:16:44.200 There are no like extra assumptions about this code, right? s164 00:16:44.200 --> 00:16:46.760 We don't even use like annotation for the function. s165 00:16:46.760 --> 00:16:49.640 It's just a function with a specified types. s166 00:16:49.640 --> 00:16:51.600 So, the types are important. s167 00:16:51.600 --> 00:16:54.320 Uh and all of those are PyDantic. s168 00:16:54.320 --> 00:17:08.839 And result you just generate uh generate the objects and the engine connects all the storages and uh warehouse together, right? s169 00:17:08.839 --> 00:17:13.480 That's how you define the parallel uh parallelization layer. s170 00:17:13.480 --> 00:17:21.079 You can say uh Uh, I just need to run it in 40 machines, right? s171 00:17:21.079 --> 00:17:25.760 That's how easy it's supposed to be for is your chance to run like a distributed computer, right? s172 00:17:25.760 --> 00:17:29.640 You just specify how many resources you need, so you got it. s173 00:17:29.640 --> 00:17:31.200 And the function itself, right? s174 00:17:31.200 --> 00:17:38.480 So, we just use it run it through generator or it can be like mapper with one-to-one function, one function. s175 00:17:38.480 --> 00:17:45.240 Uh, one file returns like one object to the database or generator, one function returns like multiple objects in database. s176 00:17:45.240 --> 00:17:49.600 And it is saved as a like data set, right? s177 00:17:49.600 --> 00:17:57.280 Uh, uh, table under the hood, uh, in your uh, database. s178 00:18:00.960 --> 00:18:06.960 Processing of those heavy files usually takes a lot of time. s179 00:18:06.960 --> 00:18:15.240 And sometimes it's very expensive because you use LLM in order to understand what is inside uh, of your binary files. s180 00:18:15.240 --> 00:18:18.920 And the last thing you want is to lose this compute. s181 00:18:18.920 --> 00:18:32.600 It's really sad when you process uh, hundreds of thousands of files and it fails in the middle because of bugs or API call issue and you want to execute everything from scratch, s182 00:18:32.600 --> 00:18:32.920 right? s183 00:18:32.920 --> 00:18:38.840 You are losing the whole half of this compute and sometimes doing this over and over again. s184 00:18:38.840 --> 00:18:46.600 Incremental update and data checkpoints it's a must-have in this data world. s185 00:18:46.600 --> 00:18:54.320 If you fail, you fix the bug, run it again and catch up all the result that you have already. s186 00:18:54.320 --> 00:19:07.600 If you got more files in the bucket, uh, you run the script again and it gets only the new files and uh, update data based on the new computer only without recomputing the s187 00:19:07.600 --> 00:19:10.200 the other stuff. s188 00:19:10.760 --> 00:19:15.240 Running tests is a very fundamental part of coding agents. s189 00:19:15.240 --> 00:19:22.920 They do this all the time for quality control, for reasoning, to better understand what is around in their universe. s190 00:19:22.920 --> 00:19:29.480 It's kind of like a hands of your agents, hands of your uh harness. s191 00:19:29.480 --> 00:19:36.600 In data, running test is super slow process. s192 00:19:36.600 --> 00:19:37.520 And how to do this? s193 00:19:37.520 --> 00:19:46.440 Because quality and accuracy of the questions it's even more important in the data question data projects compared to software projects. s194 00:19:46.440 --> 00:19:55.080 Uh as Anthropic put it in a right way in uh software projects, there's a lot of ways how to you can solve a particular problem. s195 00:19:55.080 --> 00:20:04.720 In data, there is usually only one way and only one current correct answer uh to solve the problem. s196 00:20:06.480 --> 00:20:15.640 How to answer questions very fast on your binary data mass. s197 00:20:15.640 --> 00:20:21.720 And first of all, you should stop running all this like complicated Python scripts on top of raw data. s198 00:20:21.720 --> 00:20:24.880 That's the most expensive, the slowest way of doing this. s199 00:20:24.880 --> 00:20:36.880 Instead, you need to organize layer of meta information over like data sets uh or tables around this uh raw data uh that could answer these questions very quickly. s200 00:20:36.880 --> 00:20:41.240 And this something that data industry knows for like a dozen of years. s201 00:20:41.240 --> 00:20:51.200 It's a dimensional multi-dimensional data modeling uh star schema, one big table approach, and all this like fancy theories around around this. s202 00:20:51.200 --> 00:21:01.600 And we need to use this more in order to make our unstructured meta data processing uh to be more efficient. s203 00:21:01.600 --> 00:21:05.600 And we actually incorporated these techniques inside the agents. s204 00:21:05.600 --> 00:21:21.320 And when you ask a question, instead of answering the question right away, agent ask itself, do I have a proper data sets, proper metadata to answer this question quickly in a single like s205 00:21:21.320 --> 00:21:23.360 SQL-ish query. s206 00:21:23.360 --> 00:21:27.280 And if not, it tries to build this layer. s207 00:21:27.280 --> 00:21:38.200 It tries to make sure this layer is general enough to answer not your particular question, but a set of questions related to the one that you asked. s208 00:21:38.200 --> 00:21:47.960 And that's where you are building these layers and layers of information that's could be reused by you and your teammates. s209 00:21:50.040 --> 00:22:04.280 Now, you spent a lot of time, a lot of resources to compute those useful slices of the data, and you can use those to answer some interesting questions uh efficiently s210 00:22:04.280 --> 00:22:08.080 without really running expensive compute. s211 00:22:08.080 --> 00:22:09.600 And guess what? s212 00:22:09.600 --> 00:22:14.680 People are doing the same job over and over and over again. s213 00:22:14.680 --> 00:22:20.880 You're paying double, triple, quadruple price to solving the same problem. s214 00:22:20.880 --> 00:22:34.800 So, if you discover this neutron star or black hole, it's better to share this information with people uh so they won't be wasting their resources and their time uh to doing this stuff. s215 00:22:38.240 --> 00:22:45.040 And coding agent, as well as data agent, doing this those tricks with memory a lot. s216 00:22:45.040 --> 00:22:56.720 Uh your coding agent like Copilot, Codex, PI knows a lot about your source code, about with all the indexes uh and such. s217 00:22:56.720 --> 00:23:05.920 Uh on the data harness, you need to build and provide this context uh to your agents. s218 00:23:06.280 --> 00:23:13.560 And that's not only the fact that hey, there is a data set, you need to provide a lot more information with this. s219 00:23:13.560 --> 00:23:18.560 Uh why this data set was built, kind of like a context from the session. s220 00:23:18.560 --> 00:23:25.280 Uh description of the data set, which usually enrich by LLMs, source code. s221 00:23:25.280 --> 00:23:27.920 Probably the most important part here is the source code. s222 00:23:27.920 --> 00:23:33.840 Uh that's one of the conclusion in OpenAI data agent blog post. s223 00:23:33.840 --> 00:23:40.200 Uh and this information needs to be exposed uh exposed in a way like knowledge base. s224 00:23:40.200 --> 00:23:44.520 Some way that can easily be used by agents, by people. s225 00:23:44.520 --> 00:23:48.960 Uh so you are not wasting this time over and over again. s226 00:23:48.960 --> 00:23:54.400 And the knowledge base is organized in a very traditional way, I would say. s227 00:23:54.400 --> 00:23:56.520 Uh just a set of MD files, right? s228 00:23:56.520 --> 00:24:03.720 So uh you see this is how it looks like in the in my directory, but when I get more data sets. s229 00:24:03.720 --> 00:24:08.600 So the data set that we created, that one. s230 00:24:08.600 --> 00:24:11.760 Uh this is the MD file. s231 00:24:11.760 --> 00:24:14.680 Uh so description of the data set, right? s232 00:24:14.680 --> 00:24:16.320 The session context. s233 00:24:16.320 --> 00:24:20.200 Uh so why this data set was created based on the discussion. s234 00:24:20.200 --> 00:24:27.440 Dependency to the storage, to the directory that we pointed uh to in the beginning. s235 00:24:27.440 --> 00:24:29.080 Uh preview of the data. s236 00:24:29.080 --> 00:24:37.280 Uh very useful information to kind of have a sense of the data, this schema. s237 00:24:37.280 --> 00:24:39.720 Uh some stats on the data. s238 00:24:39.720 --> 00:24:41.880 And the source code. s239 00:24:41.880 --> 00:24:52.080 Uh as we discussed, the most crucial part uh to understand uh what is data is about. s240 00:24:52.080 --> 00:24:54.880 And all these pieces together, right? s241 00:24:54.880 --> 00:24:57.920 Uh the source data, right? s242 00:24:57.920 --> 00:25:10.480 In the bucket, the source code uh in the knowledge base, and the result in the uh in the data warehouse, create like a data lineage, right? s243 00:25:10.480 --> 00:25:13.000 So, everything is connected. s244 00:25:13.000 --> 00:25:15.160 Agents knows everything. s245 00:25:15.160 --> 00:25:25.280 If you share the knowledge graph, then all your teammates already know about this uh about this data set, about the resources you spent in order to process. s246 00:25:25.280 --> 00:25:32.200 So, next time if someone ask question about this directory, uh the recompute won't happen. s247 00:25:32.200 --> 00:25:42.000 Uh agents will be using uh your your result, uh the result which based on resources that you already that you have already spent, right? s248 00:25:42.000 --> 00:25:50.880 That's the magic of the uh data harnesses when agent knows everything about your data. s249 00:25:52.760 --> 00:25:58.120 Let's put all the pieces together in a single stack. s250 00:25:58.120 --> 00:26:09.000 In the bottom of the stack, there is some huge mass of your unstructured physical data in object storages. s251 00:26:09.000 --> 00:26:24.280 No one makes sense of this data, and you need to run expensive compute LM calls to extract some meta information through the compute engine, and organize this meta information in a data sets, in a data set slices s252 00:26:24.280 --> 00:26:27.840 through the some uh data set GB. s253 00:26:27.840 --> 00:26:33.560 Knowledge base is a way how you share this information. s254 00:26:33.840 --> 00:26:45.280 This is a world when your favorite coding agents, such as Copilot, Codex, Cloud Code, do not operate efficiently. s255 00:26:45.600 --> 00:26:53.840 Their intuition pushes them in a wrong direction because laws of physics changes. s256 00:26:54.080 --> 00:26:56.920 And in order to make it, you don't use stronger models. s257 00:26:56.920 --> 00:26:59.040 Everyone use frontiers. s258 00:26:59.040 --> 00:27:02.720 Instead, you are building data harness. s259 00:27:02.720 --> 00:27:09.640 Data harness that understand the laws of this physical data. s260 00:27:09.640 --> 00:27:16.160 And that's the way how to make your favorite coding agent efficient with these problems. s261 00:27:19.600 --> 00:27:24.920 We implement some of those principles in Data Chain project, which is open source. s262 00:27:24.920 --> 00:27:30.240 So, please check it out and put some mass on your agent. s263 00:27:30.240 --> 00:27:32.440 Thank you.