When Agents Meet Physical Data: The Other Physics of Agent Harnesses - Dmitry Petrov, DataChain

AI Engineer · 27 min · 263 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:02How could intelligent work with physical data?
00:06Video recordings, sensor data, robots telemetry, and sometimes all of those combined in a single multi-model project.
00:14If you try this, you probably have seen how badly it fails.
00:18And today we'll discuss the reasons and all the physics laws behind the problems and how to fix it.
00:30This year two frontier labs published very interesting results and surprising results that agents in general are not good at data.
00:43Uh, Anthropic published that accuracy for data projects on their agents is only 21% until you add specific data harnesses to them and provide context.
00:57OpenAI published a whole layers of context, six layers of context in order to make the data agent to work.
01:07And all of those are unstructured data which lives in a very houses with tables, execution engine, and all this like luxury.
01:20In my life, I don't have this luxury unfortunately because I live in a very extreme side of the data universe, messy unstructured data.
01:32Uh, I work with data for about 10 years.
01:35I built data version control project, Git for data, and now work on data chain.
01:45In order to make agent to work for unstructured data, for physical data, we need to build not only the brain which we already have, right?
01:58It's LLM.
02:00Uh but we need to make harness data harness to for agents to understand this physical world.
02:08It should see the data properly.
02:10It should be able to run kind of giving him a leg.
02:14Uh it should be able to touch data, verify the result, run tests, and also remember the crucial important data sets, important result.
02:25So, this information could be reused uh in the future project.
02:32And the first we start from the how you see data.
02:37Uh what does it take to understand all these complicated binary files that you have in your object storages?
02:47Uh in reality, how it usually looks, uh they are just a several files, sometimes several thousand of files, right?
02:55It doesn't seems like a big deal, right?
02:57Uh but what is usually happens, those files are very complex inside.
03:04That what makes unstructured multimodal data complicated.
03:09And from 2000 because video recordings might have clips inside.
03:14Clips might have frames.
03:16Frames objects.
03:18Objects confident, type, class, label.
03:23Uh there's connection between those pieces together.
03:27It makes kind of like explosion.
03:29It's it looks like uh neutron star.
03:33It small size on the surface, right?
03:36It's just a size of a one city.
03:40But the mass of this object is tremendous.
03:42It's more than the mass of our sun.
03:45Uh and what 2000 objects, 2000 files of videos could easily generate you millions of objects inside the videos.
03:56And how people usually do deal with these problems?
03:58They usually go through like a two major steps.
04:02First step, let's put this meta information to JSON files and put it on S3 next to the images, right?
04:09And they end up with a millions of JSONs.
04:13Crazy latency, not efficiency, not consistency.
04:18And the next idea, why don't we use database?
04:22Brilliant.
04:23And the most advanced team do exactly this.
04:26Let's put a centralized database when all the meta metadata is uh in there.
04:33Great.
04:34But this way you end up with a two system with a two programming languages and all the mess around uh two different stacks.
04:42And this stacks useless for most of the researchers because they don't want to deal with this complexity.
04:51We found that the easiest way for researchers and developers to deal with the schema is Pydantic.
04:58Uh so you use the same language for the data, for the schemas, as well as code.
05:03Uh there are no SQL island in your code base.
05:07So in a that way you kind of transition from the messy world of structured data to the structure.
05:15The only thing you need to just like transpliers to the SQL uh from from Python and I'll show you uh how it works.
05:23But that's the way when you can create schemas uh and work with the schemas efficiently.
05:29I want to emphasize that the problem we are solving here it's very specific for structured data, physical data.
05:36It doesn't exist in the structured data world, right?
05:39For example, in those uh OpenAI and uh Anthropic blog posts because they work with the structured business logic, right?
05:47We work with the physical data, binaries and such.
05:52Let's see how it works.
05:55I will be showing you our open source project data chain with data harness to coding agents.
06:04So, first you need to install the tool, right?
06:07Using pip install.
06:08So, we have already done this.
06:11Then, skill needs to be installed.
06:14So, you choose the coding tool that you use.
06:20We will be using cloud code.
06:22But, we support three more different uh coding agents.
06:29And then, you run your favorite coding coding agents.
06:35Uh of course, we are skipping permissions.
06:38Uh just to make it faster, right?
06:41And then, you define the prompt and just solve your problems.
06:45So, in this case, we will be analyzing uh motions uh in video cameras.
06:54Uh that's open data set from um uh dashcam recordings.
07:00Uh video it's a very usual use case for uh physical AI projects.
07:08Uh majority of the project do include uh video recordings uh as a one of the modalities.
07:14Sometimes, the major modalities.
07:16Uh and that's one of the I would say more interesting and more challenging problems to solve uh in this uh physical AI world.
07:24Let's use uh this modality as an example.
07:28This data harness asks uh some questions from users to understand the scope better.
07:36So, in this case, it uh we need to choose the model.
07:39So, it decided to use a YOLO model and we need to choose the size.
07:43So, let's use the smallest one.
07:46Uh we need to choose the velocity.
07:51Uh so, there's a few ways how to track like speed.
07:55Uh but, let's use the simplest one.
07:58So, the second one uh adds more uh will require more time for the compute.
08:07And, granularity, so per detection frame, per track.
08:13Uh let's do just per frame in this case.
08:18And, the data set itself.
08:20So, we ask for January data, right?
08:2391 clip, but it also ask for uh if we need to extend scope and uh analyze more images.
08:32So, let's let's stay here.
08:39So, it took us 24 minutes to analyze those uh 90 videos.
08:46And, now we have all the information, we can ask uh questions about it.
08:51For example, how many um videos have people in it?
09:15This information lives uh in the database, right?
09:20As we discussed, and it can very quickly query this database and return the result, right?
09:27So, okay.
09:2992 uh oh, 82 out of 91 clips have some people uh detected, right?
09:38And uh you don't need to kind of go through all the JSON files download and analyze and parse, right?
09:48Information is here in the local database.
09:50It can easily quickly answer the question.
09:54We can even see the source code.
10:08[snorts]
10:11Okay, just that easy and nice Python code to get get data get data data set, right?
10:23Analyze apply some simple filters when label is equal to person.
10:31Count the number, right?
10:34And then see the total total number.
10:37And it goes like it runs against database so super fast.
10:41So, how big is how many records do we have here?
10:50So, that's about the size of our neutron star, right?
10:54It's not supposed to be big because we analyze only like 90 videos and 90 videos generated 100,000 of records, right?
11:06So, that's how explosion is happening.
11:09If you analyze thousands of videos, right?
11:13It goes to like a million million scale and if you go to deeper to the objects, it can easily it can easily be multiplied by 10 20 even hundreds.
11:32Let's take a look at the source code.
11:36This code was generated by the agents.
11:41Uh power agents using the harness.
11:44Uh The magic part here is the data model.
11:50So, this data class uh that was generated based on our requirements and it's a just a by dentic uh usual by dentic data model.
12:00Uh with the file.
12:03Uh right, the video file.
12:05Uh frame ID, timestamp, uh class ID, that's how you know if it's a personal car or some object, confidence score, bounding box, there a few more items here.
12:18And the object is nested.
12:22If you look at the If you take a look at the bounding box, right?
12:25There's a several columns, but files is more interesting.
12:30One.
12:31Uh file has uh path and the checksums like a version of the file, e-tag, size of the file.
12:40So, everything that's your cloud storage providers provides for you.
12:45Uh and this information is just become a row in a database uh instead of a JSON file in S3.
12:55Uh that's why you can easily uh answer uh all the uh questions.
13:01Uh all the analytical questions, right?
13:04Like before we asked how many people are there.
13:07Uh and it's just a matter of like a simple Python code that runs against the database uh to return like results super quickly.
13:18Uh that's how you can make sense of all the messy data you have on your storage, right?
13:25In a file, uh but you analyze uh you analyze this a using using regular analytical tools and two tricks.
13:39With the schemas, you can see what is inside the object.
13:42You can see the scale of the problem.
13:45But, how to do the actual heavy lifting when we when we deal with the terabytes of this messy data?
13:53You do know that the data harness have to have execution engine in order to deal with physical data.
14:05It has to work efficiently.
14:07Sometimes it spend a lot of resources and tokens if you sell them.
14:13And if it breaks somewhere in the middle, so you should be able to recover and catch up with all the data that it's already processed.
14:23So, you don't want to waste your resources.
14:25In the sequel or structured data world, execution engine is your data warehouse, right?
14:31That's obvious and easy.
14:33In an unstructured data world, you have to reinvent one.
14:37And in some teams, they use different types of orchestration tools or distributed computer on Ray or Spark.
14:46I like model of Dask which connects computer with the data structured data.
14:53But, unstructured data needs its kind of like own approach when you connect the Python and schemas with the execution engine.
15:07To simplify it, you can connect those Python functions, Pydantic schemas, input input parameters, output parameters for the function, as well as the data warehouse with all the meta information
15:24and files on the storage all together in a in a holistic experience.
15:29Uh that's what this way it will be more simple for developers to code and it might be simple for the engine to distribute jobs to different machines or different threads
15:43based on the files they process, the schema they produce.
15:46Uh so forth and so on.
15:47It's kind of like a Dask approach if you wish but for unstructured data world.
15:54To make the execution efficient, uh you need to run this uh data processing data crunching function very efficiently, right?
16:03In a parallel way or distributed way.
16:07And we use data models and the data storage models a lot.
16:14So, it kind of connects the file which is a file in your S3 bucket uh to the result uh that's a set of detection object which goes to the data warehouse, right?
16:30So, this function connects the storage and the data warehouse in a regular Python way.
16:36Uh so, that's a very typical Python code.
16:40There are no like extra assumptions about this code, right?
16:44We don't even use like annotation for the function.
16:46It's just a function with a specified types.
16:49So, the types are important.
16:51Uh and all of those are PyDantic.
16:54And result you just generate uh generate the objects and the engine connects all the storages and uh warehouse together, right?
17:08That's how you define the parallel uh parallelization layer.
17:13You can say uh Uh, I just need to run it in 40 machines, right?
17:21That's how easy it's supposed to be for is your chance to run like a distributed computer, right?
17:25You just specify how many resources you need, so you got it.
17:29And the function itself, right?
17:31So, we just use it run it through generator or it can be like mapper with one-to-one function, one function.
17:38Uh, one file returns like one object to the database or generator, one function returns like multiple objects in database.
17:45And it is saved as a like data set, right?
17:49Uh, uh, table under the hood, uh, in your uh, database.
18:00Processing of those heavy files usually takes a lot of time.
18:06And sometimes it's very expensive because you use LLM in order to understand what is inside uh, of your binary files.
18:15And the last thing you want is to lose this compute.
18:18It's really sad when you process uh, hundreds of thousands of files and it fails in the middle because of bugs or API call issue and you want to execute everything from scratch,
18:32right?
18:32You are losing the whole half of this compute and sometimes doing this over and over again.
18:38Incremental update and data checkpoints it's a must-have in this data world.
18:46If you fail, you fix the bug, run it again and catch up all the result that you have already.
18:54If you got more files in the bucket, uh, you run the script again and it gets only the new files and uh, update data based on the new computer only without recomputing the
19:07the other stuff.
19:10Running tests is a very fundamental part of coding agents.
19:15They do this all the time for quality control, for reasoning, to better understand what is around in their universe.
19:22It's kind of like a hands of your agents, hands of your uh harness.
19:29In data, running test is super slow process.
19:36And how to do this?
19:37Because quality and accuracy of the questions it's even more important in the data question data projects compared to software projects.
19:46Uh as Anthropic put it in a right way in uh software projects, there's a lot of ways how to you can solve a particular problem.
19:55In data, there is usually only one way and only one current correct answer uh to solve the problem.
20:06How to answer questions very fast on your binary data mass.
20:15And first of all, you should stop running all this like complicated Python scripts on top of raw data.
20:21That's the most expensive, the slowest way of doing this.
20:24Instead, you need to organize layer of meta information over like data sets uh or tables around this uh raw data uh that could answer these questions very quickly.
20:36And this something that data industry knows for like a dozen of years.
20:41It's a dimensional multi-dimensional data modeling uh star schema, one big table approach, and all this like fancy theories around around this.
20:51And we need to use this more in order to make our unstructured meta data processing uh to be more efficient.
21:01And we actually incorporated these techniques inside the agents.
21:05And when you ask a question, instead of answering the question right away, agent ask itself, do I have a proper data sets, proper metadata to answer this question quickly in a single like
21:21SQL-ish query.
21:23And if not, it tries to build this layer.
21:27It tries to make sure this layer is general enough to answer not your particular question, but a set of questions related to the one that you asked.
21:38And that's where you are building these layers and layers of information that's could be reused by you and your teammates.
21:50Now, you spent a lot of time, a lot of resources to compute those useful slices of the data, and you can use those to answer some interesting questions uh efficiently
22:04without really running expensive compute.
22:08And guess what?
22:09People are doing the same job over and over and over again.
22:14You're paying double, triple, quadruple price to solving the same problem.
22:20So, if you discover this neutron star or black hole, it's better to share this information with people uh so they won't be wasting their resources and their time uh to doing this stuff.
22:38And coding agent, as well as data agent, doing this those tricks with memory a lot.
22:45Uh your coding agent like Copilot, Codex, PI knows a lot about your source code, about with all the indexes uh and such.
22:56Uh on the data harness, you need to build and provide this context uh to your agents.
23:06And that's not only the fact that hey, there is a data set, you need to provide a lot more information with this.
23:13Uh why this data set was built, kind of like a context from the session.
23:18Uh description of the data set, which usually enrich by LLMs, source code.
23:25Probably the most important part here is the source code.
23:27Uh that's one of the conclusion in OpenAI data agent blog post.
23:33Uh and this information needs to be exposed uh exposed in a way like knowledge base.
23:40Some way that can easily be used by agents, by people.
23:44Uh so you are not wasting this time over and over again.
23:48And the knowledge base is organized in a very traditional way, I would say.
23:54Uh just a set of MD files, right?
23:56So uh you see this is how it looks like in the in my directory, but when I get more data sets.
24:03So the data set that we created, that one.
24:08Uh this is the MD file.
24:11Uh so description of the data set, right?
24:14The session context.
24:16Uh so why this data set was created based on the discussion.
24:20Dependency to the storage, to the directory that we pointed uh to in the beginning.
24:27Uh preview of the data.
24:29Uh very useful information to kind of have a sense of the data, this schema.
24:37Uh some stats on the data.
24:39And the source code.
24:41Uh as we discussed, the most crucial part uh to understand uh what is data is about.
24:52And all these pieces together, right?
24:54Uh the source data, right?
24:57In the bucket, the source code uh in the knowledge base, and the result in the uh in the data warehouse, create like a data lineage, right?
25:10So, everything is connected.
25:13Agents knows everything.
25:15If you share the knowledge graph, then all your teammates already know about this uh about this data set, about the resources you spent in order to process.
25:25So, next time if someone ask question about this directory, uh the recompute won't happen.
25:32Uh agents will be using uh your your result, uh the result which based on resources that you already that you have already spent, right?
25:42That's the magic of the uh data harnesses when agent knows everything about your data.
25:52Let's put all the pieces together in a single stack.
25:58In the bottom of the stack, there is some huge mass of your unstructured physical data in object storages.
26:09No one makes sense of this data, and you need to run expensive compute LM calls to extract some meta information through the compute engine, and organize this meta information in a data sets, in a data set slices
26:24through the some uh data set GB.
26:27Knowledge base is a way how you share this information.
26:33This is a world when your favorite coding agents, such as Copilot, Codex, Cloud Code, do not operate efficiently.
26:45Their intuition pushes them in a wrong direction because laws of physics changes.
26:54And in order to make it, you don't use stronger models.
26:56Everyone use frontiers.
26:59Instead, you are building data harness.
27:02Data harness that understand the laws of this physical data.
27:09And that's the way how to make your favorite coding agent efficient with these problems.
27:19We implement some of those principles in Data Chain project, which is open source.
27:24So, please check it out and put some mass on your agent.
27:30Thank you.