WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=B8l81jhvHbI NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/B8l81jhvHbI.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.280 --> 00:00:06.680 Hello everyone, and welcome to my talk, Enterprise Agents Have a Structural Problem. s2 00:00:06.680 --> 00:00:08.360 A quick introduction about myself. s3 00:00:08.360 --> 00:00:09.400 I'm Ishita Daga. s4 00:00:09.400 --> 00:00:17.240 I work as a machine learning engineer in Tesla, and I'm building enterprise agents for our organization. s5 00:00:18.960 --> 00:00:28.040 Before we get into the framework and the solution space, I actually want to talk about why these agents fail in the first place. s6 00:00:28.040 --> 00:00:41.240 So, when an agent gives a bad answer, the first reflect that we have is that we need a bigger model, we need a the latest model, we want a model with a lot of context so that s7 00:00:41.240 --> 00:00:55.240 it can hold a lot of information, or we need to add a lot of knowledge base in terms of dot MD files, documents, or MCP servers, plugins, etc. But, while all of these are fair solutions, s8 00:00:55.240 --> 00:01:00.560 they're not the answer to actually improving the data agent itself. s9 00:01:00.880 --> 00:01:10.120 Model The agent actually has these three main problems. s10 00:01:10.120 --> 00:01:11.720 One is ambiguity. s11 00:01:11.720 --> 00:01:23.520 So, it doesn't know what table is right, what column is right, which data source, or which knowledge base to access when, which one holds the source of truth, or like the cleanest information, s12 00:01:23.520 --> 00:01:26.320 which one is just a bit more flexible. s13 00:01:26.320 --> 00:01:37.360 So, we have a lot of these knowledge bases, but it's not defined which one is the best, um, or which one to use when. s14 00:01:37.360 --> 00:01:40.920 So, ambiguity, I feel, is one big problem. s15 00:01:40.920 --> 00:01:42.200 Second is staleness. s16 00:01:42.200 --> 00:01:45.560 We all know context updates so quickly. s17 00:01:45.560 --> 00:01:51.960 Um, a lot of decisions change, a lot of definition, KPIs, they change so often. s18 00:01:51.960 --> 00:01:54.640 Processes, they get updated so often. s19 00:01:54.640 --> 00:02:03.120 But, to keep our data agents accurate, we need to update this context, this context life cycle. s20 00:02:03.120 --> 00:02:08.039 And that's a second part of the problem. s21 00:02:08.039 --> 00:02:25.080 The last one is preference, which is more so capturing an individual or a team preference on what metric to use to answer a specific question, how to use a query, or what filters to use in a query. s22 00:02:25.080 --> 00:02:34.000 Although we have like a canonical query, let's say, each team has a different preference on what filters or what definitions to use. s23 00:02:34.000 --> 00:02:46.200 So, this I feel is very open-ended problem, still needs a lot of research, a lot of um industries or the frontier labs are actually working on solving this one. s24 00:02:46.440 --> 00:02:49.880 So, let's start with the first one, ambiguity. s25 00:02:49.880 --> 00:02:56.360 The agent needs to understand what source of truth to use, or what knowledge base to use. s26 00:02:56.360 --> 00:02:59.000 It can't weight all the knowledge bases equally. s27 00:02:59.000 --> 00:03:06.320 There has to be a bit more structure to the one it chooses to answer a particular question. s28 00:03:06.519 --> 00:03:19.080 In my opinion, the source of truth actually is a hierarchy, which goes from the cleanest, least flexible source of truth to the messiest, uh but most flexible, most dynamic source of truth. s29 00:03:19.080 --> 00:03:26.360 And the way to answer any question is to start from the cleanest to the messiest or the most dynamic one. s30 00:03:26.360 --> 00:03:32.320 We have divided in the sources of truth in three different buckets uh for this framework. s31 00:03:32.320 --> 00:03:44.440 The first one being the semantic layer, which is the best source of truth, a very curated list of all the different queries, KPI definitions, metric uh or how to s32 00:03:44.440 --> 00:03:56.520 calculate the metric, um the business definitions, and everything combined into just one semantic layer, which the model can use to answer a query. s33 00:03:56.520 --> 00:04:08.120 So, the agent just looks into the semantic layer, finds the closest KPI, and references all of these different data points from that KPI to actually give you an answer. s34 00:04:08.120 --> 00:04:10.520 The second part is canonical tables. s35 00:04:10.520 --> 00:04:16.519 These are most of the parametric tables or um queries, sorry. s36 00:04:16.519 --> 00:04:28.680 Where you give the agent all of a list of different parametric queries, and it sort of tries to understand which ones can it use to actually answer your question. s37 00:04:28.680 --> 00:04:42.320 It gives it a bit more flexibility in choosing or writing its own query or its own filters, etc. And it can answer a bit more wider set of questions. s38 00:04:42.480 --> 00:04:51.800 The last one is the database graph, which I feel is the most trickiest because it takes a lot of effort, but it gives you a lot of flexibility in s39 00:04:51.800 --> 00:04:54.160 the kind of questions that can be answered. s40 00:04:54.160 --> 00:05:09.520 So, the idea here is you connect each table to all the different columns uh each table to the metric that can be answered through this table and vice versa and sort of create this s41 00:05:09.520 --> 00:05:18.880 um enormous database graph that can be served to the agent and it can answer or query on top of it. s42 00:05:18.880 --> 00:05:25.240 Well, it is tricky, it is also very difficult to maintain or update. s43 00:05:25.240 --> 00:05:33.080 So, I feel if an enterprise is approaching or adding sources of truth, they should start adding the first and the second layer first. s44 00:05:33.080 --> 00:05:42.360 These are easy to set up, solve I think 80% of the problems, and then the 20% can be solved by the database graph. s45 00:05:43.160 --> 00:05:46.400 Moving on to our second problem, which is tailness. s46 00:05:46.400 --> 00:05:48.520 I think the word speaks for itself. s47 00:05:48.520 --> 00:06:05.080 But, the context gets rotten, or it gets deprecated, or processes change so often that it's hard to maintain dot MD files, or keep on updating your skills with the most latest context. s48 00:06:05.080 --> 00:06:10.800 It's just not easy, and we need a better solution for that. s49 00:06:11.120 --> 00:06:17.440 My answer to this problem is creating a context life cycle, which basically has two different components. s50 00:06:17.440 --> 00:06:24.360 The first one being embedding life data sources, or knowledge sources. s51 00:06:24.360 --> 00:06:39.360 By life, I mean something or the data sources that are going to be always updated, would be reviewed and well curated, and always provide the newest, or the most up-to-date s52 00:06:39.360 --> 00:06:40.000 data. s53 00:06:40.000 --> 00:06:53.560 This can include your GitHub, your um CRM tools, your Tableau, or DBT, semantic layers, whatever you want to include can be there, but it has to be a mandatory s54 00:06:53.560 --> 00:06:57.200 data source, like a a data source that is updated most often. s55 00:06:57.200 --> 00:07:06.800 The second part is a feedback loop, which is something that a lot of enterprise or data agents actually miss. s56 00:07:06.800 --> 00:07:12.440 The idea here is you should be able to log each and every uh event. s57 00:07:12.440 --> 00:07:22.200 And by event, I mean whenever someone says that this database is incorrect, or this definition definition is incorrect, or updated, or there's a new way to calculate a metric, s58 00:07:22.200 --> 00:07:25.080 or the there is a new filter that you need to use. s59 00:07:25.080 --> 00:07:35.640 All of these events need to be captured, logged, and used to update your data agent context. s60 00:07:35.640 --> 00:07:51.760 Once you have captured these events, the second part is to actually evaluate your data agent, and you can do that by either curating an evaluation suite with human annotated questions or human s61 00:07:51.760 --> 00:08:05.600 evaluations, or you can create an automated evaluation where you see all the questions that have been asked in the last couple of days and the actual answers and how close the new answers are to s62 00:08:05.600 --> 00:08:07.520 the actually the actual answers. s63 00:08:07.520 --> 00:08:18.960 So, I think there are a lot of ways you can do that, but a lot of teams do not focus on evaluation that much, and which is why the agents actually fail so often because you don't know s64 00:08:18.960 --> 00:08:22.720 how the agent is progressing or how is the performance. s65 00:08:22.720 --> 00:08:24.680 You don't keep a track of that. s66 00:08:24.680 --> 00:08:36.039 So, I feel creating this feedback loop or a context loop where you log, evaluate, update the context every or every so often is very important. s67 00:08:37.159 --> 00:08:51.200 The last problem is preference, which is a very subjective problem because preferences they change very often in terms of what kind of metric should be used, what's the right calculation. s68 00:08:51.200 --> 00:08:57.680 Different teams will calculate the same metric differently or um use the same query but different filters. s69 00:08:57.680 --> 00:09:07.000 So, there is a lot of subjectivity which needs to be captured, but it's very hard to do that on individual or team level. s70 00:09:07.280 --> 00:09:21.280 To give an example, um so, let's say there is team A and team B, and they don't both want to calculate the average time uh taken for a particular milestone. s71 00:09:21.280 --> 00:09:34.200 Team A calculates it by looking at the previous milestones or completion of the previous milestone to completion of the current milestone, while team B actually looks at the start of the current milestone to the start of the next milestone. s72 00:09:34.200 --> 00:09:41.600 So, while both of these are correct metrics or the correct way to calculate the metric. s73 00:09:41.600 --> 00:09:45.320 They both will give you very different answers and it's just about preference. s74 00:09:45.320 --> 00:09:54.280 We just need to understand what an individual mean when they say what's the average milestone time. s75 00:09:54.400 --> 00:10:04.840 So while this is a great question, I feel the industry still does not have a correct answer or correct way to solve the problem itself. s76 00:10:04.840 --> 00:10:07.880 I feel like there are sort of two ways to solve it. s77 00:10:07.880 --> 00:10:11.720 One is a semantic layer, which is something that I'm trying. s78 00:10:11.720 --> 00:10:22.640 Wherein you store all the different ways to calculate a metric as a semantic layer and then the individual sort of can prompt the model which one do they want to use. s79 00:10:22.640 --> 00:10:27.720 So you can solve it but still is not storing the preference of the user. s80 00:10:27.720 --> 00:10:33.400 And the second thing is you again bump into the first challenge, which is ambiguity. s81 00:10:33.400 --> 00:10:37.600 The agent doesn't know which metric is the right one. s82 00:10:37.600 --> 00:10:39.800 You still have to prompt it. s83 00:10:39.800 --> 00:10:49.760 Second part is agent memory, which is your mem zero or like storing memory.md file, but it still is not the best way. s84 00:10:49.760 --> 00:10:58.600 It stores your preference but it can't understand the distinction between two different metrics or which one to use when. s85 00:10:58.600 --> 00:11:06.440 So I feel there are ways to solve it but none of the ways are actually solving the preference problem itself. s86 00:11:06.440 --> 00:11:19.880 What we actually want is a way to route the agent to the right metric based on who what what a team or which individual is using that agent. s87 00:11:19.880 --> 00:11:29.800 So still an open-ended problem but yeah, I think something we should work on or research on a bit more. s88 00:11:29.920 --> 00:11:34.560 Which brings us to the end of this presentation. s89 00:11:34.560 --> 00:11:40.840 So just to conclude, I feel there are these three big problems, ambiguity, staleness, and preference. s90 00:11:40.840 --> 00:11:49.520 And what we really need to solve for is a better structure, a better way to manage the context and evaluate the context. s91 00:11:49.520 --> 00:11:52.840 And then preference, which is such an open-ended problem. s92 00:11:52.840 --> 00:12:04.560 It requires not just understanding how the agent works, but also embedding an individual preference, like creating a hive mind for your data agent. s93 00:12:04.560 --> 00:12:05.520 That's about it. s94 00:12:05.520 --> 00:12:07.960 Thank you so much.