Enterprise Agents Have a Structure Problem - Ishita Daga, Tesla

AI Engineer · 12 min · 94 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01Hello everyone, and welcome to my talk, Enterprise Agents Have a Structural Problem.
2. 00:06A quick introduction about myself.
3. 00:08I'm Ishita Daga.
4. 00:09I work as a machine learning engineer in Tesla, and I'm building enterprise agents for our organization.
5. 00:18Before we get into the framework and the solution space, I actually want to talk about why these agents fail in the first place.
6. 00:28So, when an agent gives a bad answer, the first reflect that we have is that we need a bigger model, we need a the latest model, we want a model with a lot of context so that
7. 00:41it can hold a lot of information, or we need to add a lot of knowledge base in terms of dot MD files, documents, or MCP servers, plugins, etc. But, while all of these are fair solutions,
8. 00:55they're not the answer to actually improving the data agent itself.
9. 01:00Model The agent actually has these three main problems.
10. 01:10One is ambiguity.
11. 01:11So, it doesn't know what table is right, what column is right, which data source, or which knowledge base to access when, which one holds the source of truth, or like the cleanest information,
12. 01:23which one is just a bit more flexible.
13. 01:26So, we have a lot of these knowledge bases, but it's not defined which one is the best, um, or which one to use when.
14. 01:37So, ambiguity, I feel, is one big problem.
15. 01:40Second is staleness.
16. 01:42We all know context updates so quickly.
17. 01:45Um, a lot of decisions change, a lot of definition, KPIs, they change so often.
18. 01:51Processes, they get updated so often.
19. 01:54But, to keep our data agents accurate, we need to update this context, this context life cycle.
20. 02:03And that's a second part of the problem.
21. 02:08The last one is preference, which is more so capturing an individual or a team preference on what metric to use to answer a specific question, how to use a query, or what filters to use in a query.
22. 02:25Although we have like a canonical query, let's say, each team has a different preference on what filters or what definitions to use.
23. 02:34So, this I feel is very open-ended problem, still needs a lot of research, a lot of um industries or the frontier labs are actually working on solving this one.
24. 02:46So, let's start with the first one, ambiguity.
25. 02:49The agent needs to understand what source of truth to use, or what knowledge base to use.
26. 02:56It can't weight all the knowledge bases equally.
27. 02:59There has to be a bit more structure to the one it chooses to answer a particular question.
28. 03:06In my opinion, the source of truth actually is a hierarchy, which goes from the cleanest, least flexible source of truth to the messiest, uh but most flexible, most dynamic source of truth.
29. 03:19And the way to answer any question is to start from the cleanest to the messiest or the most dynamic one.
30. 03:26We have divided in the sources of truth in three different buckets uh for this framework.
31. 03:32The first one being the semantic layer, which is the best source of truth, a very curated list of all the different queries, KPI definitions, metric uh or how to
32. 03:44calculate the metric, um the business definitions, and everything combined into just one semantic layer, which the model can use to answer a query.
33. 03:56So, the agent just looks into the semantic layer, finds the closest KPI, and references all of these different data points from that KPI to actually give you an answer.
34. 04:08The second part is canonical tables.
35. 04:10These are most of the parametric tables or um queries, sorry.
36. 04:16Where you give the agent all of a list of different parametric queries, and it sort of tries to understand which ones can it use to actually answer your question.
37. 04:28It gives it a bit more flexibility in choosing or writing its own query or its own filters, etc. And it can answer a bit more wider set of questions.
38. 04:42The last one is the database graph, which I feel is the most trickiest because it takes a lot of effort, but it gives you a lot of flexibility in
39. 04:51the kind of questions that can be answered.
40. 04:54So, the idea here is you connect each table to all the different columns uh each table to the metric that can be answered through this table and vice versa and sort of create this
41. 05:09um enormous database graph that can be served to the agent and it can answer or query on top of it.
42. 05:18Well, it is tricky, it is also very difficult to maintain or update.
43. 05:25So, I feel if an enterprise is approaching or adding sources of truth, they should start adding the first and the second layer first.
44. 05:33These are easy to set up, solve I think 80% of the problems, and then the 20% can be solved by the database graph.
45. 05:43Moving on to our second problem, which is tailness.
46. 05:46I think the word speaks for itself.
47. 05:48But, the context gets rotten, or it gets deprecated, or processes change so often that it's hard to maintain dot MD files, or keep on updating your skills with the most latest context.
48. 06:05It's just not easy, and we need a better solution for that.
49. 06:11My answer to this problem is creating a context life cycle, which basically has two different components.
50. 06:17The first one being embedding life data sources, or knowledge sources.
51. 06:24By life, I mean something or the data sources that are going to be always updated, would be reviewed and well curated, and always provide the newest, or the most up-to-date
52. 06:39data.
53. 06:40This can include your GitHub, your um CRM tools, your Tableau, or DBT, semantic layers, whatever you want to include can be there, but it has to be a mandatory
54. 06:53data source, like a a data source that is updated most often.
55. 06:57The second part is a feedback loop, which is something that a lot of enterprise or data agents actually miss.
56. 07:06The idea here is you should be able to log each and every uh event.
57. 07:12And by event, I mean whenever someone says that this database is incorrect, or this definition definition is incorrect, or updated, or there's a new way to calculate a metric,
58. 07:22or the there is a new filter that you need to use.
59. 07:25All of these events need to be captured, logged, and used to update your data agent context.
60. 07:35Once you have captured these events, the second part is to actually evaluate your data agent, and you can do that by either curating an evaluation suite with human annotated questions or human
61. 07:51evaluations, or you can create an automated evaluation where you see all the questions that have been asked in the last couple of days and the actual answers and how close the new answers are to
62. 08:05the actually the actual answers.
63. 08:07So, I think there are a lot of ways you can do that, but a lot of teams do not focus on evaluation that much, and which is why the agents actually fail so often because you don't know
64. 08:18how the agent is progressing or how is the performance.
65. 08:22You don't keep a track of that.
66. 08:24So, I feel creating this feedback loop or a context loop where you log, evaluate, update the context every or every so often is very important.
67. 08:37The last problem is preference, which is a very subjective problem because preferences they change very often in terms of what kind of metric should be used, what's the right calculation.
68. 08:51Different teams will calculate the same metric differently or um use the same query but different filters.
69. 08:57So, there is a lot of subjectivity which needs to be captured, but it's very hard to do that on individual or team level.
70. 09:07To give an example, um so, let's say there is team A and team B, and they don't both want to calculate the average time uh taken for a particular milestone.
71. 09:21Team A calculates it by looking at the previous milestones or completion of the previous milestone to completion of the current milestone, while team B actually looks at the start of the current milestone to the start of the next milestone.
72. 09:34So, while both of these are correct metrics or the correct way to calculate the metric.
73. 09:41They both will give you very different answers and it's just about preference.
74. 09:45We just need to understand what an individual mean when they say what's the average milestone time.
75. 09:54So while this is a great question, I feel the industry still does not have a correct answer or correct way to solve the problem itself.
76. 10:04I feel like there are sort of two ways to solve it.
77. 10:07One is a semantic layer, which is something that I'm trying.
78. 10:11Wherein you store all the different ways to calculate a metric as a semantic layer and then the individual sort of can prompt the model which one do they want to use.
79. 10:22So you can solve it but still is not storing the preference of the user.
80. 10:27And the second thing is you again bump into the first challenge, which is ambiguity.
81. 10:33The agent doesn't know which metric is the right one.
82. 10:37You still have to prompt it.
83. 10:39Second part is agent memory, which is your mem zero or like storing memory.md file, but it still is not the best way.
84. 10:49It stores your preference but it can't understand the distinction between two different metrics or which one to use when.
85. 10:58So I feel there are ways to solve it but none of the ways are actually solving the preference problem itself.
86. 11:06What we actually want is a way to route the agent to the right metric based on who what what a team or which individual is using that agent.
87. 11:19So still an open-ended problem but yeah, I think something we should work on or research on a bit more.
88. 11:29Which brings us to the end of this presentation.
89. 11:34So just to conclude, I feel there are these three big problems, ambiguity, staleness, and preference.
90. 11:40And what we really need to solve for is a better structure, a better way to manage the context and evaluate the context.
91. 11:49And then preference, which is such an open-ended problem.
92. 11:52It requires not just understanding how the agent works, but also embedding an individual preference, like creating a hive mind for your data agent.
93. 12:04That's about it.
94. 12:05Thank you so much.