{ "video": { "id": "B8l81jhvHbI", "title": "Enterprise Agents Have a Structure Problem - Ishita Daga, Tesla", "duration": 728, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "Online" }, "analysis": { "video_id": "B8l81jhvHbI", "title": "Enterprise Agents Have a Structure Problem - Ishita Daga, Tesla", "one_liner": "A Tesla ML engineer argues enterprise data agents fail not from weak models but from three structural gaps — ambiguity, staleness and preference — and prescribes a source-of-truth hierarchy (semantic layer → canonical/parametric queries → database graph) plus a logged, evaluated context lifecycle.", "summary": "Ishita Daga, an ML engineer building enterprise agents at Tesla, says the reflex to fix a bad agent answer — a bigger model, more context, more .MD files or MCP servers — is fair but not the actual fix. The real problems are structural: ambiguity (the agent doesn't know which table, column, data source or knowledge base is the source of truth), staleness (KPI definitions and processes change faster than documentation), and preference (teams calculate the same metric differently). Her answers are a ranked source-of-truth hierarchy that the agent walks from cleanest to messiest, and a context lifecycle built from always-live data sources plus a feedback loop that logs correction events and feeds an evaluation suite. She presents preference as still unsolved by either semantic layers or agent memory, and calls for routing an agent to the right metric based on who is asking.", "key_points": [ "Three structural failure modes for enterprise data agents: ambiguity (which knowledge base or table is the source of truth), staleness (context rots as KPIs, definitions and processes change), and preference (individual/team-specific metric definitions).", "An agent can't weight all knowledge bases equally; source of truth is a hierarchy ordered from cleanest/least flexible to messiest/most flexible, and questions should be answered starting at the cleanest layer.", "Layer 1 — the semantic layer: a curated set of queries, KPI definitions, metric calculations and business definitions; the agent finds the closest KPI and references its data points to answer.", "Layer 2 — canonical tables / parametric queries: a list of parametric queries the agent picks from, with more freedom to write its own filters, covering a wider question set.", "Layer 3 — the database graph: each table linked to its columns and to the metrics it can answer (and vice versa); most flexible but takes the most effort and is hardest to maintain.", "Sequencing advice: layers 1 and 2 are easy to set up and solve about 80% of the problem; the remaining 20% is what the database graph is for.", "The staleness fix is a context lifecycle with two parts: embedding 'live' mandatory data sources that are always updated, reviewed and curated (GitHub, CRM tools, Tableau, dbt semantic layers), and a feedback loop.", "The feedback loop logs every correction event — this database is wrong, this definition changed, there's a new way to calculate a metric, a new filter is needed — then feeds context updates and evaluation, either a human-annotated eval suite or automated evals comparing new answers against recent questions' actual answers.", "Preference worked example: to get average milestone time, team A measures completion of the previous milestone to completion of the current one; team B measures start of the current milestone to start of the next. Both are correct and give very different answers.", "Neither existing approach solves preference: a semantic layer storing all calculation variants still needs the user to prompt which one (falling back into ambiguity), and agent memory (mem0, a memory.md file) stores a preference but can't tell which metric to use when." ], "takeaways": [ "Stop reaching first for a bigger model, a longer context window, or more .MD files and MCP servers when an agent answers badly — fix the structure of the context instead.", "Rank your knowledge sources explicitly rather than exposing them flat, and have the agent resolve from the cleanest source down; build the semantic layer and canonical parametric queries before attempting a database graph.", "Build the feedback loop most enterprise data agents are missing: log every 'that definition is wrong' event as structured data and use it to update agent context on a cycle.", "Actually evaluate the agent — a human-annotated suite or an automated comparison against recent real questions and answers — because without it you have no idea whether performance is improving or degrading.", "Treat per-team metric preference as an open research problem, and aim for routing to the right metric based on who is asking rather than assuming a stored memory file will settle it." ], "topics": [ "agents", "enterprise", "semantic-layer", "evals", "context-engineering", "data-agents", "agent-memory", "text-to-sql" ], "tools": [ "Tesla", "MCP servers", "GitHub", "Tableau", "dbt", "mem0", "memory.md" ], "quotes": [ { "text": "when an agent gives a bad answer, the first reflect that we have is that we need a bigger model, we need a the latest model", "at": "00:28", "url": "https://www.youtube.com/watch?v=B8l81jhvHbI&t=28s" }, { "text": "In my opinion, the source of truth actually is a hierarchy, which goes from the cleanest, least flexible source of truth to the messiest, uh but most flexible, most dynamic source of truth.", "at": "03:06", "url": "https://www.youtube.com/watch?v=B8l81jhvHbI&t=186s" }, { "text": "a lot of teams do not focus on evaluation that much, and which is why the agents actually fail so often because you don't know how the agent is progressing", "at": "08:09", "url": "https://www.youtube.com/watch?v=B8l81jhvHbI&t=489s" }, { "text": "It requires not just understanding how the agent works, but also embedding an individual preference, like creating a hive mind for your data agent.", "at": "11:53", "url": "https://www.youtube.com/watch?v=B8l81jhvHbI&t=713s" } ], "words": 2046 }, "summary_url": "/#B8l81jhvHbI", "transcript": { "html": "/transcripts/B8l81jhvHbI.html", "txt": "/transcripts/B8l81jhvHbI.txt", "vtt": "/transcripts/B8l81jhvHbI.vtt" } }