{
  "video": {
    "id": "KhRjivBSwiw",
    "title": "The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef",
    "duration": 6381,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "KhRjivBSwiw",
    "title": "The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef",
    "one_liner": "Agents don't fail at data work because they can't write SQL; they fail because they lack context about grain, join safety, metric meaning and dependencies. Compute that context deterministically into a graph they can traverse, then use evals to prove the agent actually uses it instead of grepping anyway.",
    "summary": "Yoni Michael argues that the bottleneck for data agents is missing context, not the model. Typedef's 'data context layer' is a graph built mostly deterministically: SQL parsed down to the AST, tables profiled, and LLMs used only for naming and summarizing. It is refreshed atomically on every commit, and in Typedef's evals it lifts task success by almost 35 points. Brandon Callender then walks through building a code-context graph over the Qdrant Rust codebase (tree-sitter, rustdoc JSON, Neo4j, a Pydantic AI agent). His main lesson is that the agent will grep anyway: the work is getting it to use the graph and measuring that with pass^k and tool-adoption evals.",
    "key_points": [
      "Coding agents fail on data in three concrete places: impact analysis (what breaks if I change this), join recommendations (is this join safe) and safe code changes (does this edit preserve the metric's meaning). Four facts decide whether an answer is right: grain (what one row means), join safety, metric meaning and dependencies. None of them is written down anywhere the agent can read.",
      "Example: a pipeline's row count jumps 40% overnight. The agent writes valid SQL and a nice chart but blames 'more data'. The real cause was an upstream column turning a one-to-one join into one-to-many, i.e. a grain change. Grain isn't in the data, so valid SQL isn't enough.",
      "Typedef computes grain and lineage deterministically by parsing transformation SQL to the AST and using relational algebra. It verifies join-key uniqueness by profiling warehouse tables and uses the LLM only for business names, summaries and ambiguity. The result is a graph that is refreshed atomically on each commit and that the agent queries by subgraph traversal, which scales 'way past 500 models' without prompt stuffing.",
      "Typedef's eval-driven development uses real data-engineering tasks from easy to 'blind', run multiple times on isolated clones of the data environment. With the same models and tasks, the data context layer raised task success by almost 35 points. Their benchmark corpus is Mattermost's open-source dbt project (about 300–350 models) with simulated data.",
      "Brandon's three levels of context: level 1 is grep/glob raw strings; level 2 is semantic/lexical search (code RAG) that still returns a flat list; level 3 is a traversable graph with real symbol identity and typed edges. Rung 0 is tree-sitter syntax, rung 1 is name resolution via rustdoc JSON (which separates the two Qdrant PostingList structs), and rung 2 is meaning via LLM summaries with Typedef's open-source Fenic library.",
      "Given graph tools alongside grep, the agent 'ran grep anyway' because of post-training reflexes. What worked was naming the tool 'search' (the agent then does multi-keyword queries), returning results in ripgrep's format, and making sure the first call always pays off, because after an empty result the agent decides the tool is broken and never comes back.",
      "Eval results (30 questions, 5 runs, Pydantic Evals): at k=1 graph and grep agents were about tied, and pass@k was similar. pass^k (correct on all 5 runs) flipped to the graph agent being about twice as good, using about 3.1x fewer tokens. A prompt nudge raised graph-tool adoption from 34% to 72%.",
      "Brandon's earlier eval harness gave directionally right but buggy results: the search tool returned wrong file paths. He only found this by reading traces in Logfire. Agents also cheat in eval harnesses, e.g. using git to diff against main or reading a branch name that says it's a test."
    ],
    "takeaways": [
      "Be deterministic first: parse code/SQL to the AST, compute grain and lineage, and profile tables for join cardinality. Use LLMs only for classification, summarization and resolving ambiguity, and ask many small, focused questions rather than one big prompt with a 10-field schema.",
      "Store evidence rather than conclusions, persist it in a graph the agent can traverse for just the subgraph it needs, and design for incremental reindexing from day one.",
      "Own the tool surface. Curated, typed tools beat a buffet of off-the-shelf MCPs, and tools should mimic what the model already knows (call it 'search', return ripgrep-shaped output) and return something useful on the first call.",
      "Measure tool adoption from traces and report pass^k (consistency across runs), not just per-shot accuracy or pass@k. Prefer deterministic scorers over LLM-as-judge and unit-test the scorer.",
      "Harden eval harnesses against cheating, pin gold data to a fixed codebase snapshot plus injected diff, keep evals hard enough that the agent nearly fails, and read the traces: 'never trust your own scoreboard'."
    ],
    "topics": [
      "context engineering",
      "data engineering",
      "knowledge graphs",
      "agents",
      "evals",
      "tool design",
      "code intelligence",
      "semantic layer"
    ],
    "tools": [
      "Typedef",
      "Fenic",
      "Neo4j",
      "Cypher",
      "Qdrant",
      "tree-sitter",
      "rustdoc JSON",
      "Pydantic AI",
      "Pydantic Evals",
      "Logfire",
      "OpenRouter",
      "Docker Compose",
      "ripgrep",
      "dbt",
      "Airflow",
      "Snowflake",
      "Databricks",
      "Mattermost",
      "Claude Code",
      "Cursor",
      "Codex",
      "Anthropic",
      "Griffe",
      "ty",
      "SCIP",
      "ADE-bench",
      "Bird",
      "Spider",
      "Salesforce",
      "Stripe",
      "Fivetran",
      "Airbyte",
      "Palantir",
      "maturin",
      "Google Colab"
    ],
    "quotes": [
      {
        "text": "The punch line is like the the model isn't the bottleneck.",
        "at": "01:56",
        "url": "https://www.youtube.com/watch?v=KhRjivBSwiw&t=116s"
      },
      {
        "text": "if you give it access to the graph tools and GP, it's going to run GP anyway.",
        "at": "58:52",
        "url": "https://www.youtube.com/watch?v=KhRjivBSwiw&t=3532s"
      },
      {
        "text": "an elegant tool that gets reached for a third of the time is just a third of a tool.",
        "at": "82:13",
        "url": "https://www.youtube.com/watch?v=KhRjivBSwiw&t=4933s"
      },
      {
        "text": "Never trust your own scoreboard.",
        "at": "85:06",
        "url": "https://www.youtube.com/watch?v=KhRjivBSwiw&t=5106s"
      }
    ],
    "words": 23471
  },
  "summary_url": "/#KhRjivBSwiw",
  "transcript": {
    "html": "/transcripts/KhRjivBSwiw.html",
    "txt": "/transcripts/KhRjivBSwiw.txt",
    "vtt": "/transcripts/KhRjivBSwiw.vtt"
  }
}