WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=KhRjivBSwiw

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/KhRjivBSwiw.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:13.840 --> 00:00:14.480
We'll get going.

s2
00:00:14.480 --> 00:00:15.839
Uh my name is Yonyi Michael.

s3
00:00:15.839 --> 00:00:18.560
I'm one of the co-founders of a startup called Typede.

s4
00:00:18.560 --> 00:00:23.439
Uh we're here to do along with uh our founding engineer Brandon Calendarer this workshop.

s5
00:00:23.439 --> 00:00:25.519
Um we have a two-hour time slot.

s6
00:00:25.519 --> 00:00:29.920
We're going to try and keep it as interesting and fun as possible.

s7
00:00:29.920 --> 00:00:31.920
Uh, I know it's very tempting right now.

s8
00:00:31.920 --> 00:00:42.719
You guys all have your laptops out and Germany's down one- nothing in the World Cup game, but it's our promise to try and make it as fun as possible, so you don't feel the need to be streaming that.

s9
00:00:42.719 --> 00:00:53.920
Um, so, um, what we're going to talk about today, uh, is we're going to do like before we actually get into the fun, roll up your sleeves, do more of a technical workshop,

s10
00:00:53.920 --> 00:00:59.039
I wanted to walk through a little bit around the problem set that we're trying to solve here at Typedev.

s11
00:00:59.039 --> 00:01:07.920
um and kind of set the stage and give some context for concepts and principles uh that will later be applied to the actual workshop that Brandon will lead us on.

s12
00:01:07.920 --> 00:01:13.680
Um so the main idea behind this talk um is the background on typed defeaf.

s13
00:01:13.680 --> 00:01:23.840
We're building uh the enter the trust layer for data agents to be able to access and perform uh high complexity tasks on data platforms and data systems.

s14
00:01:23.840 --> 00:01:32.479
Um and so the title of the talk is the data context layer which is a core piece of infrastructure um that we've actually built to help us with that mission.

s15
00:01:32.479 --> 00:01:41.759
Um, and so, uh, a lot of it, uh, comes from the idea that perhaps maybe you guys aren't as much into the data space, but once you try and unleash,

s16
00:01:41.759 --> 00:01:53.360
uh, coding agents or software engineering agents on the data, it's actually a lot more of a nuance, complex environment, and there's a lot of things that you need to think about when building the right harness, when context engineering for these agents to be really effective.

s17
00:01:53.360 --> 00:01:54.799
That's what we'll be talking about.

s18
00:01:54.799 --> 00:01:57.759
The punch line is like the the model isn't the bottleneck.

s19
00:01:57.759 --> 00:02:00.799
The model's going to continue to grow in capabilities.

s20
00:02:00.799 --> 00:02:06.159
Um but the missing context to make these agents actually effective is the important part in this.

s21
00:02:06.159 --> 00:02:10.640
Um so this is what we call the data context layer.

s22
00:02:11.680 --> 00:02:15.840
Um all right so here's the bet that we're taking as a startup called typed defeaf.

s23
00:02:15.840 --> 00:02:26.080
Um all of us have spent let's say like uh the last two years generally in the industry um going and thinking about context engineering for coding agents right so we have like the the anthropics

s24
00:02:26.080 --> 00:02:37.519
the codeexes of the world the cursors um where you're able to understand how to map repositories effectively do retrieval and then tool design for these coding agents to be really effective.

s25
00:02:37.519 --> 00:02:46.400
um where we're falling short and where we're starting to see there's more of a need for is that the same thing needs to be done for agents that are able to access and touch the data

s26
00:02:46.400 --> 00:02:57.120
and the data is exactly where uh if an agent is confidently wrong which it often time is it can actually end up costing a lot of money right because a lot of these metrics and calculations

s27
00:02:57.120 --> 00:03:11.519
that we're asking agentic analytics to actually do wind up on things like board decks um or their metrics in a model or their pipelines that fail quietly that have really really important important features and and um and metrics and measures that they're trying to calculate.

s28
00:03:11.519 --> 00:03:17.519
Um so the core bet that we're saying is that agents are going to be core operators of these data systems.

s29
00:03:17.519 --> 00:03:25.840
In order for them to be really effective, we need to be able to ensure that when they do read and write actions, they're able to do these actions safely

s30
00:03:25.840 --> 00:03:29.920
and accurately and be able to trust these results at scale.

s31
00:03:29.920 --> 00:03:45.440
And that only really works if you're able to give the agent the reasoning layer so that it can actually understand the data system end to end very much like how a data engineering or a data team that actually has working knowledge that was able to build the system from end to end is able to also reason about

s32
00:03:45.440 --> 00:03:46.000
it.

s33
00:03:46.000 --> 00:03:50.000
So that's the goal of the data context layer here.

s34
00:03:50.400 --> 00:03:57.680
Um, so where I'm going to say what we found over let's say the last year and a half of us building in this area is that there are three general concrete

s35
00:03:57.680 --> 00:04:03.439
areas where today's let's say coding agents fail over when you're trying to get them to be effective in data.

s36
00:04:03.439 --> 00:04:05.439
The first thing is impact analysis.

s37
00:04:05.439 --> 00:04:09.280
So essentially like what breaks if I actually make this change.

s38
00:04:09.280 --> 00:04:12.159
Um the next thing is join recommendations.

s39
00:04:12.159 --> 00:04:22.160
So if the is this actually is this join actually safe for me to make across my data system and then safe code changes generally as well right safe transformation logic changes.

s40
00:04:22.160 --> 00:04:24.560
So does this edit still mean the same thing?

s41
00:04:24.560 --> 00:04:32.639
Is it preserving the meaning of this metric or is this metric definition now going to change because I'm making this data commit and this data change that's happening.

s42
00:04:32.639 --> 00:04:34.479
These aren't fringe tasks, right?

s43
00:04:34.479 --> 00:04:44.560
These are like the day-to-day uh tasks and workflows of like data teams um as they're thinking of how to actually get enterprise data to be working with AI.

s44
00:04:44.560 --> 00:04:50.160
And if your agent can't reason and do these tasks reliably, it's very hard to actually trust it.

s45
00:04:50.160 --> 00:04:55.199
It's very hard to actually trust it to give it right and even read access many times.

s46
00:04:55.199 --> 00:05:05.120
U and that's what actually matters is be able to get them to be effective core operators of our data systems and [clears throat] not just like question and answers types of agents.

s47
00:05:05.360 --> 00:05:10.880
So here's the gap the core gap that we've been able to define and like really discover through our work here.

s48
00:05:10.880 --> 00:05:16.960
Um this context that the data needs is uh is uh distributed across many different systems, right?

s49
00:05:16.960 --> 00:05:18.720
So your warehouse can have the store.

s50
00:05:18.720 --> 00:05:21.600
It stores the the rows and the actual data itself.

s51
00:05:21.600 --> 00:05:25.360
Your transformation code has the actual business logic that's really important.

s52
00:05:25.360 --> 00:05:29.520
Your BI tool stores the metrics, the dimensions, some business definitions in there.

s53
00:05:29.520 --> 00:05:38.960
And then you have your docs and Slack channels and um and Slack threads and like tribal knowledge that oftentimes lives in like people's heads as well, right?

s54
00:05:38.960 --> 00:05:41.600
But the agent actually needs something in between all of that.

s55
00:05:41.600 --> 00:05:54.240
It needs to actually understand what a model means, how it relates to other models, what breaks if I make this certain change, um and then which operations are actually safe to make, right?

s56
00:05:54.240 --> 00:06:01.199
Um and for the next let's say 15 minutes while I'm going over this like we'll be using like um a term that's called grain.

s57
00:06:01.199 --> 00:06:14.560
So for all of you like non-data folks in the room what grain actually means from like a data team perspective is um is uh um sorry is like what does each row of the table actually mean?

s58
00:06:14.560 --> 00:06:16.960
What does each row actually represent?

s59
00:06:16.960 --> 00:06:31.520
Um and so if the grain is customer by month uh and I accidentally turn it into customer by week then every downstream sum after that uh can become wrong even though the SQL compiles and everything runs correctly

s60
00:06:31.520 --> 00:06:48.319
right so that's what I mean when I talk about grain change um and so let's make this a little bit more concrete like in terms of reliability what does it actually mean if someone asks an agent why a pipeline's row count row count suddenly spiked and it jumped 40%

s61
00:06:48.319 --> 00:06:49.520
overnight.

s62
00:06:49.520 --> 00:06:52.160
Um, the agent can go and write valid SQL.

s63
00:06:52.160 --> 00:06:52.960
It runs it.

s64
00:06:52.960 --> 00:07:00.160
It could render a beautiful chart, but at the end it's going to report the wrong root cause, the wrong analysis of what actually went wrong.

s65
00:07:00.160 --> 00:07:02.880
It says, "Looks like there's more data that came in.

s66
00:07:02.880 --> 00:07:10.240
Um, but what actually happened is there was an upstream column that uh turned a onetoone join into a one to many join, right?

s67
00:07:10.240 --> 00:07:12.800
So that means the grain actually changed.

s68
00:07:12.800 --> 00:07:23.039
uh and the agent literally cannot tell a real spike like a real data spike and differentiate between that and like an actual grain change because the grain isn't in the data itself,

s69
00:07:23.039 --> 00:07:23.840
right?

s70
00:07:23.840 --> 00:07:38.000
Um and so the agent can run the query, it could get some sample rows, it could sample the data, it could read the SQL very well to understand, but it didn't know that what one row was actually supposed to mean and that now it means something completely different.

s71
00:07:38.000 --> 00:07:38.720
Right?

s72
00:07:38.720 --> 00:07:42.160
So this is why valid SQL isn't enough, right?

s73
00:07:42.160 --> 00:07:42.400
Right?

s74
00:07:42.400 --> 00:07:47.759
The semantics are actually very very important when you're trying to get an agent to be effective.

s75
00:07:49.280 --> 00:07:55.599
So then let's say like the like thinking on it, there maybe two obvious fixes for what you can do for this type of scenario.

s76
00:07:55.599 --> 00:08:01.280
And this is why neither obvious of like neither of these obvious fixes actually works, right?

s77
00:08:01.280 --> 00:08:03.759
So you can give the agent the code, right?

s78
00:08:03.759 --> 00:08:14.879
But the problem with that is if you go and give it let's say like your DBD project or airflow project it's redriving the context every time from scratch right on every single prompt which means it's probabilistic

s79
00:08:14.879 --> 00:08:23.520
um it's bounded by the context window which doesn't always work very well and then every time you start a new session you have to go and rebuild that context from scratch.

s80
00:08:23.520 --> 00:08:25.919
Then let's say okay let's try and give it database access.

s81
00:08:25.919 --> 00:08:30.080
So I'm going to connect my snowflake or datab bricks MCP to it.

s82
00:08:30.080 --> 00:08:39.680
um and instead it can see the rows and then it can pro it can it can see sample some data but it doesn't actually have an understanding of what each row actually means.

s83
00:08:39.680 --> 00:08:43.599
So it can read the numbers but not necessarily explain it right.

s84
00:08:43.599 --> 00:08:58.160
And so the first situation is one can read uh uh one can read the code but doesn't actually have memory to be able to persist the knowledge of the context that you spent a lot of time going back and forth and having bu build up and the other sees the data but doesn't actually associate the meaning behind

s85
00:08:58.160 --> 00:08:58.640
it.

s86
00:08:58.640 --> 00:09:06.160
So neither has like a persistent model of how the system actually behaves and that's the mission that we set out to try and solve.

s87
00:09:06.640 --> 00:09:17.120
So let's like take a step back and name exactly what's missing from the context that you need to actually give the agent to be able to reason effectively across these uh uh complex data changes.

s88
00:09:17.120 --> 00:09:25.440
There are four facts let's say that decide whether an answer is right and none of them are actually written down anywhere that the agent can read or infer.

s89
00:09:25.440 --> 00:09:29.680
So the first thing is what what one row actually represents in a table.

s90
00:09:29.680 --> 00:09:31.519
This is what we talked about the grain.

s91
00:09:31.519 --> 00:09:34.720
The next thing is whether a join is safe or not.

s92
00:09:34.720 --> 00:09:38.640
And then there's what a metric actually means and what relates to what.

s93
00:09:38.640 --> 00:09:38.880
Right?

s94
00:09:38.880 --> 00:09:42.956
The dependencies are very important um in data systems.

s95
00:09:42.956 --> 00:09:43.120
[snorts]

s96
00:09:43.120 --> 00:09:45.920
And so an agent can read your SQL just fine.

s97
00:09:45.920 --> 00:09:54.640
Um but it can't see these four things very easily at query time when it's the most important time for it to gather the the exact context that it needs.

s98
00:09:54.640 --> 00:10:07.519
So the goal of what we did was let's see how we can make them computable and queryable so that the agent is effectively able to grab just the context it needs just at the time it needs and not have to rebuild it during every single session.

s99
00:10:07.519 --> 00:10:11.760
So the next few slides are about how we actually went about doing that.

s100
00:10:11.760 --> 00:10:16.240
Um and so let's take for example the join idea.

s101
00:10:16.240 --> 00:10:27.680
This is like the the second fact that we talked about in the previous slide and this is because let's this is a good example because it's a cleanest example of like why syntax isn't actually enough to be able to get the agent to be effective.

s102
00:10:27.680 --> 00:10:37.360
So there's one join, we call it like an an equa join that's happening here that's written a certain way, but it could have four completely different outcomes depending on the underlying data, right?

s103
00:10:37.360 --> 00:10:46.320
So it can either preserve your rows, it can look up a dimension, or it can silently replicate every single row when it goes from like a one to end type of relationship.

s104
00:10:46.320 --> 00:10:49.760
The same thing is for this between clause that happens oftentimes in SQL.

s105
00:10:49.760 --> 00:10:55.519
So it could be a safe as of lookup or a date spine that actually explodes your grain, right?

s106
00:10:55.519 --> 00:11:03.200
So the agent can see this syntax but it can actually understand which of these scenarios is going to result because of this syntax syntax.

s107
00:11:03.200 --> 00:11:05.519
That's not really a model or capability problem.

s108
00:11:05.519 --> 00:11:11.200
That's like a context problem that we're actually trying to arm the the the agent with.

s109
00:11:11.839 --> 00:11:21.040
The fourth fact in here and this is one of the more fun ones that I like to um to show off is that um understanding of like what models what data assets relate to other ones.

s110
00:11:21.040 --> 00:11:21.360
Right?

s111
00:11:21.360 --> 00:11:37.200
So this is which models are in the revenue domain for example, which are in the product domain and those oftentimes correlate to the personas of the people that are actually trying to access or run the datas to access these uh the core metrics or numbers that are in these uh in these domains.

s112
00:11:37.200 --> 00:11:46.160
Oftentimes today like teams will organize it based on folder names and a lot of times it's tribal knowledge to understand which models relate to which um business units.

s113
00:11:46.160 --> 00:11:52.000
But you can actually discover it from the you can actually do more and actually discover it from the structure of the data.

s114
00:11:52.000 --> 00:12:02.079
So for example you can take a look at how the models actually connect looking at lineage for example table level lineage column level lineage looking at shared keys looking at shared naming.

s115
00:12:02.079 --> 00:12:13.519
So the system clusters them automatically for you into the domains so that when you go and ask the agent okay which models are in my revenue domain for the revops team

s116
00:12:13.519 --> 00:12:23.440
um it becomes a query that the agent can actually run and not a question it has to ask like you know that has to go back and forth with the human to be able to understand which are in the same domain.

s117
00:12:23.440 --> 00:12:27.600
So this is kind of the same principle for context engineering that you see over and over again.

s118
00:12:27.600 --> 00:12:39.519
It's you take something that is tribal knowledge that lives in someone's heads or lives in some documentation elsewhere and you try and actually compute it and then make it queryable for the agent.

s119
00:12:42.720 --> 00:12:45.279
Um, so how do you actually produce these facts, right?

s120
00:12:45.279 --> 00:12:58.240
And the key principle here that we talk about a lot is like you want to always be as deterministic as possible first and then only use LLMs when you're trying to resolve ambiguity or do things that models are actually really good at,

s121
00:12:58.240 --> 00:12:58.720
right?

s122
00:12:58.720 --> 00:13:03.120
And so you're able to compute grain and lineage straight from the actual SQL itself.

s123
00:13:03.120 --> 00:13:12.240
So what we do is we actually take all the transformation logic and we parse it all the way down to the as to actually understand how the data is actually structured right

s124
00:13:12.240 --> 00:13:23.279
and that way the agent can't drift from the actual implementation itself because the the basis of the graph and the context we're giving it is actually derived from the implementation itself.

s125
00:13:23.279 --> 00:13:28.959
Um and then uh and then you can do things like verify join safety against the actual data.

s126
00:13:28.959 --> 00:13:29.200
Right?

s127
00:13:29.200 --> 00:13:31.360
So is the join key really unique?

s128
00:13:31.360 --> 00:13:33.839
This is something that you can do by profiling the tables, right?

s129
00:13:33.839 --> 00:13:37.680
So you're able to go and profile the the tables that you have in your warehouse.

s130
00:13:37.680 --> 00:13:42.160
Um, so that allows the agent to not really guess about things like cardality.

s131
00:13:42.160 --> 00:13:46.480
And then you can use the LLM for things where the human readable parts of it, right?

s132
00:13:46.480 --> 00:13:57.519
So if you're trying to understand business names or summarize the utility of models or resolve any ambiguity from like the first two layers in there, that's where the LLM can really come into the picture.

s133
00:13:57.519 --> 00:14:13.360
um these facts are computed and then checked against your data so that the LLM um uh just kind of names these data assets at the end on the graph that we're building and that allows it to make sure that we're not actually drifting from the core implementation.

s134
00:14:13.360 --> 00:14:23.120
That separation matters because um it keeps the core facts tied to the actual implementation kind of like we talked about and to the data itself.

s135
00:14:23.600 --> 00:14:25.440
So that's all good and great.

s136
00:14:25.440 --> 00:14:27.040
Where does this all live?

s137
00:14:27.040 --> 00:14:29.519
What we call we call it the data context layer.

s138
00:14:29.519 --> 00:14:31.440
At the end of the day, it's a graph, right?

s139
00:14:31.440 --> 00:14:35.519
And it's computed once and then on every commit, it's refreshed atomically, right?

s140
00:14:35.519 --> 00:14:42.320
So, we're incrementally modifying the uh this context graph as data teams are making more data commits to it.

s141
00:14:42.320 --> 00:14:47.920
So, it's always representative of the the freshest state of uh your production enterprise data.

s142
00:14:47.920 --> 00:14:50.399
Um, and the reason is a graph.

s143
00:14:50.399 --> 00:14:57.839
The reason it's a graph and not just a pile of text or the robust set of markdown files is because of the access pattern, right?

s144
00:14:57.839 --> 00:15:03.920
So the agent's able to pull exactly the the subgraph it needs in one traversal.

s145
00:15:03.920 --> 00:15:14.720
Um, and it avoids things like prompt stuffing or context stuffing and it scales way past 500 models because you're not like rereading the project every time you're trying to build the context layer.

s146
00:15:14.720 --> 00:15:15.279
Right?

s147
00:15:15.279 --> 00:15:19.839
over the last few years, we've obviously seen a huge resurgence of like graph databases.

s148
00:15:19.839 --> 00:15:30.240
And I've always like wondered even like before AI like what the main utility of it is, but it turns out that models and AI is very very good at being able to to grab context and

s149
00:15:30.240 --> 00:15:33.839
be able to write cipher, be able to write a query against it and then traverse.

s150
00:15:33.839 --> 00:15:38.320
Um, and the nature of how data assets are, it's very dependency based.

s151
00:15:38.320 --> 00:15:45.360
So it yields really nicely to be able to create these uh typed objects and nodes and edges in this graph.

s152
00:15:45.360 --> 00:15:47.360
Um.

s153
00:15:48.639 --> 00:15:58.639
Oh, you guys can't see the slides.

s154
00:15:58.639 --> 00:16:00.639
Oh,

s155
00:16:03.519 --> 00:16:04.720
can you guys turn off the lights?

s156
00:16:04.720 --> 00:16:06.800
I don't know.

s157
00:16:06.800 --> 00:16:11.040
No. Oh, okay.

s158
00:16:11.040 --> 00:16:14.639
Um, oh that's better.

s159
00:16:14.639 --> 00:16:16.240
Getting a little better.

s160
00:16:16.240 --> 00:16:19.120
We dim them a little bit more.

s161
00:16:22.000 --> 00:16:22.639
Sorry guys.

s162
00:16:22.639 --> 00:16:26.800
I should have uh thought of the the black background and other colored.

s163
00:16:26.800 --> 00:16:33.120
Um, but I can happy to send this out afterwards also the deck.

s164
00:16:33.120 --> 00:16:34.800
Okay.

s165
00:16:34.800 --> 00:16:37.120
Um, apologies for that.

s166
00:16:37.120 --> 00:16:57.920
Um so um yes let's see so you can see or maybe you can't see I guess um but there there's like a sample of like a subgraph in here and you can and what you're is actually represented is that the edges carry a lot of provenence right so the edges are defining different types of relationships amongst the

s167
00:16:57.920 --> 00:17:12.079
the data assets themselves we have like for example um uh is is materialized by has a semantic definition and these are all nodes that are representing different types of data assets in the actual graph itself.

s168
00:17:12.079 --> 00:17:24.640
So there'll be like a node that represents a semantic view for example um and then there will be a node that will present like a physical table which is like the actual physical represent representation of the logical code um of the table itself.

s169
00:17:24.640 --> 00:17:29.679
Um I'll send this to you guys afterwards so you can take a closer look at it or we'll figure out how to get it to you guys.

s170
00:17:29.679 --> 00:17:36.480
Um and uh and so yeah and so this is like essentially like what a subgraph will look like.

s171
00:17:36.480 --> 00:17:45.520
Um and so if we go back and think about the use cases that are relevant that the agent can now solve because of this context layer that we have

s172
00:17:45.520 --> 00:17:59.760
um is is impact analysis is the first one right so the agent proposes a change um for modifying like a model's grain for example right before it actually does that before it writes a line of SQL to make this change it can actually walk downstream

s173
00:17:59.760 --> 00:18:09.760
the graph and ask each node that is a downstream dependency of this node it's trying to change does this actually break the grain that you have represented in this node here.

s174
00:18:09.760 --> 00:18:23.840
So here it's able to catch that um uh a monthly AR measure would get repeated across weekly rows if I'm trying to go from the monthly grain change to the weekly grain change and that downstream there's some ARR that's actually summing

s175
00:18:23.840 --> 00:18:40.880
all of the different um values for this on a monthly basis which would result in let's say the metric having like a 4x um of the actual value amount that it's actually accurate right and then it could land on let's say everything downstream on like a board level dashboard that you have because the context graph is inclusive

s176
00:18:40.880 --> 00:18:42.080
of your entire data system.

s177
00:18:42.080 --> 00:18:46.080
Everything from like source systems all the way down to the BI layer in the dashboards layer.

s178
00:18:46.080 --> 00:18:51.840
So the agent has visibility and able to walk the lineage all the way across the the data life cycle if you will.

s179
00:18:51.840 --> 00:18:54.400
So the key part is this all happens pre-flight, right?

s180
00:18:54.400 --> 00:18:55.360
It's not at build time.

s181
00:18:55.360 --> 00:18:56.880
It's not in a production incident.

s182
00:18:56.880 --> 00:19:01.840
It knows a blast radius before it actually even acts on it.

s183
00:19:01.840 --> 00:19:04.400
And then the second use case here is join recommendations, right?

s184
00:19:04.400 --> 00:19:05.440
So this is fact two.

s185
00:19:05.440 --> 00:19:18.960
Um so this is like the agent wants to attribute the the exercise here is like the agent wants to attribute a arr by marketing channel right so it doesn't pick a path because of like column names that it's able to understand

s186
00:19:18.960 --> 00:19:35.600
it actually checks whether each join key is actually unique and this is done because we're profiling the actual tables themselves um and it's verified against the data right so then it has the recommendation of what the actual safe join path is to make this happen while flagging the ones that will actually cause like a fan out or

s187
00:19:35.600 --> 00:19:43.440
like a 7x um in in rows or one that will suddenly like silently drop 12% of the rows downstream as well.

s188
00:19:43.440 --> 00:19:49.919
So we're making recommendations with actual reasoning and evidence is the key here and it's grounded in the data itself.

s189
00:19:49.919 --> 00:19:51.520
And then safe code changes, right?

s190
00:19:51.520 --> 00:19:52.400
This is the third fact.

s191
00:19:52.400 --> 00:19:55.120
Oftentimes you'll find like business definitions.

s192
00:19:55.120 --> 00:19:58.640
The reality is they're oftentimes nested in the SQL itself, right?

s193
00:19:58.640 --> 00:20:10.480
And so if you're asking an agent to let's say tidy up some measure or some metric that you have and it removes a predicate um syntactically that will work right the the SQL will still compile

s194
00:20:10.480 --> 00:20:24.480
um but what it's actually happening and it it could could look harmless but the predicate was actually a very important part of the business definition itself right so here in this example it's dropping the definition of um of paid right so it it's

s195
00:20:24.480 --> 00:20:34.480
silently redefining what the metric uh the metric to include trial churn uh across all the way downstream three dashboards that are actually happening representing this metric.

s196
00:20:34.480 --> 00:20:45.120
Um so this context layer actually catches this uh change in business definitions and business metrics before it's actually implemented.

s197
00:20:45.120 --> 00:20:51.919
Um and so this is like if you're making the agent ser like have these as like first class constraints.

s198
00:20:51.919 --> 00:20:55.200
It's not just a comment that is in some YAML file or something like that.

s199
00:20:55.200 --> 00:20:56.000
That's the key here.

s200
00:20:56.000 --> 00:21:03.360
And that's the line between like a a syntactic refactor and a change that a human can actually trust.

s201
00:21:03.679 --> 00:21:06.400
So all good and great, but does it actually work?

s202
00:21:06.400 --> 00:21:07.120
Right?

s203
00:21:07.120 --> 00:21:09.200
Um and so you have to you have to measure this.

s204
00:21:09.200 --> 00:21:12.159
Anytime you're building anything agentic, you need to be able to measure it.

s205
00:21:12.159 --> 00:21:16.559
Um and so we practice like eval driven development, which is a very important part of it.

s206
00:21:16.559 --> 00:21:35.200
We're essentially going and creating a bunch of like real production data engineering use cases and tasks and workflows that range in difficulty from like easy to blind and they each run multiple times and they're is they're each running on isolated clones of the actual data environment which is a very important part so that there's no actual leakage

s207
00:21:35.200 --> 00:21:40.799
that happens across these environments and then we have pass fail that's running checking the actual output of it.

s208
00:21:40.799 --> 00:21:44.559
We have some LLM as judge doing like impact analysis things as well.

s209
00:21:44.559 --> 00:21:54.159
Um, but the key findings here is that although it's the same models, it's the same task, the only variable that's changing is where the agent has the access to the data context layer.

s210
00:21:54.159 --> 00:21:57.440
And so you see a material lift um when it has this right context.

s211
00:21:57.440 --> 00:22:02.320
It's able to reason across these really complex changes and the task success rate goes up tremendously.

s212
00:22:02.320 --> 00:22:06.720
It's almost 35 points higher when it's armed with the data context layer.

s213
00:22:06.720 --> 00:22:14.880
And this exact benchmark is obviously going to keep growing and keep evolving as we're working with more teams to figure out their kind of um bespoke use cases in there.

s214
00:22:14.880 --> 00:22:16.799
Um but the signal is consistent, right?

s215
00:22:16.799 --> 00:22:31.840
The better context um the more rich context you're able to give the agent, the outcome changes tremendously than just let's say like swapping models or adding more generic like tools, MCP tools um to to the model itself.

s216
00:22:32.240 --> 00:22:46.240
So, I'll wrap up here, but before I wrap up, the the five patterns, and I'll hand it off to Brandon, obviously, but the five patterns that are even if you're not building this exact system, these are some key takeaways that we've really learned the hard way actually over the last like year of us building this, right?

s217
00:22:46.240 --> 00:22:50.400
The first thing is like deterministic always when you can, right?

s218
00:22:50.400 --> 00:22:57.200
Don't ever try and take like your entire DBT project or your Airflow logic and just throw it to the model to be able to try and do these tough complex tasks.

s219
00:22:57.200 --> 00:22:58.960
That that won't work, right?

s220
00:22:58.960 --> 00:23:05.360
Um what you really want to do is be able to get a really deep and detailed understanding of the transformation logic in a deterministic way.

s221
00:23:05.360 --> 00:23:11.520
So parsing it all the way down to the as to understand the structure the relationship the dependencies are very very important.

s222
00:23:11.520 --> 00:23:15.840
And then things like being able to compute grain right like that's something that you can use.

s223
00:23:15.840 --> 00:23:24.559
We spent a lot of time refining and like using like relational algebra to actually calculate and compute for each stage of the data life cycle.

s224
00:23:24.559 --> 00:23:35.760
Um and then use the model when you need to when it's like summarizing models or summarizing the different data assets or categorizing or classifying models are really great for that.

s225
00:23:35.760 --> 00:23:37.919
The second thing is store evidence not conclusion.

s226
00:23:37.919 --> 00:23:44.799
So it's oftentimes good to just keep the signals and then let the agent be able to reason because these models are very capable.

s227
00:23:44.799 --> 00:23:45.440
They're very smart.

s228
00:23:45.440 --> 00:23:47.919
They can reason at query time effectively.

s229
00:23:47.919 --> 00:23:53.440
And then the third thing is that traversal always beats prompt stuffing or context stuffing, right?

s230
00:23:53.440 --> 00:23:57.200
um it's going to want to let it pull the subgraph that it needs.

s231
00:23:57.200 --> 00:24:00.240
Don't dump the entire repository at runtime.

s232
00:24:00.240 --> 00:24:07.520
It'll pull the subgraph and be able to traverse to figure out how to solve the task at hand because all the dependencies and everything is clearly mapped out there

s233
00:24:07.520 --> 00:24:18.640
and the context when naturally you'll have be running more efficient AI because the context is going to be a lot smaller every time um the task is trying to be solved by these agents.

s234
00:24:18.640 --> 00:24:21.279
And then the fourth thing is like own the tool surface.

s235
00:24:21.279 --> 00:24:29.200
So like what we found is a curated a curated type tool set always beats like a buffet of just like off-the-shelf MCPS.

s236
00:24:29.200 --> 00:24:39.200
Um even like for the MCPS that we do have that give our agent access to Snowflake MCP or data bricks MP we found ourselves like rewriting a lot of it to kind of match the capabilities and functionality

s237
00:24:39.200 --> 00:24:41.600
of what we're trying to accomplish.

s238
00:24:41.600 --> 00:24:56.960
Um and then um let's say the fifth thing is we talked about like eval right this is uh the most important thing you can't really build a reliable agent regardless of what the agent is doing if you can't measure it you can't learn from anything you can't measure

s239
00:24:56.960 --> 00:25:07.760
um and so invest in like the eval infrastructure early and often and it's a continuous like living breathing uh piece of your software that you have to always maintain and up and and update

s240
00:25:07.760 --> 00:25:10.320
um and run continuously.

s241
00:25:10.559 --> 00:25:14.799
So to close uh this portion of the workshop, this hasn't been a real workshop so far.

s242
00:25:14.799 --> 00:25:18.159
It's just me kind of rambling, but we'll get to the fun part now.

s243
00:25:18.159 --> 00:25:24.960
Um what we found is the agents don't really fail at data engineering or data task because they can't read or write SQL.

s244
00:25:24.960 --> 00:25:25.520
They do that.

s245
00:25:25.520 --> 00:25:27.039
They do a very good job of that.

s246
00:25:27.039 --> 00:25:33.520
They fail because they don't have or they don't understand what the data actually means, how it flows, or what breaks when it changes.

s247
00:25:33.520 --> 00:25:33.760
Right?

s248
00:25:33.760 --> 00:25:45.919
So if you make that that context really a first class citizen um you extract it, you compute it, you persist it, and then the agent's able to query it in a really efficient way because you have a graph.

s249
00:25:45.919 --> 00:25:53.279
Uh then you're able to cross a line between from an agent that only reads your code to one that's actually ready to like operate your data system.

s250
00:25:53.279 --> 00:25:57.600
And that's like what we've built with this like data context layer.

s251
00:25:57.600 --> 00:25:58.159
All right.

s252
00:25:58.159 --> 00:26:03.120
Um so thank you for that like little brief talk um that we've given here.

s253
00:26:03.120 --> 00:26:11.440
I want to hand it over to Brandon who's actually going to like start walking you guys through more of like a hands-on workshop that carries over some of the concepts we talked about in here.

s254
00:26:11.440 --> 00:26:21.919
Um, I won't take any questions out because I want to make sure that we have enough time for Brandon to go through his entire workshop, but I'll be around like after this and whenever like feel free to reach out to me.

s255
00:26:21.919 --> 00:26:24.080
I'm happy to chat more about this.

s256
00:26:24.080 --> 00:26:26.320
Thank you.

s257
00:26:27.738 --> 00:26:29.738
[applause]

s258
00:26:31.440 --> 00:26:33.600
Good job.

s259
00:26:35.600 --> 00:26:38.000
Okie dokie.

s260
00:26:41.600 --> 00:26:42.960
How is everybody doing today?

s261
00:26:42.960 --> 00:26:45.039
You're doing good.

s262
00:26:45.039 --> 00:26:47.039
Fantastic.

s263
00:27:02.080 --> 00:27:05.440
Sorry, give me one moment.

s264
00:27:07.520 --> 00:27:08.159
All righty.

s265
00:27:08.159 --> 00:27:15.039
So, the topic that I'm going to talk about today um is uh definitely related to what Yonyi was talking about.

s266
00:27:15.039 --> 00:27:21.120
We're going to try to bring these concepts together in a form that we can actually do in uh the rest of our time here.

s267
00:27:21.120 --> 00:27:22.559
Um so, my name is Brandon.

s268
00:27:22.559 --> 00:27:23.600
I'm at Typed.

s269
00:27:23.600 --> 00:27:33.600
Um, and you're going to hear a lot of talks here at the conference this week about how agent is only as good as the context you give it and that building that context is in fact the actual job.

s270
00:27:33.600 --> 00:27:37.760
And so this is basically a hands-on version of that exact same lesson.

s271
00:27:37.760 --> 00:27:46.400
Um, over the next 90 minutes, we're going to build a context layer for a coding agent together, a real one over a real example codebase.

s272
00:27:46.400 --> 00:27:48.799
We're going to be using the quadrant codebase today.

s273
00:27:48.799 --> 00:27:58.640
Um, and then I'm going to show you uh the last part, which is that even if you build when you build these amazing tools, um the agent's mostly going to ignore it unless you help it along the way.

s274
00:27:58.640 --> 00:28:03.360
Um, getting it to actually use the tool is going to be uh where the work really is.

s275
00:28:03.360 --> 00:28:16.640
Um, so before we go ahead and dive in, um, if you want to go ahead and follow along, this QR code links to my GitHub repository, which does have

s276
00:28:16.640 --> 00:28:18.559
all of the relevant code.

s277
00:28:18.559 --> 00:28:23.840
Um, that repository has a few Jupyter notebooks for people to follow along with.

s278
00:28:23.840 --> 00:28:27.200
Again, don't worry if you don't want to have set that set everything up.

s279
00:28:27.200 --> 00:28:32.320
The dependencies there are uh docker compose or docker for running neo4j.

s280
00:28:32.320 --> 00:28:37.200
Um and if you do want to try out the llm portions of it live, you can use an open router key.

s281
00:28:37.200 --> 00:28:38.480
However, that's not required.

s282
00:28:38.480 --> 00:28:43.919
It'll all run offline if you desire or you can just follow along with me uh as I build it up here.

s283
00:28:43.919 --> 00:28:55.120
Um so, uh I'm going to leave this up for just a little bit longer and then I'm going to start doing a little bit of talking for people to be able to set things up if they want to and then we'll dive into the notebooks themselves.

s284
00:28:55.120 --> 00:28:58.159
Okay, everyone got the scan in.

s285
00:28:58.159 --> 00:29:00.080
Ready to go?

s286
00:29:00.080 --> 00:29:02.799
Give it one more minute.

s287
00:29:06.159 --> 00:29:09.760
All right, perfect.

s288
00:29:09.760 --> 00:29:15.840
So, let's start where every coding agent starts, which is text search.

s289
00:29:15.840 --> 00:29:19.919
Um, I'm going to ask about this type for this strct called posting list.

s290
00:29:19.919 --> 00:29:21.840
Uh, it's a real type in quadrant.

s291
00:29:21.840 --> 00:29:24.559
Um, so when the agent goes, it runs rip grip.

s292
00:29:24.559 --> 00:29:30.159
Um, and you see great, it found that we have these two different um, answers that are exact matches.

s293
00:29:30.159 --> 00:29:32.000
They're both exactly posting list.

s294
00:29:32.000 --> 00:29:33.840
Well, we see it has a dozen more look alikes.

s295
00:29:33.840 --> 00:29:40.000
You know, we have a posting list view, a posting list iterator, etc., etc. Your agent's going to do the same thing.

s296
00:29:40.000 --> 00:29:40.880
It's going to GP.

s297
00:29:40.880 --> 00:29:44.240
It's going to get this wall back and it's going to pick one, right?

s298
00:29:44.240 --> 00:29:45.120
It may pick the right one.

s299
00:29:45.120 --> 00:29:45.919
It may pick the wrong one.

s300
00:29:45.919 --> 00:29:47.200
It's going to explore that one first.

s301
00:29:47.200 --> 00:29:51.120
Maybe it has to go back, backtrack, read the next one to actually figure out what you want, right?

s302
00:29:51.120 --> 00:29:55.200
But it doesn't really have any means of knowing which posting list you want, right?

s303
00:29:55.200 --> 00:30:03.919
It it it can't the text itself doesn't have any concept of identity or you know which posting list is actually you know meaningful.

s304
00:30:03.919 --> 00:30:15.360
Um this the the things you care about you know what methods use it where it's used what crate that it's in that's not necessarily just in the text it's in the actual structure of the code itself.

s305
00:30:15.360 --> 00:30:20.640
Um, so we just watched Yonyi make this, you know, a very similar case for data.

s306
00:30:20.640 --> 00:30:28.799
And it's it's problematic to have the the agent redive the entire system for every run with no memory.

s307
00:30:28.799 --> 00:30:32.000
Um, because for one thing, it's going to be expensive, right?

s308
00:30:32.000 --> 00:30:35.279
It's going to go through it's going to have to reread the same set of files every time.

s309
00:30:35.279 --> 00:30:40.000
If you ask it the same question five times, it's going to go back and read all those same files every time, right?

s310
00:30:40.000 --> 00:30:43.600
But the important thing is that it's fragile, right?

s311
00:30:43.600 --> 00:30:52.240
when it goes back and it reads those five files, it might read them in a different order next time or it might read one file first that gives it the wrong answer and sends on a goose chase to another file.

s312
00:30:52.240 --> 00:30:52.799
Right?

s313
00:30:52.799 --> 00:31:03.200
So, it's that issue of actually having this thing be deterministic and not as fragile uh that we're really trying to accomplish when we're trying to engineer context for these kinds of agents, right?

s314
00:31:03.200 --> 00:31:09.840
The goal isn't necessarily to get the agent to be a lot more correct or a lot more efficient, even though that's sometimes the case.

s315
00:31:09.840 --> 00:31:12.720
The real goal is to have it be a lot more deterministic, right?

s316
00:31:12.720 --> 00:31:18.399
because we're trying to apply as much determinism as we can to this very non-deterministic system.

s317
00:31:18.399 --> 00:31:24.640
Um without you know this persistent trustworthy model of the code, it's going to guess um every time.

s318
00:31:24.640 --> 00:31:31.679
And you know what we see is that over time the models have been getting better and better and better at guessing, right?

s319
00:31:31.679 --> 00:31:35.600
You know, we can all see what happened when we ran cloud code a year ago versus running it now, right?

s320
00:31:35.600 --> 00:31:37.360
It's completely different experience.

s321
00:31:37.360 --> 00:31:54.399
So like if we want to build a context layer for code, we basically want to follow this exact same recipe um but um just at a slightly different uh level, we're going to scale this down so that we can build a workshop and you can leave it something that you can actually run today

s322
00:31:54.399 --> 00:31:58.320
um instead of trying to replicate all of our uh our entire product up here.

s323
00:31:58.320 --> 00:32:09.039
So we're basically going to take um that that concept this exact this extract persist and query um to basically go ahead and start extracting information out of our codebase.

s324
00:32:09.039 --> 00:32:16.480
So we're going to pull out information about the symbols you know every every type every trait every function uh with its real identity and location.

s325
00:32:16.480 --> 00:32:23.360
Um we're going to the relationships between these different symbols get uh edges inside of our graph that we can traverse.

s326
00:32:23.360 --> 00:32:28.559
These get computed once and they end up being uh queried many times.

s327
00:32:28.559 --> 00:32:37.039
This is the same uh overall approach as what we do when we're tackling a sprawling data architecture just with a small enough example to run in this room.

s328
00:32:37.039 --> 00:32:41.919
Um and so yeah, that's basically what I was just saying.

s329
00:32:41.919 --> 00:32:44.399
Uh so we're going to send it up today.

s330
00:32:44.399 --> 00:32:49.840
So uh this is going to be a concept that you're going to hear about a lot during the conference.

s331
00:32:49.840 --> 00:32:52.000
Um I thought it was I thought it was really cool.

s332
00:32:52.000 --> 00:32:57.760
I I brought up I made the slide of three levels of context and look at the schedule and someone has a talk about four levels of context tomorrow.

s333
00:32:57.760 --> 00:33:04.720
So I feel a bit preemptively oneuped but we're still going to go through this today because I think it's still uh pretty useful for this context here.

s334
00:33:04.720 --> 00:33:09.039
Um so you can think about context of a coding agent in sort of this terms of levels.

s335
00:33:09.039 --> 00:33:10.480
Uh that's how I like to think about it.

s336
00:33:10.480 --> 00:33:12.240
So level one is sort of what you just saw.

s337
00:33:12.240 --> 00:33:16.240
It's the GP, it's the glob raw strings, no structure, right?

s338
00:33:16.240 --> 00:33:20.399
Level two is where you get, you know, most of the code rag solutions that we have today, right?

s339
00:33:20.399 --> 00:33:24.720
you get um semantic search, you index the code, you embed it, right?

s340
00:33:24.720 --> 00:33:31.279
You do some nice lexical search, semantic search, you get ranked results, right, of all the, you know, the locations that this this thing was mentioned in.

s341
00:33:31.279 --> 00:33:34.640
But at the end of the day, that's still going to be a flat list, right?

s342
00:33:34.640 --> 00:33:37.279
It's going to be a flat list that gives you more relevant results.

s343
00:33:37.279 --> 00:33:39.360
It's going to be still a flat list.

s344
00:33:39.360 --> 00:33:44.000
It's not going to show you the relationships between these different code, right?

s345
00:33:44.000 --> 00:33:51.440
So the the top level or or the highest level at level three is in our opinion a traversible graph.

s346
00:33:51.440 --> 00:33:57.440
Um something that links the symbols with real identity and adds typed edges between them that you can actually traverse.

s347
00:33:57.440 --> 00:34:07.519
Um so I think for the next part of this workshop we're going to go through and build up through these different levels and by the end we'll get to an agent that can answer questions about the codebase

s348
00:34:07.519 --> 00:34:09.679
um using this traversible graph.

s349
00:34:09.679 --> 00:34:15.520
Um, and uh, yeah, let's uh, let's let's lay it out.

s350
00:34:15.520 --> 00:34:24.000
So, as we're building this, there's going to be a few different uh, stepping stones or rungs to get to that level three understanding of how we do things.

s351
00:34:24.000 --> 00:34:27.040
So, rung zero is sort of the syntactic level.

s352
00:34:27.040 --> 00:34:31.280
First, you need to actually parse the code that you're actually trying to trying to understand.

s353
00:34:31.280 --> 00:34:35.119
And so, the easiest way to do that is with something that's language agnostic like treesitter.

s354
00:34:35.119 --> 00:34:43.839
And that gets you a very basic high-level ASD breakdown of the different um parts of your codebase and how things interact.

s355
00:34:43.839 --> 00:34:48.159
Um, however, that still doesn't really have any idea how to deal with collisions, right?

s356
00:34:48.159 --> 00:34:56.800
Posting list, the symbol that's in posting list.rs and the posting list, the symbol that's in, you know, quant.rs are two different posting lists.

s357
00:34:56.800 --> 00:34:59.280
They're still going to collide at rung zero.

s358
00:34:59.280 --> 00:35:02.800
So, the name of the game is going to be rung one.

s359
00:35:02.800 --> 00:35:06.560
So resolution is the one thing that GP can't really do, right?

s360
00:35:06.560 --> 00:35:12.320
It's the thing that turns those two ambiguous posting lists into two distinct identifiable types, right?

s361
00:35:12.320 --> 00:35:14.640
So with Rust, it's a little bit cheating.

s362
00:35:14.640 --> 00:35:19.200
You kind of get wrong one for free because Rust compiler will do a lot of this work for you.

s363
00:35:19.200 --> 00:35:26.160
And uh you know, we'll basically be using Rust do.json to give us all that information basically out of the box.

s364
00:35:26.160 --> 00:35:27.920
Now, is this cheating?

s365
00:35:27.920 --> 00:35:28.800
Yes.

s366
00:35:28.800 --> 00:35:37.839
Um I picked rest on purpose because I needed to build this workshop and you know building an entire parser for a whole other language or trying to build something out is a little bit harder.

s367
00:35:37.839 --> 00:35:48.560
However, at the end I will talk about a more useful case of this on Python and there's an example notebook in the repo for you guys to peruse on that if we don't get to it uh during this workshop today.

s368
00:35:49.520 --> 00:35:58.320
Um the next layer uh that we're there is a there is a notebook for I'm not sure we'll get to today is rung two which is the actual meaning behind

s369
00:35:58.320 --> 00:36:00.079
um these different things that we're extracting.

s370
00:36:00.079 --> 00:36:00.480
Right?

s371
00:36:00.480 --> 00:36:02.960
So these first two layers are solely deterministic.

s372
00:36:02.960 --> 00:36:08.240
We're basically always trying to be ground our understanding in as deterministic a base approach as possible.

s373
00:36:08.240 --> 00:36:17.680
So we build out these first two layers using tree sitter and rest.json JSON and then if we have some time we'll explore um how we build out that rung two layer like how do we take that information

s374
00:36:17.680 --> 00:36:27.200
feed it into an LLM to classify and summarize and do the things that the LLM are good at as opposed to just trying to throw everything in there and asking it to make sense of it.

s375
00:36:27.839 --> 00:36:30.160
Um fantastic.

s376
00:36:30.160 --> 00:36:52.640
So, now we're going to go ahead and open up our uh notebook 01 and going to be confused.

s377
00:37:05.520 --> 00:37:06.800
Bring this over here.

s378
00:37:06.800 --> 00:37:11.839
Oh, actually it's already here.

s379
00:37:13.760 --> 00:37:15.760
Okay.

s380
00:37:17.440 --> 00:37:17.760
All right.

s381
00:37:17.760 --> 00:37:18.560
Can folks see this?

s382
00:37:18.560 --> 00:37:18.880
Okay.

s383
00:37:18.880 --> 00:37:20.400
Or did I bring them up on your side?

s384
00:37:20.400 --> 00:37:22.400
Great.

s385
00:37:22.480 --> 00:37:34.000
So this uh this notebook, what this notebook is going to do is it's just going to walk through the process of building up those uh those levels of different context for our

s386
00:37:34.000 --> 00:37:35.359
uh our quadrant project.

s387
00:37:35.359 --> 00:37:37.440
We're not going to be indexing the entire codebase today.

s388
00:37:37.440 --> 00:37:38.560
It's a little bit big.

s389
00:37:38.560 --> 00:37:42.960
Um we're taking a look at uh a sampling of I think five crates.

s390
00:37:42.960 --> 00:37:48.000
Um it gives us a good corpus of code to look over uh for our testing here.

s391
00:37:48.000 --> 00:37:52.880
Um, so we can start going ahead and running these cells.

s392
00:37:52.880 --> 00:37:55.599
And so what we do first, we just do a bunch of setup.

s393
00:37:55.599 --> 00:38:06.800
Um, and now that the setup is good, um, we can go ahead and start looking at the, um, yes.

s394
00:38:07.040 --> 00:38:09.920
Ah, it is in the GitHub.

s395
00:38:09.920 --> 00:38:15.760
Yes, I can put up the link again to that real quick.

s396
00:38:22.480 --> 00:38:24.880
bring that up.

s397
00:38:24.880 --> 00:38:28.079
Um, so the link is up here.

s398
00:38:32.480 --> 00:38:37.839
Uh, if you don't want to follow on locally, there are collab links in each of the different um, things there.

s399
00:38:37.839 --> 00:38:38.320
Yes.

s400
00:38:38.320 --> 00:38:44.880
Uh, for the collab for the notebook one, if you do want to use the graph, you will need a Neo4j or a login.

s401
00:38:44.880 --> 00:38:50.560
Okay, that's just a that's just a special thing for that.

s402
00:38:51.520 --> 00:38:52.960
Okie dokie.

s403
00:38:52.960 --> 00:39:02.800
So, we can see here um there's a bunch of additional code in here to actually do all of the um parsing inside of uh Rust itself.

s404
00:39:02.800 --> 00:39:07.599
I'm not going to necessarily go over that today, but it's all available in the repo for you at your leisure.

s405
00:39:07.599 --> 00:39:16.400
Um we just have a magical function here today uh extract rust gate that will uh pull out the information from running trees sitter across the rust codebase.

s406
00:39:16.400 --> 00:39:21.680
Uh and then another one that will go ahead and do the same thing but for our information from uh rest dock.

s407
00:39:21.680 --> 00:39:31.680
So we can see here that from the first couple of crates we've pulled out um a bunch of symbols from freesitter and then a bunch of symbols uh resolved versions of those symbols from

s408
00:39:31.680 --> 00:39:34.000
Rustto do.

s409
00:39:38.480 --> 00:39:46.640
So now we can go ahead and start running some code that's going to start trying to understand why we're differentiating between rung one and rung zero.

s410
00:39:46.640 --> 00:39:58.880
So we can see when we ask about something like posting list in rung zero we see that okay we see this exported version of posting list we don't know actually where that originates from though we can't resolve that backward

s411
00:39:58.880 --> 00:40:02.000
into the actual um location where it was defined.

s412
00:40:02.000 --> 00:40:03.920
Um, so that's what rung one is for, right?

s413
00:40:03.920 --> 00:40:06.160
It's going to be able to say this is posting list.

s414
00:40:06.160 --> 00:40:13.200
This canonical location is here at posting list posting list posting list and that actual location is at this path at this line, right?

s415
00:40:13.200 --> 00:40:19.599
So this is what's going to be able to let us figure out when things get reexported outside of a crate, link it back to the original crate.

s416
00:40:19.599 --> 00:40:27.280
The agent's going to be able to traverse all the relationships between those different things um once we actually build out the graph itself.

s417
00:40:27.280 --> 00:40:32.800
Um so now we'll just build very quickly the level two that I was talking about before.

s418
00:40:32.800 --> 00:40:44.960
Um where we have a kind of index where we build a uh very you know where we basically take all the symbols and then we let um a function go that just does a full text search.

s419
00:40:44.960 --> 00:40:45.359
Right?

s420
00:40:45.359 --> 00:40:48.960
So we do a search and we get the same problem that we saw in grap.

s421
00:40:48.960 --> 00:40:49.280
Right?

s422
00:40:49.280 --> 00:40:52.400
We can search for posting list and we can get very useful information.

s423
00:40:52.400 --> 00:40:54.079
We can see this is the reexport.

s424
00:40:54.079 --> 00:40:54.960
This is the strruct.

s425
00:40:54.960 --> 00:40:56.079
This is the strruct.

s426
00:40:56.079 --> 00:41:00.319
But it's still not going to tell us the linkages between those different things, right?

s427
00:41:00.319 --> 00:41:10.079
And the agent can infer that and it can, you know, make the make the figure out, oh, probably that if the posting list posting list is a strruct, it's probably reexported up here.

s428
00:41:10.079 --> 00:41:11.680
So that's probably the basis of it.

s429
00:41:11.680 --> 00:41:21.920
But you don't want it to have to make that logical leap every single time because it's going to be that one time where it gets it wrong and then it's going to end up on a completely different wild gase.

s430
00:41:23.680 --> 00:41:29.839
So our next step here, we build out the we start building out the actual graph um itself.

s431
00:41:29.839 --> 00:41:42.160
So we take that all those different um uh symbols and uh information that we just extracted, we transform that into a bunch of nodes and edges that we can populate into our Neo4j

s432
00:41:42.160 --> 00:41:42.880
graph.

s433
00:41:42.880 --> 00:41:48.160
Um and then we go ahead and actually um load all of that in.

s434
00:41:48.160 --> 00:41:52.640
So once that's in the graph, right, we can even go into Neo4j.

s435
00:41:52.640 --> 00:42:02.319
Um we can see we have you know a beautiful little graph of u references to all the different parts of our uh of our of our codebase here.

s436
00:42:02.319 --> 00:42:18.800
Um so now that we have this sort of graph, we can build tools that specifically harness the capabilities of the graph to give our agent access to something that's going to make it easier for it to do its job.

s437
00:42:18.800 --> 00:42:29.119
So if we build a tool and we want to say okay I want to look at a function and I want to see what's going to change what's going to break if I change this function right so we can build this function

s438
00:42:29.119 --> 00:42:36.960
blast radius where we say okay you pass in the qualified name of something the canonical name it's going to go in and it's going to try to figure out

s439
00:42:36.960 --> 00:42:54.240
walk down the graph through those different edges that we defined try to figure out exactly um what depends on this thing what will break if this thing gets changed right and so in that sense you can ask the agent to answer a question that would have taken multiple grapes across multiple different files and

s440
00:42:54.240 --> 00:42:59.119
in theory um the agent can just answer it in one in one go.

s441
00:42:59.119 --> 00:43:07.520
So of course that didn't work because why would it why would it work live demos everybody?

s442
00:43:11.119 --> 00:43:18.960
Okay, so you can see here when we say we're going to pass the blast radius to this strruct, we can say, okay, cool.

s443
00:43:18.960 --> 00:43:25.040
It's linked here via it gets returned by this um there's a method that does this um it gets returned here.

s444
00:43:25.040 --> 00:43:33.040
So we can see all the linkages that we're able to establish um just by having the content inside of the graph.

s445
00:43:34.319 --> 00:43:38.800
So finally um we get to resolve our collision problem.

s446
00:43:38.800 --> 00:43:51.440
Um now if we say we have two different posting lists and we want to see the difference between them um we can call our new tools blast radius for this posting list and this posting list and we get a different set of results.

s447
00:43:51.440 --> 00:43:57.520
Changing one posting list is not going to in fact the same level of changes if you impact change the other posting list.

s448
00:43:57.520 --> 00:44:12.000
So even though these things have the same underlying name, the actual canonical resolution of them results in us being able to more definitively say what's going to change when you break this thing more than just going a grap reading all the files and then trying to guess.

s449
00:44:12.000 --> 00:44:22.560
Um so now we can actually take what we built so far and build a very small limited function agent to actually use this new tooling.

s450
00:44:22.560 --> 00:44:30.240
Um now that we have uh this blast radius tool um we can wire up a very quick little agent.

s451
00:44:30.240 --> 00:44:32.160
Um this agent is very simple.

s452
00:44:32.160 --> 00:44:46.480
It's just a pyantic agent um that uh is just configured with a system prompt that says you have the ability to uh run these different tools and we wrote some very basic tools for it to be able to use blast radius the implementers

s453
00:44:46.480 --> 00:44:51.599
methods of um that work with very basic cipher queries in order to get the information out.

s454
00:44:51.599 --> 00:44:57.760
So once you have this information in the graph, writing the queries to actually pull it out becomes quite simple.

s455
00:44:57.760 --> 00:45:07.440
Um if you can express the idea that you want to uh do then it's pretty easy to write cipher or the equivalent language in whatever graph you're using um to actually pull that data out.

s456
00:45:07.440 --> 00:45:12.240
And that's a sign that you have created the right schema for the job.

s457
00:45:12.240 --> 00:45:21.440
So now if you go ahead and run this agent, this agent will actually spend a few turns um calling the tools and trying to figure things out.

s458
00:45:21.440 --> 00:45:24.480
Uh and then it will go ahead and uh try to answer our question.

s459
00:45:24.480 --> 00:45:32.800
So for now if we can say what breaks if I change posting list tell me each dependent symbol with its file line and then it will tell me exactly

s460
00:45:32.800 --> 00:45:38.000
uh what will break um based on the blast radius finding and we can see here what tools it called.

s461
00:45:38.000 --> 00:45:45.599
It started with search did a couple of searches before calling the blast radius uh and then doing another blast radius call.

s462
00:45:45.599 --> 00:45:55.440
So we can see that if you have an agent that is uh skilled enough in tool usage um it can use these new tools to uh build out its understanding

s463
00:45:55.440 --> 00:45:57.280
uh of the codebase.

s464
00:45:57.280 --> 00:46:06.240
Now this part is a uh more of an interactive section where you can build whatever tool that you would like.

s465
00:46:06.240 --> 00:46:12.800
Um, how this basically works is we can write a tool that basically executes arbitrary arbitrary cipher.

s466
00:46:12.800 --> 00:46:21.839
Um, so here we're wr writing some cipher that says find me all of the um symbols within this file path um and then just lay them out.

s467
00:46:21.839 --> 00:46:23.839
So this tool will do just that.

s468
00:46:23.839 --> 00:46:28.880
It will find all of the symbols in this file path and lay them out for me.

s469
00:46:28.880 --> 00:46:32.880
Um, yes.

s470
00:46:32.880 --> 00:46:39.280
So then if I wanted to write a new tool, I wouldn't even necessarily have to know cipher, right?

s471
00:46:39.280 --> 00:46:44.960
Um I could give the agent access to a system prompt that has a schema in it, right?

s472
00:46:44.960 --> 00:46:48.480
That tells the agent, okay, these are the different nodes, these are the different edges.

s473
00:46:48.480 --> 00:47:00.640
And I can write um I can write my um my requests in natural language like I want to see uh every strct that implements a trait along with the trait that it implements, right?

s474
00:47:00.640 --> 00:47:07.040
And if I run um if I run that uh it'll take a second because the agent's going to actually try to figure it out.

s475
00:47:07.040 --> 00:47:07.839
And it does.

s476
00:47:07.839 --> 00:47:16.079
It actually generates this cipher query that it can run and it can go ahead and pull that information um out of the uh out of the graph.

s477
00:47:16.079 --> 00:47:21.280
Um we're not going to get to uh rung two today, but we can get a little taste of rung two.

s478
00:47:21.280 --> 00:47:24.000
So so far everything has been very deterministic.

s479
00:47:24.000 --> 00:47:26.640
Um you get your structure and you get your resolution.

s480
00:47:26.640 --> 00:47:30.319
Um that third rung I was talking about was the actual semantic meaning.

s481
00:47:30.319 --> 00:47:35.440
um you can actually have an LLM summarize these symbols and uh embed them.

s482
00:47:35.440 --> 00:47:38.800
And uh typed defaf actually has an open source library for this called fenic.

s483
00:47:38.800 --> 00:47:44.000
It's a dataf frame library for u marrying traditional dataf frame operations and LLMs.

s484
00:47:44.000 --> 00:47:51.119
Um so here we're just doing a quick configuration of fenick where we say we're going to have it talk to open router.

s485
00:47:51.119 --> 00:47:53.040
Um we're going to pull in some information.

s486
00:47:53.040 --> 00:48:01.760
We're going to say uh we're going to try to classify this specific rust where we're going to pull in the kind and the qualified name and the dock string

s487
00:48:01.760 --> 00:48:13.440
and then we're going to ask in one concise sentence what is this rust item for and so when we run fenic over this over this corpus over these rows we get back um the summarized outputs from the LLM.

s488
00:48:13.440 --> 00:48:21.839
So in the long run, right, the idea would be to bake this information up into your graph so that this information is also available um for the agent.

s489
00:48:21.839 --> 00:48:25.599
And the nice thing about this is that this is auditable when you generate it.

s490
00:48:25.599 --> 00:48:31.839
So if you're looking at this and you say, "Hey, this explanation of what this function does is wrong, we can go ahead and actually get that fixed."

s491
00:48:31.839 --> 00:48:42.079
If the agent on the fly figures out what that function does and it's wrong, you're not going to know that it's wrong until much later down the line when it writes code that doesn't work.

s492
00:48:42.880 --> 00:48:46.319
Um, awesome.

s493
00:48:56.000 --> 00:48:57.040
Yeah, sure.

s494
00:48:57.040 --> 00:48:59.599
Um, that was uh notebook one.

s495
00:48:59.599 --> 00:49:08.160
We have a little bit more stuff to talk about, but I'm happy to take any questions about the content that was there and I'll put it back up real quick.

s496
00:49:09.119 --> 00:49:10.800
Yeah, I see two in the back.

s497
00:49:10.800 --> 00:49:12.160
will come back with a with a microphone

s498
00:49:12.160 --> 00:49:14.160
testing.

s499
00:49:14.720 --> 00:49:15.040
Yeah.

s500
00:49:15.040 --> 00:49:24.160
Is the is the graph reestablished at runtime when whenever you're uh building these a uh traversing them?

s501
00:49:24.160 --> 00:49:25.520
Define runtime.

s502
00:49:25.520 --> 00:49:30.559
runtime as in uh when the PR or CI/CD comes through or

s503
00:49:30.559 --> 00:49:45.839
yeah in an ideal world I mean we don't build this product for a living but like in an ideal world that would be uh cached and saved in the a canonical form as like say the gold version based on main for example right um and then as you tweak stuff and modify things locally you need to probably

s504
00:49:45.839 --> 00:49:52.480
maintain a local copy of that that would ideally be incrementally refreshed so you wouldn't have to do it all up front

s505
00:49:54.160 --> 00:49:58.160
yeah I My question is similar.

s506
00:49:58.160 --> 00:50:20.319
How do you uh think about the cost of keeping these structures up to date versus you know whatever the increase in performance and fidelity um because at least for me you know I'm using AI to make

s507
00:50:20.319 --> 00:50:22.079
uh large changes.

s508
00:50:22.079 --> 00:50:23.599
So

s509
00:50:23.599 --> 00:50:33.359
um so it's hard for me to know um offhand you know whether it would really be better if you have to maintain these structures.

s510
00:50:33.359 --> 00:50:34.079
Yeah, absolutely.

s511
00:50:34.079 --> 00:50:35.960
I think I think the key to that is

s512
00:50:35.960 --> 00:50:36.319
[snorts]

s513
00:50:36.319 --> 00:50:50.079
um a lot of these problems can be boiled down and incrementalized because a lot of the things that you're doing is you you don't necessarily have to reindex the things that you've already indexed if they're not changing, right?

s514
00:50:50.079 --> 00:50:56.880
And when you're making broad sweeping changes across your codebase, like you are changing a lot of files, but there also are a lot of files you're not changing, right?

s515
00:50:56.880 --> 00:51:08.800
So if we can be smart about incremental reindexing, making sure we're only keeping track of what's changed, only reindexing that part, then what we've seen so far is that the trade-off is in fact worth it.

s516
00:51:08.800 --> 00:51:20.800
Especially as the context size of what you need to keep indexed grows, it actually becomes more useful because the cost of doing it live every time just becomes even more expensive.

s517
00:51:20.800 --> 00:51:22.079
Um, thank you very much.

s518
00:51:22.079 --> 00:51:23.040
That was super useful.

s519
00:51:23.040 --> 00:51:38.079
Um, I I'm I'm wondering how it works on kind of like a real code base in a in a messy language where uh for example you have dynamic dispatch or like not not all types are available at you know at compile time or before sorry when you just analyze code.

s520
00:51:38.079 --> 00:51:43.040
So have you tried like what are the limitations any any any tips there?

s521
00:51:43.040 --> 00:51:43.760
Yes.

s522
00:51:43.760 --> 00:51:49.280
Um we I do have a third notebook here which is talking about that in particular.

s523
00:51:49.280 --> 00:51:56.800
Um which is basically how you do this thing on a language like Python which doesn't have the same uh uh easy access to the types as graph as as rust.

s524
00:51:56.800 --> 00:51:58.720
Um we use we can use other tooling.

s525
00:51:58.720 --> 00:52:01.440
We can use tools like griff and same type sitter.

s526
00:52:01.440 --> 00:52:06.880
Uh types will still give you a and griff will give you static resolution of some of the other stuff like the types.

s527
00:52:06.880 --> 00:52:13.200
You can use something like tai uh to do type checking as well um to gain more index there.

s528
00:52:13.200 --> 00:52:17.119
Yes, it does become exponentially harder with different languages.

s529
00:52:17.119 --> 00:52:20.960
Um, so the and I will talk about that a little bit later on.

s530
00:52:20.960 --> 00:52:24.160
Um, but yes, the answer to your question is yes, I did think about it.

s531
00:52:24.160 --> 00:52:30.079
It is a much harder problem which is why I chose to solve a slightly easier problem today.

s532
00:52:31.359 --> 00:52:33.359
Yeah.

s533
00:52:59.200 --> 00:53:02.026
I don't think you should be building this to be clear.

s534
00:53:02.026 --> 00:53:02.480
[laughter]

s535
00:53:02.480 --> 00:53:06.880
This this is this is a this is a worked example for educational purposes.

s536
00:53:06.880 --> 00:53:16.079
um people build these kind of coding uh graph things like on the reg on the regular and I wouldn't try to build one from scratch um just for doing code analysis.

s537
00:53:16.079 --> 00:53:21.359
Uh I don't know if that answers your your question and I think they are worth it to build.

s538
00:53:21.359 --> 00:53:24.240
I don't know if it's worth it for everyone to build their own.

s539
00:53:24.240 --> 00:53:34.160
I think there is probably a and there are a bunch of folks in this game already that are are doing this sort of you know analysis just on the code layer.

s540
00:53:34.160 --> 00:53:41.280
Um, so I I would say it's probably not worth it for a single person to build if for the purpose of being a product.

s541
00:53:41.280 --> 00:53:47.760
It is probably worth it for someone to build for the purpose of learning about how to build an agent and how to build tooling and how to build context graphs.

s542
00:53:47.760 --> 00:53:52.480
Um, I think the scale of this problem is like where it that's where it's more useful.

s543
00:53:52.480 --> 00:53:54.079
That answer your question?

s544
00:53:54.079 --> 00:53:55.119
Awesome.

s545
00:53:55.119 --> 00:54:00.640
So uh my question is about the data context layer uh that you talked about earlier.

s546
00:54:00.640 --> 00:54:07.839
Uh so uh I'm assuming that that's useful in a for a like a developer.

s547
00:54:07.839 --> 00:54:15.119
So you develop this this context layer and it helps uh produce uh better SQL procedures whatnot.

s548
00:54:15.119 --> 00:54:22.960
Um so uh two questions number one um does it have you seen if it helps at runtime as well?

s549
00:54:22.960 --> 00:54:26.400
Uh like inference time uh that's number one.

s550
00:54:26.400 --> 00:54:32.319
And number two, um do you have any benchmark results to demonstrate?

s551
00:54:32.319 --> 00:54:32.880
Yeah, absolutely.

s552
00:54:32.880 --> 00:54:36.960
So what Yonyi was talking about in his talk, um we do have benchmark results for that.

s553
00:54:36.960 --> 00:54:54.400
Um what we do is we've uh we've tried to find a representative data architectures which is kind of hard because you can build an easy tour example like in your in your in your and say oh my agent aces 100% of my tests on my 30 model corpus and it's like okay that's not very impressive.

s554
00:54:54.400 --> 00:54:59.280
Um, so what we tried to do is find uh open source examples of data architecture that was larger.

s555
00:54:59.280 --> 00:55:03.839
Um, so we ended up settling on using uh I don't know if anyone's heard of Mattermost.

s556
00:55:03.839 --> 00:55:06.880
They're a instant response and chat company.

s557
00:55:06.880 --> 00:55:12.319
Um, they actually open source their entire DBT project with like several 300 350 models in it.

s558
00:55:12.319 --> 00:55:19.680
Um, so we're actually able to pull that down, simulate data in there, and then run evaluations against specified scenarios inside of that.

s559
00:55:19.680 --> 00:55:23.760
Um, asking it to do a specific task, break something in there, and then ask it to fix it.

s560
00:55:23.760 --> 00:55:28.559
ask it uh build out a set of you know new tasks for it to build new things.

s561
00:55:28.559 --> 00:55:38.480
And we're able to see that uh by building out our evaluation layer and layering out you know evaluations of how correct it is, how complete it is, um how

s562
00:55:38.480 --> 00:55:39.680
efficient it is, right?

s563
00:55:39.680 --> 00:55:51.280
We are able to see that uh with the appropriate context our agents are able to do a better job uh overall and in certain particular categories uh to uh like

s564
00:55:51.280 --> 00:55:55.599
accomplish those same tasks better than just cloud code running alone.

s565
00:55:55.599 --> 00:55:56.319
Um

s566
00:55:56.319 --> 00:56:01.599
and what about the earlier question or the other question about inference time?

s567
00:56:01.599 --> 00:56:06.640
So uh it does so you're talking about like when the agent is actually running.

s568
00:56:06.640 --> 00:56:08.319
Yes, when the agent is actually running

s569
00:56:08.319 --> 00:56:14.960
that that that's when we find the most use out of it because when the agent is running, it's going to be querying all of the tools that we built and

s570
00:56:14.960 --> 00:56:17.680
that's where the real value is coming into play, right?

s571
00:56:17.680 --> 00:56:23.520
If the agent's able to query all this stuff at runtime, it doesn't have to rederive it all at one time, right?

s572
00:56:23.520 --> 00:56:33.280
So, um, so then have you tried this with like NL2SQL benchmarks like Bird or Spider or one of those?

s573
00:56:33.280 --> 00:56:33.680
Yeah.

s574
00:56:33.680 --> 00:56:36.000
So, I mean, yes.

s575
00:56:36.000 --> 00:56:38.559
And no like yes we have tried it on some of those benchmarks.

s576
00:56:38.559 --> 00:56:44.319
I think the the tricky bit is that like the goal of the product isn't necessarily to write analytical SQL.

s577
00:56:44.319 --> 00:56:48.160
The goal of what we're trying to build for people is more on the data engineering side.

s578
00:56:48.160 --> 00:56:55.760
So it's I think you know it's it's sort of a very common thing to say oh we're going to build text to SQL and we're going to build that.

s579
00:56:55.760 --> 00:56:57.359
I think everyone's trying to build that, right?

s580
00:56:57.359 --> 00:57:10.480
But it's I think it's more important to say we want to build the tools that are going to make it possible for you to actually trust text to SQL, not necessarily that we're going to be able to, you know, come in and write better analytics queries for you.

s581
00:57:10.480 --> 00:57:18.079
Like we want to make sure that when you are actually going in and doing your day-to-day work of maintaining your data architecture that we're going to be able to help you with that.

s582
00:57:18.079 --> 00:57:20.720
Did that answer your question?

s583
00:57:25.040 --> 00:57:27.200
All righty.

s584
00:57:27.760 --> 00:57:32.640
So, popping back into if I can find my mouse.

s585
00:57:33.520 --> 00:57:36.480
Popping back into the slides.

s586
00:57:37.359 --> 00:57:53.359
So what we saw there at the end, right, is that when we uh we can actually go ahead and tell these different posting lists apart.

s587
00:57:53.359 --> 00:57:58.640
Um because we're able to uh query our new graph context.

s588
00:57:58.640 --> 00:58:06.559
Um and we're able to actually figure out all the dependencies of a given symbol with one hop uh without any regrabing.

s589
00:58:06.559 --> 00:58:18.720
Um and so how the query works you know it matches on the qualifi the fully qualified name and then it goes and finds all the dependencies and that's it guys great workshop we're done for the No.

s590
00:58:18.720 --> 00:58:20.400
So the thing is like this is cool.

s591
00:58:20.400 --> 00:58:21.040
This is real.

s592
00:58:21.040 --> 00:58:22.319
This tool is really great right?

s593
00:58:22.319 --> 00:58:26.799
This tool can do in one call what it would certainly take multiple graphs to do.

s594
00:58:26.799 --> 00:58:31.440
I don't think you could accomplish this in one grab even if you did the perfect grab, right?

s595
00:58:31.440 --> 00:58:38.240
But uh the problem is the agent you have to actually go and give it to an agent.

s596
00:58:38.240 --> 00:58:49.839
So when you give this agent this tool and you give it this agent this tool and you give it alongside all the existing tools the agent already has and you know we have this really cool tool, we've seen it do this amazing stuff

s597
00:58:49.839 --> 00:58:51.520
and it ran GP anyway.

s598
00:58:51.520 --> 00:58:57.359
Um, if you give it access to the graph tools and GP, it's going to run GP anyway.

s599
00:58:57.359 --> 00:59:04.240
And we gave it amazing tools, a precomputed traversible model of the codebase, and it just ran grip anyway.

s600
00:59:04.240 --> 00:59:12.319
It just globbed and read files and and ran grip and it would sometimes, you know, poke the graph and get confused and then get back to, you know, waiting through the graph stuff.

s601
00:59:12.319 --> 00:59:13.920
And this is where no one really tells you.

s602
00:59:13.920 --> 00:59:16.720
Everyone's like, "Oh, he built these cool sexy tools and they do this amazing stuff."

s603
00:59:16.720 --> 00:59:19.440
But the agent has to actually use them, right?

s604
00:59:19.440 --> 00:59:23.040
you have to do the prompt engineering and the rest of the hard work and the evaluations.

s605
00:59:23.040 --> 00:59:25.119
That's the rest that's the hard part, right?

s606
00:59:25.119 --> 00:59:26.240
You can build that notebook.

s607
00:59:26.240 --> 00:59:29.440
You can build the your amazing graph context in an afternoon.

s608
00:59:29.440 --> 00:59:32.079
The hard part is actually making the agent use it.

s609
00:59:32.079 --> 00:59:37.440
Um like the uh yes.

s610
00:59:37.440 --> 00:59:39.200
Um and why does it do this?

s611
00:59:39.200 --> 00:59:40.480
Well, it's because of the post- training.

s612
00:59:40.480 --> 00:59:44.079
The models are trained and designed to call these tools, right?

s613
00:59:44.079 --> 00:59:49.359
There are so many circumstances in their training where the answer is to GP and read, right?

s614
00:59:49.359 --> 00:59:54.880
And you're not going to be able to necessarily fight that um just by making a better tool.

s615
00:59:54.880 --> 00:59:58.400
You can make a better tool, but the agent's muscle memory isn't going to let it use it.

s616
00:59:58.400 --> 00:59:59.920
It's very similar to humans, right?

s617
00:59:59.920 --> 01:00:07.119
You can build humans a better tool and if you try to force them to use it, you're going to get they're going to not be happy about it.

s618
01:00:07.119 --> 01:00:08.319
This may be a little bit topical.

s619
01:00:08.319 --> 01:00:08.880
I don't know.

s620
01:00:08.880 --> 01:00:18.079
Um, but so the the important thing is that like you're not necessarily competing on the merits of your tool being better than all the tools inside of the toolb

s621
01:00:18.079 --> 01:00:20.000
tool set of the agent, right?

s622
01:00:20.000 --> 01:00:22.960
You're trying to fight against its own reflexes.

s623
01:00:22.960 --> 01:00:25.599
So how do you actually do that, right?

s624
01:00:25.599 --> 01:00:31.280
Because I don't think there's a question anymore that yes, if it could query the graph, it would be more efficient and better.

s625
01:00:31.280 --> 01:00:34.960
We'll prove that later, but how do we actually get it to do that?

s626
01:00:34.960 --> 01:00:41.119
And the answer is to not really fight the the prior experience of the model, but to kind of lean into it, right?

s627
01:00:41.119 --> 01:00:45.680
We we make the graph feel more like the model is using grab, right?

s628
01:00:45.680 --> 01:00:53.280
You know, you you you take this you take a a tool that maybe called query search or query graph or query whatever and you just call it search, right?

s629
01:00:53.280 --> 01:00:55.040
The agent will use it in a certain kind of way.

s630
01:00:55.040 --> 01:01:00.960
What I found is that if you call a tool search, the agent will basically always do multi keyword search into it with no training.

s631
01:01:00.960 --> 01:01:04.400
So you have to you have to kind of meet the model halfway, right?

s632
01:01:04.400 --> 01:01:15.680
You take the model, you name it search, you account for it to adjust for how the model is going to query it and then you just, you know, you get back the results from the graph and you return the results from the graph in the same form as rip grab, right?

s633
01:01:15.680 --> 01:01:19.839
You take the same tool structure and response as rip grip and you return it back to the model, right?

s634
01:01:19.839 --> 01:01:21.200
Because that's what the model's been trained on.

s635
01:01:21.200 --> 01:01:29.920
It knows how to handle this stuff and you're not going to get it to completely change its own self um just by prompting, right?

s636
01:01:29.920 --> 01:01:44.720
And the last thing is that you need to have uh your tools pay off in the first call uh every time because when something when the agent goes down a path and it finds a a dead end or something that it thinks is a dead end.

s637
01:01:44.720 --> 01:01:47.200
It called search didn't get any results.

s638
01:01:47.200 --> 01:01:50.240
Well, sometimes it's going to think search is broken.

s639
01:01:50.240 --> 01:01:53.680
If I'm getting no results and like I there should be results, maybe search is broken.

s640
01:01:53.680 --> 01:01:54.799
I should try another tool, right?

s641
01:01:54.799 --> 01:01:56.480
It's not going to go back and use that tool again.

s642
01:01:56.480 --> 01:01:58.880
And if it does, maybe it tries to use a different way.

s643
01:01:58.880 --> 01:02:01.280
you can't really necessarily predict that, right?

s644
01:02:01.280 --> 01:02:08.240
So, you have to do your best to make sure that your tools are returning something that's super useful uh right away.

s645
01:02:08.240 --> 01:02:13.599
Otherwise, they're just going to go back to GP and they are uh never going to come back.

s646
01:02:13.599 --> 01:02:17.200
So, the second notebook is uh shorter.

s647
01:02:17.200 --> 01:02:25.760
Um it's a little bit of a more focused example and this is all about evals, right?

s648
01:02:25.760 --> 01:02:31.440
How do you actually show that the agent is doing what you think it is supposed to be doing, right?

s649
01:02:31.440 --> 01:02:39.280
And how do you actually um prove that your tool is better than the existing tooling that may already exist, right?

s650
01:02:39.280 --> 01:02:42.960
So um all of this stuff is already pre-cached.

s651
01:02:42.960 --> 01:02:43.599
I did this run.

s652
01:02:43.599 --> 01:02:51.280
If you do want to run the notebook locally, you can absolutely set sample size to um something greater than zero and run the eval live.

s653
01:02:51.280 --> 01:02:52.240
It will take a long time.

s654
01:02:52.240 --> 01:02:53.599
Don't say I didn't warn you.

s655
01:02:53.599 --> 01:02:59.920
Um but um I did run this overnight with uh five retries and uh 30 questions in my email set.

s656
01:02:59.920 --> 01:03:04.160
So that's where these results are coming from for this uh for this notebook.

s657
01:03:04.160 --> 01:03:15.920
So the first thing that we do um after we do this setup is um we basically uh set up our our test cases, right?

s658
01:03:15.920 --> 01:03:19.599
So we're going to have an agent with um our new graph tools.

s659
01:03:19.599 --> 01:03:23.599
So we're going to give the agent access to the same model and our graph tools.

s660
01:03:23.599 --> 01:03:31.440
We're going to have our control agent that's going to have access to the same model and a set of file system probing tools that's maintained by paid by dantic.

s661
01:03:31.440 --> 01:03:38.160
Um so uh that's going to be our two test agents and then we're going to build up a corpus of questions.

s662
01:03:38.160 --> 01:03:40.160
Um each question is a real task.

s663
01:03:40.160 --> 01:03:45.280
We're going to figure out okay questions like what public types break if I change posting list?

s664
01:03:45.280 --> 01:03:46.400
There are two posting lists.

s665
01:03:46.400 --> 01:03:47.599
What do I use?

s666
01:03:47.599 --> 01:03:49.039
You know this kinds of thing.

s667
01:03:49.039 --> 01:03:55.839
um where we try to you know say okay this this should be a question that should be really good for the graph this is a question that's probably great for for for GP

s668
01:03:55.839 --> 01:04:10.240
um you know there's a control questions that are probably be good for both right so the gold data for this is actually we do pull the mind data from the graph we do cross reference it back to the source code so it's not that we're polluting the context of the gold with the output of the graph we

s669
01:04:10.240 --> 01:04:24.480
do have to cross reference it to make sure that the gold isn't cheating um so now we can go ahead and load up our uh our gold and we can basically start trying to set up the actual evaluation structure itself.

s670
01:04:24.480 --> 01:04:29.280
Um and so this is what lays out what's basically a good eval question versus a poor eval question.

s671
01:04:29.280 --> 01:04:35.920
Um you want an eval question that can be in an ideal world your eval should be able to be deterministically scored.

s672
01:04:35.920 --> 01:04:45.119
I think LM is judge is really fantastic for a lot of use cases but it tends to be overused when you really should be trying as hard as possible to have your scoring be as deterministic as possible.

s673
01:04:45.119 --> 01:04:51.200
It's going to reduce variance in your runs and it's going to reduce cost in your runs and it's going to make you happier in the long term.

s674
01:04:51.200 --> 01:05:01.440
Um, so you want to create questions where the answers are pullable from the you you're able to structure an answer to them that's gold that works.

s675
01:05:01.440 --> 01:05:03.680
You're able to check the answer against the gold.

s676
01:05:03.680 --> 01:05:10.240
Um, you're not just asking the agent to provide its opinion and then asking another agent to counterman that opinion with its opinion.

s677
01:05:10.240 --> 01:05:13.039
Those questions don't really tend to ever work out as emails.

s678
01:05:13.039 --> 01:05:16.000
Um, all right.

s679
01:05:16.000 --> 01:05:18.160
So, this is this is some craft text.

s680
01:05:18.160 --> 01:05:19.200
You don't need to worry about this too much.

s681
01:05:19.200 --> 01:05:23.760
This is just basically building out the um the the test cases that I was saying.

s682
01:05:23.760 --> 01:05:29.039
This is the deterministic part where we check to see if it cites all the evidence that we expect it to site.

s683
01:05:29.039 --> 01:05:37.839
Um, we're basically just creating a list of examples, creating a list of uh uh things it should be citing, and then checking to see if it's citing all of them.

s684
01:05:37.839 --> 01:05:43.760
For the eval themselves, we're using a configuration in Fidantic Evals just because it's really easy to set up.

s685
01:05:43.760 --> 01:05:53.920
Um, how this works is we build out uh our two different agents and then we build out um the agent runner itself and then we build out an evaluator, right?

s686
01:05:53.920 --> 01:06:00.160
So, this is what actually will evaluate the agent based on um the given tasks that we have in front of it.

s687
01:06:00.160 --> 01:06:06.799
We'll basically compare what the agent has said to uh what we expect it to for each given task, right?

s688
01:06:06.799 --> 01:06:12.640
And that way we're able to compare the responses and the results between our prep agent and our graph agent.

s689
01:06:12.640 --> 01:06:12.799
Right?

s690
01:06:12.799 --> 01:06:25.280
So that's how we're able to establish um one you know what is the actual utility of our skills and two um how do we uh what what can we improve or or what can we do better

s691
01:06:25.280 --> 01:06:27.039
as time goes on.

s692
01:06:27.039 --> 01:06:31.760
So um go ahead and run.

s693
01:06:32.000 --> 01:06:32.319
Right.

s694
01:06:32.319 --> 01:06:35.920
So I'm not going to go ahead and run the eval again because that would take about 30 minutes.

s695
01:06:35.920 --> 01:06:47.200
But um we can see that we run the eval the first time and we run it with uh k equals 1. So we just say for every task in our data set

s696
01:06:47.200 --> 01:06:53.200
onehot graph agent graph agent and we do that the graph agent doesn't do very well.

s697
01:06:53.200 --> 01:07:02.160
You can see that the the graph agent it does edge out the the graph agent a little bit but it's not really consistent and it's not really like very

s698
01:07:02.160 --> 01:07:04.559
it's not what we wanted, right?

s699
01:07:04.559 --> 01:07:11.599
But what we have to understand is that the goal wasn't ever necessarily to build something that's going to be more correct than Grap.

s700
01:07:11.599 --> 01:07:17.520
The goal because all of us that use Grap on a day-to-day basis when we use our coding agents know that it's very good at what it does.

s701
01:07:17.520 --> 01:07:19.039
Like it uses GP for a reason.

s702
01:07:19.039 --> 01:07:21.760
It's not going to be wrong a lot of the time, right?

s703
01:07:21.760 --> 01:07:31.280
But the important part is not that it's right the this one time I tried it, but that when I try it 10, five, 10, 15, 20 times in a row that it gets it right every time, right?

s704
01:07:31.280 --> 01:07:33.920
So that's how we need to actually start sampling these tests.

s705
01:07:33.920 --> 01:07:40.720
Now when we say we run these same set of tests over the same corpus five times to get our our pass at K metrics.

s706
01:07:40.720 --> 01:07:43.280
Now we get a slightly different um answer.

s707
01:07:43.280 --> 01:07:43.680
Right?

s708
01:07:43.680 --> 01:07:45.119
So we have two different statistics.

s709
01:07:45.119 --> 01:07:51.440
Our pass at K and our pass hat K. Our pass at K says across these five runs was this ever correct?

s710
01:07:51.440 --> 01:07:52.079
Right?

s711
01:07:52.079 --> 01:07:54.240
In that case they're still they're very similar.

s712
01:07:54.240 --> 01:07:56.640
Actually graph has a little bit of an edge there.

s713
01:07:56.640 --> 01:08:02.640
Um the graph doesn't sample any of the private variables only the the private part of the crates only the public.

s714
01:08:02.640 --> 01:08:06.559
So there's some control questions that the graph will always do better on.

s715
01:08:06.559 --> 01:08:15.440
Um but what's really interesting is when you change the metric and say is it doing the right job on every run and when you do that the numbers completely flip right

s716
01:08:15.440 --> 01:08:22.560
now all of a sudden the graph agent is twice as good at doing the correct thing all five runs than the than the graph agent.

s717
01:08:22.560 --> 01:08:23.279
Right?

s718
01:08:23.279 --> 01:08:30.480
Um, and we can see that within that ad that's doing that stuff and it's doing it more consistently, it's also doing it more efficiently, right?

s719
01:08:30.480 --> 01:08:37.920
We can see that over the course of all our different runs, on average, the graph approach took about three and a 3.1 times fewer tokens.

s720
01:08:37.920 --> 01:08:41.839
The last part is, okay, how do we actually get the agent to use the graph?

s721
01:08:41.839 --> 01:08:48.560
This is where your evals come into place because you basically have to write an eval that says, hey, examine all the tools in my trace.

s722
01:08:48.560 --> 01:08:51.440
How often is my agent using the tools that I want it to be using?

s723
01:08:51.440 --> 01:08:54.000
And then you can get your sort of adoption numbers, right?

s724
01:08:54.000 --> 01:08:59.199
So that's how you can get to a place where you can say, "Oh, I wrote a prompt that didn't have any instructions in it to use the tools.

s725
01:08:59.199 --> 01:09:01.839
I got 34% of tool usage.

s726
01:09:01.839 --> 01:09:02.640
I went back.

s727
01:09:02.640 --> 01:09:03.759
I changed my prompt.

s728
01:09:03.759 --> 01:09:06.000
Now I have 72% tool usage."

s729
01:09:06.000 --> 01:09:06.159
Right?

s730
01:09:06.159 --> 01:09:13.120
So the eval are the only way that you're going to be able to actually understand that the agent is using the tools that you want it to use.

s731
01:09:13.120 --> 01:09:16.239
Um, this is just a test of that exact concept.

s732
01:09:16.239 --> 01:09:22.719
we have uh two different prompts that are both trying to answer the same question with the same tool set.

s733
01:09:22.719 --> 01:09:30.560
Um if you if you nudge the prompt uh if you just naive it will just go ahead and grap um and then we'll start doing some other stuff.

s734
01:09:30.560 --> 01:09:37.638
If you nudge it the ideal world is that it will go and actually run the search and use get symbol to go and figure out the actual answer.

s735
01:09:37.638 --> 01:09:38.239
[snorts]

s736
01:09:38.239 --> 01:09:45.920
And so the recipe for building, you know, really solid, honest evales is to mine real task candidates out of your corpus.

s737
01:09:45.920 --> 01:09:49.359
Uh build a deterministic as much as you can, an anchored scoreer.

s738
01:09:49.359 --> 01:09:52.319
Uh and unit test the scorer itself first.

s739
01:09:52.319 --> 01:09:53.759
Um I like pantic eval.

s740
01:09:53.759 --> 01:09:58.640
There's many many other eval things out there, but the core structure of them is all going to be the same.

s741
01:09:58.640 --> 01:10:01.840
You have a data set, you have cases, you have an evaluator.

s742
01:10:01.840 --> 01:10:08.640
Um, make sure you run your evals many times over and reports pass at K, pass at K, uh, and all your different metrics.

s743
01:10:08.640 --> 01:10:21.360
And, uh, make sure that you are, um, putting in examples of things that your agent may not necessarily be good at because you don't want it to pass 100%.

s744
01:10:21.360 --> 01:10:24.560
It's not a useful if your e if your agent's passing your email 100%.

s745
01:10:24.560 --> 01:10:25.840
Your email is too easy.

s746
01:10:25.840 --> 01:10:28.800
Your agent should be close to failing your emails.

s747
01:10:28.800 --> 01:10:34.080
um like your emails are your chance to see where the metal hits the road and what your agent's going to do in as hard as possible circumstances.

s748
01:10:34.080 --> 01:10:41.840
So it doesn't really do a good anyone any service by just uh giving it very easy and very simple evals.

s749
01:10:43.280 --> 01:10:50.480
All right, anyone have any questions on this eval notebook before I jump back into the slides for the last part of the talk here?

s750
01:10:54.320 --> 01:11:02.480
Uh I I was just wondering in the example when or when you were discussing how you the agent would often use GP instead of graph.

s751
01:11:02.480 --> 01:11:04.960
So you had to retool it to be this search tool.

s752
01:11:04.960 --> 01:11:15.679
Uh was there ever any uh uh like attempt to take the graph and distill it into a file structure maybe like with some loss but a file structure that then GP would naturally

s753
01:11:15.679 --> 01:11:18.480
search over so you didn't have to define any separate tool at all.

s754
01:11:18.480 --> 01:11:18.719
Yeah.

s755
01:11:18.719 --> 01:11:20.000
Not today.

s756
01:11:20.000 --> 01:11:26.640
um just because uh there was a bit of a time crunch but I think that's definitely a reasonable approach.

s757
01:11:26.640 --> 01:11:35.040
I mean I think that's what you know that's corporate llm wiki you know with all these different things where like you can get a lot of these different benefits just from the markdown corpus

s758
01:11:35.040 --> 01:11:45.120
but like I and I think that that's that's definitely true and I think that you would still get a lot of the same like benefit out of it but I don't

s759
01:11:45.120 --> 01:12:00.640
like in the markdown like the ele wiki that kind of stuff right I still think that that your problem is going to be like you want real like edges and provenence between them right and like those connection connections are where it's going to be really hard because even when you do something like Grain or something like that,

s760
01:12:00.640 --> 01:12:01.199
right?

s761
01:12:01.199 --> 01:12:06.800
It yeah, it builds all the stuff in markdown, but it does build a graph on top of it for you to see the timeline and the connections and stuff, right?

s762
01:12:06.800 --> 01:12:18.719
So, I think at some level, you know, if your task is based in needing to understand the relationship between these different things, then at some level you're you will want some kind of graph.

s763
01:12:18.719 --> 01:12:24.080
Even though I do think that you can get a very similar level of of of of success out of out of an approach like that.

s764
01:12:24.080 --> 01:12:26.080
Yeah.

s765
01:12:30.560 --> 01:12:39.679
Um, do you think there's anything missing from a setup like this for letting non-technical, non-data engineer folks like kick off a PR with this as like the guide through a codebase?

s766
01:12:39.679 --> 01:12:45.040
Have you tried that or like is there a something you'd add before you empower people to do it that way?

s767
01:12:45.040 --> 01:12:47.760
Oh, like for this as a as a tool?

s768
01:12:47.760 --> 01:12:48.000
Yeah.

s769
01:12:48.000 --> 01:12:48.400
Oh, yeah.

s770
01:12:48.400 --> 01:12:52.080
I think um that's a good question.

s771
01:12:52.080 --> 01:12:52.400
Yes.

s772
01:12:52.400 --> 01:13:03.920
I I think the the the easiest part I mean would probably be like starting to hook it in as an MCP to the tools that you're already using.

s773
01:13:03.920 --> 01:13:15.600
So you would have you know this code graph in addition to your your cloud code and you would ask your cloud code to call the graph and do those other things and then you would write skills in order for it to actually use that that that properly.

s774
01:13:15.600 --> 01:13:18.080
I think that would be the the best approach there.

s775
01:13:18.080 --> 01:13:35.440
Um, but I would think that yeah, if you're less technical or non-technical, I I would think that, you know, the the the goal would be that you would if you want to engineer the agent to be using these things in the context of actual coding, you would promote these prompt engineering things I'm talking about here into skills

s776
01:13:35.440 --> 01:13:37.360
and give those to your coding agent.

s777
01:13:37.360 --> 01:13:37.600
Yeah.

s778
01:13:37.600 --> 01:13:46.159
Like that PR lands at the engineer for the first time with like a halfdeent first pass as a result of being more deterministically guided by this.

s779
01:13:46.159 --> 01:13:46.960
Yes.

s780
01:13:46.960 --> 01:13:47.199
Yeah.

s781
01:13:47.199 --> 01:13:54.880
And and I think that the the deterministic part is helpful because you can like like we were talking about before with for the question in the back like you can incrementally

s782
01:13:54.880 --> 01:13:57.120
just diff over what's changed, right?

s783
01:13:57.120 --> 01:14:10.239
Analyze that deterministically and get a really really fast conclusion back about what's what's happening and what's been changing since the last time as opposed to waiting for the agent to go and read everything and look at all the changes and and give you back that that same diff.

s784
01:14:10.800 --> 01:14:11.440
That's one back there.

s785
01:14:11.440 --> 01:14:13.679
I think

s786
01:14:15.440 --> 01:14:28.000
um did you uh experiment or think through like different uh levels of resolution like one thing I was thinking of um in some of the earlier talks is like you do

s787
01:14:28.000 --> 01:14:34.400
next token prediction and you will have the prefix I think people said which is just like

s788
01:14:34.400 --> 01:14:45.360
you know it's all at the level of like this is the whole string of everything in the codebase here you have the symbols but like the symbols is a certain granularity,

s789
01:14:45.360 --> 01:14:59.520
there would be higher levels of granularity that are imminent in the codebase and making those things searchable and accessible to an agent, you know, might have payoffs and I was wondering if you thought about that or if you have thoughts on how you'd approach that.

s790
01:14:59.520 --> 01:15:00.880
It's a fantastic question.

s791
01:15:00.880 --> 01:15:02.000
I did think about this.

s792
01:15:02.000 --> 01:15:06.320
There are uh and I will actually talk about it at the end of the at the end of the the presentation.

s793
01:15:06.320 --> 01:15:08.640
Um there was definitely some follow-ups I would do here.

s794
01:15:08.640 --> 01:15:16.960
Um if you were actually trying to build this um you would uh uh use something like SCIP to actually trace the call graphs inside of the codebase.

s795
01:15:16.960 --> 01:15:24.239
You can actually see what is calling what and how that interacts because you can track um like symbols and how they move across the codebase.

s796
01:15:24.239 --> 01:15:35.840
But if you want to actually track, you know, what is calling what and how it's all connected, that is the higher level that you're talking about and you need a you need a slightly augmented approach, another rung of of of deterministic parsing.

s797
01:15:35.840 --> 01:15:40.560
And you can do that um with uh there's like a Rust or SEIP.

s798
01:15:40.560 --> 01:15:44.320
Um there's similar SEP outputs for like TypeScript or or Python.

s799
01:15:44.320 --> 01:15:47.062
I was going to do all of that too, but I I ran out of time.

s800
01:15:47.062 --> 01:15:49.062
[laughter]

s801
01:15:50.159 --> 01:15:56.239
But to answer your question, yes, there's uh there's definitely more structure we can extract from a given codebase.

s802
01:15:56.239 --> 01:16:05.199
And that's sort of what we try to do at Type Def is find all the different ways that we can pull as much context as possible out of the data that we have.

s803
01:16:06.000 --> 01:16:21.840
Um I have a quick question for how would you say like this concept of the data context layer um combines with like something like a data catalog like is it

s804
01:16:21.840 --> 01:16:27.440
the data catalog would help feed a graph are they kind of redundant to both have in an enterprise

s805
01:16:27.440 --> 01:16:44.000
I don't think they're redundant I think they answer different questions um your data catalog is a a sort of fixed understanding of exactly what is present in your data warehouse which I think is a kind of a different

s806
01:16:44.000 --> 01:17:01.360
ask than what the context graph is trying to extract right um I think they they coexist you read information from the catalog to form the the context graph um so what you'll try to do is basically say okay I have my say my dbt project that has you know a bunch of models in it those models build

s807
01:17:01.360 --> 01:17:06.400
out uh uh tables and views in Snowflake, right?

s808
01:17:06.400 --> 01:17:08.960
So we may say we're going to ingest that DBT project.

s809
01:17:08.960 --> 01:17:10.239
We're going to pull in all that SQL.

s810
01:17:10.239 --> 01:17:16.640
We're going to parse it, link everything together, and then we're going to say, oh, okay, this model is corresponding to this table.

s811
01:17:16.640 --> 01:17:21.120
Let me go query the catalog, get information about that table, pull it into the graph as well.

s812
01:17:21.120 --> 01:17:40.719
So we can say okay this model is uh producing this physical table and this physical table has these columns and these columns have these these profiles and that information all comes from the catalog right so that catalog of information the information schema the profiling information all that stuff that's essential to what we do and that's all in

s813
01:17:40.719 --> 01:17:45.760
the catalog so I would say they coexist more than it's a replacement

s814
01:17:54.788 --> 01:17:55.280
[clears throat]

s815
01:17:55.280 --> 01:18:01.280
to try to bring this back to like data engineering and this concept of like eval.

s816
01:18:01.280 --> 01:18:10.480
Do you think about the same way in terms of like asking a set of like data questions and then seeing how those react uh to using the tool or not using the tool?

s817
01:18:10.480 --> 01:18:11.360
Yes.

s818
01:18:11.360 --> 01:18:11.600
Yeah.

s819
01:18:11.600 --> 01:18:11.840
Yeah.

s820
01:18:11.840 --> 01:18:12.239
Precisely.

s821
01:18:12.239 --> 01:18:13.120
Oh, so do you have more?

s822
01:18:13.120 --> 01:18:14.000
Sorry.

s823
01:18:14.000 --> 01:18:14.480
Sorry.

s824
01:18:14.480 --> 01:18:15.760
Oh, I didn't know if you were done.

s825
01:18:15.760 --> 01:18:17.360
Um yes absolutely.

s826
01:18:17.360 --> 01:18:30.000
Um we do use uh like I was saying we basically try to find like as as real a data corpus as we can and then we try to find multiple different kinds of scenarios

s827
01:18:30.000 --> 01:18:42.560
right so you have your basic fix a bug um where the scenario is we'll inject a diff into the codebase that introduces a bug right and then you'll ask the agent to go and try and fix this bug

s828
01:18:42.560 --> 01:18:51.360
um and then you can you can test you know that it looked at the right places that it checked the right things that when it fixed the bug, it didn't fix it upstream or fix it downstream, right?

s829
01:18:51.360 --> 01:18:52.560
By putting a band-aid on it, right?

s830
01:18:52.560 --> 01:18:55.280
It actually found the right model and fixed the right issue.

s831
01:18:55.280 --> 01:18:57.679
And you can make those tests easier or harder, right?

s832
01:18:57.679 --> 01:19:02.719
Like you can you can change the prompt language to say, "Oh, I think there's a problem with model XYZ."

s833
01:19:02.719 --> 01:19:09.199
Or you can say for the harder example, my down my dashboard Y is having this issue where this number is doubled.

s834
01:19:09.199 --> 01:19:10.159
I don't know why.

s835
01:19:10.159 --> 01:19:18.560
So forcing the agent to work backward from what that depends on upward through the tree to understand like where the actual root cause the issue is coming from and then

s836
01:19:18.560 --> 01:19:22.239
asserting that it actually goes in and fixes that problem instead of leaving it be.

s837
01:19:22.239 --> 01:19:27.040
Um so that's what's really hard and the problem is that the agents are always going to cheat.

s838
01:19:27.040 --> 01:19:31.120
Um so in your eval harness you have to be really careful about hardening it.

s839
01:19:31.120 --> 01:19:40.320
Um like for example your agent if you if you if you say that oh my test case is the agent will check out a branch and then it will go and try to fix the problem.

s840
01:19:40.320 --> 01:19:47.679
The agent will very happily cheat and use git to go back to main and try to figure out how you introduce the problem in this branch from from there.

s841
01:19:47.679 --> 01:19:52.719
Um and it'll go and very happily find the branch name that says this is a test and be like oh this is a test.

s842
01:19:52.719 --> 01:19:53.760
Oh I can cheat on this.

s843
01:19:53.760 --> 01:20:02.560
Um, so you have to be very careful about hardening your your your harness to actually uh be representative of the tasks that uh you want your agent to do.

s844
01:20:02.560 --> 01:20:04.719
Um, because that's always going to be the danger there.

s845
01:20:04.719 --> 01:20:09.679
For these evals, do you find people like encoding them from scratch for the most part or like

s846
01:20:09.679 --> 01:20:11.920
there are there are some existing frameworks for this?

s847
01:20:11.920 --> 01:20:14.000
Um, ADE bench is is a great one.

s848
01:20:14.000 --> 01:20:18.239
Um, it's probably the current industry standard for this kind of like data engineering benchmarks.

s849
01:20:18.239 --> 01:20:30.560
They have a very similar approach um for uh uh creating problems in repos using diffs and then having the agent go and read data from the warehouse and read data from the code and try to fix the problem.

s850
01:20:30.560 --> 01:20:33.199
Um so those do exist.

s851
01:20:33.199 --> 01:20:36.000
People are wiring them up and I think it's becoming more and more of a thing.

s852
01:20:36.000 --> 01:20:42.560
I think the harder part is always the scaffolding around it because for code you just have your codebase but you need a running database.

s853
01:20:42.560 --> 01:20:46.080
You need it to be you know populated with the right buggy data.

s854
01:20:46.080 --> 01:20:48.080
you need that data to be internally consistent.

s855
01:20:48.080 --> 01:20:53.600
So you don't want the agent to be spinning around trying to figure out, oh, oh, why are the numbers from 2024?

s856
01:20:53.600 --> 01:20:56.239
It's we only generated 2024 data in our sample set.

s857
01:20:56.239 --> 01:20:59.920
So, um, I think you run into those kind of issues a lot.

s858
01:20:59.920 --> 01:21:02.640
Um, which is why people are trying to standardize it in AD.

s859
01:21:02.640 --> 01:21:09.280
I think the hard part is building out a benchmark like that that other people can use that actually has a big enough data set.

s860
01:21:09.280 --> 01:21:13.679
That's going to be the tricky part and that's not something that has been done yet.

s861
01:21:13.679 --> 01:21:16.000
Uh, at least to me.

s862
01:21:16.000 --> 01:21:18.480
to my satisfaction.

s863
01:21:18.640 --> 01:21:25.920
Um but yeah, the uh you know, we get back the the this is just a summarization of the same results we just saw.

s864
01:21:25.920 --> 01:21:45.679
Um, we see that as we uh increase the amount of ability, as we tell, as we get the agent to use the tools more, um, the agent is able to be more consistent in its ability to answer the same set of questions and is able to answer those same set of questions um, more efficiently.

s865
01:21:45.679 --> 01:21:54.560
Um, and an example of why it's quite important to uh, run these eval multiple times.

s866
01:21:54.560 --> 01:21:58.880
It's not about your per shot accuracy, it's about your determinism.

s867
01:21:58.880 --> 01:22:04.560
Um, and the number that actually matters was how do you actually make sure that your agent is using the new tooling.

s868
01:22:04.560 --> 01:22:11.040
And so you really want to make sure that you're getting your adoption numbers up and that you're measuring your adoption of the tools because that's really what's going to get you

s869
01:22:11.040 --> 01:22:13.440
um the right the last part of the way, right?

s870
01:22:13.440 --> 01:22:17.360
You know, an elegant tool that gets reached for a third of the time is just a third of a tool.

s871
01:22:17.360 --> 01:22:24.000
Um, and every capability win that you can get from better tools is gated by this number.

s872
01:22:24.400 --> 01:22:31.120
Um, so this is why we built the eval, not to prove that the tool is good, but to prove that it's not going unused.

s873
01:22:31.120 --> 01:22:35.760
So the hard part about this whole talk was basically admitting that graph works really, really well.

s874
01:22:35.760 --> 01:22:46.880
Um, I did all this work and I built this context graph and I ran the evals and it was a tie and it was like it was kind of frustrating and you kind of want to keep optimizing and building the thing and trying to get more and more out of it and

s875
01:22:46.880 --> 01:22:48.320
it's kind of a trap, right?

s876
01:22:48.320 --> 01:22:52.880
That instinct is like to build more I think can sometimes lead to a sunk cost, right?

s877
01:22:52.880 --> 01:22:54.800
like the agent has its own reflex.

s878
01:22:54.800 --> 01:22:57.040
I have mine what I think it should be doing.

s879
01:22:57.040 --> 01:23:03.199
And like we're we're trying to meet in the middle, right, and actually figure out a way to move forward with this, right?

s880
01:23:03.199 --> 01:23:09.120
And it's like, oh, you think that maybe it's just that you just need more testing or more eval or or or more something, right?

s881
01:23:09.120 --> 01:23:17.679
And you end up with this this false feeling of I just need more rigor and more tests and then the agent will be better at using the tools.

s882
01:23:17.679 --> 01:23:26.400
But you really maybe you need to ask, hey, why is GP working so well as opposed to saying, why isn't it using my tools?

s883
01:23:26.400 --> 01:23:30.080
Well, maybe the answer is that GP actually does work really well already, right?

s884
01:23:30.080 --> 01:23:33.199
So maybe you're not going to win by fighting it.

s885
01:23:33.199 --> 01:23:37.360
Um maybe you're going to have to lean into it just a little bit.

s886
01:23:37.360 --> 01:23:39.760
So the agent is going to GP anyway.

s887
01:23:39.760 --> 01:23:45.360
Uh so when you keep trying to out engineer GP, um you're just going to have a kind of frustrating time.

s888
01:23:45.360 --> 01:23:55.280
Um, so if you can design as with the grain as much as possible, help the agents lean into what they want to do anyway, uh, it will help you out quite a bit.

s889
01:23:55.280 --> 01:24:06.000
Um, I've now done multiple talks where I I I had a conclusion, finished the talk, I said, "Let me rerun one set of evals and then I had to redo the whole talk because the conclusions were completely different."

s890
01:24:06.000 --> 01:24:14.400
Um, what had happened was I wrote my evals and I built this whole harness and I built it wrong and I got answers that were directionally correct.

s891
01:24:14.400 --> 01:24:17.920
They said that graph beat GP, but I wanted to look into it a little deeper.

s892
01:24:17.920 --> 01:24:25.840
And when you look into it a little deeper, you'll find that oh, that tool that we made, it returns the path to the file, but it's not like the right path.

s893
01:24:25.840 --> 01:24:30.880
So sometimes when they call search, it would find the wrong path and it would go down a wrong path to trying to read these files.

s894
01:24:30.880 --> 01:24:35.120
So you actually have to go in and you read all of your different traces, right?

s895
01:24:35.120 --> 01:24:38.960
Go into logfire, make sure or go into wherever you're you're you're you're sending your traces in, right?

s896
01:24:38.960 --> 01:24:55.520
and make sure that you're actually reading them because you're not supposed to just if you just look at your scoreboard, you might get answers that look directionally correct or or that look great for you, but they may be burying underlying issues that are actually much larger and and a much bigger problem inside of your inside of your

s897
01:24:55.520 --> 01:24:56.560
architecture.

s898
01:24:56.560 --> 01:25:00.880
Um, so the fix is never just to make the number higher or nicer.

s899
01:25:00.880 --> 01:25:02.639
It's to build a harder eval.

s900
01:25:02.639 --> 01:25:06.239
Uh, read the traces, find the artifact, harden the harness, and then rerun.

s901
01:25:06.239 --> 01:25:08.480
Never trust your own scoreboard.

s902
01:25:08.480 --> 01:25:12.080
Um and yeah, these are just some of the other thoughts I had.

s903
01:25:12.080 --> 01:25:14.800
The you know, the the models are only getting better.

s904
01:25:14.800 --> 01:25:25.040
Um so, you know, you can you we can continue to believe that the models are going to have less variance and more drift towards being able to uh do the work that we want them to do.

s905
01:25:25.040 --> 01:25:28.800
So, the easy part may actually be building the context, building the tools.

s906
01:25:28.800 --> 01:25:39.440
The hard part is really going to be building out your eval loop, building out your monitoring observability, and making sure that your agent is actually doing the work that you want it to do with the tools you want to be using.

s907
01:25:39.440 --> 01:25:48.480
Um, and so all the stuff is there, the the score, the loop, uh, it's all in the repo if anyone wants to play with that um on their own time.

s908
01:25:48.480 --> 01:25:52.400
And uh, yeah, so this is what I was talking about before.

s909
01:25:52.400 --> 01:25:54.400
You know, we did I did cheat by using Rust.

s910
01:25:54.400 --> 01:25:57.040
Um, the Rust compiler already did run one, right?

s911
01:25:57.040 --> 01:25:58.080
It's homework.

s912
01:25:58.080 --> 01:26:06.400
uh picked the language that was free because the lesson was you can build a well-engineered version of something uh on a relatively easy instance of a problem.

s913
01:26:06.400 --> 01:26:14.480
Uh and in general, if you can get things to work on a slightly easier problem, it's slightly easier to go and scale that back up to a larger issue.

s914
01:26:14.480 --> 01:26:20.239
Um so we now can say, oh wow, we have this really easy thing that we did for Rust.

s915
01:26:20.239 --> 01:26:21.360
I want to do for Python.

s916
01:26:21.360 --> 01:26:22.960
And Python is just harder, right?

s917
01:26:22.960 --> 01:26:32.560
like you can't really if you look at this these lines of code you're not going to get anywhere right if you GP client you're going to get stuck in a land of 50,000

s918
01:26:32.560 --> 01:26:45.440
clients you can use static analysis to to load this stuff and things like that but you know you're going to have like a a difficult time and so this is where a code graph really earned its keep right in a case like this where the code bases are larger

s919
01:26:45.440 --> 01:26:55.679
and the the languages are are less well structured a code graph is actually even more effective than grap in that case Because for Rust, you know, I want to see what implements this trait.

s920
01:26:55.679 --> 01:26:56.080
Okay, cool.

s921
01:26:56.080 --> 01:26:57.760
I just search imple, right?

s922
01:26:57.760 --> 01:26:58.880
Like it's really easy.

s923
01:26:58.880 --> 01:27:01.520
But for Python, that becomes a lot harder, right?

s924
01:27:01.520 --> 01:27:05.679
So that's where um you really would need to build something out that's more robust.

s925
01:27:05.679 --> 01:27:10.719
Um but the structure of the graph and the structure of the tools doesn't necessarily have to change, right?

s926
01:27:10.719 --> 01:27:14.320
It's the structure of the way that we ingest and load the data.

s927
01:27:14.320 --> 01:27:19.760
And if you make that modular, then you can build that out with for as many languages or as many projects as you want, right?

s928
01:27:19.760 --> 01:27:32.719
So that's kind of the underlying structure for like how type def works is we try to build out as many of these different injest processes as we can to pull data from many different places and then sort of synthesize it all into one.

s929
01:27:32.719 --> 01:27:34.080
Um there is another notebook for this.

s930
01:27:34.080 --> 01:27:44.400
I'm not going to go through it now because we've we've all been here for a little while and um but feel free to go through it yourselves and uh try to learn more if you want.

s931
01:27:44.400 --> 01:27:49.360
Um, last thing I was talking about was uh basically the meaning rung.

s932
01:27:49.360 --> 01:27:55.040
Um, the structure is what's telling you what's connecting to what, but the meaning is telling you sort of why.

s933
01:27:55.040 --> 01:27:58.880
Uh, so there's comments in code and dock strings and all the other information.

s934
01:27:58.880 --> 01:28:08.080
And folding all of that information in helps us understand the intent that the coders had when they wrote the code in addition to the actual structure of the code that currently exists.

s935
01:28:08.080 --> 01:28:24.639
So um we would basically want to add in small uh passes like this to say basically summarize the information from the code um and uh understand it at a at a rowbyrow basis so that search can now answer what the code does in addition to

s936
01:28:24.639 --> 01:28:26.320
um its location.

s937
01:28:26.320 --> 01:28:38.239
Um, and the important part when you're building something like that, um, you're building this sort of auditable semantic analysis that's going to go on top of your deterministic analysis.

s938
01:28:38.239 --> 01:28:43.120
Um, the important part is to ask many small questions and not one big question.

s939
01:28:43.120 --> 01:28:45.760
The answer isn't writing, you know, one really.

s940
01:28:45.760 --> 01:28:58.719
So when we do analysis in SQL, right, we don't just write one really really good prompt and throw your SQL at it and say, "Hey, uh, here's a pedantic model with 10 fields like give me all the information you can out of the SQL, right?

s941
01:28:58.719 --> 01:28:59.520
It's not going to work."

s942
01:28:59.520 --> 01:29:01.199
Um, a model is going to hallucinate.

s943
01:29:01.199 --> 01:29:02.880
It's going to invent information.

s944
01:29:02.880 --> 01:29:05.840
It's going to not collect as many as much information as it can.

s945
01:29:05.840 --> 01:29:19.120
You want to basically instead of asking the model 10 things about a chunk of code at once, you ask the question, you ask the model 10 different questions about 10 different smaller chunks of the code that answer specific questions.

s946
01:29:19.120 --> 01:29:19.600
Right?

s947
01:29:19.600 --> 01:29:25.120
You parse the structure deterministically and then ask many small focused questions that play to the model strengths.

s948
01:29:25.120 --> 01:29:25.760
Right?

s949
01:29:25.760 --> 01:29:33.280
Instead of asking the model to look at an entire block of SQL and tell you what all the columns mean, you parse the SQL and you say, "Hey, here's one column.

s950
01:29:33.280 --> 01:29:33.920
Here's its type.

s951
01:29:33.920 --> 01:29:34.880
Here's the comments."

s952
01:29:34.880 --> 01:29:44.000
you know, is this a measure, is this a dimension, is this blah blah blah, then at that point, you can ask the agent to do classification and summarization, and those are the things that it's good at, right?

s953
01:29:44.000 --> 01:29:50.480
And if you want to weigh it more, um, you give it a more you give it more, um, prompt input to that single focus question.

s954
01:29:50.480 --> 01:29:52.239
You don't try to cram more questions in.

s955
01:29:52.239 --> 01:29:56.000
You give it more context for it to be able to answer that single question with.

s956
01:29:56.000 --> 01:30:00.320
So, small, grounded, auditable, semantic operations over our structure.

s957
01:30:00.320 --> 01:30:04.080
Um, that's what our library fenic is for, and that's what we use it for as part of typed.

s958
01:30:04.080 --> 01:30:11.840
Um, so that last ladder, the last rung in the ladder, which is basically we have all this stuff layered in, we know how everything links together.

s959
01:30:11.840 --> 01:30:13.360
Now, what does it actually mean?

s960
01:30:13.360 --> 01:30:20.880
Now, we can start pulling some of that semantic understanding of what things actually mean by using the LLMs in the last layer.

s961
01:30:20.880 --> 01:30:24.719
Um, yes, so as I was talking about before, there's a lot more that you can do here.

s962
01:30:24.719 --> 01:30:35.840
you actually want to build a coding uh codebase graph analysis agent um you can use you can leverage call graphs to understand who's calling this and who references this right and

s963
01:30:35.840 --> 01:30:43.679
you know it can read the comments inside and outside of the code and it can fold in pros from outside of the code it can fold in GitHub issues and and and

s964
01:30:43.679 --> 01:30:54.480
your your your linear issues and your ticket and stuff so that's how you build this into something that's actually more useful and and and grounded is you start small you start with the smallest possible thing you can do which is how do

s965
01:30:54.480 --> 01:31:03.120
extract structure out of this codebase and then you move on to how do I take that structure and layer into it the semantic understanding that's really going to help

s966
01:31:03.120 --> 01:31:15.920
um build out a substrate that's useful for agents to actually work inside of this space and make changes um and that's sort of what the uh where the end goal is there

s967
01:31:15.920 --> 01:31:25.600
so anyway thanks so much guys uh this has been my talk the agent will grip anyway um I'm Brandon calendar from typed defev here is the link again for the workshop and my LinkedIn.

s968
01:31:25.600 --> 01:31:26.400
So, uh, thanks.

s969
01:31:26.400 --> 01:31:31.600
I can take any questions or or any other things, but thanks so much everybody.

s970
01:31:32.238 --> 01:31:34.238
[applause]

s971
01:31:36.159 --> 01:31:37.679
Any any questions?

s972
01:31:37.679 --> 01:31:41.360
I can pass around the mic for one last round.

s973
01:31:45.040 --> 01:31:54.960
Have you tried this kind of workflow like across repositories where things might be either microservices or like going across HTTP boundaries um with any success?

s974
01:31:54.960 --> 01:32:03.679
Uh not particularly no. I think where where this would where this would come into play is on a single repository.

s975
01:32:03.679 --> 01:32:07.520
Now we do that at scale for our like larger data architecture product.

s976
01:32:07.520 --> 01:32:09.120
So I think it can work.

s977
01:32:09.120 --> 01:32:14.560
um you basically just need to the hard part there is obviously figuring out the boundaries between the different things.

s978
01:32:14.560 --> 01:32:29.679
So you basically need to write boundary classifiers that basically try to understand um I was trying to do an example with Python and Rust like with maturin so you could have a similar example of that but I didn't have time to do that but you basically would need to write some way of

s979
01:32:29.679 --> 01:32:37.040
understanding when rust called code is calling Python or vice versa right to try to like build out that linking layer between them.

s980
01:32:37.040 --> 01:32:38.560
Um, so yes, you could do it.

s981
01:32:38.560 --> 01:32:42.560
I just think it would it would require quite a bit more work.

s982
01:32:44.560 --> 01:32:47.040
Any other questions?

s983
01:32:49.280 --> 01:32:50.639
Sorry.

s984
01:32:50.639 --> 01:32:53.040
I got you.

s985
01:32:57.760 --> 01:33:01.520
Cursor does a lot of the um level two stuff.

s986
01:33:01.520 --> 01:33:11.040
So like when you see like cursor doing searching through your codebase, it's usually doing semantic search like vector embedding search um and that's that's that that does work really well.

s987
01:33:11.040 --> 01:33:17.040
Like I think that there's a reason why there's all these like code rag products that do um that do semantic or retrieval, right?

s988
01:33:17.040 --> 01:33:18.560
And I think that that's that's really great.

s989
01:33:18.560 --> 01:33:27.920
It's just that you know that approach does get you the ability to find all the places in the code that you want to then examine with the agent.

s990
01:33:27.920 --> 01:33:31.199
But you still have to go and examine all those places with the agent.

s991
01:33:31.199 --> 01:33:31.600
Right?

s992
01:33:31.600 --> 01:33:39.600
The goal of this approach is if I want to be able to in one shot say I know that I'm looking at this function right here, right?

s993
01:33:39.600 --> 01:33:46.960
I basically want to be able to do the IDE click on the thing to see all its call sites but within my agent without my agent needing to grab everywhere.

s994
01:33:46.960 --> 01:33:49.280
That make sense?

s995
01:33:51.360 --> 01:33:53.199
Any other questions?

s996
01:33:53.199 --> 01:33:54.400
All right, one over here.

s997
01:33:54.400 --> 01:33:56.880
One sec.

s998
01:34:00.239 --> 01:34:08.000
I imagine half the fun of building this is that you end up with uh I don't know, sort of a a vector of the graph that you can recall.

s999
01:34:08.000 --> 01:34:08.719
Yeah.

s1000
01:34:08.719 --> 01:34:19.600
Um, I'm really curious about caching and storage as you really start thinking about scaling this given that uh making sure the agent gets it right on the first call is so key in steering.

s1001
01:34:19.600 --> 01:34:20.400
Yeah, that makes sense.

s1002
01:34:20.400 --> 01:34:33.520
I think um as you as you start to scale out your um your graph database, you're going to start finding that it's uh it can get very hairy very fast.

s1003
01:34:33.520 --> 01:34:35.440
And you're you're absolutely correct.

s1004
01:34:35.440 --> 01:34:47.840
Um, so I think storage is less of a concern, I think, because like these like yes, these graphs do tend to uh get a little sprawly, especially as they grow and they're harder to kind of garbage collect for, but

s1005
01:34:47.840 --> 01:35:00.560
they do um I'm less concerned about storage and more concerned about like the amount of compute that it takes to actually run like graph analysis across all these different things.

s1006
01:35:00.560 --> 01:35:08.080
Um, so I think the as you try to scale it up, I think the main thing you need to focus on like I was saying earlier is the incrementalism

s1007
01:35:08.080 --> 01:35:21.360
like figuring out how to rebuild this thing without regenerating all of it I think is probably the most important part and that's something we fight with at typed death for trying to build this you know at a larger scale across a large data architecture

s1008
01:35:21.360 --> 01:35:30.080
because you don't want to have to go and query your warehouse you know 500 times again right you don't want to have to go check you know compile and all the SQL code again right?

s1009
01:35:30.080 --> 01:35:33.120
You want to be able to update the stuff as it's coming in.

s1010
01:35:33.120 --> 01:35:40.400
So, I think the most important way to get around that is design for incremental indexing first as a first class citizen.

s1011
01:35:40.400 --> 01:35:47.040
Um, that's kind of the only way you'll be able to get to a point where you have to deal with the other scaling problems.

s1012
01:35:47.040 --> 01:35:56.000
I I think you have to deal with the problem of reyncing the information long before you have to deal with the actual scaling problem of how to store it or how to compute across it all.

s1013
01:35:57.360 --> 01:35:59.360
Yeah.

s1014
01:36:09.280 --> 01:36:17.040
I found that for us the building of the golden data set and maintaining it is always the biggest problem.

s1015
01:36:17.040 --> 01:36:25.280
Yeah, I I agree with you 100% to that and I I that's that is one I don't have a great answer for.

s1016
01:36:25.280 --> 01:36:32.800
I don't think because I I think I think so.

s1017
01:36:32.800 --> 01:36:34.239
I have two answers to that actually.

s1018
01:36:34.239 --> 01:36:47.119
One of them is that like the the gold data set isn't necessarily if I was if I was actually building this and I was testing it on something, I would pick a specific snapshot

s1019
01:36:47.119 --> 01:36:49.119
of that codebase, right?

s1020
01:36:49.119 --> 01:36:56.880
So I could say that this is my snapshot and that when I run my evals, I have a deterministically set of my gold questions, right?

s1021
01:36:56.880 --> 01:36:58.880
That's that's kind of a copout to your question, right?

s1022
01:36:58.880 --> 01:37:01.199
But that's what I would do if I was actually designing this, right?

s1023
01:37:01.199 --> 01:37:03.760
Is I would I was I would design this against a snapshot.

s1024
01:37:03.760 --> 01:37:08.159
So my eval questions are deterministic and aren't changing over time, right?

s1025
01:37:08.159 --> 01:37:15.040
Because the in an ideal world, right, your agent is going to get better and better at answering the same questions.

s1026
01:37:15.040 --> 01:37:21.600
But changing the questions based on changing the corpus doesn't actually help you evaluate if your agents are getting better.

s1027
01:37:21.600 --> 01:37:21.840
Right?

s1028
01:37:21.840 --> 01:37:26.000
The best way to evaluate if your agents are getting better is have them do the exact same task again.

s1029
01:37:26.000 --> 01:37:26.639
Right?

s1030
01:37:26.639 --> 01:37:35.199
So it's sort of your answer is you don't because you if for the purpose of evals your gold should be constant against a constant corpus.

s1031
01:37:35.199 --> 01:37:36.159
Does that make sense?

s1032
01:37:36.159 --> 01:37:45.280
Now if you're testing live yes like you need to be able to figure out new ways of testing it against your codebase but I think that and like your fixed eval set are different things.

s1033
01:37:45.280 --> 01:37:47.119
Does that make sense?

s1034
01:37:47.119 --> 01:37:47.360
Yeah.

s1035
01:37:47.360 --> 01:37:49.520
But like

s1036
01:37:49.679 --> 01:37:49.840
Yeah.

s1037
01:37:49.840 --> 01:38:01.360
Yeah, but I mean like if you so if we go with the codebase example and let's say you have a kind of like a complicated bug that is like a function that is calling a different function and

s1038
01:38:01.360 --> 01:38:15.600
there's like a in one of those functions there's like a bug or there's like a if like you said and then you have an eval that tests that if this bug was introduced then the agent went to this and this function and fixed it but then

s1039
01:38:15.600 --> 01:38:20.320
you do a refactoring and then this function is actually not even used anymore.

s1040
01:38:20.320 --> 01:38:22.480
So then the eval breaks.

s1041
01:38:22.480 --> 01:38:24.000
Oh, I see what you're saying.

s1042
01:38:24.000 --> 01:38:37.199
So in that case, I I think it's still I think man is still the same because what what I'm saying is that I when I'm building my eval, I would say my eval is a combination of a specific commit that like my code is at, right?

s1043
01:38:37.199 --> 01:38:46.639
The specific diff that I want to introduce to see if the agent can fix it and then the resultant set of changes from that prompt, right?

s1044
01:38:46.639 --> 01:38:48.480
that should always stay constant.

s1045
01:38:48.480 --> 01:38:56.480
Like I I I see what you're saying because like yeah, the code's going to change and it's not going to need to fix that issue anymore, but that's not what you're testing,

s1046
01:38:56.480 --> 01:38:56.800
right?

s1047
01:38:56.800 --> 01:39:01.040
You're not testing the agents ability to react to the new code.

s1048
01:39:01.040 --> 01:39:08.159
You're testing the agents ability to make that change based on that circumstance that it was in when you wrote the eval, right?

s1049
01:39:08.159 --> 01:39:11.119
Like that task I don't think has changed.

s1050
01:39:11.119 --> 01:39:16.560
even if like it's not relevant anymore, it's still testing something that's useful.

s1051
01:39:16.560 --> 01:39:19.040
Does that make sense?

s1052
01:39:20.560 --> 01:39:22.560
Question.

s1053
01:39:23.280 --> 01:39:24.159
Great talk.

s1054
01:39:24.159 --> 01:39:24.719
Loved it.

s1055
01:39:24.719 --> 01:39:29.520
Um this kind of reminds me of ontology um

s1056
01:39:29.520 --> 01:39:33.199
classes and this is what Palunteer is making a killing out of actually.

s1057
01:39:33.199 --> 01:39:33.760
Yeah.

s1058
01:39:33.760 --> 01:39:44.560
So um once you build the ontology and the dependency um can we talk with the LLM in these higher constructs now?

s1059
01:39:44.560 --> 01:39:44.880
Yes.

s1060
01:39:44.880 --> 01:39:46.639
With these classes.

s1061
01:39:46.639 --> 01:39:50.639
And is this is essentially what you were kind of insinuating at this point in time.

s1062
01:39:50.639 --> 01:39:50.880
Yes.

s1063
01:39:50.880 --> 01:39:53.040
So that that's that's basically our product, right?

s1064
01:39:53.040 --> 01:40:05.440
Is is is that concept of we we build um we build the context graph, we build the ontology layer on top of it where we try to organize all the information around your data architecture, where things are pointed, where things live, what things mean.

s1065
01:40:05.440 --> 01:40:12.880
Uh and then we give the agent access to that tool set to be able to answer questions and and make changes within your data architecture.

s1066
01:40:15.920 --> 01:40:18.159
Excuse me.

s1067
01:40:22.800 --> 01:40:23.199
All right.

s1068
01:40:23.199 --> 01:40:28.239
So, this may be a little naive, but so going back to kind of the product you guys are building, what you demonstrate here.

s1069
01:40:28.239 --> 01:40:32.639
So, a lot of stuff you showed here is very similar to what Neo4j showed at an earlier workshop,

s1070
01:40:32.639 --> 01:40:34.960
which is like, okay, here's a knowledge graph.

s1071
01:40:34.960 --> 01:40:45.679
You as the developer with domain expertise are going to build a knowledge graph in this area and they're going to provide it out to an agent and then make that agent query cipher to get some semantic information.

s1072
01:40:45.679 --> 01:40:52.639
I'm responsible for I'll call it a sprawling data state where there's a lot of you know thirdparty software as a service.

s1073
01:40:52.639 --> 01:40:54.239
We've got other things that are internal.

s1074
01:40:54.239 --> 01:40:57.760
My team is responsible for you know bringing in a lot of information for those things.

s1075
01:40:57.760 --> 01:41:00.159
Some of them are highly configurable like a salesforce.

s1076
01:41:00.159 --> 01:41:08.320
You bring the information in you put in the data warehouse etc etc and eventually you have business intelligence on the other side and also interaction.

s1077
01:41:08.320 --> 01:41:19.840
So the goal is how can I get my pretty small team of pretty good people to define the semantic relationships between all of this stuff in one place one time

s1078
01:41:19.840 --> 01:41:36.880
with good version control so that then I can both provide context and tools to AI agents for mostly analytic work but also write back occasionally and documentation for human beings that have to use these systems whether that's onboarding or in one place

s1079
01:41:36.880 --> 01:41:40.480
I haven't figured out the right way to do that yet.

s1080
01:41:40.480 --> 01:41:41.600
Let's chat.

s1081
01:41:41.600 --> 01:41:43.280
So, are are you gonna are you going to give me that?

s1082
01:41:43.280 --> 01:41:46.000
Because you also said like you're not going to do the data catalog,

s1083
01:41:46.000 --> 01:41:46.400
right?

s1084
01:41:46.400 --> 01:41:57.520
No, we're not gonna Well, the thing is I think I think what what I what I mean to say is we're not going to be limited to the data catalog because we're not going to be reading just your Snowflake catalog, just your, you know, just your warehouse information.

s1085
01:41:57.520 --> 01:42:09.679
The goal is to be reading from your Salesforce, from your from your from your fiverr, from your your um your air bite, from your whatever, from all the transfer mechanisms that you have, your airflow, your whatever.

s1086
01:42:09.679 --> 01:42:16.159
Incorporate all of that into a single understood context tree uh context graph that we can actually traverse.

s1087
01:42:16.159 --> 01:42:21.360
And then from there, the goal is to actually um distill out of that the things that you were talking about.

s1088
01:42:21.360 --> 01:42:25.840
The the main problem with semantic layers is no one wants to maintain a semantic layer, right?

s1089
01:42:25.840 --> 01:42:34.000
It's kind of a dirty word because once you build one, it's all really cool and and sick and sexy, but then you have to actually keep updating it every time you add models and it's it's a nightmare.

s1090
01:42:34.000 --> 01:42:37.360
So, the goal of of typed defe is you really just it was great.

s1091
01:42:37.360 --> 01:42:45.520
Thank you for like gave a great understanding of exactly what we want the product to be able to do, which is exactly what you said is to be able to

s1092
01:42:45.520 --> 01:42:49.360
reason across all those different layers that you have in your data architecture.

s1093
01:42:49.360 --> 01:42:56.960
build off a single graph that you can actually trace how data is flowing from Salesforce all the way down to your BI layer, right?

s1094
01:42:56.960 --> 01:43:11.040
And then when your agent asks when you ask your agent a question like hey um where is my like where is AR being coming from it's able to say oh yeah great that originates up here in the Stripe data actually and then it gets joined into your Salesforce lead data down here um to like figure out you

s1095
01:43:11.040 --> 01:43:14.880
know this this thing down here which is where the actual data is being calculated.

s1096
01:43:14.880 --> 01:43:21.119
Yeah, like that that lineage is very useful, but and for us it's, you know, partially for the data team that's doing this development.

s1097
01:43:21.119 --> 01:43:21.920
Great.

s1098
01:43:21.920 --> 01:43:24.800
But I want that to eventually be accessible to the business, right?

s1099
01:43:24.800 --> 01:43:27.119
So they can look at their report and we're like, that's not right.

s1100
01:43:27.119 --> 01:43:27.520
What the hell?

s1101
01:43:27.520 --> 01:43:35.920
And then rather than coming to my team and to yell at me, they can say, "Oh, well that's actually owned by these people on this system and this record and like this is the person I have to talk to."

s1102
01:43:35.920 --> 01:43:36.320
So

s1103
01:43:36.320 --> 01:43:37.280
keep it all together.

s1104
01:43:37.280 --> 01:43:38.239
Agree 100% with you.

s1105
01:43:38.239 --> 01:43:47.920
And I think that what we're trying to like one thing that the LLM sort of let us do at a high level that wasn't necessarily possible for is be that kind of translation

s1106
01:43:47.920 --> 01:43:58.080
layer that sits between the business folks the the accountants the analysts right the and then the data analyst the the engineers or whatever things like that right because what we're saying is

s1107
01:43:58.080 --> 01:44:07.920
what we want to say is okay that business analyst says hey that number in that revenue dashboard is wrong right so the process for them to fix that would be

s1108
01:44:07.920 --> 01:44:12.159
Okay, I need to go file a ticket somewhere that gets sat in some system.

s1109
01:44:12.159 --> 01:44:16.400
Someone picks it up, triages it, tries to figure out who owns that thing, right?

s1110
01:44:16.400 --> 01:44:17.760
Sends it over to them.

s1111
01:44:17.760 --> 01:44:18.960
They had to talk to the original.

s1112
01:44:18.960 --> 01:44:20.239
So I it's crazy, right?

s1113
01:44:20.239 --> 01:44:24.320
I worked at Salesforce for this, so I I get a little bit of that on a high level.

s1114
01:44:24.320 --> 01:44:32.560
Um, so the goal with type defeaf, right, is for able that that that business analyst to be able to go in and say, "Hey, my dashboard is broken, right?"

s1115
01:44:32.560 --> 01:44:34.000
They can put in the name of the dashboard.

s1116
01:44:34.000 --> 01:44:37.360
We can pull up the dashboard, you know, in that session where they are.

s1117
01:44:37.360 --> 01:44:40.800
we can trace back and say, "Oh, yes, I think I see what's the problem here.

s1118
01:44:40.800 --> 01:44:43.760
It's probably that, you know, I'm multiple layers back, this is happening.

s1119
01:44:43.760 --> 01:44:47.199
I can detect that this team owns this this model up here.

s1120
01:44:47.199 --> 01:44:55.679
I can go ahead and file that ticket for you with the resolution information already encaptured in it because I already can see what the root cause is because I looked back three layers and saw that this select was wrong, right?

s1121
01:44:55.679 --> 01:45:00.960
I can trace that back to a change that landed last week, you know, oh, that's why this broke here, right?

s1122
01:45:00.960 --> 01:45:06.719
And oh, we can see when that happened because there was an airflow run over here that actually populated that data, right?

s1123
01:45:06.719 --> 01:45:19.280
That's what we're trying to get to is that unified vision of how everything works from runtime through lineage and being able for the agent to actually answer those kind of questions and sit between those different layers and

s1124
01:45:19.280 --> 01:45:31.199
make that process more efficient because if if instead of that ticket sitting around and bouncing between five different teams, if it's immediately understood understood what the problem is and that gets routed to the right team right away, that team can even just

s1125
01:45:31.199 --> 01:45:38.239
in an ideal world where our product is ubiquitous and works perfectly, that team can just say, "Yeah, that solution sounds good check plus and then it goes and implements it.

s1126
01:45:38.239 --> 01:45:38.480
Right?

s1127
01:45:38.480 --> 01:45:39.920
So that's that's the dream, right?

s1128
01:45:39.920 --> 01:45:44.639
So but that that's kind of what we're trying to to get to and we're we are definitely part of the way there.

s1129
01:45:44.639 --> 01:45:47.520
So be happy to chat.

s1130
01:45:49.199 --> 01:45:49.760
All right.

s1131
01:45:49.760 --> 01:45:52.400
Any final questions?

s1132
01:45:52.880 --> 01:45:53.360
Cool.

s1133
01:45:53.360 --> 01:45:56.239
We'll be around if you guys want to connect with us on LinkedIn or anytime.

s1134
01:45:56.239 --> 01:45:58.320
We love chatting about context graphs generally.

s1135
01:45:58.320 --> 01:46:02.080
So thank you guys so much for coming through and we'll catch you guys later.

s1136
01:46:02.080 --> 01:46:04.320
Thanks everyone.

s1137
01:46:18.463 --> 01:46:20.463
[music]