WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=spxoDAfh5ig

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/spxoDAfh5ig.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.880 --> 00:00:13.200
Okay.

s2
00:00:13.200 --> 00:00:14.080
So, I'm I'm Peter.

s3
00:00:14.080 --> 00:00:15.120
I'm from Unblocked.

s4
00:00:15.120 --> 00:00:18.640
Um, over here is my colleague Brandon.

s5
00:00:18.640 --> 00:00:22.560
And today, uh, well, first of all, I just want to talk a little bit about what we do.

s6
00:00:22.560 --> 00:00:25.039
So, uh, we build a context engine.

s7
00:00:25.039 --> 00:00:27.599
Our company's name is Unblocked.

s8
00:00:27.599 --> 00:00:33.440
And uh we deliver organizational context to humans and increasingly of course agents.

s9
00:00:33.440 --> 00:00:35.920
Um so the session was originally a couple hours.

s10
00:00:35.920 --> 00:00:38.640
It's it's now one hour so it's quite condensed.

s11
00:00:38.640 --> 00:00:42.559
U which means that I'm going to step through the workshop pretty quickly.

s12
00:00:42.559 --> 00:00:53.440
If you don't get uh you know all the steps nailed down as we go don't worry about it because the repos up online um and you can run through it at your leisure um afterwards.

s13
00:00:53.440 --> 00:01:04.720
Um, so the thesis here is that rag and I'm just going to say that rag in our parlance today is going to be semantic similarity search or vector retrieval.

s14
00:01:04.720 --> 00:01:14.640
Okay, rag is only half of retrieval and today we're going to build the other half which is turning na natural language into real database queries.

s15
00:01:14.640 --> 00:01:19.439
So here's the again the QR code for setup um if people haven't already done so.

s16
00:01:19.439 --> 00:01:25.759
I'm gonna now skip um on to the next thing and I'm just going to briefly bring up hopefully that renders.

s17
00:01:25.759 --> 00:01:26.320
No, it won't.

s18
00:01:26.320 --> 00:01:27.280
Of course.

s19
00:01:27.280 --> 00:01:30.560
Give me one second to switch.

s20
00:01:32.640 --> 00:01:34.640
Whoops.

s21
00:01:37.520 --> 00:01:39.200
And hopefully people can see this.

s22
00:01:39.200 --> 00:01:40.560
Uh this is the repo here.

s23
00:01:40.560 --> 00:01:42.720
So setup steps are pretty simple.

s24
00:01:42.720 --> 00:01:50.320
Um there are a few sort of uh dependencies up here, prerexs that that uh you'll need to run through.

s25
00:01:50.320 --> 00:02:05.920
Um and the the course throughout the course of the workshop, we're going to be running through a series of pull requests uh and the network is basically gone or at least isn't running properly.

s26
00:02:05.920 --> 00:02:08.799
Let's try this one instead.

s27
00:02:16.400 --> 00:02:19.200
Well, the good thing is we don't need network for this talk.

s28
00:02:19.200 --> 00:02:23.840
Um, so hopefully you got everything before we started.

s29
00:02:23.840 --> 00:02:26.720
Um, because this one's obviously not going to work.

s30
00:02:26.720 --> 00:02:32.800
But at any rate, uh, we're going to be following a series of six pull requests and they stack on top of each other.

s31
00:02:32.800 --> 00:02:34.000
So, I'll be There we go.

s32
00:02:34.000 --> 00:02:36.720
I'll be referencing these as we go.

s33
00:02:36.720 --> 00:02:41.280
Um, and I'll just go back to here so that everyone can see that.

s34
00:02:41.920 --> 00:02:43.440
And back to the keynote.

s35
00:02:43.440 --> 00:02:47.040
Now I'm gonna have to extend it again.

s36
00:02:47.040 --> 00:02:49.040
Sorry.

s37
00:02:58.640 --> 00:03:01.120
There we go.

s38
00:03:01.920 --> 00:03:02.239
All right.

s39
00:03:02.239 --> 00:03:03.280
Hopefully people can see that.

s40
00:03:03.280 --> 00:03:04.000
Great.

s41
00:03:04.000 --> 00:03:12.080
So um our agents need context and context is scattered across every tool that engineering teams live in.

s42
00:03:12.080 --> 00:03:26.239
So you're going to find it in PRs, review comments, um issues and tickets like in linear and jura docs [clears throat] and wikis, notion confluence, and of course in conversations and places like Slack and Teams.

s43
00:03:26.239 --> 00:03:30.959
And all of that is semantically queryable, but it's it also has structure.

s44
00:03:30.959 --> 00:03:50.319
It has authors, dates, uh, states, labels, relationships, etc. And you know, the the instinct I I love to say this like even two years ago is ancient history in the realm of AI, but the instinct even two years ago was let's just dump this all in a vector store and retrieve it and rerank it and hopefully we

s45
00:03:50.319 --> 00:03:52.080
get the right thing.

s46
00:03:52.080 --> 00:03:54.480
Um, but that's that's not going to work.

s47
00:03:54.480 --> 00:03:59.840
So I'm just going to step back and and kind of relate this to the larger discussion about context.

s48
00:03:59.840 --> 00:04:11.280
um in in the context of a context engine that serves organizational context um this is this is made up of more than just code as I said we're connecting everything together

s49
00:04:11.280 --> 00:04:26.160
so the scattered context uh on the left in all your code and PRs docs etc goes in and the idea is to get you know grounded context that relates to each other out the other side that's specifically tailored to the task at hand

s50
00:04:26.160 --> 00:04:34.800
um and in that model uh we have to care not just about the semantic relationships, but of course the structural relationships that connect all the data.

s51
00:04:35.600 --> 00:04:43.199
So again, rag is about embedding the query, finding docs with similar meaning, and then we take that stuff and we stuff it into the prompt, right?

s52
00:04:43.199 --> 00:04:59.120
Well, um, that shines for content questions like find the code, find the PR that changed it, find the Slack thread that debated it, but um, and if you wrap it in an agent loop, you can get pretty far with that.

s53
00:04:59.120 --> 00:05:06.960
And you can also wire this thing up to like um, you know, tools that can recurse files and it acts as a good jump off point.

s54
00:05:06.960 --> 00:05:17.120
you get a good semantic head start and then from there your agent can recurse over the actual file system in GitHub um and you can get pretty far with that.

s55
00:05:17.120 --> 00:05:22.960
But in our experience um our this is not really the way things go in practice.

s56
00:05:22.960 --> 00:05:29.600
engineers often ask questions like this PRs that I merged last week or my team merged last week.

s57
00:05:29.600 --> 00:05:40.400
Um they're they're basically, you know, hunting for something in particular and they know where the starting point is, but they don't know where the destination is, but they do know the relationships.

s58
00:05:42.320 --> 00:05:46.479
So, as an example, PRs I merged last week, let's think about this in a rag context.

s59
00:05:46.479 --> 00:05:58.160
you'd you'd you know embed that sentence, you'd find the nearest vectors, you'd rerank it top K. Uh you get PRs that mention merging about a talk last week, but

s60
00:05:58.160 --> 00:06:08.479
uh you wouldn't get the actual relationships like the author, the merge state, um and whether it was merged last week actually.

s61
00:06:08.479 --> 00:06:17.039
So what you could end up with is the wrong answer where you pull up PR that's semantically related but it was merged three years ago.

s62
00:06:17.039 --> 00:06:24.000
Um and so that doesn't actually answer the question because there's a structural there's a temporal aspect to that question.

s63
00:06:24.000 --> 00:06:33.120
And so retrieval has two halves here semantic and again today we're going to be talking about this structured queries.

s64
00:06:34.400 --> 00:06:35.199
And here's the key.

s65
00:06:35.199 --> 00:06:38.960
We're going to take uh the model doesn't actually answer the question in this paradigm.

s66
00:06:38.960 --> 00:06:43.440
It writes the query that accesses the database.

s67
00:06:45.919 --> 00:06:46.319
Okay.

s68
00:06:46.319 --> 00:06:52.720
So, um it would be naive to say that generating the query is the only thing that we have to do.

s69
00:06:52.720 --> 00:07:04.080
Um in order to generate the query and to do it reliably um and and to take care of maybe even some security concerns, we need to look at all four of these points here.

s70
00:07:04.080 --> 00:07:10.000
So that first of all the LM needs to be aware of what the schema is before uh creating the query.

s71
00:07:10.000 --> 00:07:13.680
It needs to understand uh some of the relationships in the data.

s72
00:07:13.680 --> 00:07:28.160
So for example, if you pass a natural language query that says show me Peter's open PRs, um then it's not going to know what to do because it'll synthesize a query that says Peter when really the GitHub login name might be different.

s73
00:07:28.160 --> 00:07:30.960
So it needs to be able to map that across.

s74
00:07:30.960 --> 00:07:36.560
Um, and of course you're going to have some uh untrusted queries coming out the other end.

s75
00:07:36.560 --> 00:07:45.840
Um, maybe using the function operator in the in parlance which might execute JavaScript directly on the server and we probably don't want that.

s76
00:07:45.840 --> 00:07:52.560
Um, and then finally, if there are errors the first time around, we want to be able to self-correct those things.

s77
00:07:52.560 --> 00:07:56.000
So here's here's what this looks like in one single shot.

s78
00:07:56.000 --> 00:07:58.319
What we want is something that looks like this.

s79
00:07:58.319 --> 00:08:02.319
Um it it maps uh my colleagueu's name is Rashine.

s80
00:08:02.319 --> 00:08:05.360
His user login name is is Rashab.

s81
00:08:05.360 --> 00:08:14.080
Um and then we want to be able to map things like the state and get maybe uh operators like limit and sort.

s82
00:08:16.479 --> 00:08:18.960
So uh this is what we're going to build.

s83
00:08:18.960 --> 00:08:28.960
We're going to start with a full UX, a stubbed engine, um, that is not capable to start with, and we're going to gradually build this up.

s84
00:08:28.960 --> 00:08:31.120
And then at step three, the thing will come alive.

s85
00:08:31.120 --> 00:08:36.000
And so we'll actually be able to issue queries to it and see what it does.

s86
00:08:36.159 --> 00:08:43.599
And uh, if you hadn't seen that before, now is another good time to scan that and get it in.

s87
00:08:46.959 --> 00:08:49.519
I'm here to help you.

s88
00:09:00.000 --> 00:09:02.000
Yeah.

s89
00:09:03.040 --> 00:09:03.360
Yes.

s90
00:09:03.360 --> 00:09:04.800
Any repo is fine.

s91
00:09:04.800 --> 00:09:07.360
Doesn't matter which one.

s92
00:09:07.360 --> 00:09:13.200
Um and if if you want to take a shortcut, you can just point an agent at the thing and let it set everything up for you.

s93
00:09:13.200 --> 00:09:15.519
That works.

s94
00:09:16.720 --> 00:09:19.839
Okay, let's move.

s95
00:09:19.839 --> 00:09:21.680
So, this is what we're going to do to start with.

s96
00:09:21.680 --> 00:09:25.360
Um, by now you you probably should have run this ingestion thing.

s97
00:09:25.360 --> 00:09:26.480
Don't worry if you haven't.

s98
00:09:26.480 --> 00:09:29.760
Um, again, like you can revisit this at the end.

s99
00:09:29.760 --> 00:09:36.320
I'm just going to quickly show what the the kind of barebones interface looks like for this thing.

s100
00:09:36.640 --> 00:09:37.839
So, now this thing is running.

s101
00:09:37.839 --> 00:09:41.120
Now, if I type anything in here, this is obviously not going to work.

s102
00:09:41.120 --> 00:09:43.839
Um, and what you should see is something like that.

s103
00:09:43.839 --> 00:09:46.080
Um, there's also a tab for chat.

s104
00:09:46.080 --> 00:09:51.200
This is a fully implemented agent uh that can uh do work for you.

s105
00:09:51.200 --> 00:09:54.480
So, we'll come back to that in a sec.

s106
00:09:58.000 --> 00:10:01.839
And so, this this is the contract in the code if you wanted to read.

s107
00:10:01.839 --> 00:10:03.760
Um, we're going to don't worry about this.

s108
00:10:03.760 --> 00:10:05.519
We're going to skip through this.

s109
00:10:05.519 --> 00:10:10.240
Um, but basically like that's the entry point and we're going to be building around that.

s110
00:10:10.240 --> 00:10:14.240
So the first thing we want to talk about is schema discovery.

s111
00:10:14.240 --> 00:10:24.320
Um during ingestion, if you if you look at the way that the ingestion script worked, it it actually had to build out a schema in order to download the content and store it.

s112
00:10:24.320 --> 00:10:27.760
Um but you can imagine that you may not know the schema.

s113
00:10:27.760 --> 00:10:34.720
So the ingestion may be happening in a service elsewhere and you want to build like a fully dynamic schema discovery engine.

s114
00:10:34.720 --> 00:10:36.640
Um schemalis.

s115
00:10:36.640 --> 00:10:38.480
So there's no schema to read.

s116
00:10:38.480 --> 00:10:42.800
um unlike you know PG or something where we could read the table definitions.

s117
00:10:42.800 --> 00:10:52.320
Um so what we'll do is we'll sample the documents to infer the schema and that's the whole trick.

s118
00:10:52.320 --> 00:10:54.399
So the we're just going to reconstruct it.

s119
00:10:54.399 --> 00:10:58.720
We're going to take um all the nodes and merge them down.

s120
00:10:58.720 --> 00:11:03.680
So this is going to sample 300 documents, collapse it into a single tree.

s121
00:11:03.680 --> 00:11:08.399
uh nulls are skipped because missing fields won't be able to constrain the type.

s122
00:11:08.399 --> 00:11:14.000
And then uh we're going to do something kind of interesting here with enum types.

s123
00:11:17.360 --> 00:11:23.920
And so now uh with with enum types, we can do an interesting sort of procedural trick.

s124
00:11:23.920 --> 00:11:25.600
This is just one thing.

s125
00:11:25.600 --> 00:11:36.640
I would suggest using procedural tricks to to do this type of thing because you could pass it through an LLM and do extraction and that works u but it's fairly non-deterministic.

s126
00:11:36.640 --> 00:11:48.079
So you always need some kind of structural um harness around it that is running procedural code to either validate the things that you pull out or to actually do the extraction itself.

s127
00:11:48.079 --> 00:11:54.240
And in this sample project um it's the procedural code that's doing the extraction.

s128
00:11:55.760 --> 00:11:59.600
And let's go and show that.

s129
00:12:00.480 --> 00:12:08.480
So we'll do and then run it again.

s130
00:12:09.600 --> 00:12:12.720
And now what we get is this thing here.

s131
00:12:12.720 --> 00:12:22.240
So the schema ran uh the the discovery process ran and it created a schema for us that outlines what the pull request schema looks like in GitHub.

s132
00:12:22.240 --> 00:12:25.120
It even created some indexes for us.

s133
00:12:25.120 --> 00:12:36.800
Now, this is like marked down on purpose because we're going to take this schema and pass it to the LLM when we synthesize the actual uh query.

s134
00:12:40.000 --> 00:12:45.920
Now, the second step is the thing that I referred to a few slides ago about identity resolution.

s135
00:12:45.920 --> 00:12:51.920
And this is because like my lo GitHub login um is similar but it's not quite the same.

s136
00:12:51.920 --> 00:13:03.519
And so if I were to ask a question like um hey find all the PRs by Peter um it if without doing some sort of identity mapping it would just say

s137
00:13:03.519 --> 00:13:07.200
you know it would just match across Peter and it's not going to find anything.

s138
00:13:07.200 --> 00:13:09.284
So we have to be able to solve that.

s139
00:13:09.284 --> 00:13:11.284
[snorts]

s140
00:13:11.440 --> 00:13:14.480
And again, um, we're going to do this.

s141
00:13:14.480 --> 00:13:19.040
At first blush, this seems like we're coupling to the schema a little bit here.

s142
00:13:19.040 --> 00:13:24.079
Um, but you know, it is absolutely possible to determine this dynamically.

s143
00:13:24.079 --> 00:13:30.240
And you can have uh your LLM essentially synthesize the mappings for you.

s144
00:13:32.000 --> 00:13:35.600
So, but in in our case, we're just going to do these simple little tricks.

s145
00:13:35.600 --> 00:13:38.720
Um very simple sort of fuzzy matching.

s146
00:13:38.720 --> 00:13:40.959
Um Levenstein distance is fine.

s147
00:13:40.959 --> 00:13:47.440
You can just like match across the names and uh there you go.

s148
00:13:50.160 --> 00:13:54.560
Oh, actually I want to show you one thing before we get on to that one.

s149
00:13:54.560 --> 00:13:57.199
So let's do the same thing again.

s150
00:13:57.199 --> 00:13:59.920
We can pop up

s151
00:14:01.440 --> 00:14:03.040
Oh yeah, I'm sorry.

s152
00:14:03.040 --> 00:14:05.040
Absolutely.

s153
00:14:05.279 --> 00:14:07.519
Tell me when it's big.

s154
00:14:07.519 --> 00:14:08.240
Good.

s155
00:14:08.240 --> 00:14:08.639
Thanks.

s156
00:14:08.639 --> 00:14:11.040
Good call out.

s157
00:14:11.600 --> 00:14:12.480
Okay.

s158
00:14:12.480 --> 00:14:15.040
So, let's run our thing again.

s159
00:14:15.040 --> 00:14:20.800
And now, um Oh, actually, I have to I have to wait till the next step.

s160
00:14:20.800 --> 00:14:21.920
I apologize.

s161
00:14:21.920 --> 00:14:24.720
That's not going to be wired up just yet.

s162
00:14:24.720 --> 00:14:31.199
So, we'll go on to the thing that will really give us the payoff here, which is step three.

s163
00:14:36.720 --> 00:14:42.000
So this is the actual query plan step and it's it's it's arguably sort of the simplest part.

s164
00:14:42.000 --> 00:14:44.480
It's the LM part.

s165
00:14:44.480 --> 00:14:54.399
Um and what we do here is we pass the schema give it a tool and force it to call that tool for name extraction and synthesis.

s166
00:14:54.399 --> 00:14:56.959
Um and then it will execute the plan.

s167
00:14:56.959 --> 00:15:00.959
There's one other thing which is the temporal aspect.

s168
00:15:00.959 --> 00:15:08.560
uh LLMs are actually pretty good about generating temporal queries if you pass them the current date and time.

s169
00:15:08.560 --> 00:15:14.079
So that's another important thing to do in the in the prompt.

s170
00:15:18.639 --> 00:15:21.120
And so here this is the name extraction part.

s171
00:15:21.120 --> 00:15:25.519
We're going to force that tool call to happen in this case.

s172
00:15:28.880 --> 00:15:33.440
And at this point I am going to show you the the payoff for this.

s173
00:15:33.440 --> 00:15:47.680
So there we go.

s174
00:15:47.680 --> 00:15:51.600
So now I can do things like this.

s175
00:15:52.320 --> 00:15:59.279
open PRS by Peter and bam.

s176
00:15:59.279 --> 00:16:13.920
So what we'll see here is that it uh generated this um this match clause and uh you can't see it from here but it it mapped Peter to my GitHub user login puery

s177
00:16:13.920 --> 00:16:17.759
and generated the query and that's the outcome.

s178
00:16:22.320 --> 00:16:31.600
So that's cool, but um we need to get on to the next step because even though we are able now to query the data, we have some safety concerns.

s179
00:16:31.600 --> 00:16:48.480
Um it could be the case that it produces untrusted output and so you know it could use the wear or function clauses and literally run JavaScript on the server which means that this is now a prompt injection point.

s180
00:16:48.480 --> 00:16:56.639
Um, you could also have hidden fields in your schema like ID or metadata fields that leak internals.

s181
00:16:56.639 --> 00:17:01.759
So, we need to sanitize um, and we don't deny list these things.

s182
00:17:01.759 --> 00:17:03.040
We allow list them.

s183
00:17:03.040 --> 00:17:05.039
So, it's deny by default.

s184
00:17:05.039 --> 00:17:12.240
You want to create basically an allow list of allowed operators and uh, you know, potentially an allow list of allowed fields.

s185
00:17:12.240 --> 00:17:16.240
In our case, this is dynamic schema discovery, so we're not going to do that.

s186
00:17:16.240 --> 00:17:21.600
Um, but we could have some fields that we know about like metadata fields.

s187
00:17:21.600 --> 00:17:27.039
And this is a a useful thing for like tenant uh tenant isolation.

s188
00:17:27.039 --> 00:17:32.480
So if you have a a uh collection that houses multiple tenants, I don't recommend that.

s189
00:17:32.480 --> 00:17:43.440
you should probably separate your collections by tenant, but if you did, then you could have a metadata field that contains the tenant ID and you would want to hide that

s190
00:17:43.440 --> 00:17:49.520
no that um information from the LLM, but also validate the output.

s191
00:17:49.520 --> 00:17:58.240
So if it outputs that metadata field, you need to wipe it out and then you need to inject your own metadata field underneath.

s192
00:18:05.120 --> 00:18:08.720
And again, like technically this is it's not actually that difficult to do.

s193
00:18:08.720 --> 00:18:10.000
You just have to write the code.

s194
00:18:10.000 --> 00:18:11.679
It's all procedural.

s195
00:18:11.679 --> 00:18:16.640
Um I I wouldn't recommend using LLMs for this part.

s196
00:18:19.679 --> 00:18:26.320
And so again, allow list, don't deny or deny everything, I should say.

s197
00:18:33.120 --> 00:18:38.400
Now the last part is um we're going to this kind of dovetales into the next step.

s198
00:18:38.400 --> 00:18:43.679
At the end of validation we're obviously going to return an error and it's going to get dumped out.

s199
00:18:43.679 --> 00:18:46.559
And right now that error is just it's just there.

s200
00:18:46.559 --> 00:18:49.520
We're not going to do anything with it just yet.

s201
00:18:49.520 --> 00:18:52.000
But I will show you now what this looks like.

s202
00:18:52.000 --> 00:19:04.000
Um, oops.

s203
00:19:04.000 --> 00:19:09.679
Stay on that one and run our service.

s204
00:19:10.000 --> 00:19:20.320
So now I'm going to write run another query that checks for authentication in the title and we've disallowed regular expressions.

s205
00:19:20.320 --> 00:19:24.400
So now what we can see is that it rejects the query.

s206
00:19:35.919 --> 00:19:42.880
And finally, this is the thing that kind of ties it all together is the retry loop.

s207
00:19:47.280 --> 00:19:50.799
The plan can fail in a couple of different ways, right?

s208
00:19:50.799 --> 00:19:52.640
Um, it can fail validation.

s209
00:19:52.640 --> 00:19:59.520
So maybe the uh the LM actually outputs things that are illegal according to our validator.

s210
00:19:59.520 --> 00:20:02.960
Or it could actually fail during execution.

s211
00:20:02.960 --> 00:20:09.840
Um, so for example, let's imagine that it it mixed an inclusion exclusion project.

s212
00:20:09.840 --> 00:20:13.280
Um, then that's obviously going to fail.

s213
00:20:13.280 --> 00:20:15.039
And then but we can self-correct.

s214
00:20:15.039 --> 00:20:20.480
We can take that error and pass it back into the LLM and say hey that was wrong.

s215
00:20:20.480 --> 00:20:27.120
Um maybe you can do better up to some number of max attempts of course.

s216
00:20:30.080 --> 00:20:41.200
So I'm going to show now what this looks like back over to our project so we can punch up a level.

s217
00:20:41.200 --> 00:20:41.840
Whoops.

s218
00:20:41.840 --> 00:20:43.840
Sorry.

s219
00:20:45.520 --> 00:20:46.400
You have to forgive me.

s220
00:20:46.400 --> 00:20:50.480
I actually have to read this off of the screen out here.

s221
00:20:56.159 --> 00:20:57.840
Okay.

s222
00:20:57.840 --> 00:21:01.440
So now if we run that same query.

s223
00:21:09.760 --> 00:21:10.240
Perfect.

s224
00:21:10.240 --> 00:21:14.159
So, what we can see is that it took several attempts to get it right.

s225
00:21:14.159 --> 00:21:22.240
Um, by the third attempt though, there's a bit of a problem here because we said no regular expressions, no FTS.

s226
00:21:22.240 --> 00:21:29.360
Um, so now it said we're going to just use match, but now it's an exact match.

s227
00:21:29.360 --> 00:21:41.760
So it tries to find authentication fails which means there's one more thing that we can do which is this.

s228
00:21:42.480 --> 00:21:45.440
We'll add full text search.

s229
00:21:49.679 --> 00:21:50.799
It's pretty basic.

s230
00:21:50.799 --> 00:21:59.760
Um we need to create an index obviously and we'll combine it with author and state filters uh in the same match.

s231
00:21:59.760 --> 00:22:07.039
And so now we should be able to answer a question like this.

s232
00:22:07.919 --> 00:22:09.120
And let's see what happens.

s233
00:22:09.120 --> 00:22:14.000
And see if it is actually going to do what I said it's going to do.

s234
00:22:29.840 --> 00:22:30.480
Bam.

s235
00:22:30.480 --> 00:22:32.880
There we go.

s236
00:22:36.000 --> 00:22:48.559
And so now the the final payoff here is if we go into the chat tab, we can ask some sophistic more sophisticated questions that um you know an agent can actually reason about behind the scenes.

s237
00:22:48.559 --> 00:23:00.880
So you can say things like um did we work on anything authentication related last week.

s238
00:23:10.799 --> 00:23:14.400
So here we can see that it generated the query.

s239
00:23:14.400 --> 00:23:18.000
Now it's pulling the data and kind of reasoning about it.

s240
00:23:18.000 --> 00:23:20.400
There we go.

s241
00:23:22.400 --> 00:23:24.720
That's it.

s242
00:23:35.679 --> 00:23:38.880
So, just a quick summary of what we built.

s243
00:23:38.880 --> 00:23:41.200
We're basically like reaching the end.

s244
00:23:41.200 --> 00:23:44.000
There's going to leave lots of time for questions.

s245
00:23:44.000 --> 00:23:49.120
So just to recap here, um there are questions that that rag can't touch.

s246
00:23:49.120 --> 00:23:54.080
All those structural questions around things like state or temporality.

s247
00:23:54.080 --> 00:24:02.080
Um you don't get the right data out when you ask those types of questions with rag, traditional rag.

s248
00:24:02.080 --> 00:24:06.400
So the same engine behind the query playground and the chat agent, uh they're they're the same.

s249
00:24:06.400 --> 00:24:09.039
So if you guys want to play with that after, go ahead.

s250
00:24:09.039 --> 00:24:11.679
Um it's it's the same backing thing.

s251
00:24:11.679 --> 00:24:20.000
The only difference is of course that uh the chat agent actually does reasoning whereas the query interface is just for queries.

s252
00:24:20.159 --> 00:24:25.440
So just to to recount on this here are the reusable ideas here.

s253
00:24:25.440 --> 00:24:36.480
Schema aware prompting dynamic schema discovery identity resolution validation and error retries.

s254
00:24:36.480 --> 00:24:40.720
The error loop is actually one of the most important things.

s255
00:24:45.279 --> 00:24:55.520
And so we built something that's capable of filtering, providing aggregations, it's time scoped, has full text search.

s256
00:24:55.520 --> 00:25:00.159
Um, it's validated, it self-corrects, and it powers a full-blown chat agent.

s257
00:25:00.159 --> 00:25:03.679
So you could take this into your organization and just deploy.

s258
00:25:03.679 --> 00:25:14.400
I I would recommend maybe working a little bit more on the validation harnesses before deploying this internally, but um yeah, it'll it'll answer all your questions about your PRs.

s259
00:25:14.720 --> 00:25:16.559
And that's it.

s260
00:25:16.559 --> 00:25:18.962
So, we've left lots of time for questions now.

s261
00:25:18.962 --> 00:25:19.760
[laughter]

s262
00:25:19.760 --> 00:25:22.880
Did Did everyone uh was everyone able to actually set it up?

s263
00:25:22.880 --> 00:25:24.480
Just curious.

s264
00:25:24.480 --> 00:25:26.000
No. Okay.

s265
00:25:26.000 --> 00:25:33.120
If if you want uh you can come and fetch Brandon and I afterwards, we'll just kind of linger outside here.

s266
00:25:33.120 --> 00:25:35.679
Oh, at the and at the P16 booth as well.

s267
00:25:35.679 --> 00:25:42.240
Yeah, but that that starts later today at around four o'clock and we'll be there all week.

s268
00:25:42.240 --> 00:25:42.640
Cool.

s269
00:25:42.640 --> 00:25:44.799
Any questions?

s270
00:25:44.799 --> 00:25:45.440
Yes,

s271
00:25:45.440 --> 00:25:46.000
I have a question.

s272
00:25:46.000 --> 00:25:49.520
I think you did a great job breaking down like basics.

s273
00:25:49.520 --> 00:25:50.000
Yeah,

s274
00:25:50.000 --> 00:25:58.960
a lot of my mind goes like okay so at scale how does for example you you generate the schema and you put it into the system prompt how does that work at scale?

s275
00:25:58.960 --> 00:26:02.880
It's like also depending on the size of your company, how are you fitting all this data?

s276
00:26:02.880 --> 00:26:06.559
How do you choose what data and like how long does it take to process all this?

s277
00:26:06.559 --> 00:26:07.440
What's the cost?

s278
00:26:07.440 --> 00:26:08.480
That's where my mind is.

s279
00:26:08.480 --> 00:26:09.440
Yeah, it's a good question.

s280
00:26:09.440 --> 00:26:13.600
So, you know, um I'm just going to repeat the question so everyone can can hear it.

s281
00:26:13.600 --> 00:26:19.200
The demo showed obviously a fairly contrived example where you're just focusing on pull requests.

s282
00:26:19.200 --> 00:26:21.679
Um and so the question is how do you scale this?

s283
00:26:21.679 --> 00:26:36.559
Like how do you do this across multiple different collections, multiple different tables, etc. Um, so you you can add uh another LM step on top where uh this the schema collections have tiny little summaries and descriptions

s284
00:26:36.559 --> 00:26:43.120
and then uh depending on what the query is, you can load the right schema for that query.

s285
00:26:43.120 --> 00:26:44.720
LMS are surprisingly good at this.

s286
00:26:44.720 --> 00:26:47.840
They weren't a year ago, but now they're very good at it.

s287
00:26:47.840 --> 00:26:51.200
Um, so that's kind of the way to shave the context thing down.

s288
00:26:51.200 --> 00:26:53.919
And that's to your question about scaling things out.

s289
00:26:53.919 --> 00:26:57.600
Um, was there another aspect of scaling that you were interested in?

s290
00:26:57.600 --> 00:27:01.200
Getting all the data in and I guess like part of that is like cost.

s291
00:27:01.200 --> 00:27:03.039
Yeah, it's livveness and stuff too.

s292
00:27:03.039 --> 00:27:08.720
How long does it take to do like what is what data should I pick to put into this?

s293
00:27:08.720 --> 00:27:10.000
Yeah, it's it's a good question.

s294
00:27:10.000 --> 00:27:14.799
So unblocked um at unblocked like we're continuously ingesting content.

s295
00:27:14.799 --> 00:27:19.679
Uh we get web hook callbacks for all the events for example in in PRs.

s296
00:27:19.679 --> 00:27:27.520
And so every time there's a change, we get that web hbook event, we ingest the content, we rem um and store it in the database.

s297
00:27:27.520 --> 00:27:34.000
Um that that's just really just like a distributed systems question about how you scale that out from an ingestion level.

s298
00:27:34.000 --> 00:27:36.240
Um you know, storage, same thing.

s299
00:27:36.240 --> 00:27:38.240
So yeah,

s300
00:27:38.240 --> 00:27:40.559
so maybe this will make it more concrete.

s301
00:27:40.559 --> 00:27:49.760
So what if you would say like these are the top three data sources like someone seems like GitHub maybe like something

s302
00:27:49.840 --> 00:28:01.919
um so Slack for sure a lot of the like why questions and and decision- making happens in Slack um people uh in the context of this sort of like structural query

s303
00:28:01.919 --> 00:28:14.399
um people usually think about conversations that they've had in the past um and they'll be able to query unblocked and say Oh yeah, you know, roughly three weeks ago, I was having a conversation with so- and so.

s304
00:28:14.399 --> 00:28:17.600
And so that that's a perfect thing for structural query.

s305
00:28:17.600 --> 00:28:23.520
Um yeah, like I would say that um Slack is the most important for why questions.

s306
00:28:23.520 --> 00:28:28.320
Um a lot of the why decision-m is also embedded in PRs.

s307
00:28:28.320 --> 00:28:36.559
PRs uh have all of that like organizational history in the comments and that's just basically black hole, right?

s308
00:28:36.559 --> 00:28:44.640
So uh you can basically run over that data before you serve it and and kind of mine it for interesting insights.

s309
00:28:44.640 --> 00:28:53.279
So for example at unblocked we we create these things called best practice memories which are distilled from PR comments in the past.

s310
00:28:53.279 --> 00:28:54.080
Does that make sense?

s311
00:28:54.080 --> 00:28:54.480
Yeah.

s312
00:28:54.480 --> 00:28:55.760
Yep.

s313
00:28:55.760 --> 00:28:56.320
Any else?

s314
00:28:56.320 --> 00:28:57.440
Yeah.

s315
00:28:57.440 --> 00:28:59.440
Question.

s316
00:29:02.880 --> 00:29:11.840
directly into how this building.

s317
00:29:25.900 --> 00:29:26.320
[snorts]

s318
00:29:26.320 --> 00:29:28.399
Yeah, I mean you you can definitely do that.

s319
00:29:28.399 --> 00:29:34.000
So if you have a structured if you have a structured query table, you could express that through MCP.

s320
00:29:34.000 --> 00:29:39.360
You can even do that dynamically by updating the tool every time the schema changes behind the scenes.

s321
00:29:39.360 --> 00:29:44.880
Um this this type of thing is showing like what happens on the other end of that MCP call.

s322
00:29:44.880 --> 00:29:55.760
So you you could you know have an MCP interface that just takes natural language queries for example and then translate that all on the back end or you could provide a direct interface to the database.

s323
00:29:55.760 --> 00:29:59.919
It depends on like what the customer interface looks like, but yeah.

s324
00:29:59.919 --> 00:30:01.919
Yeah.

s325
00:30:02.159 --> 00:30:04.159
Yeah.

s326
00:30:21.120 --> 00:30:22.159
Yep.

s327
00:30:22.159 --> 00:30:32.880
Second part is better.

s328
00:30:32.880 --> 00:30:35.200
Thoughts around

s329
00:30:40.480 --> 00:30:48.080
uh how to mean how to combine it or let me just make sure I understand your last question.

s330
00:30:50.720 --> 00:30:51.120
Yeah.

s331
00:30:51.120 --> 00:30:54.080
feel like this is like

s332
00:30:54.080 --> 00:30:54.720
No, not at all.

s333
00:30:54.720 --> 00:30:56.720
No, no, they're absolutely complimentary.

s334
00:30:56.720 --> 00:30:57.200
Yeah.

s335
00:30:57.200 --> 00:30:57.440
Yeah.

s336
00:30:57.440 --> 00:31:04.799
I mean, you can have a full text search that does this kind of um you know, like that's full text search, right?

s337
00:31:04.799 --> 00:31:06.399
So, it'll identify keywords.

s338
00:31:06.399 --> 00:31:19.919
You still need some kind of mechanism to to do like the actual semantic similarity search because there you all have ideas and sentences that are encoded that aren't not ne necessarily textual matches.

s339
00:31:19.919 --> 00:31:24.000
So you want you want that like conceptual matching, right?

s340
00:31:24.000 --> 00:31:31.360
So as I said um at the beginning like there's similarity search is actually really powerful for jump off points.

s341
00:31:31.360 --> 00:31:43.279
So you can for example start with a vector search and then from there give it access to tools like this because then it'll actually have like some interesting starting data

s342
00:31:43.279 --> 00:31:46.880
and help it form queries to go and and fetch the data.

s343
00:31:46.880 --> 00:31:53.440
So yeah that that's that's something that we do and so sorry the first part of your question was

s344
00:32:03.279 --> 00:32:15.360
so you take um you take you know in in anyways like these are all documents and they're they kind of like form into JSON so you can merge those things down and look for certain tokens

s345
00:32:15.360 --> 00:32:20.159
and then you you know, extract the tokens and then that's how you distill the types.

s346
00:32:20.159 --> 00:32:26.640
So you you can look through the source code and you'll see that at least like this iteration of that algorithm is pretty hacky, right?

s347
00:32:26.640 --> 00:32:38.720
It'll do things like look to make sure that the character count is between two and eight for uh for type information and then for enumerations it's it's looking for particular patterns as well.

s348
00:32:38.720 --> 00:32:40.720
Yeah.

s349
00:32:54.399 --> 00:32:56.399
Yes.

s350
00:33:05.039 --> 00:33:09.519
Now, that that's a very overloaded term, so I'll just make sure I understand exactly what you're talking about.

s351
00:33:09.519 --> 00:33:22.000
Do you mean hybrid search in terms of um a provider that does semantic similarity and then also does a lexical search and then does rank fusion?

s352
00:33:34.880 --> 00:33:35.039
Yeah.

s353
00:33:35.039 --> 00:33:35.440
Oh, yeah.

s354
00:33:35.440 --> 00:33:35.600
Yeah.

s355
00:33:35.600 --> 00:33:37.200
No. So, so we we do that.

s356
00:33:37.200 --> 00:33:37.519
Yes.

s357
00:33:37.519 --> 00:33:38.399
The answer is yes.

s358
00:33:38.399 --> 00:33:38.960
Yeah.

s359
00:33:38.960 --> 00:33:40.960
Yeah.

s360
00:33:53.840 --> 00:33:57.840
Uh just empiric empirical trial and error.

s361
00:33:57.840 --> 00:34:01.840
And so I mean there's there's no easy solution to this.

s362
00:34:01.840 --> 00:34:09.440
Um be behind the scenes like at unblocks we're we're using a lot of static schemas because we want to get the correctness absolutely nailed down.

s363
00:34:09.440 --> 00:34:22.720
This this was like a just a technique that I wanted to show because um you know for exploration of data or experimental um uh for experiments or either to wire up like arbitrary

s364
00:34:22.720 --> 00:34:24.560
databases this is a good technique.

s365
00:34:24.560 --> 00:34:36.240
So we we got this through some trial and iteration with customers and you know we we'd ingest a whole bunch of contents from the document store and we'd get

s366
00:34:36.240 --> 00:34:45.679
responses back from customers that said oh you missed the thing that I was searching for and we'd dig into it and we'd figure out why and we kind of got to the sample size that made the most sense.

s367
00:34:56.000 --> 00:34:58.000
Yeah,

s368
00:34:58.400 --> 00:34:59.520
it's absolutely true.

s369
00:34:59.520 --> 00:35:01.359
Yeah, definitely.

s370
00:35:01.359 --> 00:35:03.280
You're always actually gonna you're going to miss something.

s371
00:35:03.280 --> 00:35:06.320
There's always going to be some document that has a little edgy thing.

s372
00:35:06.320 --> 00:35:08.320
Yeah.

s373
00:35:09.119 --> 00:35:11.440
Anyone else?

s374
00:35:12.079 --> 00:35:13.599
I thought I saw a question over there.

s375
00:35:13.599 --> 00:35:15.839
Oh, yep.

s376
00:35:21.200 --> 00:35:22.560
Sorry, can you say that first part again?

s377
00:35:22.560 --> 00:35:30.400
There's mention that you understand the schema.

s378
00:35:30.400 --> 00:35:37.440
Uh yeah, what we do is we force a tool to be called for not for schema discovery because that happens at the start.

s379
00:35:37.440 --> 00:35:40.240
It's for name extraction.

s380
00:35:48.160 --> 00:35:52.240
we so we have a validator that watches what the agent's doing.

s381
00:35:52.240 --> 00:35:57.040
So like if it doesn't call the the first part which is name extraction.

s382
00:35:57.040 --> 00:35:59.200
Um so there's there's actually two techniques for this.

s383
00:35:59.200 --> 00:36:06.880
One is you create a tool that does the the query and as an argument to the tool you accept a username.

s384
00:36:06.880 --> 00:36:11.839
So you can fail right there if the username is not correct or malformed.

s385
00:36:11.839 --> 00:36:20.320
Um, if you want to do name extraction uh via tool call, then you have to watch what the agent does.

s386
00:36:20.320 --> 00:36:26.400
And then if it doesn't call it, you just basically insert a user message that says you forgot to call the extraction tool.

s387
00:36:26.400 --> 00:36:28.800
Do so now.

s388
00:36:28.800 --> 00:36:30.800
Yeah.

s389
00:36:32.960 --> 00:36:34.960
Yeah.

s390
00:36:40.160 --> 00:36:42.960
How do you handle the changes?

s391
00:37:04.720 --> 00:37:05.040
Yeah.

s392
00:37:05.040 --> 00:37:14.480
So remember like for this particular example the schema is is completely unknown and so as documents get added um that schema could update.

s393
00:37:14.480 --> 00:37:26.560
So you your your choice is either you know on a sampling interval you you rerun the schema generation um in our case at unblock the schema is is static and it's tied to our codebase

s394
00:37:26.560 --> 00:37:28.400
and we control the schema actually.

s395
00:37:28.400 --> 00:37:41.920
So if the schema of if GitHub decides to change the schema um that of course rarely happens because that would have serious implications for their customers but if they did then we would just adopt whatever changes are made

s396
00:37:41.920 --> 00:37:44.960
statically and then obviously deploy that through CI.

s397
00:37:44.960 --> 00:37:46.960
Yeah.

s398
00:37:50.320 --> 00:37:52.640
Anyone else?

s399
00:37:55.040 --> 00:37:55.440
Awesome.

s400
00:37:55.440 --> 00:37:59.040
I think we can uh reclaim some time in that case.

s401
00:37:59.040 --> 00:38:00.736
Thanks.

s402
00:38:00.736 --> 00:38:02.736
[applause]

s403
00:38:16.456 --> 00:38:18.456
[music]