Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked

AI Engineer · 38 min · 403 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:12Okay.
2. 00:13So, I'm I'm Peter.
3. 00:14I'm from Unblocked.
4. 00:15Um, over here is my colleague Brandon.
5. 00:18And today, uh, well, first of all, I just want to talk a little bit about what we do.
6. 00:22So, uh, we build a context engine.
7. 00:25Our company's name is Unblocked.
8. 00:27And uh we deliver organizational context to humans and increasingly of course agents.
9. 00:33Um so the session was originally a couple hours.
10. 00:35It's it's now one hour so it's quite condensed.
11. 00:38U which means that I'm going to step through the workshop pretty quickly.
12. 00:42If you don't get uh you know all the steps nailed down as we go don't worry about it because the repos up online um and you can run through it at your leisure um afterwards.
13. 00:53Um, so the thesis here is that rag and I'm just going to say that rag in our parlance today is going to be semantic similarity search or vector retrieval.
14. 01:04Okay, rag is only half of retrieval and today we're going to build the other half which is turning na natural language into real database queries.
15. 01:14So here's the again the QR code for setup um if people haven't already done so.
16. 01:19I'm gonna now skip um on to the next thing and I'm just going to briefly bring up hopefully that renders.
17. 01:25No, it won't.
18. 01:26Of course.
19. 01:27Give me one second to switch.
20. 01:32Whoops.
21. 01:37And hopefully people can see this.
22. 01:39Uh this is the repo here.
23. 01:40So setup steps are pretty simple.
24. 01:42Um there are a few sort of uh dependencies up here, prerexs that that uh you'll need to run through.
25. 01:50Um and the the course throughout the course of the workshop, we're going to be running through a series of pull requests uh and the network is basically gone or at least isn't running properly.
26. 02:05Let's try this one instead.
27. 02:16Well, the good thing is we don't need network for this talk.
28. 02:19Um, so hopefully you got everything before we started.
29. 02:23Um, because this one's obviously not going to work.
30. 02:26But at any rate, uh, we're going to be following a series of six pull requests and they stack on top of each other.
31. 02:32So, I'll be There we go.
32. 02:34I'll be referencing these as we go.
33. 02:36Um, and I'll just go back to here so that everyone can see that.
34. 02:41And back to the keynote.
35. 02:43Now I'm gonna have to extend it again.
36. 02:47Sorry.
37. 02:58There we go.
38. 03:01All right.
39. 03:02Hopefully people can see that.
40. 03:03Great.
41. 03:04So um our agents need context and context is scattered across every tool that engineering teams live in.
42. 03:12So you're going to find it in PRs, review comments, um issues and tickets like in linear and jura docs [clears throat] and wikis, notion confluence, and of course in conversations and places like Slack and Teams.
43. 03:26And all of that is semantically queryable, but it's it also has structure.
44. 03:30It has authors, dates, uh, states, labels, relationships, etc. And you know, the the instinct I I love to say this like even two years ago is ancient history in the realm of AI, but the instinct even two years ago was let's just dump this all in a vector store and retrieve it and rerank it and hopefully we
45. 03:50get the right thing.
46. 03:52Um, but that's that's not going to work.
47. 03:54So I'm just going to step back and and kind of relate this to the larger discussion about context.
48. 03:59um in in the context of a context engine that serves organizational context um this is this is made up of more than just code as I said we're connecting everything together
49. 04:11so the scattered context uh on the left in all your code and PRs docs etc goes in and the idea is to get you know grounded context that relates to each other out the other side that's specifically tailored to the task at hand
50. 04:26um and in that model uh we have to care not just about the semantic relationships, but of course the structural relationships that connect all the data.
51. 04:35So again, rag is about embedding the query, finding docs with similar meaning, and then we take that stuff and we stuff it into the prompt, right?
52. 04:43Well, um, that shines for content questions like find the code, find the PR that changed it, find the Slack thread that debated it, but um, and if you wrap it in an agent loop, you can get pretty far with that.
53. 04:59And you can also wire this thing up to like um, you know, tools that can recurse files and it acts as a good jump off point.
54. 05:06you get a good semantic head start and then from there your agent can recurse over the actual file system in GitHub um and you can get pretty far with that.
55. 05:17But in our experience um our this is not really the way things go in practice.
56. 05:22engineers often ask questions like this PRs that I merged last week or my team merged last week.
57. 05:29Um they're they're basically, you know, hunting for something in particular and they know where the starting point is, but they don't know where the destination is, but they do know the relationships.
58. 05:42So, as an example, PRs I merged last week, let's think about this in a rag context.
59. 05:46you'd you'd you know embed that sentence, you'd find the nearest vectors, you'd rerank it top K. Uh you get PRs that mention merging about a talk last week, but
60. 05:58uh you wouldn't get the actual relationships like the author, the merge state, um and whether it was merged last week actually.
61. 06:08So what you could end up with is the wrong answer where you pull up PR that's semantically related but it was merged three years ago.
62. 06:17Um and so that doesn't actually answer the question because there's a structural there's a temporal aspect to that question.
63. 06:24And so retrieval has two halves here semantic and again today we're going to be talking about this structured queries.
64. 06:34And here's the key.
65. 06:35We're going to take uh the model doesn't actually answer the question in this paradigm.
66. 06:38It writes the query that accesses the database.
67. 06:45Okay.
68. 06:46So, um it would be naive to say that generating the query is the only thing that we have to do.
69. 06:52Um in order to generate the query and to do it reliably um and and to take care of maybe even some security concerns, we need to look at all four of these points here.
70. 07:04So that first of all the LM needs to be aware of what the schema is before uh creating the query.
71. 07:10It needs to understand uh some of the relationships in the data.
72. 07:13So for example, if you pass a natural language query that says show me Peter's open PRs, um then it's not going to know what to do because it'll synthesize a query that says Peter when really the GitHub login name might be different.
73. 07:28So it needs to be able to map that across.
74. 07:30Um, and of course you're going to have some uh untrusted queries coming out the other end.
75. 07:36Um, maybe using the function operator in the in parlance which might execute JavaScript directly on the server and we probably don't want that.
76. 07:45Um, and then finally, if there are errors the first time around, we want to be able to self-correct those things.
77. 07:52So here's here's what this looks like in one single shot.
78. 07:56What we want is something that looks like this.
79. 07:58Um it it maps uh my colleagueu's name is Rashine.
80. 08:02His user login name is is Rashab.
81. 08:05Um and then we want to be able to map things like the state and get maybe uh operators like limit and sort.
82. 08:16So uh this is what we're going to build.
83. 08:18We're going to start with a full UX, a stubbed engine, um, that is not capable to start with, and we're going to gradually build this up.
84. 08:28And then at step three, the thing will come alive.
85. 08:31And so we'll actually be able to issue queries to it and see what it does.
86. 08:36And uh, if you hadn't seen that before, now is another good time to scan that and get it in.
87. 08:46I'm here to help you.
88. 09:00Yeah.
89. 09:03Yes.
90. 09:03Any repo is fine.
91. 09:04Doesn't matter which one.
92. 09:07Um and if if you want to take a shortcut, you can just point an agent at the thing and let it set everything up for you.
93. 09:13That works.
94. 09:16Okay, let's move.
95. 09:19So, this is what we're going to do to start with.
96. 09:21Um, by now you you probably should have run this ingestion thing.
97. 09:25Don't worry if you haven't.
98. 09:26Um, again, like you can revisit this at the end.
99. 09:29I'm just going to quickly show what the the kind of barebones interface looks like for this thing.
100. 09:36So, now this thing is running.
101. 09:37Now, if I type anything in here, this is obviously not going to work.
102. 09:41Um, and what you should see is something like that.
103. 09:43Um, there's also a tab for chat.
104. 09:46This is a fully implemented agent uh that can uh do work for you.
105. 09:51So, we'll come back to that in a sec.
106. 09:58And so, this this is the contract in the code if you wanted to read.
107. 10:01Um, we're going to don't worry about this.
108. 10:03We're going to skip through this.
109. 10:05Um, but basically like that's the entry point and we're going to be building around that.
110. 10:10So the first thing we want to talk about is schema discovery.
111. 10:14Um during ingestion, if you if you look at the way that the ingestion script worked, it it actually had to build out a schema in order to download the content and store it.
112. 10:24Um but you can imagine that you may not know the schema.
113. 10:27So the ingestion may be happening in a service elsewhere and you want to build like a fully dynamic schema discovery engine.
114. 10:34Um schemalis.
115. 10:36So there's no schema to read.
116. 10:38um unlike you know PG or something where we could read the table definitions.
117. 10:42Um so what we'll do is we'll sample the documents to infer the schema and that's the whole trick.
118. 10:52So the we're just going to reconstruct it.
119. 10:54We're going to take um all the nodes and merge them down.
120. 10:58So this is going to sample 300 documents, collapse it into a single tree.
121. 11:03uh nulls are skipped because missing fields won't be able to constrain the type.
122. 11:08And then uh we're going to do something kind of interesting here with enum types.
123. 11:17And so now uh with with enum types, we can do an interesting sort of procedural trick.
124. 11:23This is just one thing.
125. 11:25I would suggest using procedural tricks to to do this type of thing because you could pass it through an LLM and do extraction and that works u but it's fairly non-deterministic.
126. 11:36So you always need some kind of structural um harness around it that is running procedural code to either validate the things that you pull out or to actually do the extraction itself.
127. 11:48And in this sample project um it's the procedural code that's doing the extraction.
128. 11:55And let's go and show that.
129. 12:00So we'll do and then run it again.
130. 12:09And now what we get is this thing here.
131. 12:12So the schema ran uh the the discovery process ran and it created a schema for us that outlines what the pull request schema looks like in GitHub.
132. 12:22It even created some indexes for us.
133. 12:25Now, this is like marked down on purpose because we're going to take this schema and pass it to the LLM when we synthesize the actual uh query.
134. 12:40Now, the second step is the thing that I referred to a few slides ago about identity resolution.
135. 12:45And this is because like my lo GitHub login um is similar but it's not quite the same.
136. 12:51And so if I were to ask a question like um hey find all the PRs by Peter um it if without doing some sort of identity mapping it would just say
137. 13:03you know it would just match across Peter and it's not going to find anything.
138. 13:07So we have to be able to solve that.
139. 13:09[snorts]
140. 13:11And again, um, we're going to do this.
141. 13:14At first blush, this seems like we're coupling to the schema a little bit here.
142. 13:19Um, but you know, it is absolutely possible to determine this dynamically.
143. 13:24And you can have uh your LLM essentially synthesize the mappings for you.
144. 13:32So, but in in our case, we're just going to do these simple little tricks.
145. 13:35Um very simple sort of fuzzy matching.
146. 13:38Um Levenstein distance is fine.
147. 13:40You can just like match across the names and uh there you go.
148. 13:50Oh, actually I want to show you one thing before we get on to that one.
149. 13:54So let's do the same thing again.
150. 13:57We can pop up
151. 14:01Oh yeah, I'm sorry.
152. 14:03Absolutely.
153. 14:05Tell me when it's big.
154. 14:07Good.
155. 14:08Thanks.
156. 14:08Good call out.
157. 14:11Okay.
158. 14:12So, let's run our thing again.
159. 14:15And now, um Oh, actually, I have to I have to wait till the next step.
160. 14:20I apologize.
161. 14:21That's not going to be wired up just yet.
162. 14:24So, we'll go on to the thing that will really give us the payoff here, which is step three.
163. 14:36So this is the actual query plan step and it's it's it's arguably sort of the simplest part.
164. 14:42It's the LM part.
165. 14:44Um and what we do here is we pass the schema give it a tool and force it to call that tool for name extraction and synthesis.
166. 14:54Um and then it will execute the plan.
167. 14:56There's one other thing which is the temporal aspect.
168. 15:00uh LLMs are actually pretty good about generating temporal queries if you pass them the current date and time.
169. 15:08So that's another important thing to do in the in the prompt.
170. 15:18And so here this is the name extraction part.
171. 15:21We're going to force that tool call to happen in this case.
172. 15:28And at this point I am going to show you the the payoff for this.
173. 15:33So there we go.
174. 15:47So now I can do things like this.
175. 15:52open PRS by Peter and bam.
176. 15:59So what we'll see here is that it uh generated this um this match clause and uh you can't see it from here but it it mapped Peter to my GitHub user login puery
177. 16:13and generated the query and that's the outcome.
178. 16:22So that's cool, but um we need to get on to the next step because even though we are able now to query the data, we have some safety concerns.
179. 16:31Um it could be the case that it produces untrusted output and so you know it could use the wear or function clauses and literally run JavaScript on the server which means that this is now a prompt injection point.
180. 16:48Um, you could also have hidden fields in your schema like ID or metadata fields that leak internals.
181. 16:56So, we need to sanitize um, and we don't deny list these things.
182. 17:01We allow list them.
183. 17:03So, it's deny by default.
184. 17:05You want to create basically an allow list of allowed operators and uh, you know, potentially an allow list of allowed fields.
185. 17:12In our case, this is dynamic schema discovery, so we're not going to do that.
186. 17:16Um, but we could have some fields that we know about like metadata fields.
187. 17:21And this is a a useful thing for like tenant uh tenant isolation.
188. 17:27So if you have a a uh collection that houses multiple tenants, I don't recommend that.
189. 17:32you should probably separate your collections by tenant, but if you did, then you could have a metadata field that contains the tenant ID and you would want to hide that
190. 17:43no that um information from the LLM, but also validate the output.
191. 17:49So if it outputs that metadata field, you need to wipe it out and then you need to inject your own metadata field underneath.
192. 18:05And again, like technically this is it's not actually that difficult to do.
193. 18:08You just have to write the code.
194. 18:10It's all procedural.
195. 18:11Um I I wouldn't recommend using LLMs for this part.
196. 18:19And so again, allow list, don't deny or deny everything, I should say.
197. 18:33Now the last part is um we're going to this kind of dovetales into the next step.
198. 18:38At the end of validation we're obviously going to return an error and it's going to get dumped out.
199. 18:43And right now that error is just it's just there.
200. 18:46We're not going to do anything with it just yet.
201. 18:49But I will show you now what this looks like.
202. 18:52Um, oops.
203. 19:04Stay on that one and run our service.
204. 19:10So now I'm going to write run another query that checks for authentication in the title and we've disallowed regular expressions.
205. 19:20So now what we can see is that it rejects the query.
206. 19:35And finally, this is the thing that kind of ties it all together is the retry loop.
207. 19:47The plan can fail in a couple of different ways, right?
208. 19:50Um, it can fail validation.
209. 19:52So maybe the uh the LM actually outputs things that are illegal according to our validator.
210. 19:59Or it could actually fail during execution.
211. 20:02Um, so for example, let's imagine that it it mixed an inclusion exclusion project.
212. 20:09Um, then that's obviously going to fail.
213. 20:13And then but we can self-correct.
214. 20:15We can take that error and pass it back into the LLM and say hey that was wrong.
215. 20:20Um maybe you can do better up to some number of max attempts of course.
216. 20:30So I'm going to show now what this looks like back over to our project so we can punch up a level.
217. 20:41Whoops.
218. 20:41Sorry.
219. 20:45You have to forgive me.
220. 20:46I actually have to read this off of the screen out here.
221. 20:56Okay.
222. 20:57So now if we run that same query.
223. 21:09Perfect.
224. 21:10So, what we can see is that it took several attempts to get it right.
225. 21:14Um, by the third attempt though, there's a bit of a problem here because we said no regular expressions, no FTS.
226. 21:22Um, so now it said we're going to just use match, but now it's an exact match.
227. 21:29So it tries to find authentication fails which means there's one more thing that we can do which is this.
228. 21:42We'll add full text search.
229. 21:49It's pretty basic.
230. 21:50Um we need to create an index obviously and we'll combine it with author and state filters uh in the same match.
231. 21:59And so now we should be able to answer a question like this.
232. 22:07And let's see what happens.
233. 22:09And see if it is actually going to do what I said it's going to do.
234. 22:29Bam.
235. 22:30There we go.
236. 22:36And so now the the final payoff here is if we go into the chat tab, we can ask some sophistic more sophisticated questions that um you know an agent can actually reason about behind the scenes.
237. 22:48So you can say things like um did we work on anything authentication related last week.
238. 23:10So here we can see that it generated the query.
239. 23:14Now it's pulling the data and kind of reasoning about it.
240. 23:18There we go.
241. 23:22That's it.
242. 23:35So, just a quick summary of what we built.
243. 23:38We're basically like reaching the end.
244. 23:41There's going to leave lots of time for questions.
245. 23:44So just to recap here, um there are questions that that rag can't touch.
246. 23:49All those structural questions around things like state or temporality.
247. 23:54Um you don't get the right data out when you ask those types of questions with rag, traditional rag.
248. 24:02So the same engine behind the query playground and the chat agent, uh they're they're the same.
249. 24:06So if you guys want to play with that after, go ahead.
250. 24:09Um it's it's the same backing thing.
251. 24:11The only difference is of course that uh the chat agent actually does reasoning whereas the query interface is just for queries.
252. 24:20So just to to recount on this here are the reusable ideas here.
253. 24:25Schema aware prompting dynamic schema discovery identity resolution validation and error retries.
254. 24:36The error loop is actually one of the most important things.
255. 24:45And so we built something that's capable of filtering, providing aggregations, it's time scoped, has full text search.
256. 24:55Um, it's validated, it self-corrects, and it powers a full-blown chat agent.
257. 25:00So you could take this into your organization and just deploy.
258. 25:03I I would recommend maybe working a little bit more on the validation harnesses before deploying this internally, but um yeah, it'll it'll answer all your questions about your PRs.
259. 25:14And that's it.
260. 25:16So, we've left lots of time for questions now.
261. 25:18[laughter]
262. 25:19Did Did everyone uh was everyone able to actually set it up?
263. 25:22Just curious.
264. 25:24No. Okay.
265. 25:26If if you want uh you can come and fetch Brandon and I afterwards, we'll just kind of linger outside here.
266. 25:33Oh, at the and at the P16 booth as well.
267. 25:35Yeah, but that that starts later today at around four o'clock and we'll be there all week.
268. 25:42Cool.
269. 25:42Any questions?
270. 25:44Yes,
271. 25:45I have a question.
272. 25:46I think you did a great job breaking down like basics.
273. 25:49Yeah,
274. 25:50a lot of my mind goes like okay so at scale how does for example you you generate the schema and you put it into the system prompt how does that work at scale?
275. 25:58It's like also depending on the size of your company, how are you fitting all this data?
276. 26:02How do you choose what data and like how long does it take to process all this?
277. 26:06What's the cost?
278. 26:07That's where my mind is.
279. 26:08Yeah, it's a good question.
280. 26:09So, you know, um I'm just going to repeat the question so everyone can can hear it.
281. 26:13The demo showed obviously a fairly contrived example where you're just focusing on pull requests.
282. 26:19Um and so the question is how do you scale this?
283. 26:21Like how do you do this across multiple different collections, multiple different tables, etc. Um, so you you can add uh another LM step on top where uh this the schema collections have tiny little summaries and descriptions
284. 26:36and then uh depending on what the query is, you can load the right schema for that query.
285. 26:43LMS are surprisingly good at this.
286. 26:44They weren't a year ago, but now they're very good at it.
287. 26:47Um, so that's kind of the way to shave the context thing down.
288. 26:51And that's to your question about scaling things out.
289. 26:53Um, was there another aspect of scaling that you were interested in?
290. 26:57Getting all the data in and I guess like part of that is like cost.
291. 27:01Yeah, it's livveness and stuff too.
292. 27:03How long does it take to do like what is what data should I pick to put into this?
293. 27:08Yeah, it's it's a good question.
294. 27:10So unblocked um at unblocked like we're continuously ingesting content.
295. 27:14Uh we get web hook callbacks for all the events for example in in PRs.
296. 27:19And so every time there's a change, we get that web hbook event, we ingest the content, we rem um and store it in the database.
297. 27:27Um that that's just really just like a distributed systems question about how you scale that out from an ingestion level.
298. 27:34Um you know, storage, same thing.
299. 27:36So yeah,
300. 27:38so maybe this will make it more concrete.
301. 27:40So what if you would say like these are the top three data sources like someone seems like GitHub maybe like something
302. 27:49um so Slack for sure a lot of the like why questions and and decision- making happens in Slack um people uh in the context of this sort of like structural query
303. 28:01um people usually think about conversations that they've had in the past um and they'll be able to query unblocked and say Oh yeah, you know, roughly three weeks ago, I was having a conversation with so- and so.
304. 28:14And so that that's a perfect thing for structural query.
305. 28:17Um yeah, like I would say that um Slack is the most important for why questions.
306. 28:23Um a lot of the why decision-m is also embedded in PRs.
307. 28:28PRs uh have all of that like organizational history in the comments and that's just basically black hole, right?
308. 28:36So uh you can basically run over that data before you serve it and and kind of mine it for interesting insights.
309. 28:44So for example at unblocked we we create these things called best practice memories which are distilled from PR comments in the past.
310. 28:53Does that make sense?
311. 28:54Yeah.
312. 28:54Yep.
313. 28:55Any else?
314. 28:56Yeah.
315. 28:57Question.
316. 29:02directly into how this building.
317. 29:25[snorts]
318. 29:26Yeah, I mean you you can definitely do that.
319. 29:28So if you have a structured if you have a structured query table, you could express that through MCP.
320. 29:34You can even do that dynamically by updating the tool every time the schema changes behind the scenes.
321. 29:39Um this this type of thing is showing like what happens on the other end of that MCP call.
322. 29:44So you you could you know have an MCP interface that just takes natural language queries for example and then translate that all on the back end or you could provide a direct interface to the database.
323. 29:55It depends on like what the customer interface looks like, but yeah.
324. 29:59Yeah.
325. 30:02Yeah.
326. 30:21Yep.
327. 30:22Second part is better.
328. 30:32Thoughts around
329. 30:40uh how to mean how to combine it or let me just make sure I understand your last question.
330. 30:50Yeah.
331. 30:51feel like this is like
332. 30:54No, not at all.
333. 30:54No, no, they're absolutely complimentary.
334. 30:56Yeah.
335. 30:57Yeah.
336. 30:57I mean, you can have a full text search that does this kind of um you know, like that's full text search, right?
337. 31:04So, it'll identify keywords.
338. 31:06You still need some kind of mechanism to to do like the actual semantic similarity search because there you all have ideas and sentences that are encoded that aren't not ne necessarily textual matches.
339. 31:19So you want you want that like conceptual matching, right?
340. 31:24So as I said um at the beginning like there's similarity search is actually really powerful for jump off points.
341. 31:31So you can for example start with a vector search and then from there give it access to tools like this because then it'll actually have like some interesting starting data
342. 31:43and help it form queries to go and and fetch the data.
343. 31:46So yeah that that's that's something that we do and so sorry the first part of your question was
344. 32:03so you take um you take you know in in anyways like these are all documents and they're they kind of like form into JSON so you can merge those things down and look for certain tokens
345. 32:15and then you you know, extract the tokens and then that's how you distill the types.
346. 32:20So you you can look through the source code and you'll see that at least like this iteration of that algorithm is pretty hacky, right?
347. 32:26It'll do things like look to make sure that the character count is between two and eight for uh for type information and then for enumerations it's it's looking for particular patterns as well.
348. 32:38Yeah.
349. 32:54Yes.
350. 33:05Now, that that's a very overloaded term, so I'll just make sure I understand exactly what you're talking about.
351. 33:09Do you mean hybrid search in terms of um a provider that does semantic similarity and then also does a lexical search and then does rank fusion?
352. 33:34Yeah.
353. 33:35Oh, yeah.
354. 33:35Yeah.
355. 33:35No. So, so we we do that.
356. 33:37Yes.
357. 33:37The answer is yes.
358. 33:38Yeah.
359. 33:38Yeah.
360. 33:53Uh just empiric empirical trial and error.
361. 33:57And so I mean there's there's no easy solution to this.
362. 34:01Um be behind the scenes like at unblocks we're we're using a lot of static schemas because we want to get the correctness absolutely nailed down.
363. 34:09This this was like a just a technique that I wanted to show because um you know for exploration of data or experimental um uh for experiments or either to wire up like arbitrary
364. 34:22databases this is a good technique.
365. 34:24So we we got this through some trial and iteration with customers and you know we we'd ingest a whole bunch of contents from the document store and we'd get
366. 34:36responses back from customers that said oh you missed the thing that I was searching for and we'd dig into it and we'd figure out why and we kind of got to the sample size that made the most sense.
367. 34:56Yeah,
368. 34:58it's absolutely true.
369. 34:59Yeah, definitely.
370. 35:01You're always actually gonna you're going to miss something.
371. 35:03There's always going to be some document that has a little edgy thing.
372. 35:06Yeah.
373. 35:09Anyone else?
374. 35:12I thought I saw a question over there.
375. 35:13Oh, yep.
376. 35:21Sorry, can you say that first part again?
377. 35:22There's mention that you understand the schema.
378. 35:30Uh yeah, what we do is we force a tool to be called for not for schema discovery because that happens at the start.
379. 35:37It's for name extraction.
380. 35:48we so we have a validator that watches what the agent's doing.
381. 35:52So like if it doesn't call the the first part which is name extraction.
382. 35:57Um so there's there's actually two techniques for this.
383. 35:59One is you create a tool that does the the query and as an argument to the tool you accept a username.
384. 36:06So you can fail right there if the username is not correct or malformed.
385. 36:11Um, if you want to do name extraction uh via tool call, then you have to watch what the agent does.
386. 36:20And then if it doesn't call it, you just basically insert a user message that says you forgot to call the extraction tool.
387. 36:26Do so now.
388. 36:28Yeah.
389. 36:32Yeah.
390. 36:40How do you handle the changes?
391. 37:04Yeah.
392. 37:05So remember like for this particular example the schema is is completely unknown and so as documents get added um that schema could update.
393. 37:14So you your your choice is either you know on a sampling interval you you rerun the schema generation um in our case at unblock the schema is is static and it's tied to our codebase
394. 37:26and we control the schema actually.
395. 37:28So if the schema of if GitHub decides to change the schema um that of course rarely happens because that would have serious implications for their customers but if they did then we would just adopt whatever changes are made
396. 37:41statically and then obviously deploy that through CI.
397. 37:44Yeah.
398. 37:50Anyone else?
399. 37:55Awesome.
400. 37:55I think we can uh reclaim some time in that case.
401. 37:59Thanks.
402. 38:00[applause]
403. 38:16[music]