Beyond RAG: A Relational Context Engine That Cuts Token Burn — Unblocked

AI Engineer · 19 min · 203 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:12Are we up?
3. 00:13Okay.
4. 00:15Can you hear me?
5. 00:16Just like this just fine, everyone.
6. 00:18Good.
7. 00:18Okay.
8. 00:19Um, so we're we're going to be doing a bit of a speedun here because uh uh we've only got 20 minutes, but um I'm I'm Peter from Unblocked.
9. 00:28And at Unblocked, we build a context engine.
10. 00:32So I'm going to show you a couple of things.
11. 00:34The talk's going to be split into two parts.
12. 00:36Um I'm going to show you what a context engine is and how it can add value to your organization.
13. 00:41And then we're going to actually go and build a component of a context engine.
14. 00:46And I'm going to show you a couple of open source projects that you can uh go and play with after this.
15. 00:54So there's the agenda.
16. 00:56What's a context engine?
17. 00:57How does it deliver value?
18. 00:59And then we'll build.
19. 01:01Um so let let's just hop back in time maybe a few years to the times before AI agents.
20. 01:08And in those days um you were the context layer.
21. 01:14Imagine that you're a new employee to an organization and you're just trying to get to grips with what's what's happening.
22. 01:21What you usually end up having to do is look at the look at a whole bunch of code, rip around documentation systems just trying to find the information that you need to get your work done.
23. 01:31Um, and not to mention like all of the the sort of shared history that's built up over time through an organization dealing with incidents and the battle scars that come with it.
24. 01:43Um, so what a context engine does is it takes all of that institutional knowledge and it pulls it in and what you get the scattered context that goes into the context engine and out the other end
25. 01:57you get grounded context which means that the context is all interrelated to each other.
26. 02:03It's intent specific.
27. 02:05It's personalized and it's permissions aware.
28. 02:12And so I'm going to show you exactly what I mean by that.
29. 02:14Now let's imagine that I'm going to go into here into Claude and I'm going to show you a couple of interesting sessions that I had.
30. 02:27So um I asked Claude here, this is without a context engine attached to create a plan to optimize a component in our layer called the source mark engine.
31. 02:39And uh I'm sure you're all familiar with what Claude does when it starts a new task.
32. 02:44It's like a new employee.
33. 02:45It doesn't understand anything about your your codebase or your organization.
34. 02:49So the first thing it does is it reps around your codebase and and tries to figure out what's going on.
35. 02:55And so you can kind of see that here.
36. 02:57And you know it figures it out eventually or at least you know it confidently projects that it's figured it out.
37. 03:04Um, and so it provides kind of like a a recommendation for me there.
38. 03:09But now I'm going to show you what it looks like when you attach a context engine to the back end.
39. 03:14So now what it does is it calls the unblock context engine and unblock now delivers a very comprehensive but taskspecific context bundle for this task.
40. 03:26And what you can see is that it it's referencing not only the source code but also you know notion documents that talk about the architecture um pull requests slack conversations that were had about optimizing
41. 03:39this particular component um and the the the net out is that you get this really tight context bundle with all these links.
42. 03:47So now uh if if um you know Claude wants to go back and fetch more information, it has all these great jump off points.
43. 03:55But the really interesting thing is is this.
44. 03:58So here I spent um you know a dollar 29 on this and it took about a minute and 30 seconds.
45. 04:08Um that was with the context engine but without I'm spending up to you know 260 and 3 minutes.
46. 04:15So what you see essentially is a 50% reduction in cost which is pretty incredible.
47. 04:21Um now I want to show you another interesting demo um which is uh an interesting session that my colleague Whoops sorry bring this out here.
48. 04:32It's an interesting um session that my colleague had with unblock the other week.
49. 04:37So what he we also have a code review product and Richie works on that and he discovered that the number of issues that the code review product was surfacing
50. 04:47was was starting to drop.
51. 04:49So he just asked unblocked and unblocked was able to pull this information um and and you know narrow it down very quickly to uh a likely cause which was that we switched from Opus 48 or Opus 46 to Opus 48 and the behavior characteristics
52. 05:08behind the scenes um you know caused all those issues to drop and uh so what he did is he asked on block to just fix it and now on locked
53. 05:20um was able to just create a PR behind the scenes and it has all that context and it knows what to do.
54. 05:25But the really cool part is you can see that it created this description and it's it's just remarkable like it it um understood the PR that caused the regression in the first place
55. 05:37and then it found a Slack conversation where the conver where where this was being discussed and pulled on some of the data here to help inform um the fix which which is just nuts.
56. 05:53So, I'm going to go back to wh I think I lost my keynote presentation here.
57. 05:57Let's see.
58. 05:58There we go.
59. 06:03So, you have the same problem with agents, right?
60. 06:07Cost.
61. 06:07Oh, sorry.
62. 06:08This is the wrong
63. 06:09[laughter]
64. 06:09It's the wrong keynote.
65. 06:17There we go.
66. 06:20So now I think what we're going to do is we're just going to get right into the building part because we're we're quite short on time today.
67. 06:25Um and so I'm just going to summarize this.
68. 06:28You've got context scattered all over the place.
69. 06:30You've got context in PRs, reviews, comments, issues and tickets, docs, conversations.
70. 06:37Um and and all of that is uh semantically queryable, but it's also structured and rag nails the content part.
71. 06:48Um, it it can be it can handle things like content questions, find the code, the PR that changed it, the Slack thread that's debating it, and if you wrap that stuff in an agent loop,
72. 07:00it can go really really far.
73. 07:02You give an agent loop tools to do file search after this and it it's very powerful but it doesn't handle some of the other type of uh structural questions like PRs that I merged last week
74. 07:15who reviewed payments the most open PRs etc. Um, and these are queries.
75. 07:22They're not semantic searches.
76. 07:25So, I want to just show you again, um, going back to Richie's example here that he started the the conversation with unblock by asking it a query based question because there's a temporal element.
77. 07:39Can you show me a chart of ratio weekly since January 1st?
78. 07:43Okay.
79. 07:44So, in order to do this, it's not going to be able to just pull this information from a vector store because that's not where that information is encoded.
80. 07:51So, it's it's going to have to do that um through some sort of query engine.
81. 07:59And so, you know, retrieval has two halves essentially.
82. 08:02We've got semantic retrieval and that can handle those types of semantic based questions.
83. 08:07But today, we're going to work on this the structured part.
84. 08:11We're going to turn structured queries or natural language queries into structured queries.
85. 08:20But generating the queries is actually the easy part.
86. 08:22The hard part is the scaffolding that sits around it.
87. 08:25So you need to do things like understand the schema so you can pass it to the LLM so it generates the a tightly bound query.
88. 08:33Uh you need to also be able to map identities.
89. 08:36If I ask a question like open PR is by Rashene, um it's not going to know who Rashene is because his identity in GitHub might be completely different.
90. 08:46So we have to be able to map that somehow.
91. 08:50And then when you get the query out of the LLM, it's untrusted.
92. 08:54It could have uh operations that are dangerous.
93. 08:57It could cross tenant boundaries.
94. 08:59And so you have to have some way of validating that the query is is uh correct and safe.
95. 09:05And then finally, uh, even though you do that validation, there could be errors.
96. 09:11Um, and when you actually do the query against the database, there could be errors there as well.
97. 09:15And so we can actually get the LLM to try to correct itself if it gets into that situation.
98. 09:21So this is what we're aiming for here.
99. 09:22We want to go to uh to this.
100. 09:25We want to be able to say something like open PR is by machine and then we want to get this kind of query on the back of that.
101. 09:34So, just take a sec here.
102. 09:35U this is the open- source repo that we're going to be building against.
103. 09:41And I'm going to jump right into what that looks like actually.
104. 09:48So, this is the repo here.
105. 09:50It's called the document query engine.
106. 09:53And I want to show you what it looks like when we just boot it up the first time.
107. 09:56So, I can say, you know, PR is by Peter at ground zero.
108. 10:02Oh, actually we're actually not running it.
109. 10:04So, I'll go back.
110. 10:07Let's run it first.
111. 10:15Now I can ask my question.
112. 10:19So, when you start it's going to have it's not going to be wired up at all.
113. 10:22It's going to produce nothing.
114. 10:23So, we're going to just gradually build this thing up here.
115. 10:25Okay.
116. 10:29First thing we're going to talk about is automatic schema discovery.
117. 10:33This is completely dynamic.
118. 10:36is schemalist.
119. 10:37Not and not all databases are schemalist, but the same kind of principle can apply.
120. 10:42You can dynamically discover the schema at runtime.
121. 10:45Um and so what what the technique involves is sampling a whole bunch of documents in the document store and then deriving a schema from that.
122. 10:55And uh you know I just want to show you like one tricky part of that which is inferring enums.
123. 11:00Um now the the the outcome from this is that you can use an LLM as a Thor hammer um and it can do a pretty good job but actually
124. 11:12you get uh really really far just using traditional procedural methods.
125. 11:17So you can you know don't always fall back to an LLM to do this kind of what looks like non-deterministic work.
126. 11:24This is actually quite deterministic this kind of thing.
127. 11:28So now that we've discovered schema, I'm just going to show you what that looks like in our demo.
128. 11:33Whoops.
129. 11:34First we have to level up to the next and then run.
130. 11:44So now we've discovered the schema from our um GitHub collection and this is just pull requests.
131. 11:53Okay, so it has not only the schema here but also um indexes that will help improve the performance of queries.
132. 12:05The next step is identity resolutions that thing I talked about about mapping you know users to uh real identities and um again this is this can be solved just using simple procedural tricks.
133. 12:19So um that that's actually more or less how we do it.
134. 12:23Uh and underneath the hooded unblocks when we take usernames or identities from different systems and map them across um you can do this kind of fuzzy matching and then in the end if you have to you can pass a list of names to an LLM and have it act as a tiebreaker.
135. 12:45And finally, uh, the last step before we can actually show the cool stuff is this.
136. 12:51It's the synthesis part.
137. 12:53And roughly speaking, this is actually the easiest part.
138. 12:57We just take the schema that we dynamically resolve from earlier.
139. 13:01We take the current date and time, uh, maybe some information about the user that's asking the question, so it can easily resolve questions like my PRs and not just Peter's PRs.
140. 13:13Um and then we pass it to the LLM and ask it to synthesize a query for us.
141. 13:18And the outcome uh will look something like this.
142. 13:24Let's get to this one.
143. 13:29So now I can go back to my engine and try to run that query again.
144. 13:37And it's going to pull the documents for me now and generate this query.
145. 13:45Now the next step I I was talking about was was validation.
146. 13:48This is actually the most important thing.
147. 13:50Um your your uh query could contain really dangerous operators like the function operator for example which would execute raw JavaScript um on the DB server which is definitely not ideal.
148. 14:03Um, but there could be other things that you want to do with this.
149. 14:06Like for example, you might have hidden fields that you don't want uh to be surfaced either to the LM or for the LM to to derive and execute.
150. 14:15If you have metadata fields, for example, that contain um tenant IDs and you want to isolate each query to a specific tenant, you're better off making sure that the LM doesn't interact with those fields at all and then just constraining it behind the scenes.
151. 14:31you can inject those fields underneath.
152. 14:36And that's kind of what that looks like.
153. 14:40So again, we're going to just quickly show the payoff for that and what that looks like.
154. 14:47Let's go up to our validation step.
155. 14:50Oops.
156. 14:52Run it again.
157. 14:53So if I now ask a question like this PRs with authentication in the title that is probably going to drive some kind of like regular expression thing and there we go.
158. 15:04Okay.
159. 15:05So uh our validation engine said sorry regular expressions aren't allowed you're getting bounced and now we're not allowed to do that.
160. 15:17So validation is cool.
161. 15:18Um but what do we do with that?
162. 15:22you know, we got an error.
163. 15:23Um, but what we can do instead is just take that error and just loop it back in to the LLM and have the LLM kind of self-correct.
164. 15:34And that's kind of what it that's all it is.
165. 15:36That's all it looks like from a technical standpoint.
166. 15:38It's just a loop with some sort of maximum retry.
167. 15:45And so if I try this again, bounce up to my retries and run that.
168. 15:57Now, if I try that query again.
169. 16:00Oops, that's the wrong one.
170. 16:03Let's try this one.
171. 16:12Now what you can see is that it actually took a couple attempts.
172. 16:16Um but it got there in the end.
173. 16:18However, um it constrained it down to use match and unfortunately this is an exact title match.
174. 16:25We don't have any PRs with that exact title.
175. 16:28So we need one more thing which is this full text search.
176. 16:35So now if we add full text search to our engine um hopefully the outcome that we get is something that looks a little bit like this.
177. 16:44So let's try it one more time.
178. 17:03Awesome.
179. 17:06So there you go.
180. 17:07We got we went from start to finish there.
181. 17:09This is all open source.
182. 17:10So you can take a look at how it's implemented and play around with it yourself.
183. 17:17And this is what we built.
184. 17:18Something that can effectively handle all of the uh other half of rag.
185. 17:24So, anything that has to do with uh structural based data, you can have filters, aggregations, it's time scoped, and it selfcorrects.
186. 17:35But what's really cool is that it also powers a chat agent.
187. 17:40And so if I go over here and I say something like um what authentication related things have we've been working on last week.
188. 18:00Now you get this whole thing that will show you exactly what it's doing, all the queries that it creates.
189. 18:08And now it has the data and it's just kind of reasoning about it behind the scenes.
190. 18:11And there we go.
191. 18:14Very cool.
192. 18:19So there you are.
193. 18:20Structured queryable context for agents and humans.
194. 18:25Now there's just one last thing.
195. 18:27Um on the description you probably saw that we were promising a context engine simulator.
196. 18:33If you don't want to wire up unblocked uh right away because you've got some constraints in your in your environment, um we have an open source project that will allow you to do this entirely
197. 18:44locally yourself.
198. 18:46And so the idea is uh it will create a context bundle for you on a taskbytask basis and then take that context bundle and run an AB test against the same task
199. 18:58and show you the benefits of having all of that context at your fingertips for your agents.
200. 19:04And that's the uh PR code for that if you want that.
201. 19:16All right.
202. 19:18And one last thing, if you want some coconuts, come by the booth.
203. 19:26Thank you.