{ "video": { "id": "spxoDAfh5ig", "title": "Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked", "duration": 2299, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "spxoDAfh5ig", "title": "Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked", "one_liner": "RAG (semantic vector search) is only half of retrieval. The other half is having the LLM write a real database query instead of answering itself, wrapped in schema discovery, identity resolution, allow-list validation and an error-feedback retry loop.", "summary": "Peter Werry of Unblocked argues that engineers' questions about organizational context, like \"PRs I merged last week\", depend on structure (author, state, dates, relationships) that vector similarity search can't capture. A semantically similar PR merged three years ago is a wrong answer. Across six stacked PRs he builds a natural-language-to-database-query engine over GitHub pull requests. It discovers the schema by sampling documents, maps names to GitHub logins, has the LLM generate the query through a forced tool call, validates that query against an allow list, retries with error feedback, and adds full-text search. The same engine powers a query playground and a chat agent that reasons over the results.", "key_points": [ "In the talk, \"RAG\" means semantic similarity search / vector retrieval. It works well for content questions (find the code, the PR that changed it, the Slack thread that debated it) and as a starting point for an agent. It fails on structural and temporal questions such as \"PRs I merged last week\".", "Core paradigm: the model doesn't answer the question. It writes the query that accesses the database.", "Dynamic schema discovery for a schemaless store: sample 300 documents, merge them into a single tree, skip nulls (a missing field can't constrain the type), and infer enum types with procedural code. The schema is emitted as markdown, with suggested indexes, so it can be passed to the LLM.", "Identity resolution: \"Peter\" must map to the GitHub login (puery). Simple fuzzy matching such as Levenshtein distance is enough, though an LLM could also synthesize the mappings.", "Query planning: pass the schema, force a tool call for name extraction, and include the current date and time in the prompt. LLMs are \"pretty good\" at temporal queries when given it.", "Safety: generated queries are untrusted. Where/function clauses could run JavaScript on the server, which makes them a prompt-injection point, and hidden ID or metadata fields could leak internals. Allow-list operators (deny by default) and, where possible, fields. For multi-tenant collections, hide the tenant ID field from the LLM, wipe it if the LLM outputs it, and inject your own. Separate collections per tenant is still recommended.", "Retry loop: queries fail validation or execution (e.g. mixing inclusion and exclusion projection). Feeding the error back to the LLM, up to a maximum number of attempts, lets it self-correct, and the speaker called this one of the most important pieces. In the demo, banning regex left it with an exact match on \"authentication\", which was fixed by adding a full-text index combined with author and state filters.", "Q&A: to scale to many collections, add an LLM step that picks the relevant schema from short collection summaries. Unblocked ingests continuously via webhooks and uses static schemas in production for correctness. Slack is the most important source for \"why\" questions, and PR comments are mined into \"best practice memories\". Vector search and structured queries are complementary, and Unblocked also does hybrid lexical + semantic search with rank fusion." ], "takeaways": [ "Don't send structured, temporal or state-based questions through vector search. Have the LLM generate a database query against an explicit schema, with the current date and time in the prompt.", "Do extraction and validation with deterministic procedural code rather than LLMs. Allow-list query operators and fields (deny by default), and strip and re-inject tenant or metadata fields yourself.", "Build an error-feedback retry loop with a maximum number of attempts so validation and execution errors go back to the LLM for self-correction.", "Resolve human names to system identities (e.g. GitHub logins) before querying. Enforce this either through a query tool that takes a validated username argument, or by watching the agent and inserting a \"you forgot to call the extraction tool\" message.", "Use vector search as a semantic starting point, then give the agent structured-query tools. For many collections, route to the right schema with short collection descriptions." ], "topics": [ "rag", "text-to-query", "context-engineering", "schema-discovery", "query-validation", "agents", "retrieval", "security" ], "tools": [ "Unblocked", "GitHub", "Slack", "Microsoft Teams", "Linear", "Jira", "Notion", "Confluence", "MCP", "Levenshtein distance" ], "quotes": [ { "text": "rag is only half of retrieval and today we're going to build the other half which is turning na natural language into real database queries.", "at": "01:06", "url": "https://www.youtube.com/watch?v=spxoDAfh5ig&t=66s" }, { "text": "the model doesn't actually answer the question in this paradigm. It writes the query that accesses the database.", "at": "06:36", "url": "https://www.youtube.com/watch?v=spxoDAfh5ig&t=396s" }, { "text": "and we don't deny list these things. We allow list them. So, it's deny by default.", "at": "16:59", "url": "https://www.youtube.com/watch?v=spxoDAfh5ig&t=1019s" }, { "text": "The error loop is actually one of the most important things.", "at": "24:36", "url": "https://www.youtube.com/watch?v=spxoDAfh5ig&t=1476s" } ], "words": 5950 }, "summary_url": "/#spxoDAfh5ig", "transcript": { "html": "/transcripts/spxoDAfh5ig.html", "txt": "/transcripts/spxoDAfh5ig.txt", "vtt": "/transcripts/spxoDAfh5ig.vtt" } }