Citation Needed: Provenance for LLM-Built Knowledge Graphs — Daniel Chalef, Zep AI

AI Engineer · 20 min · 194 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12So, LLMs are really great at pulling together data from many sources.
00:17Uh, but they do so non sorry, they do so non-deterministically.
00:23They interpret and synthesize data generating a summary, an extracted fact, uh a structured record.
00:31And this output artifact may not appear verbatim in the source inputs.
00:36Synthesis often destroys the paper trail of how these outputs were originated.
00:42And I'm going to be talking today about provenence, which is tracing how an artifact was built and why.
00:51Legal compliance often demands provenence, but it's also useful for debugging.
00:58Deciding which sources you trust and which artifacts to delete.
01:03And solving this at scale presents a real engineering challenge.
01:10If I can get my next slide going here.
01:14So my team and I built Graffiti uh the open-source temporal graph framework and Zep, our enterprise agent memory infrastructure is built on graffiti.
01:26Our customers derive context or agent memory for many user touch points.
01:32Those could be chat, but not only chat.
01:35Often it's voice transcripts, email, business data.
01:39And our customers have struggled with provenence.
01:42Where did this fact come from?
01:45What is the veracity of this fact?
01:47And over the next few slides, I'll share how we engineered solutions to this problem.
01:55So here's an stylized failure mode.
01:59An agent retrieves context about a patient.
02:03So this is a healthcare scenario.
02:06And what comes back is a clean confident fact.
02:09Patient has a penicellin allergy.
02:13And the context was synthesized from three sources.
02:16A lengthy EHR record, electronic health record, a PDF lab report, and something a patient typed into an AI intake chat.
02:30If the agent presents the fact to a doctor in treat in a treatment scenario without clearly indicating the source was from the patient themselves, it may mislead the doctor.
02:43When an agent retrieves context, can we point to the exact source and its veracity?
02:51For complex agent applications, the answer is often no. So, I can imagine you're probably thinking, um, but can't we just store like a source ID on the fact?
03:07This can work well in structured data warehouses or data lakes.
03:11A pipeline outputs one value copied or mutated deterministically, and the sources are known and easily marked.
03:20But with context pipelines run by LLMs, this breaks in several ways.
03:26You prompt an LLM with several sources.
03:29Many facts are each synthesized from one or more of the sources.
03:35Somebody like J. Smith and John Smith are merged into a single entity, one identity.
03:42And John's facts are derived from many different places.
03:46So new data might invalidate old facts.
03:50The store keeps changing underneath your pointer and an appendon log which often might you know might come to mind here gets very hard to manage at scale as there's so many changes occurring.
04:05So lineage needs to be an evolving set and survive mutation.
04:13So sets of links between facts and their sources can be modeled on a graph as relationships.
04:20So provenence in a context store containing facts is a knowledge graph.
04:27We have three in this example we have three source data in graffiti that turned episodes.
04:34We have two entities extracted from the episodes patient and penicellin and an edge between them.
04:44This graph triple the two entities and the edge can be hydrated as a fact.
04:50Patient has a penicellin allergy.
04:54Tracing a fact to its source is just a graph walk.
05:01So it's pretty simple and easy to map source to fact on the first right but keeping it correct while the graph changes can be really hard when new data
05:18uh so for example when two entities merge the merged entity needs to keep all source links from both otherwise we silently drop a source and we lose lineage.
05:31And when new data contradicts existing data, mutating it, we need to capture this lineage too.
05:41In the rightmost card, a fact is rendered invalid by new data.
05:45And in graffiti, an invalid date is added to the mutated edge.
05:54And the source episodes that resulted in the edge mutating are noted against the fact.
06:06So gaining graffiti, the relationship between source data and derived artifacts such as facts is easily modeled on the graph.
06:16With metadata projection, we can also model classifications that span many different episodes.
06:24and facts derived from them.
06:26And so I'll give you an example here.
06:30In in the prior healthc care scenario, episodes may originate from an EHR record and have an EHR tag associate associated with them, but not all records are.
06:43And so on ingestion, we tag the episodes with the EHR tag.
06:50All subsequent entities and facts derived from the episode inherit the tag.
06:58And so if the agent wants to retrieve only facts from verified clinical sources, it's very simple to filter for the appropriate tag as we walk the graph.
07:11So one tagging action at ingestion supports evaluating the veracity of a fact.
07:22But what if the fact is three parents or more?
07:26Here we have a verified flag as our metadata and in this case one parent is not verified.
07:34So is the fact verified for the allergy flag which could be a life and death situation.
07:42the agent missing it, missing that particular flag could be a deadly mistake.
07:49So not retrieving the fact and any source of the three should block that prescription being issued.
07:58But for something like a consent on file for procedure fact, the mistake is operating on unverified consent.
08:08So the patient hasn't actually given their consent and every parent needs to be verified.
08:14So every single episode should have that tag.
08:18So the facts are very similar shapes.
08:21They have three three parent episodes but opposite policies.
08:26And here graffiti or the underlying store exposes that choice.
08:32It exposes which of the episodes have the gra the particular tag, but your agent needs to execute or apply your business rules.
08:45So that's not necessarily something that we bake into the graph.
08:49[snorts]
08:49It's situational.
08:52Another situation where lineage is really important, we may have to delete source data due to retention policies or right to be forgotten requests.
09:05So privacy compliance and this is really tricky because if we have context derived from multiple sources, how do we do so?
09:16Mapping lineage here is really useful.
09:19We know which facts are derived from the source data we intend on deleting.
09:25But what if only some of the source data needs to be deleted but not all?
09:31So in this example, we need to delete the intake chart data.
09:36So what the patient filled in, which is only one of three source data.
09:43In graffiti's model, the allergy fact survives the deletion and that's because there are two parents parent episode still supporting the fact.
09:56However, the contact preference fact is deleted as it was derived solely from the deleted source data.
10:07So, the rule is pretty simple here and it's easier to apply because the link exists.
10:12A fact is only deleted if no remaining episodes support it.
10:19So to sum it all up, deriving context is lossy and generative.
10:26Lineage needs to be built in to the data structure, engineered into the data structure, which is a graph, not logged afterwards.
10:37And in graffiti we keep the sources verbatim and we link everything back everything derived from those sources back to the source.
10:47And provenence offers many benefits to users of graffiti.
10:53You have compliance built in which makes your chief compliance officer very happy.
10:59You can verify a fact based on its sources so you understand veracity.
11:05Should I trust this fact?
11:08It's easy to debug where something came from.
11:12So, why do I have this fact?
11:14How was it generated?
11:15And also determining what to delete.
11:18And most of what I've covered today is in the graffiti framework.
11:23So, you can go to uh the graffiti repo on GitHub and I have a little uh QR code QR code later that you can zap um and try it out.
11:37So, by the way, as an aside, lineage and provenence is expensive.
11:43Graph construction is really expensive uh in the way that graffiti does it.
11:48And so we've put significant effort into reducing cost and latency of generating graph artifacts.
11:57And I'd be happy to speak to how we've done that uh in in the Q&A.
12:01So thanks for attending.
12:03Um if you'd like to learn a little bit more about Zep or Graffiti, you can zap the QR codes.
12:10Uh Zep is on the left and graffiti on the right.
12:14And I don't know if we're doing Q&A here or outside.
12:19Okay, happy to do Q&A.
12:21Just repeat the question.
12:23Yeah,
12:24we have time.
12:25Anybody have a question?
12:28Yeah, one right from the front.
12:34How do we mutate the graph at the edge?
12:41Oh, to account for weight changes in relevancy.
12:45Um that is some structure that we've actually built into Zap not into graffiti and um what we do is we do have for that a some tracing that we do which is kind of x of the graph.
13:02So not all of the um provenence is in the graph.
13:08Sorry, you
13:10if you have 50 edges,
13:16uh, sorry, is was that a question?
13:17If you have 50.
13:18Oh, there you go.
13:19I said I I use 50.
13:22You use 50.
13:23Yeah,
13:2450 edges.
13:25Um, so so you know, Zep is able to uh look at provenence across those but using a separate data structure from the graph.
13:36Okay.
13:36Thank you.
13:37Thank you
13:38for that particular problem.
13:43So uh does the agent also create the edges edge types and the entities itself or how does it resolve those edges types?
13:51So in graffiti you can search across the entire graph.
13:55Uh it has um vector similarity search against various textual artifacts um full text search as well as graph relational operations things like BFS um it depends on the underlying
14:10uh graph database that's used and so your agent can walk the graph it can search semantically etc. And obviously from anywhere you hit in the graph you're then able to understand the provenence
14:24of a particular artifact that you've hit.
14:29Uh I find it very uh fascinating this temporal support out right out of the gravity database.
14:38Can you uh tell just a little more um under the hood what are those really the the episodes are nodes in the graph just like the other nodes and how do you extract so I saw the API that let's say I give some
14:54uh ad ed episode how do we extract information from the ad episode call under the hood like large language models and and edges
15:04yeah yeah so uh yes episodes are an entity on the graph uh or a node on the graph.
15:13Um it makes sense to model them that way.
15:17Um in Zep and graffiti we have various derived artifacts that are um nodes on the graph as well because they too need to have lineage and we need to understand how they were derived.
15:32Um, and in terms of how the ad episode method works, there's a pretty complicated uh, pipeline that gets run on um, uh, episode ingestion and I'll just give it the very high level
15:50uh, uh, outline for you.
15:53So there's a structured extraction extracting entities and the relationships between them and candidate facts and those are the materialized or hydrated fact triples.
16:06So two entities and a fact.
16:09Um and it's structured as uh a fact is structured as um um uh subject well subject uh verb object.
16:21Um and after that there is a uh deconliction process that runs dduplication and deconliction process.
16:30We dduplicate entities and we deconlict facts because there might be existing facts in the graph that are going to be mutated by a new learned fact.
16:41So Daniel loves Adidas shoes.
16:44Three months later, Daniel's shoes fell apart.
16:47He sends it back to those the shoes back to the return application and he sends a nasty gram along with it.
16:53We now Daniel returned shoes as a fact and Daniel was unhappy about Adidas.
16:59We need to invalidate the Daniel loves Adidas shoes fact and so that is part of that pipeline as well.
17:06A lot of what we do uses LLMs but we try very hard not to use LLMs in this process as well.
17:14So where we're able to deploy more traditional information retrieval techniques, more traditional NLP techniques, uh looking at things like entropy and a bunch of other, you know, using um sim hash and a bunch of other approaches to ddupe,
17:31uh we do so uh far cheaper, far faster, deter more more far more deterministic.
17:38So hopefully that answer your questions.
17:41Yep.
17:42Uh thank you for the great talk.
17:44Um just wanted to ask a question.
17:46So it seems like a common theme these days in uh memory systems is more filebased memory and wikis and knowledge bases and uh I'm just wondering have is Zep working on something like that and also
18:01uh could some of the ideas here be represented in that paradigm?
18:05Yeah.
18:07uh markdown suffers from provenence.
18:11File-based um file-based uh memory starts to break down with provenence.
18:18It's very difficult when you m mutate lines in a file to understand the lineage or the provenence of why those changes occurred.
18:30Um, not only that, but in multi- aent, multi-user, and multi-source scenarios, it can be very challenging to manage markdown files at scale.
18:42I think they work really well um for desktop usage.
18:47Uh, they sometimes work well in uh agentic use cases that are server based, not necessarily desktop or single user, single agent scenarios.
18:58Um but what we found is that it's um they just break down with the types of enterprise problems that we solving in particularly in particular provenence as an example.
19:10Does that answer your question?
19:14Oh more I don't know how much time we have left but maybe one more
19:19uh Daniel thank you.
19:20Um question on the right bag.
19:24How do you do that explicitly or implicitly?
19:27Um, how do you create the facts?
19:29Do you ask the LLM to summarize the conversation or at every turn you do that?
19:34And and which component does it, Zep or?
19:37Uh,
19:37yeah.
19:37So, so we do um as part we actually as part of the extraction, we've managed to get um a singleshot extraction working that extracts entities and the relationships between them
19:52and facts.
19:55and we're able to do so really cheaply as a consequence.
19:59Um, and so yes, we're using an LLM for that.
20:04Uh, we do have a reflection step or some reflection built in to ensure that the things that we've retrieved um are actually accurate um as well as to do some other stuff around
20:18uh uh more more richness to lineage.
20:22So why did something change?
20:24Not just this was related but also why did it change.
20:30Uh that's uh partly in graffiti partly in zap.
20:35Yeah.
20:36All right.
20:36Well, thank you everybody.