WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=H7puB0RwJMM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/H7puB0RwJMM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.960 --> 00:00:17.920
So, LLMs are really great at pulling together data from many sources.

s3
00:00:17.920 --> 00:00:23.039
Uh, but they do so non sorry, they do so non-deterministically.

s4
00:00:23.039 --> 00:00:31.279
They interpret and synthesize data generating a summary, an extracted fact, uh a structured record.

s5
00:00:31.279 --> 00:00:36.559
And this output artifact may not appear verbatim in the source inputs.

s6
00:00:36.559 --> 00:00:42.800
Synthesis often destroys the paper trail of how these outputs were originated.

s7
00:00:42.800 --> 00:00:51.840
And I'm going to be talking today about provenence, which is tracing how an artifact was built and why.

s8
00:00:51.840 --> 00:00:58.000
Legal compliance often demands provenence, but it's also useful for debugging.

s9
00:00:58.000 --> 00:01:03.199
Deciding which sources you trust and which artifacts to delete.

s10
00:01:03.199 --> 00:01:08.799
And solving this at scale presents a real engineering challenge.

s11
00:01:10.560 --> 00:01:14.479
If I can get my next slide going here.

s12
00:01:14.479 --> 00:01:26.159
So my team and I built Graffiti uh the open-source temporal graph framework and Zep, our enterprise agent memory infrastructure is built on graffiti.

s13
00:01:26.159 --> 00:01:32.079
Our customers derive context or agent memory for many user touch points.

s14
00:01:32.079 --> 00:01:35.200
Those could be chat, but not only chat.

s15
00:01:35.200 --> 00:01:39.600
Often it's voice transcripts, email, business data.

s16
00:01:39.600 --> 00:01:42.240
And our customers have struggled with provenence.

s17
00:01:42.240 --> 00:01:45.200
Where did this fact come from?

s18
00:01:45.200 --> 00:01:47.759
What is the veracity of this fact?

s19
00:01:47.759 --> 00:01:54.720
And over the next few slides, I'll share how we engineered solutions to this problem.

s20
00:01:55.600 --> 00:01:59.439
So here's an stylized failure mode.

s21
00:01:59.439 --> 00:02:03.119
An agent retrieves context about a patient.

s22
00:02:03.119 --> 00:02:06.000
So this is a healthcare scenario.

s23
00:02:06.000 --> 00:02:09.679
And what comes back is a clean confident fact.

s24
00:02:09.679 --> 00:02:13.280
Patient has a penicellin allergy.

s25
00:02:13.280 --> 00:02:16.959
And the context was synthesized from three sources.

s26
00:02:16.959 --> 00:02:30.080
A lengthy EHR record, electronic health record, a PDF lab report, and something a patient typed into an AI intake chat.

s27
00:02:30.080 --> 00:02:43.920
If the agent presents the fact to a doctor in treat in a treatment scenario without clearly indicating the source was from the patient themselves, it may mislead the doctor.

s28
00:02:43.920 --> 00:02:51.440
When an agent retrieves context, can we point to the exact source and its veracity?

s29
00:02:51.440 --> 00:03:07.040
For complex agent applications, the answer is often no. So, I can imagine you're probably thinking, um, but can't we just store like a source ID on the fact?

s30
00:03:07.280 --> 00:03:11.519
This can work well in structured data warehouses or data lakes.

s31
00:03:11.519 --> 00:03:20.480
A pipeline outputs one value copied or mutated deterministically, and the sources are known and easily marked.

s32
00:03:20.480 --> 00:03:26.560
But with context pipelines run by LLMs, this breaks in several ways.

s33
00:03:26.560 --> 00:03:29.920
You prompt an LLM with several sources.

s34
00:03:29.920 --> 00:03:35.440
Many facts are each synthesized from one or more of the sources.

s35
00:03:35.519 --> 00:03:42.239
Somebody like J. Smith and John Smith are merged into a single entity, one identity.

s36
00:03:42.239 --> 00:03:46.959
And John's facts are derived from many different places.

s37
00:03:46.959 --> 00:03:50.480
So new data might invalidate old facts.

s38
00:03:50.480 --> 00:04:05.040
The store keeps changing underneath your pointer and an appendon log which often might you know might come to mind here gets very hard to manage at scale as there's so many changes occurring.

s39
00:04:05.040 --> 00:04:11.519
So lineage needs to be an evolving set and survive mutation.

s40
00:04:13.439 --> 00:04:20.959
So sets of links between facts and their sources can be modeled on a graph as relationships.

s41
00:04:20.959 --> 00:04:27.120
So provenence in a context store containing facts is a knowledge graph.

s42
00:04:27.520 --> 00:04:34.400
We have three in this example we have three source data in graffiti that turned episodes.

s43
00:04:34.400 --> 00:04:44.000
We have two entities extracted from the episodes patient and penicellin and an edge between them.

s44
00:04:44.000 --> 00:04:50.960
This graph triple the two entities and the edge can be hydrated as a fact.

s45
00:04:50.960 --> 00:04:54.240
Patient has a penicellin allergy.

s46
00:04:54.240 --> 00:04:59.520
Tracing a fact to its source is just a graph walk.

s47
00:05:01.520 --> 00:05:18.639
So it's pretty simple and easy to map source to fact on the first right but keeping it correct while the graph changes can be really hard when new data

s48
00:05:18.639 --> 00:05:31.919
uh so for example when two entities merge the merged entity needs to keep all source links from both otherwise we silently drop a source and we lose lineage.

s49
00:05:31.919 --> 00:05:41.120
And when new data contradicts existing data, mutating it, we need to capture this lineage too.

s50
00:05:41.120 --> 00:05:45.680
In the rightmost card, a fact is rendered invalid by new data.

s51
00:05:45.680 --> 00:05:54.800
And in graffiti, an invalid date is added to the mutated edge.

s52
00:05:54.800 --> 00:06:04.000
And the source episodes that resulted in the edge mutating are noted against the fact.

s53
00:06:06.319 --> 00:06:16.000
So gaining graffiti, the relationship between source data and derived artifacts such as facts is easily modeled on the graph.

s54
00:06:16.720 --> 00:06:24.319
With metadata projection, we can also model classifications that span many different episodes.

s55
00:06:24.319 --> 00:06:26.960
and facts derived from them.

s56
00:06:26.960 --> 00:06:30.240
And so I'll give you an example here.

s57
00:06:30.240 --> 00:06:43.919
In in the prior healthc care scenario, episodes may originate from an EHR record and have an EHR tag associate associated with them, but not all records are.

s58
00:06:43.919 --> 00:06:50.720
And so on ingestion, we tag the episodes with the EHR tag.

s59
00:06:50.720 --> 00:06:58.400
All subsequent entities and facts derived from the episode inherit the tag.

s60
00:06:58.400 --> 00:07:11.039
And so if the agent wants to retrieve only facts from verified clinical sources, it's very simple to filter for the appropriate tag as we walk the graph.

s61
00:07:11.039 --> 00:07:20.240
So one tagging action at ingestion supports evaluating the veracity of a fact.

s62
00:07:22.720 --> 00:07:26.880
But what if the fact is three parents or more?

s63
00:07:26.880 --> 00:07:34.639
Here we have a verified flag as our metadata and in this case one parent is not verified.

s64
00:07:34.639 --> 00:07:42.800
So is the fact verified for the allergy flag which could be a life and death situation.

s65
00:07:42.800 --> 00:07:49.280
the agent missing it, missing that particular flag could be a deadly mistake.

s66
00:07:49.280 --> 00:07:58.800
So not retrieving the fact and any source of the three should block that prescription being issued.

s67
00:07:58.879 --> 00:08:08.879
But for something like a consent on file for procedure fact, the mistake is operating on unverified consent.

s68
00:08:08.879 --> 00:08:14.560
So the patient hasn't actually given their consent and every parent needs to be verified.

s69
00:08:14.560 --> 00:08:18.720
So every single episode should have that tag.

s70
00:08:18.720 --> 00:08:21.440
So the facts are very similar shapes.

s71
00:08:21.440 --> 00:08:26.240
They have three three parent episodes but opposite policies.

s72
00:08:26.240 --> 00:08:32.640
And here graffiti or the underlying store exposes that choice.

s73
00:08:32.640 --> 00:08:45.120
It exposes which of the episodes have the gra the particular tag, but your agent needs to execute or apply your business rules.

s74
00:08:45.120 --> 00:08:49.246
So that's not necessarily something that we bake into the graph.

s75
00:08:49.246 --> 00:08:49.839
[snorts]

s76
00:08:49.839 --> 00:08:52.160
It's situational.

s77
00:08:52.480 --> 00:09:05.200
Another situation where lineage is really important, we may have to delete source data due to retention policies or right to be forgotten requests.

s78
00:09:05.200 --> 00:09:16.720
So privacy compliance and this is really tricky because if we have context derived from multiple sources, how do we do so?

s79
00:09:16.720 --> 00:09:19.360
Mapping lineage here is really useful.

s80
00:09:19.360 --> 00:09:25.680
We know which facts are derived from the source data we intend on deleting.

s81
00:09:25.680 --> 00:09:31.839
But what if only some of the source data needs to be deleted but not all?

s82
00:09:31.839 --> 00:09:36.720
So in this example, we need to delete the intake chart data.

s83
00:09:36.720 --> 00:09:43.839
So what the patient filled in, which is only one of three source data.

s84
00:09:43.839 --> 00:09:56.560
In graffiti's model, the allergy fact survives the deletion and that's because there are two parents parent episode still supporting the fact.

s85
00:09:56.560 --> 00:10:07.040
However, the contact preference fact is deleted as it was derived solely from the deleted source data.

s86
00:10:07.040 --> 00:10:12.399
So, the rule is pretty simple here and it's easier to apply because the link exists.

s87
00:10:12.399 --> 00:10:18.480
A fact is only deleted if no remaining episodes support it.

s88
00:10:19.760 --> 00:10:26.480
So to sum it all up, deriving context is lossy and generative.

s89
00:10:26.880 --> 00:10:36.480
Lineage needs to be built in to the data structure, engineered into the data structure, which is a graph, not logged afterwards.

s90
00:10:37.360 --> 00:10:47.360
And in graffiti we keep the sources verbatim and we link everything back everything derived from those sources back to the source.

s91
00:10:47.360 --> 00:10:53.120
And provenence offers many benefits to users of graffiti.

s92
00:10:53.120 --> 00:10:59.120
You have compliance built in which makes your chief compliance officer very happy.

s93
00:10:59.120 --> 00:11:05.440
You can verify a fact based on its sources so you understand veracity.

s94
00:11:05.440 --> 00:11:08.160
Should I trust this fact?

s95
00:11:08.160 --> 00:11:12.000
It's easy to debug where something came from.

s96
00:11:12.000 --> 00:11:14.320
So, why do I have this fact?

s97
00:11:14.320 --> 00:11:15.920
How was it generated?

s98
00:11:15.920 --> 00:11:18.800
And also determining what to delete.

s99
00:11:18.800 --> 00:11:23.120
And most of what I've covered today is in the graffiti framework.

s100
00:11:23.120 --> 00:11:37.920
So, you can go to uh the graffiti repo on GitHub and I have a little uh QR code QR code later that you can zap um and try it out.

s101
00:11:37.920 --> 00:11:43.200
So, by the way, as an aside, lineage and provenence is expensive.

s102
00:11:43.200 --> 00:11:48.560
Graph construction is really expensive uh in the way that graffiti does it.

s103
00:11:48.560 --> 00:11:57.040
And so we've put significant effort into reducing cost and latency of generating graph artifacts.

s104
00:11:57.040 --> 00:12:01.760
And I'd be happy to speak to how we've done that uh in in the Q&A.

s105
00:12:01.760 --> 00:12:03.920
So thanks for attending.

s106
00:12:03.920 --> 00:12:10.480
Um if you'd like to learn a little bit more about Zep or Graffiti, you can zap the QR codes.

s107
00:12:10.480 --> 00:12:14.000
Uh Zep is on the left and graffiti on the right.

s108
00:12:14.000 --> 00:12:19.200
And I don't know if we're doing Q&A here or outside.

s109
00:12:19.519 --> 00:12:21.600
Okay, happy to do Q&A.

s110
00:12:21.600 --> 00:12:23.040
Just repeat the question.

s111
00:12:23.040 --> 00:12:24.480
Yeah,

s112
00:12:24.480 --> 00:12:25.279
we have time.

s113
00:12:25.279 --> 00:12:27.920
Anybody have a question?

s114
00:12:28.000 --> 00:12:31.279
Yeah, one right from the front.

s115
00:12:34.639 --> 00:12:40.320
How do we mutate the graph at the edge?

s116
00:12:41.519 --> 00:12:45.200
Oh, to account for weight changes in relevancy.

s117
00:12:45.200 --> 00:13:02.399
Um that is some structure that we've actually built into Zap not into graffiti and um what we do is we do have for that a some tracing that we do which is kind of x of the graph.

s118
00:13:02.399 --> 00:13:08.399
So not all of the um provenence is in the graph.

s119
00:13:08.399 --> 00:13:10.720
Sorry, you

s120
00:13:10.880 --> 00:13:13.680
if you have 50 edges,

s121
00:13:16.000 --> 00:13:17.680
uh, sorry, is was that a question?

s122
00:13:17.680 --> 00:13:18.480
If you have 50.

s123
00:13:18.480 --> 00:13:19.760
Oh, there you go.

s124
00:13:19.760 --> 00:13:22.720
I said I I use 50.

s125
00:13:22.720 --> 00:13:23.839
You use 50.

s126
00:13:23.839 --> 00:13:24.800
Yeah,

s127
00:13:24.800 --> 00:13:25.680
50 edges.

s128
00:13:25.680 --> 00:13:36.240
Um, so so you know, Zep is able to uh look at provenence across those but using a separate data structure from the graph.

s129
00:13:36.240 --> 00:13:36.639
Okay.

s130
00:13:36.639 --> 00:13:37.360
Thank you.

s131
00:13:37.360 --> 00:13:38.160
Thank you

s132
00:13:38.160 --> 00:13:41.040
for that particular problem.

s133
00:13:43.839 --> 00:13:51.440
So uh does the agent also create the edges edge types and the entities itself or how does it resolve those edges types?

s134
00:13:51.440 --> 00:13:55.440
So in graffiti you can search across the entire graph.

s135
00:13:55.440 --> 00:14:10.959
Uh it has um vector similarity search against various textual artifacts um full text search as well as graph relational operations things like BFS um it depends on the underlying

s136
00:14:10.959 --> 00:14:24.639
uh graph database that's used and so your agent can walk the graph it can search semantically etc. And obviously from anywhere you hit in the graph you're then able to understand the provenence

s137
00:14:24.639 --> 00:14:28.480
of a particular artifact that you've hit.

s138
00:14:29.760 --> 00:14:38.320
Uh I find it very uh fascinating this temporal support out right out of the gravity database.

s139
00:14:38.320 --> 00:14:54.399
Can you uh tell just a little more um under the hood what are those really the the episodes are nodes in the graph just like the other nodes and how do you extract so I saw the API that let's say I give some

s140
00:14:54.399 --> 00:15:04.800
uh ad ed episode how do we extract information from the ad episode call under the hood like large language models and and edges

s141
00:15:04.800 --> 00:15:13.680
yeah yeah so uh yes episodes are an entity on the graph uh or a node on the graph.

s142
00:15:13.680 --> 00:15:17.360
Um it makes sense to model them that way.

s143
00:15:17.360 --> 00:15:32.480
Um in Zep and graffiti we have various derived artifacts that are um nodes on the graph as well because they too need to have lineage and we need to understand how they were derived.

s144
00:15:32.480 --> 00:15:50.959
Um, and in terms of how the ad episode method works, there's a pretty complicated uh, pipeline that gets run on um, uh, episode ingestion and I'll just give it the very high level

s145
00:15:50.959 --> 00:15:53.040
uh, uh, outline for you.

s146
00:15:53.040 --> 00:16:06.880
So there's a structured extraction extracting entities and the relationships between them and candidate facts and those are the materialized or hydrated fact triples.

s147
00:16:06.880 --> 00:16:09.839
So two entities and a fact.

s148
00:16:09.839 --> 00:16:21.519
Um and it's structured as uh a fact is structured as um um uh subject well subject uh verb object.

s149
00:16:21.519 --> 00:16:30.720
Um and after that there is a uh deconliction process that runs dduplication and deconliction process.

s150
00:16:30.720 --> 00:16:41.600
We dduplicate entities and we deconlict facts because there might be existing facts in the graph that are going to be mutated by a new learned fact.

s151
00:16:41.600 --> 00:16:44.959
So Daniel loves Adidas shoes.

s152
00:16:44.959 --> 00:16:47.040
Three months later, Daniel's shoes fell apart.

s153
00:16:47.040 --> 00:16:53.199
He sends it back to those the shoes back to the return application and he sends a nasty gram along with it.

s154
00:16:53.199 --> 00:16:59.360
We now Daniel returned shoes as a fact and Daniel was unhappy about Adidas.

s155
00:16:59.360 --> 00:17:06.160
We need to invalidate the Daniel loves Adidas shoes fact and so that is part of that pipeline as well.

s156
00:17:06.160 --> 00:17:14.720
A lot of what we do uses LLMs but we try very hard not to use LLMs in this process as well.

s157
00:17:14.720 --> 00:17:31.200
So where we're able to deploy more traditional information retrieval techniques, more traditional NLP techniques, uh looking at things like entropy and a bunch of other, you know, using um sim hash and a bunch of other approaches to ddupe,

s158
00:17:31.200 --> 00:17:38.160
uh we do so uh far cheaper, far faster, deter more more far more deterministic.

s159
00:17:38.640 --> 00:17:41.360
So hopefully that answer your questions.

s160
00:17:41.360 --> 00:17:42.880
Yep.

s161
00:17:42.880 --> 00:17:44.640
Uh thank you for the great talk.

s162
00:17:44.640 --> 00:17:46.640
Um just wanted to ask a question.

s163
00:17:46.640 --> 00:18:01.360
So it seems like a common theme these days in uh memory systems is more filebased memory and wikis and knowledge bases and uh I'm just wondering have is Zep working on something like that and also

s164
00:18:01.360 --> 00:18:05.840
uh could some of the ideas here be represented in that paradigm?

s165
00:18:05.840 --> 00:18:07.360
Yeah.

s166
00:18:07.360 --> 00:18:11.919
uh markdown suffers from provenence.

s167
00:18:11.919 --> 00:18:18.320
File-based um file-based uh memory starts to break down with provenence.

s168
00:18:18.320 --> 00:18:30.000
It's very difficult when you m mutate lines in a file to understand the lineage or the provenence of why those changes occurred.

s169
00:18:30.000 --> 00:18:42.720
Um, not only that, but in multi- aent, multi-user, and multi-source scenarios, it can be very challenging to manage markdown files at scale.

s170
00:18:42.720 --> 00:18:47.600
I think they work really well um for desktop usage.

s171
00:18:47.600 --> 00:18:58.000
Uh, they sometimes work well in uh agentic use cases that are server based, not necessarily desktop or single user, single agent scenarios.

s172
00:18:58.000 --> 00:19:10.080
Um but what we found is that it's um they just break down with the types of enterprise problems that we solving in particularly in particular provenence as an example.

s173
00:19:10.080 --> 00:19:12.960
Does that answer your question?

s174
00:19:14.160 --> 00:19:19.120
Oh more I don't know how much time we have left but maybe one more

s175
00:19:19.280 --> 00:19:20.640
uh Daniel thank you.

s176
00:19:20.640 --> 00:19:24.240
Um question on the right bag.

s177
00:19:24.240 --> 00:19:27.120
How do you do that explicitly or implicitly?

s178
00:19:27.120 --> 00:19:29.520
Um, how do you create the facts?

s179
00:19:29.520 --> 00:19:34.080
Do you ask the LLM to summarize the conversation or at every turn you do that?

s180
00:19:34.080 --> 00:19:37.039
And and which component does it, Zep or?

s181
00:19:37.039 --> 00:19:37.520
Uh,

s182
00:19:37.520 --> 00:19:37.760
yeah.

s183
00:19:37.760 --> 00:19:52.799
So, so we do um as part we actually as part of the extraction, we've managed to get um a singleshot extraction working that extracts entities and the relationships between them

s184
00:19:52.799 --> 00:19:55.120
and facts.

s185
00:19:55.120 --> 00:19:59.520
and we're able to do so really cheaply as a consequence.

s186
00:19:59.520 --> 00:20:04.000
Um, and so yes, we're using an LLM for that.

s187
00:20:04.000 --> 00:20:18.640
Uh, we do have a reflection step or some reflection built in to ensure that the things that we've retrieved um are actually accurate um as well as to do some other stuff around

s188
00:20:18.640 --> 00:20:22.480
uh uh more more richness to lineage.

s189
00:20:22.480 --> 00:20:24.000
So why did something change?

s190
00:20:24.000 --> 00:20:28.960
Not just this was related but also why did it change.

s191
00:20:30.080 --> 00:20:35.440
Uh that's uh partly in graffiti partly in zap.

s192
00:20:35.440 --> 00:20:36.000
Yeah.

s193
00:20:36.000 --> 00:20:36.320
All right.

s194
00:20:36.320 --> 00:20:38.799
Well, thank you everybody.