WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=XPj3mIKEtI4

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/XPj3mIKEtI4.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.400 --> 00:00:14.559
Let's go ahead and get started.

s2
00:00:14.559 --> 00:00:17.600
I see some folks wanting to to come on in.

s3
00:00:17.600 --> 00:00:19.520
So, feel free to to come in.

s4
00:00:19.520 --> 00:00:26.160
Uh, this session is going to be all about actionable knowledge and context graphs.

s5
00:00:26.160 --> 00:00:27.359
So, my name is Will.

s6
00:00:27.359 --> 00:00:31.359
I'm a product manager at Neo Forj.

s7
00:00:31.359 --> 00:00:38.719
Um there's been a lot of discussion in this conference so far about loops, right?

s8
00:00:38.719 --> 00:00:41.840
The uh React loop, the Ralph loops.

s9
00:00:41.840 --> 00:00:45.840
Sometimes I feel like we're in maybe like a an amnesic loop, right?

s10
00:00:45.840 --> 00:00:50.559
where our our agents go through this uh reasoning phase acting.

s11
00:00:50.559 --> 00:00:57.680
They successfully complete some task but then kind of forget uh what they've learned.

s12
00:00:57.680 --> 00:01:10.640
And memory systems today are largely focused around uh embedding text then at retrieval time sort of finding the most relevant data to to stuff into the context, right?

s13
00:01:10.640 --> 00:01:21.360
Um so so something like this we find you know similar chunks of data in in our our corpus we inject that into the context window and trust that the agent is able to

s14
00:01:21.360 --> 00:01:34.799
do something useful with that right now the the challenge that we have is you know recall isn't really like actionable knowledge so one challenge we run into is like for example

s15
00:01:34.799 --> 00:01:46.079
we don't have a canonical representation of a thing right so if we have you know three different ways that we refer to Dr. Nwin, Robert N, the cardiologist, right?

s16
00:01:46.079 --> 00:01:53.840
Like depending on the the context of this discussion, we need to have some canonical representation of the thing, right?

s17
00:01:53.840 --> 00:02:01.680
So really agents need knowledge that is both connected, typed and traceable, right?

s18
00:02:01.680 --> 00:02:02.880
Not just retrievable.

s19
00:02:02.880 --> 00:02:10.399
So retrieval is is uh just a part of the problem when we're talking about uh agent memory.

s20
00:02:10.399 --> 00:02:14.720
So at NearJ this is how we think of agent memory.

s21
00:02:14.720 --> 00:02:22.239
We think of it as a connected graph composed of short-term, long-term and reasoning memory.

s22
00:02:22.239 --> 00:02:25.840
Uh we'll take a look at uh this in a bit more detail.

s23
00:02:25.840 --> 00:02:31.840
Uh but bear with me on on this idea of context graphs for agent memory.

s24
00:02:31.840 --> 00:02:32.080
Right?

s25
00:02:32.080 --> 00:02:35.040
So what are a few of the the key components here?

s26
00:02:35.040 --> 00:02:40.160
Well, one is going from unstructured data to a knowledge graph, right?

s27
00:02:40.160 --> 00:02:45.280
Going from the this process of agent messages, right?

s28
00:02:45.280 --> 00:02:55.760
Both user and assistant messages going through some entity extraction process where we're identifying what are the entities uh and how are they connected.

s29
00:02:55.760 --> 00:03:01.599
um doing this in a graph where we have strong types, right?

s30
00:03:01.599 --> 00:03:07.920
We have a a relationship that describes how these entities are related.

s31
00:03:08.640 --> 00:03:15.760
The entity resol resolution phase is one of the most important parts of building up this this knowledge graph, right?

s32
00:03:15.760 --> 00:03:21.040
Understanding and making sure that you have a canonical representation of the thing.

s33
00:03:21.040 --> 00:03:35.040
Uh so when we're talking about Dr. win we know this is a provider his name the role that he has uh and so on and a shared ontology is an important piece of this right so having some

s34
00:03:35.040 --> 00:03:45.120
description of your data model of the domain that you're working with this is one of the key pieces uh making sure that you are successful in going from that unstructured

s35
00:03:45.120 --> 00:03:56.959
text data to uh to a knowledge graph so for folks that have that have worked with memory uh in agents in the past like this this should look a little familiar.

s36
00:03:56.959 --> 00:04:10.560
Uh one thing though that I think is really important when we're talking about agent memory and actual knowledge is this idea of the reasoning graph right so um if we saw in our

s37
00:04:10.560 --> 00:04:24.000
our uh sort of three components of agent memory here reasoning memory is a first class citizen right we've talked about short-term and long-term these are the messages the entities extracted but what about the actual actions

s38
00:04:24.000 --> 00:04:32.479
that the agent takes what about the the reasoning um we want to make sure that we're capturing that because that is an important piece as well.

s39
00:04:32.479 --> 00:04:45.199
We want to make sure that we're storing thinking the facts not just uh not just the data right so how an agent decided and this is typically represented as decision traces

s40
00:04:45.199 --> 00:04:58.639
right so every decision uh that our agent makes is linked to uh the reasoning grounded in evidence right we have policies um that that we're modeling explicitly and we understand

s41
00:04:58.639 --> 00:05:04.560
for that execution plan the tools the agent called the results of those tool calls.

s42
00:05:04.560 --> 00:05:10.560
Uh we also want to capture things like how many tokens did this burn, the the timing that this took and so on.

s43
00:05:10.560 --> 00:05:16.000
This is all part of uh capturing the reasoning memory.

s44
00:05:16.000 --> 00:05:26.400
And this is important like not just so that a single agent can get better the next time you do it to do the exact same thing, but rather think about systems where you have

s45
00:05:26.400 --> 00:05:33.759
hundreds or or thousands of agents that have some shared grouping of of tools that they have access to, right?

s46
00:05:33.759 --> 00:05:43.199
So we're able to persist these agent runs, these these decision traces in the graph and then share that with other agents in our organization, right?

s47
00:05:43.199 --> 00:05:53.759
so that we have one shared context graph uh that enables our agents to essentially learn from uh each other in this shared memory piece.

s48
00:05:53.759 --> 00:05:54.160
Okay.

s49
00:05:54.160 --> 00:05:56.400
So that that that's the memory piece.

s50
00:05:56.400 --> 00:06:05.919
Like this this is um uh a system that we've built at at Near Forj, a pattern that that lots of folks are are following for working uh with agent memory.

s51
00:06:05.919 --> 00:06:08.800
But what about this idea of of actionable knowledge, right?

s52
00:06:08.800 --> 00:06:12.400
How do we enable our agents to to take action?

s53
00:06:12.400 --> 00:06:13.280
Uh memory.

s54
00:06:13.280 --> 00:06:17.360
One way to think of memory is is this is a good representation of what happened, right?

s55
00:06:17.360 --> 00:06:23.199
We know um the people we were talking about, how they're connected.

s56
00:06:23.199 --> 00:06:25.680
we have the these decision traces.

s57
00:06:25.680 --> 00:06:31.039
Um the next step is typically creating skills, right?

s58
00:06:31.039 --> 00:06:34.240
How many people are are using skills with their agents today?

s59
00:06:34.240 --> 00:06:35.120
Mo most folks.

s60
00:06:35.120 --> 00:06:35.680
Cool.

s61
00:06:35.680 --> 00:06:39.199
How many people have have written skills yourself?

s62
00:06:39.199 --> 00:06:39.440
Cool.

s63
00:06:39.440 --> 00:06:40.240
Mo most folks.

s64
00:06:40.240 --> 00:06:43.120
How many folks have had an agent write the skill for you?

s65
00:06:43.120 --> 00:06:43.600
Right.

s66
00:06:43.600 --> 00:06:44.319
Cool.

s67
00:06:44.319 --> 00:06:44.639
Yeah.

s68
00:06:44.639 --> 00:06:59.440
So that that's what we're we're talking about here is going from this memory graph, this context graph to how can we use that to create like grounded uh executable

s69
00:06:59.440 --> 00:06:59.919
skills.

s70
00:06:59.919 --> 00:07:13.840
So if you're not familiar with uh with a skill, this is um an open standard anthropic um put this out uh agentskills.io IO I think is where the um open standard is hosted.

s71
00:07:13.840 --> 00:07:18.800
And the basic idea here is that we have some uh some some metadata, right?

s72
00:07:18.800 --> 00:07:21.440
Some description of like what this skill is about.

s73
00:07:21.440 --> 00:07:24.319
Uh and then this progressive disclosure, right?

s74
00:07:24.319 --> 00:07:26.960
So I I have lots of more detailed information.

s75
00:07:26.960 --> 00:07:32.960
These are often in like markdown files, references that we can um progressively choose.

s76
00:07:32.960 --> 00:07:40.160
If if we're going down this path of one piece of the skill, we can uh retrieve and load that data uh into context.

s77
00:07:40.160 --> 00:07:40.479
Right?

s78
00:07:40.479 --> 00:07:52.639
So this is this then gives us some like sharable unit uh that we can then uh take the skill, package it up uh and reuse that across different agents.

s79
00:07:52.639 --> 00:08:02.160
But skills have a a similar challenge to some of the issues that we saw uh previously when when we were talking about uh memory, right?

s80
00:08:02.160 --> 00:08:06.080
is that we're we're typically working with pros.

s81
00:08:06.080 --> 00:08:17.680
Um, and so we're still often limited in in this uh challenge of understanding is this the canonical thing that we're talking about, making sure that we have like debugable

s82
00:08:17.680 --> 00:08:19.520
steps and and output.

s83
00:08:19.520 --> 00:08:33.680
Um and so to address some of these challenges that uh we're seeing with skills uh some folks on the NeoRaj research team um have done some interesting research and published some work on APE

s84
00:08:33.680 --> 00:08:37.599
uh graph representation for learning and governing agent skills.

s85
00:08:37.599 --> 00:08:40.399
Um this is a screenshot from the the paper.

s86
00:08:40.399 --> 00:08:43.200
Zach is uh is here somewhere.

s87
00:08:43.200 --> 00:08:46.720
If he's not here, he's he's at um the NearJ booth today.

s88
00:08:46.720 --> 00:08:49.519
So definitely uh chat with Zach if you're interested in this.

s89
00:08:49.519 --> 00:09:01.680
But you can think of this uh APE is essentially an extension of the agent skills um protocol with some additional metadata that now treats your skills as typed execution

s90
00:09:01.680 --> 00:09:02.240
graphs.

s91
00:09:02.240 --> 00:09:02.640
Right?

s92
00:09:02.640 --> 00:09:06.959
So we have uh steps modeled as nodes.

s93
00:09:06.959 --> 00:09:18.160
We have uh very um very strict schema that governs how we describe that how these uh steps are actionable.

s94
00:09:18.160 --> 00:09:18.480
Right?

s95
00:09:18.480 --> 00:09:31.839
So think of think of this as a way of of representing um a skill as a graph broken up into steps that are executable with a schema governed description.

s96
00:09:31.839 --> 00:09:44.080
Um that's the the ape protocol and this is from the uh the benchmark that was used in the the paper found that yeah like this actually matters when when they applied this to

s97
00:09:44.080 --> 00:09:58.880
um skills bench saw like a significant increase in uh task successfully completed when looking at applying uh the a protocol to human uh curated uh skills.

s98
00:10:00.000 --> 00:10:02.240
So that looks interesting.

s99
00:10:02.240 --> 00:10:09.600
How can we leverage some of those uh some of those ideas, some of that research for skill distillation?

s100
00:10:09.600 --> 00:10:15.120
Uh so essentially what what we want to do is take this this memory graph, right?

s101
00:10:15.120 --> 00:10:27.440
This context graph that's maybe scoped to a a workspace or scoped to a a project in an organization and we want to distill that into a skill, but not just like a markdown file.

s102
00:10:27.440 --> 00:10:32.399
We want this to be grounded in actual data that we've observed.

s103
00:10:32.399 --> 00:10:35.519
We want this to be like deterministic, right?

s104
00:10:35.519 --> 00:10:40.320
We want uh well understood procedural steps in our graph.

s105
00:10:40.320 --> 00:10:42.720
And we want to be able to govern this over time, right?

s106
00:10:42.720 --> 00:10:52.000
If if the underlying data that makes up our skill changes in memory, we want to be able to understand that and and know about that.

s107
00:10:52.000 --> 00:10:55.200
Um grounding is is an important piece here, right?

s108
00:10:55.200 --> 00:10:59.360
making sure that the the data that makes up our skill is grounded in our memory system.

s109
00:10:59.360 --> 00:11:02.160
But that that's that's not enough for it to be useful, right?

s110
00:11:02.160 --> 00:11:05.120
There are other heruristics that we need to look at here.

s111
00:11:05.120 --> 00:11:18.079
So, uh things like coverage, are we making sure that the uh steps and descriptions of our um throughout our skill are grounded across the skill?

s112
00:11:18.079 --> 00:11:18.720
Coherence.

s113
00:11:18.720 --> 00:11:20.079
Coherence is interesting.

s114
00:11:20.079 --> 00:11:36.320
This is a way that we can detect uh maybe if the um information the piece the subgraph going into the skill distillation is split across multiple topics and we can suggest well you may want to create multiple skills here

s115
00:11:36.320 --> 00:11:37.680
um and so on.

s116
00:11:37.680 --> 00:11:42.640
So those are are some of the pieces that lead up to generating the skill.

s117
00:11:42.640 --> 00:11:45.680
Uh skill governance is an important piece that I mentioned.

s118
00:11:45.680 --> 00:11:54.720
Making sure that we can understand uh as that skill changes uh as the data changes are are we able to update the skill?

s119
00:11:54.720 --> 00:12:06.240
And with dynamic loading, you can think of uh having sort of like a governed skills registry that allows us to retrieve uh for any agent the the most recent up-to-date skill,

s120
00:12:06.240 --> 00:12:14.720
understand uh if that skill may have been uh stale or or or drifted, right?

s121
00:12:15.760 --> 00:12:16.079
Cool.

s122
00:12:16.079 --> 00:12:22.399
So, we've implemented this in uh the NEFJ agent memory service or we call NAMS.

s123
00:12:22.399 --> 00:12:27.839
This is um an agent memory as a service as part of our NearJ labs effort.

s124
00:12:27.839 --> 00:12:33.279
Um we also have open source tooling around that that implements these patterns.

s125
00:12:33.279 --> 00:12:38.880
And essentially the way this works is we we have a a workspace scoped context graph, right?

s126
00:12:38.880 --> 00:12:44.160
That has our three types of agent memory, short-term, long-term reasoning memory.

s127
00:12:44.160 --> 00:12:51.360
Um we have uh background workers that are capable of going through this distillation process.

s128
00:12:51.360 --> 00:12:54.079
We'll take a look at at what this looks like in a minute.

s129
00:12:54.079 --> 00:13:08.160
Um, and then having some background curation process that is still making sure to take care of that governance to surface when uh our skills become stale based on the data that we have in the memory system.

s130
00:13:08.160 --> 00:13:12.560
Um, this is the the pipeline that that we go through to generate these.

s131
00:13:12.560 --> 00:13:15.920
Um, the I won't go through this in in detail.

s132
00:13:15.920 --> 00:13:20.880
The the one thing that we're trying to point out here is that most of these steps are deterministic.

s133
00:13:20.880 --> 00:13:32.560
Um we're really only leveraging an LLM here uh for synthesizing some of the claims uh generating some of the the text that we use for some of the skill description.

s134
00:13:32.560 --> 00:13:42.079
Uh and then the other thing I want to call out here is that the first piece is really identifying the scope like where what is your starting scope for

s135
00:13:42.079 --> 00:13:43.680
distilling one of these skills.

s136
00:13:43.680 --> 00:13:45.200
Is it around a certain entity?

s137
00:13:45.200 --> 00:13:50.560
the subgraph around that is it a specific conversation uh and that sort of thing.

s138
00:13:50.560 --> 00:14:01.839
And so we can decompose uh the the way we represent the skill and make sure that each of these components is grounded again in underlying data.

s139
00:14:01.839 --> 00:14:07.680
And that's the piece that we're looking at for if that skill uh essentially becomes stale.

s140
00:14:07.680 --> 00:14:15.360
If any of that underlying data changes, is removed or uh becomes uh contradictory.

s141
00:14:16.320 --> 00:14:16.880
Um, cool.

s142
00:14:16.880 --> 00:14:20.480
And then we can also, as I mentioned before, we can also compose skills, right?

s143
00:14:20.480 --> 00:14:23.279
So, this is where that coherence piece comes in.

s144
00:14:23.279 --> 00:14:26.560
We use um graph algorithms like community detection, right?

s145
00:14:26.560 --> 00:14:37.120
So, if we're mixing uh topics from multiple communities, that can be an indication that we need to decompose our skill and we'll catch that in um the skill governance piece.

s146
00:14:37.120 --> 00:14:37.519
Cool.

s147
00:14:37.519 --> 00:14:39.199
So, I've got a few minutes left here.

s148
00:14:39.199 --> 00:14:41.440
Let's see what this looks like.

s149
00:14:41.440 --> 00:14:45.120
Um so this is NAMS the near agent memory service.

s150
00:14:45.120 --> 00:14:48.880
Um this is this is uh free currently.

s151
00:14:48.880 --> 00:14:51.760
Anyone can can sign in and and try this out.

s152
00:14:51.760 --> 00:14:55.440
Um this is what the the dashboard looks like.

s153
00:14:55.440 --> 00:15:09.360
The the basic idea here is that we have um REST API and MCP tools that we can expose to our agents that map to um ingesting, retrieving, working with long-term,

s154
00:15:09.360 --> 00:15:11.839
short-term reasoning memory.

s155
00:15:11.839 --> 00:15:16.720
Um we're going through this entity extraction and resolution process.

s156
00:15:16.720 --> 00:15:23.920
So I can uh look at some graph representation of my uh agent memory here.

s157
00:15:23.920 --> 00:15:26.800
I I I can traverse that and and so on.

s158
00:15:26.800 --> 00:15:32.160
Um we can look at entities that have been flagged.

s159
00:15:32.160 --> 00:15:36.560
Um one important piece here is this idea of an ontology.

s160
00:15:36.560 --> 00:15:39.519
So here we're using a healthcare ontology.

s161
00:15:39.519 --> 00:15:46.079
So we we're going to be working with data about you know encounters, providers, uh that sort of thing.

s162
00:15:46.079 --> 00:15:51.360
Uh and then I have ingested a bunch of conversations here.

s163
00:15:51.360 --> 00:15:57.839
So we can see the conversations that have been um ingested related to a healthcare agent, right?

s164
00:15:57.839 --> 00:16:01.759
And so now we're ready to distill a skill.

s165
00:16:01.759 --> 00:16:09.920
Um and we said the the first piece is to decide like what is the the scope of that of that skill.

s166
00:16:09.920 --> 00:16:13.040
We can do this for you know an entire workspace.

s167
00:16:13.040 --> 00:16:15.120
That's often not what the case we want.

s168
00:16:15.120 --> 00:16:22.079
We can do this around um you know a certain entity specific conversations a a class in the ontology.

s169
00:16:22.079 --> 00:16:26.880
Let's do this around um our most recent conversation.

s170
00:16:26.880 --> 00:16:39.360
And we're going to see this is going to kick off um in the the distillation queue and go through and fetch the data, go through that sevenstage pipeline and um construct the skill

s171
00:16:39.360 --> 00:16:42.800
as a graph and then package that up for us.

s172
00:16:42.800 --> 00:16:44.399
Um here's one.

s173
00:16:44.399 --> 00:16:51.440
While this is running, let's just take a look at this guy.

s174
00:16:51.440 --> 00:16:53.839
So, here's one uh that we ran previously.

s175
00:16:53.839 --> 00:16:58.320
This is uh was run on a patient intake uh conversation.

s176
00:16:58.320 --> 00:17:09.679
And you can see here that we've essentially extracted out the steps that make up the skill to run from a patient intake um all the way through uh charting for the patient.

s177
00:17:09.679 --> 00:17:21.919
But we can see each one of these steps is grounded in the actual tool calls and the entities that constructed the underlying components of the skill.

s178
00:17:21.919 --> 00:17:27.039
And we packaged this up with skill um MD file.

s179
00:17:27.039 --> 00:17:35.520
Uh if we downloaded this, we would this would be packaged up with uh other references and and and so on.

s180
00:17:35.520 --> 00:17:43.280
uh following that progressive disclosure uh standard that we use with agent skills.

s181
00:17:43.280 --> 00:17:43.679
Cool.

s182
00:17:43.679 --> 00:17:56.000
So that was a a quick look at kind of how we think about agent memory as part of this context graph uh with nearj and I'll leave up some uh resources.

s183
00:17:56.000 --> 00:17:57.679
You can c grab the slides here.

s184
00:17:57.679 --> 00:18:02.080
There's a link uh to the slides and a QR code.

s185
00:18:02.080 --> 00:18:14.240
Um, the nearf agent memory service that I mentioned um is listed here as well as lots of documentation uh and resources for some of our open source tooling.

s186
00:18:14.240 --> 00:18:14.960
So that's it.

s187
00:18:14.960 --> 00:18:21.919
I'm out of time, but we have a NearJ booth uh so I will be there as well as lots of other folks from the NearJ team.

s188
00:18:21.919 --> 00:18:22.799
So we'll see you there.

s189
00:18:22.799 --> 00:18:25.120
Thanks folks.