Turning Agent Memory Into Skills That Work — Will Lyon, Neo4j

AI Engineer · 18 min · 189 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Let's go ahead and get started.
00:14I see some folks wanting to to come on in.
00:17So, feel free to to come in.
00:19Uh, this session is going to be all about actionable knowledge and context graphs.
00:26So, my name is Will.
00:27I'm a product manager at Neo Forj.
00:31Um there's been a lot of discussion in this conference so far about loops, right?
00:38The uh React loop, the Ralph loops.
00:41Sometimes I feel like we're in maybe like a an amnesic loop, right?
00:45where our our agents go through this uh reasoning phase acting.
00:50They successfully complete some task but then kind of forget uh what they've learned.
00:57And memory systems today are largely focused around uh embedding text then at retrieval time sort of finding the most relevant data to to stuff into the context, right?
01:10Um so so something like this we find you know similar chunks of data in in our our corpus we inject that into the context window and trust that the agent is able to
01:21do something useful with that right now the the challenge that we have is you know recall isn't really like actionable knowledge so one challenge we run into is like for example
01:34we don't have a canonical representation of a thing right so if we have you know three different ways that we refer to Dr. Nwin, Robert N, the cardiologist, right?
01:46Like depending on the the context of this discussion, we need to have some canonical representation of the thing, right?
01:53So really agents need knowledge that is both connected, typed and traceable, right?
02:01Not just retrievable.
02:02So retrieval is is uh just a part of the problem when we're talking about uh agent memory.
02:10So at NearJ this is how we think of agent memory.
02:14We think of it as a connected graph composed of short-term, long-term and reasoning memory.
02:22Uh we'll take a look at uh this in a bit more detail.
02:25Uh but bear with me on on this idea of context graphs for agent memory.
02:31Right?
02:32So what are a few of the the key components here?
02:35Well, one is going from unstructured data to a knowledge graph, right?
02:40Going from the this process of agent messages, right?
02:45Both user and assistant messages going through some entity extraction process where we're identifying what are the entities uh and how are they connected.
02:55um doing this in a graph where we have strong types, right?
03:01We have a a relationship that describes how these entities are related.
03:08The entity resol resolution phase is one of the most important parts of building up this this knowledge graph, right?
03:15Understanding and making sure that you have a canonical representation of the thing.
03:21Uh so when we're talking about Dr. win we know this is a provider his name the role that he has uh and so on and a shared ontology is an important piece of this right so having some
03:35description of your data model of the domain that you're working with this is one of the key pieces uh making sure that you are successful in going from that unstructured
03:45text data to uh to a knowledge graph so for folks that have that have worked with memory uh in agents in the past like this this should look a little familiar.
03:56Uh one thing though that I think is really important when we're talking about agent memory and actual knowledge is this idea of the reasoning graph right so um if we saw in our
04:10our uh sort of three components of agent memory here reasoning memory is a first class citizen right we've talked about short-term and long-term these are the messages the entities extracted but what about the actual actions
04:24that the agent takes what about the the reasoning um we want to make sure that we're capturing that because that is an important piece as well.
04:32We want to make sure that we're storing thinking the facts not just uh not just the data right so how an agent decided and this is typically represented as decision traces
04:45right so every decision uh that our agent makes is linked to uh the reasoning grounded in evidence right we have policies um that that we're modeling explicitly and we understand
04:58for that execution plan the tools the agent called the results of those tool calls.
05:04Uh we also want to capture things like how many tokens did this burn, the the timing that this took and so on.
05:10This is all part of uh capturing the reasoning memory.
05:16And this is important like not just so that a single agent can get better the next time you do it to do the exact same thing, but rather think about systems where you have
05:26hundreds or or thousands of agents that have some shared grouping of of tools that they have access to, right?
05:33So we're able to persist these agent runs, these these decision traces in the graph and then share that with other agents in our organization, right?
05:43so that we have one shared context graph uh that enables our agents to essentially learn from uh each other in this shared memory piece.
05:53Okay.
05:54So that that that's the memory piece.
05:56Like this this is um uh a system that we've built at at Near Forj, a pattern that that lots of folks are are following for working uh with agent memory.
06:05But what about this idea of of actionable knowledge, right?
06:08How do we enable our agents to to take action?
06:12Uh memory.
06:13One way to think of memory is is this is a good representation of what happened, right?
06:17We know um the people we were talking about, how they're connected.
06:23we have the these decision traces.
06:25Um the next step is typically creating skills, right?
06:31How many people are are using skills with their agents today?
06:34Mo most folks.
06:35Cool.
06:35How many people have have written skills yourself?
06:39Cool.
06:39Mo most folks.
06:40How many folks have had an agent write the skill for you?
06:43Right.
06:43Cool.
06:44Yeah.
06:44So that that's what we're we're talking about here is going from this memory graph, this context graph to how can we use that to create like grounded uh executable
06:59skills.
06:59So if you're not familiar with uh with a skill, this is um an open standard anthropic um put this out uh agentskills.io IO I think is where the um open standard is hosted.
07:13And the basic idea here is that we have some uh some some metadata, right?
07:18Some description of like what this skill is about.
07:21Uh and then this progressive disclosure, right?
07:24So I I have lots of more detailed information.
07:26These are often in like markdown files, references that we can um progressively choose.
07:32If if we're going down this path of one piece of the skill, we can uh retrieve and load that data uh into context.
07:40Right?
07:40So this is this then gives us some like sharable unit uh that we can then uh take the skill, package it up uh and reuse that across different agents.
07:52But skills have a a similar challenge to some of the issues that we saw uh previously when when we were talking about uh memory, right?
08:02is that we're we're typically working with pros.
08:06Um, and so we're still often limited in in this uh challenge of understanding is this the canonical thing that we're talking about, making sure that we have like debugable
08:17steps and and output.
08:19Um and so to address some of these challenges that uh we're seeing with skills uh some folks on the NeoRaj research team um have done some interesting research and published some work on APE
08:33uh graph representation for learning and governing agent skills.
08:37Um this is a screenshot from the the paper.
08:40Zach is uh is here somewhere.
08:43If he's not here, he's he's at um the NearJ booth today.
08:46So definitely uh chat with Zach if you're interested in this.
08:49But you can think of this uh APE is essentially an extension of the agent skills um protocol with some additional metadata that now treats your skills as typed execution
09:01graphs.
09:02Right?
09:02So we have uh steps modeled as nodes.
09:06We have uh very um very strict schema that governs how we describe that how these uh steps are actionable.
09:18Right?
09:18So think of think of this as a way of of representing um a skill as a graph broken up into steps that are executable with a schema governed description.
09:31Um that's the the ape protocol and this is from the uh the benchmark that was used in the the paper found that yeah like this actually matters when when they applied this to
09:44um skills bench saw like a significant increase in uh task successfully completed when looking at applying uh the a protocol to human uh curated uh skills.
10:00So that looks interesting.
10:02How can we leverage some of those uh some of those ideas, some of that research for skill distillation?
10:09Uh so essentially what what we want to do is take this this memory graph, right?
10:15This context graph that's maybe scoped to a a workspace or scoped to a a project in an organization and we want to distill that into a skill, but not just like a markdown file.
10:27We want this to be grounded in actual data that we've observed.
10:32We want this to be like deterministic, right?
10:35We want uh well understood procedural steps in our graph.
10:40And we want to be able to govern this over time, right?
10:42If if the underlying data that makes up our skill changes in memory, we want to be able to understand that and and know about that.
10:52Um grounding is is an important piece here, right?
10:55making sure that the the data that makes up our skill is grounded in our memory system.
10:59But that that's that's not enough for it to be useful, right?
11:02There are other heruristics that we need to look at here.
11:05So, uh things like coverage, are we making sure that the uh steps and descriptions of our um throughout our skill are grounded across the skill?
11:18Coherence.
11:18Coherence is interesting.
11:20This is a way that we can detect uh maybe if the um information the piece the subgraph going into the skill distillation is split across multiple topics and we can suggest well you may want to create multiple skills here
11:36um and so on.
11:37So those are are some of the pieces that lead up to generating the skill.
11:42Uh skill governance is an important piece that I mentioned.
11:45Making sure that we can understand uh as that skill changes uh as the data changes are are we able to update the skill?
11:54And with dynamic loading, you can think of uh having sort of like a governed skills registry that allows us to retrieve uh for any agent the the most recent up-to-date skill,
12:06understand uh if that skill may have been uh stale or or or drifted, right?
12:15Cool.
12:16So, we've implemented this in uh the NEFJ agent memory service or we call NAMS.
12:22This is um an agent memory as a service as part of our NearJ labs effort.
12:27Um we also have open source tooling around that that implements these patterns.
12:33And essentially the way this works is we we have a a workspace scoped context graph, right?
12:38That has our three types of agent memory, short-term, long-term reasoning memory.
12:44Um we have uh background workers that are capable of going through this distillation process.
12:51We'll take a look at at what this looks like in a minute.
12:54Um, and then having some background curation process that is still making sure to take care of that governance to surface when uh our skills become stale based on the data that we have in the memory system.
13:08Um, this is the the pipeline that that we go through to generate these.
13:12Um, the I won't go through this in in detail.
13:15The the one thing that we're trying to point out here is that most of these steps are deterministic.
13:20Um we're really only leveraging an LLM here uh for synthesizing some of the claims uh generating some of the the text that we use for some of the skill description.
13:32Uh and then the other thing I want to call out here is that the first piece is really identifying the scope like where what is your starting scope for
13:42distilling one of these skills.
13:43Is it around a certain entity?
13:45the subgraph around that is it a specific conversation uh and that sort of thing.
13:50And so we can decompose uh the the way we represent the skill and make sure that each of these components is grounded again in underlying data.
14:01And that's the piece that we're looking at for if that skill uh essentially becomes stale.
14:07If any of that underlying data changes, is removed or uh becomes uh contradictory.
14:16Um, cool.
14:16And then we can also, as I mentioned before, we can also compose skills, right?
14:20So, this is where that coherence piece comes in.
14:23We use um graph algorithms like community detection, right?
14:26So, if we're mixing uh topics from multiple communities, that can be an indication that we need to decompose our skill and we'll catch that in um the skill governance piece.
14:37Cool.
14:37So, I've got a few minutes left here.
14:39Let's see what this looks like.
14:41Um so this is NAMS the near agent memory service.
14:45Um this is this is uh free currently.
14:48Anyone can can sign in and and try this out.
14:51Um this is what the the dashboard looks like.
14:55The the basic idea here is that we have um REST API and MCP tools that we can expose to our agents that map to um ingesting, retrieving, working with long-term,
15:09short-term reasoning memory.
15:11Um we're going through this entity extraction and resolution process.
15:16So I can uh look at some graph representation of my uh agent memory here.
15:23I I I can traverse that and and so on.
15:26Um we can look at entities that have been flagged.
15:32Um one important piece here is this idea of an ontology.
15:36So here we're using a healthcare ontology.
15:39So we we're going to be working with data about you know encounters, providers, uh that sort of thing.
15:46Uh and then I have ingested a bunch of conversations here.
15:51So we can see the conversations that have been um ingested related to a healthcare agent, right?
15:57And so now we're ready to distill a skill.
16:01Um and we said the the first piece is to decide like what is the the scope of that of that skill.
16:09We can do this for you know an entire workspace.
16:13That's often not what the case we want.
16:15We can do this around um you know a certain entity specific conversations a a class in the ontology.
16:22Let's do this around um our most recent conversation.
16:26And we're going to see this is going to kick off um in the the distillation queue and go through and fetch the data, go through that sevenstage pipeline and um construct the skill
16:39as a graph and then package that up for us.
16:42Um here's one.
16:44While this is running, let's just take a look at this guy.
16:51So, here's one uh that we ran previously.
16:53This is uh was run on a patient intake uh conversation.
16:58And you can see here that we've essentially extracted out the steps that make up the skill to run from a patient intake um all the way through uh charting for the patient.
17:09But we can see each one of these steps is grounded in the actual tool calls and the entities that constructed the underlying components of the skill.
17:21And we packaged this up with skill um MD file.
17:27Uh if we downloaded this, we would this would be packaged up with uh other references and and and so on.
17:35uh following that progressive disclosure uh standard that we use with agent skills.
17:43Cool.
17:43So that was a a quick look at kind of how we think about agent memory as part of this context graph uh with nearj and I'll leave up some uh resources.
17:56You can c grab the slides here.
17:57There's a link uh to the slides and a QR code.
18:02Um, the nearf agent memory service that I mentioned um is listed here as well as lots of documentation uh and resources for some of our open source tooling.
18:14So that's it.
18:14I'm out of time, but we have a NearJ booth uh so I will be there as well as lots of other folks from the NearJ team.
18:21So we'll see you there.
18:22Thanks folks.