CrabRAG: Why Automated Assistants Need Graph Memory, Not More Tokens — Stephen Chin, Neo4j

AI Engineer · 20 min · 244 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12My name's Stephen Chin.
00:14I run the developer relations team here at Neo4j.
00:21And I'm excited to talk to you about something we've all come to love, our our crustacean friends.
00:29So, we have um um open claw mascot.
00:33We have a bunch of other crustaceans, and we're going to we're going to focus on one member of the crustacean family.
00:40I I I love crabs, so our little boy Crab D. And I think in the in the journey to to figure out how to apply agents, how to do things which are more autonomous,
00:53we're all looking for ways where we can get better results, more accurate answers, and to actually capture all of this.
01:00But, the tools kind of work against us.
01:02So, um here is our our friend Crab D. He's a he's a personal assistant.
01:08Very happy, very eager.
01:09He wants to to help us out with our lives, maybe to help us to code, to help us to, you know, manage our email, to do different things.
01:18But, he's got a problem.
01:22And our poor boy Crab D has a very bad memory.
01:27He wakes up every day, and his memory file flips, and now it's a new day, and he forgets everything from yesterday.
01:32Does this happen to you where you you wake up, and you're using open claw, and suddenly it's on a new set of memory files, and remembers nothing that you actually did the previous day?
01:44He's got a lot of tools at his disposal.
01:46I mean, we love giving our agents tools, but sometimes he doesn't pick the right tool for the job.
01:53I don't think either of these are going to help him drink his his bowl of soup.
01:56So, that's not the tool which he was looking to to reach for.
02:03And a little bit forgetful at times.
02:05So, you know, I think I don't remember everybody I meet, but I'm pretty good at faces.
02:10Like if I've if I've met you before, I recognize faces.
02:12It's like pleased to meet you.
02:15Um Krabdi is not as good at that.
02:17So, very forgetful.
02:19It's like you're re-teaching it every day to do the same sort of tasks and we want agents which are more helpful, which are able to do more for us.
02:26So, let's dig into how Krabdi actually works.
02:29So, it's basically a a memory loop, right?
02:33So, we're we're prompting, we're thinking about the response, maybe calling tools, observing what happens.
02:40But, the hard part is the memory.
02:43The hard part is what you put in context, what you're recalling from.
02:47And the way which you have memory structured in most tools is an example of um how Open Claw structures things.
02:55Is you have a solved at MD for your agent's memory.
02:58You have maybe um memory files.
03:00You have different tool files.
03:01You have daily memory files.
03:04Now, if you look at this, there's one thing which is in common with all of these, they're just markdown files.
03:10So, mark markdown files are great.
03:12That's easy for us to read.
03:14Like we can we can look through it.
03:15We can quickly figure out what's not needed and compact them.
03:19Um they're intentionally small for agents because you have a limited context window and also you need to keep the right things at the top of the context.
03:28But, if your whole memory is a bunch of markdown files, you're wasting a lot of tokens.
03:35So, um my my average agents are loading up at least 100k in tokens for each round.
03:43Um they're doing a lot of skills.
03:45They're adding a lot of things into the context constantly.
03:47It's very repetitive because they they basically load up everything in the hopes that something will be useful in the context.
03:55At small scale that works where you get the results you want with a high quality model.
04:00It doesn't work at large scale and I'm going to show a demo of large scale where we take open claw and we let it run loose on my home lab.
04:10So, um high demo risk, but a lot of fun and um a classic digital twin scenario.
04:16So, I think we'll have we'll have a lot of fun here.
04:20Um anybody use Hermes agent at all?
04:23Okay.
04:23I'm a big fan of Hermes agent.
04:25Um I think it's got a much better memory system.
04:29It kind of at the end of each task it goes and it reflects and it adds back in new skills or new things which it needs.
04:36So, um it's a really powerful system.
04:39And um you know, again, we're relying a lot on markdown files.
04:46Skills are just basically markdown files.
04:49But we can teach the agent to do a lot of things with skills and it can it can get the right skill if it gets loaded up and then good things happen.
04:56But sometimes we don't get the right skill loaded up.
05:00So, our our poor boy crab D here is not going to get that clam.
05:04He just doesn't have the open clam shell skill.
05:07Lots of shrimp, no clams.
05:10Maybe you pick the wrong skill for the job and suddenly you're you're jet skiing on the on the beach.
05:16Right?
05:16This this is not this is not going to get him very far.
05:20And sometimes you you might get that clam open, but then you don't have the skill to eat them.
05:30So, skills you need to have the right skills, the right chain of skills.
05:34Um actually we have an awesome project by one of the Neo4j folks which is just Bennet as an Arvix paper which is a graph for skills.
05:43So, that's an exciting way of like like figuring out what the right skills are.
05:47But, maybe we can do better.
05:50So, um Goose is a project that's part of the um Agentic AI Foundation.
05:55It's a new foundation which MCP is part of.
05:58Um Anthropic is backing this.
06:00We're We're also a member of this.
06:02So, it's a it's a great automation tool for a lot of enterprise workflows.
06:07You can also use it kind of like a personal assistant.
06:10It relies heavily on MCP as the layer.
06:13Over 70 MCP extensions.
06:15And what it does is it treats memory just like another MCP server.
06:23So, this is great, right?
06:24It's It's pluggable.
06:25You can call different commands on it to retrieve memories, remember memories, um forget memories.
06:32Memories are just plain files on disk.
06:34So, now you can manipulate them.
06:37So, same great idea, same fundamental problem.
06:40We're We're storing the memory.
06:42We're storing the memory of agents as markdown files on disks.
06:46And again, you end up with what if What if you pick the wrong tool for the job, the wrong paddle?
06:55Now, in this case, if you pick the wrong paddle, you're genius cuz you've invented the most the fastest rising sport in the US, which is pickleball.
07:04Um actually, the origin of pickleball was was um a family wanted to create a new game and they just took what they had around the house, a badminton court,
07:13and um made up the rules along the way.
07:16So, creation can be good when you have the wrong tools.
07:20Maybe you remember everything, but it's too much.
07:22It's too much weight because you can't actually solve the problem.
07:25So, our poor friend Goose here is encumbered by too many notes, too many memories.
07:33Or, most dangerously, now you have MCP tools, you're one step away from calling the forget command and just wiping out your own memory.
07:44Okay, so we want to be able to do better than this.
07:48So, vector databases, right?
07:52So, we can store everything, we can create embeddings for it.
07:54Now, we actually have a a database, we can store it in a vector database.
07:59So, this is great.
08:00I mean, you have to pick the right vector database.
08:02Um and then now you can do similarity searches, so you can pull back information which is which is relevant.
08:09So, we're doing much better.
08:11We have a large repository of knowledge, we can pull back related information.
08:15Um Open Claw comes with PGVector out of the box.
08:18Given an embedding, you can just start using this.
08:21Um LanceDB is a great option.
08:22I'm going to use both of those in my demo.
08:26But, the challenge here is similar what what vectors give you, which is similarity in vector space, is not the same as actual relationships.
08:37And so, you get hallucinations, you get a lot of problems when you're relying solely on vector lookup as the answer.
08:44And it compounds with more complex scenarios when you're doing things like like I'm going to show you an example of a digital twin.
08:50When you're doing things which are very complex, they they just don't scale and you make silly mistakes like this is obviously is not what poor crab D wanted to munch into and it's a very expensive lunch for him.
09:06Also, it's sometimes impossible to get to the answer even though you have all the facts because those large multi-hop reasoning chains don't work on similarity searches.
09:18They're also very expensive on traditional relational databases.
09:25And often things look similar but they're not exactly the same.
09:30And this is one of the problems with the responses you get from a vector database is you suffer from getting facts which are related in some way and they're not your shell.
09:42And you don't you don't want to take the wrong shell out of the locker room.
09:45That's that's very unfortunate.
09:46So, enter graphs.
09:48Graphs are a great way of finding the relationships, finding those identities, mapping out the paths, getting that full chain.
09:57And they're built for this sort of connected data.
10:00So, now that you have first class nodes, which are the the circles, edges, those are the relationships between different objects, and then you can put properties on top of graphs to store information.
10:12You can also store embeddings in your graph and that gives you a way to both use vectors and graphs together.
10:19Um architecturally, the demo I'm going to show you is um both a vector search and a graph search.
10:25So, it uses the vector search to get the seed nodes where it starts the traversal.
10:31And then it uses a graph search pulling the the nearest neighbors and then ranking those by how related they are.
10:38And this gives you this complex multi-hop queries to solve more difficult, more domain-specific problems and to figure out where that where that reef is that we want to get to with all the
10:50the tasty um the tasty junk food across the ocean.
10:56And graphs are they're accurate.
10:58So, they give you very precise information.
11:02Explainable because you can look at the graph which got returned.
11:06And auditable because now you can actually say these are the this is the context.
11:11This is the part of the graph which resulted in that answer.
11:14So, it's very powerful and it gives you more tools as a developer where if you're not getting the right answer, you know where it's coming from.
11:23You can actually see and introspect the graph and you can change how you're doing extraction, you can reduce duplicate nodes in the graph, and then you can get to and converge very quickly on a great answer.
11:35If you're not a graph expert, guess what?
11:38Claude is.
11:39Claude can write Cypher better than I can.
11:41Claude can extract build entity extractors and it can do pretty much everything you need to do to get started with graphs today as long as you know the the basic kind of model for what you want to accomplish.
11:52And that's what I'm going to cover in the demo.
11:53So, we're going to do have Claude write each action into the graph as he works.
11:58We're going to follow up by traversing, not re-reading it.
12:02And then in a fresh session, we will get the results we want to get out.
12:06Now, what I did for this um high-stakes demo is I over the past week or two, I took my home lab as the demo environment, did a full digital twin
12:17as a graph and I have two separate environments built off the same original markdown files.
12:23One is a vector database store, that's our our A test.
12:27And the second is a graph store, that's our B test.
12:30And the graph store is built on top of um Cogney.
12:35So, I'm using Cogney, which is a startup.
12:39Um they do amazing stuff in the memory space.
12:41They've a Neo4j backend.
12:44This is my the structure.
12:45So, we have a bunch of Proxmox servers in my my home lab.
12:49It's really a couple computers around my desk.
12:52And I built a separate VLAN for the demo, so it's segmented off my real network.
12:58So, it was trained on real network for my network, but now it's it's cut off.
13:02It can only answer from memory.
13:04It can't actually look up the hosts and I can't get dynamic information.
13:08So, let's see how it does in a live demo.
13:16Okay.
13:17So, All right.
13:24Here we have our our Grab Rack cockpit.
13:27Um and I have five different questions queued up with schematics.
13:32You can see this is the same home lab schematic that you saw earlier in the slides.
13:38And um let's let's start with this one.
13:40So, WRN exposed end-of-life soft when exposed end-of-life software.
13:45So, we're going to basically we're going to try to find out if there's anything on my network which is exposed to the network, the the internet, the WAN that's running out-of-date software which put my home lab at risk, right?
13:57So, if if somebody can attack the home lab.
13:59And um you can see here that [clears throat] there there is some servers um Tinkster Lands, which is my daughter's Minecraft server.
14:07It's running Oh my god, Debian 8 Jessie.
14:10And let's see how the the two agents did in looking this up.
14:14Okay.
14:15So, we got the vector response back.
14:17Couldn't find specific details.
14:20Excluded by policy for more precise information.
14:23Yadada yada.
14:24Source is separately.
14:25Okay, that's not that's not very helpful.
14:27Now, on the graph side, it's done a bunch of Cypher queries.
14:30Here are the Cypher queries it's fired off.
14:33Um this is the graph traversal.
14:35And the the color coding on the graph traversal is these blue guys.
14:41These are the seed nodes.
14:42So, this came from a a vector look up and a ranking, but it didn't stop there.
14:46It does the one hop traversals.
14:49Those are all the gray nodes.
14:51Some of the nodes get highlighted in green and those are the ones which which won and got into context.
14:58And you can see the answer here.
15:00So, guest name Tinksterlin, exactly as expected.
15:04Um OS version out of date and it's flagging So, so it gives us very precise actionable information.
15:11And so that's the difference between same same exact data.
15:15One is a vector store, one is a graph store, and you can see the difference where the the vector store is having a lot of trouble pulling the information out the relevant information out.
15:25Okay, let's try another one just for fun.
15:27Um Let's see, expose 0.0.0.0 management ports.
15:33That's that's bad.
15:35So, um basically you don't want your management ports on the network exposed to the, you know, the world.
15:44And there's a bunch of these.
15:45So, I have a a new Matrix server I set up and also HAProxy which are exposed to the internet.
15:51That's bad.
15:52The rest of these, like my Cogny demo, my Open Claw instance, those are inside the LAN.
15:56You need to get into the LAN to access them.
15:58That's That's what you want.
16:00Okay, and let's see how the two agents did in identifying this.
16:05So, the memory search returned some information and it's telling me check services configuration expects pfSense rule.
16:14So, it told me to go do the job for it.
16:17Um okay, and then on the graph memory side found an open port exposed to WAN, HAProxy and OpenVPN, which are the are the two we expected.
16:26Now, this you can see the shape of this graph is entirely different from the previous one.
16:31And what it did is it it actually found the node for for my router, the pfSense router, and it was able to follow that directly to all of the results which related to it
16:41and then give us a like a very precise answer.
16:49All right.
16:50So, um So, now So, now we've seen our little boy Krab D with his certified Neo4j developer t-shirt is able to do a lot more, right?
17:08Now, he's able to follow that full chain, crack, eat, do the next thing.
17:14So, he's getting his he's getting his clams.
17:16He's helping me fix all the security holes in my network.
17:20Um oh, by the way, I I patched all those security holes after the demo.
17:24Um so, this was good for me, too.
17:26It found a bunch of security holes in my home lab, and then I I went and patched them later.
17:30And um now we have an agent which actually can do interesting things.
17:34Now, if you can imagine like I have a three or four node home lab at home, if if you have a big enterprise which has a huge data center, if you're doing things in financial
17:45services where you have like a huge set of companies and customer records you're trying to do, if you're doing anything at at large scale where it doesn't fit into the 1 million context window
17:56of the modern models, you really need a better memory system than just throwing things in markdown files.
18:03Our little boy crab recognizes all crew, all the crustacean friends.
18:10And he's read the book.
18:12Um so, we just finished my my co-authors and I, Michael Hunger and Jesús Barrasa, finished Graph RAG the Definitive Guide.
18:20The full book is out on on early release.
18:22It'll be published um in a couple months once they finish the editorial process, but super excited about this.
18:28It's got information not only on Graph RAG, but also on building memory, on different industry vertical use cases, on agents.
18:37So, it's kind of the the whole umbrella if you're building on top of graph solutions, how you need to build applications, the technologies you need end to end.
18:49And then finally, a great free resource which everybody in this room can take advantage of is Neo4j um's Graph Academy.
19:00So, it's free online training, um dev.neo4j.com/ga-rag or the QR code below there.
19:07And um we have courses on doing agent memory, doing context graphs, and everything you need to get know to get started and to do some of the amazing stuff which I showed you
19:19on stage today.
19:20So, thank you so much for coming to the the kickoff talk for the graph track.
19:25[applause]
19:30You're in the right place for all of the content from graph experts.
19:35Andreas Kollegger, my colleague and I crafted a great set of speakers from industry to experts.
19:40People really know about graph technology.
19:43So, hang out here, find out more, and then you can see me in the Neo4j booth.
19:47Thank you.
20:03[music]