Make Your AI Agents Predictable With Graph Engineering - Zachary Blumenfeld, Neo4j

AI Engineer · 20 min · 198 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Uh we're going to talk about graph engineering today.
00:14How many people have heard about that term graph engineering?
00:18Okay, so so fair amount of the room.
00:19And then just so I understand, how many people here are familiar with Neo4j and have used Neo4j?
00:26Okay, so we got a few people.
00:27So that's good for uh just letting me know who we're talking to today.
00:36Um so just some history on some of the verbage that's being used.
00:41Uh there was a tweet that went out by Peter Steinberger, I think earlier this year in the summer, uh that talked about graph engineering and it was almost like is this sort of a new discipline that's happening as this evolution
00:54from prompt engineering and context engineering.
00:57Uh and there's a lot of hype around this.
01:00Um and the name for this is new, but the mechanisms are not really new.
01:04They we've always known in computer science generally about state machines and workflow graphs and knowledge graphs and all these different things.
01:12What's changing is really how we think about them in reference to agents.
01:18Um and what I'm going to talk to you about today is just how we think about graph engineering.
01:23I'm going to offer you a little bit of a new perspective on this.
01:26So if we actually try to define what graph engineering is, I think a good definition is it's just the practice of designing agent work as an explicit graph.
01:36So where nodes do work and edges effectively route state between them.
01:42And I tend to think of it in terms of three different types of graphs.
01:47We have our general execution graphs, which you're probably very familiar with.
01:52So these are the workflow graphs with routers and agents as actual nodes that happen within side of an agent harness.
02:00But, there's other ways that we can think about graph engineering, too, that overlap with this.
02:05So, that includes context graphs.
02:07These are decision traces, instances of what's happened, and how it connects to other data inside of your domain.
02:15And then there's also something that I'd call more procedural graphs.
02:19So, this is almost like having a skill or procedural knowledge as a graph, where the steps might not be actual agents, but they might be steps inside of uh uh a way of doing something, basically a procedure.
02:32So, different calls, maybe a step for a decision model, steps for different agents, whether to escalate, validation steps, and these sorts of things.
02:42And I'm really going to focus on these last two, because I think we don't talk about them enough, and I think by understanding these, you can actually get a little bit at what the first graph is trying to do as well, but in a very portable and efficient way.
02:57So, graph engineering lets agents really take on bigger, more complex work, and that's why we care about it.
03:04And really, if you were to split it up, I think in terms of the benefits that a graph offers agents in this uh context, it would be predictability so being able to constrain a path that an agent takes,
03:16making sure that the the answer is basically accurate from a task that it's doing um and having an explicit graph, the ability to govern, control, and guide what the agent does, and audit it, so you can understand what it's doing and make it better next time.
03:32And then the third one, which is going to be interesting that we'll talk about, is how do we actually make graph engineering portable?
03:38And instead of thinking about a graph as something that happens in the runtime, it could be something that you can store and actually move between different runtimes and have inside of a knowledge layer,
03:49which gives you more interoperability and allows you to have a very simple stack.
03:54Um and in the runtime, when we think about the graph, it's really like a piece of orchestration.
03:58So, we're probably all of us or a large portion of us are probably familiar with LangGraph, ADK, um and all these other agent frameworks.
04:08Um even to a certain extent Anthropic's multi-agent research and and some other stuff fits inside of graph engineering.
04:14Um and what I'm going to talk about today is actually what happens when we take that sort of graph and we put it next to our data as context graphs and some other things that I was talking about before.
04:26Um and really when you do this, as I was saying, you add this portability layer, you add the ability to take all of this knowledge and make it work
04:36basically with any type of runtime.
04:39So, that's what I'm going to go over today.
04:41Um and this is what we're doing at Neo4j is we're defining this knowledge layer and this knowledge layer sits between systems of record that you would ordinarily retrieve data from.
04:52So, Data Bricks, BigQuery, Snowflake, these types of places.
04:55It could be documentation, it could be Obsidian, these these markdown files.
05:00Um and then can we create standard protocols and artifacts that live inside of that layer that then you can take a graph engineering process and make it available to any runtime or any agent.
05:14Um and the way our product works, just so you know at Neo4j, uh is we have our core database and then we have a ton of other AI stuff on top of it.
05:24Uh so, the database and the analytics layer, um I'm going to be showing some of this throughout my talk, uh but what I'm really going to be focusing on is this top layer where we talk about sort of this agentic brain or these other memory tools
05:38that wrap around the database.
05:40So, just know that as I go through, I'm going to be showing you data being loaded into the database.
05:44That's going to constitute our knowledge layer.
05:46It's going to come from multiple different sources.
05:49Um but a lot of the sort of extensions of that tooling and some of the experimental things that I'm going to show you exist as sort of this graph AI that hooks into that and loads data in and out of that thing.
06:00[snorts]
06:01Um, so, what we're doing at Neo4j, and kind of how we're thinking about this: we start with this concept of a context graph, which has different types of memory
06:12from short-term memory which is going to be like your conversation history your long-term memory, being entities that are extracted from that or from your data, and then reasoning or decision trees inside of that.
06:25And one of the interesting things that we've been looking at from a research perspective, and this really started out not so much at that graph, but actually at individual skill level.
06:35There was a paper that we wrote earlier this year that we published in a workshop, and we were really trying to ask the question, can we take procedural knowledge,
06:45whether it be from past decisions or even something like a skill like we have here, and compile it down to a graph?
06:53And what we came up with was a process of taking written material, often a skill MD file, and the scripts and the references and assets, and moving that into an actual graph structure, which is what you see on that other side.
07:07And the result of doing this and giving the agent the access to that graph structure, which has different nodes again representing inside of the skill decisions, different routing points, different scripts that have to be executed,
07:22taking pros and compiling them down to to determine the six scripts.
07:26When we do that and we force the agent to sort of run that graph, what we see is increases in uh task reward and performance.
07:34So, this was uh from a while ago.
07:37I I'm still [clears throat] working on updating it to the latest models, um but we were able to use a smaller model.
07:43In this case, I think it was Sonnet, um, and now we're we're on to Haiku, as a matter of fact.
07:49But, uh, we were seeing, even for really challenging tasks over diverse fields in physics and computer science and, um, various security, uh, physical training, a whole a whole ton of things,
08:02um, is this overall increase in task reward, um, performance.
08:08Then we were able to switch over to this.
08:10And our PMs, uh, Will Lein, who works in our Labs team, actually took this idea and said, "Well, if we can do that with skills, maybe we can do it, too, with these memory graphs."
08:21So, you can actually take that context graph that I was showing you earlier and distill a skill from that and have it still be connected in the same graph
08:32over to all of your past decision traces and data.
08:35And so, then you get those performance benefits, but you also have all this provenance that goes back from every step.
08:41It's versioned, it's tracked, and you have governance over time where you can assess drift, you can understand where certain tool calls are coming from and their history.
08:50A human can go through actually inside of NAMS, Neo4j Agent Memory Service, which is our reference architecture for this.
08:56There's a QR code there, and there will be another one at the end, um, and be able to understand, like, exactly how this works, edit it, change it, do all sorts of cool stuff with it.
09:08Um, so again, what we're talking about now are these artifacts that you can move around.
09:12I mean, this thing makes a skill, right?
09:14And so, any agent harness that can take that skill can now be running these graphs and doing all of this logic.
09:21Um, and we have various customers that are using us for graph engineering types of tasks.
09:29I'll go quickly because we're we're starting to run out of time.
09:33Um, and even though they're not may not be using exactly the Labs project that they that we have, uh, we have consultancies which are working with other asset managers,
09:42uh, for tough problems in compliance.
09:45Uh, so being able to write audit reports, for example, where you have data that might come from multiple sources.
09:51How do you automate that, but do it in a way that's governed and trustworthy?
09:54So, you have compliance documentation.
09:57You might have entities with different ticker symbols and names from different databases, and those all need to be resolved.
10:03And so, giving your agent these graphs that express the logic that you need to do on besides the data basically allows you to automate those things in a trustworthy way.
10:13Um and we have other people uh such as Uber that do this with configuration graphs.
10:18Uh, so basically, if you have, um, you can imagine if you have drivers across the country, uh, compliance and regulations for different areas geographically vary greatly, um, and they come from this sprawl of microservices
10:31that you need to pull data from to figure out like, "Okay, well, what is it for each area, for each component of a car, for example?"
10:38Um and so being able to express that logic and have it sit aside the data um also helps them operationalize some of these things effectively.
10:47So, I'm going to jump to a demo now.
10:50It I'm going to warn you in advance: it's going to be quite a lot, and I'm going to jump through it kind of quickly, um, but it's very interesting.
10:56And what I'm going to show you is how we can use that AIP library that I was talking about before, uh, to compile a skill and actually get a task that was not governed very well and took
11:08very long to run, basically cut its cost by like 100% and then run it a lot, lot faster.
11:14Um and of course, it's uh it's a demo, it's not a formal research proposal, but you can go there and look at the code.
11:19It's all It's all very transparent.
11:22Uh so, this is going to be an example for a loan um a loan approval process.
11:29So, in a side of a loan approval process, there's lots of policies that have to be followed, obviously.
11:35I'm not a financial expert, but many of you in the room are.
11:38Um and so there's probably going to be a policy doc.
11:41There's going to be a systems of steps that have to be followed exactly, and then you need your agent to be able to explain those steps very clearly, articulate them, and really get the answer right as if it was a human reviewing.
11:53Um and the first thing you can do, like I was saying before, and I don't have time to probably run this all live, and the internet's going to be slow.
12:00So, luckily I I have this preloaded.
12:03Um and basically, the question that was asked was should we approve this new loan for this Rosa's Bakery business?
12:09And in this case, you know, we have a business plan that's in a PDF that was uploaded.
12:16Um and that is, let's see, um the business plan is the orange data.
12:22So, if I was to scroll out here, you can see it here in the graph.
12:26That connects also to data that we pulled from Data Bricks, from past transactions, lending policy data, and past decisions that were actually made.
12:36And with that, the agent can crawl over all that information and basically realize that this person either had a loan before or there was a mistake before, something happened before,
12:47and come up with an audit trail for that and basically post a solution uh very, very quickly.
12:53[snorts]
12:53Um and this is a great start.
12:55Just in terms of like resolving data and being able to follow policies and understand how things connect, uh this is great.
13:02One of the issues with this is, and I can show you a graph kind of of what this looks like.
13:07This is an example of me running more of these policies.
13:11Um and if I zoom in, I have a conversation here.
13:16And then I have my my reasoning trace that exists inside of this graph.
13:19And all of these steps are time-stamped and they have tool calls.
13:23Uh what can happen is if I uh look at this happening twice.
13:30I think this is the right query.
13:32Um this is this is two decision chains.
13:35It's going to be a little bit hard to see.
13:36Let me zoom in.
13:38Um they they're of different lengths, you'll notice.
13:41So, one is longer than the other.
13:43Um in this case, it was cuz it found a precedent in one.
13:47Um basically, something else that had happened before.
13:50But the the problem with this is that if you run these over and over again, there's nothing actually like enforcing that the agent follows an exact path.
13:58You might have a skill or something that will say like do these things.
14:02Uh but then there's other problems where it's like there's these decision steps where it's approve or deny is a simple one, or approve it or send to a human to review.
14:10And those decision steps as well come with confidence scores.
14:14Uh or you say like if you're confident about it, if you're not confident about something, but those decisions aren't calibrated.
14:21So, it would be great to like even take some of this and send it to something like a type safe, you know, Jev type of model as well to be able to do some of those things.
14:30So, what you can do is you can scrape all of this and you can compile it into a graph shape skill, which is what we did here.
14:37And that graph shape skill, if I go here, you're not going to be able to see it.
14:41But that's okay that you don't see a lot of the detail here.
14:45The important part is that we sort of compiled all the information down from those successful runs that we did, even though they were a little bit inconsistent, and we put them into this graph YAML shape
14:56inside of this skill.
14:58And this skill, even though you probably have trouble seeing, has scripts, it has assets here, it has sometimes other reference resources.
15:07It doesn't for this one.
15:09It also has source documentation around the policy that you're supposed to do for the loan.
15:15So, there's So, there's basically a ton of stuff in here for traceability, but then also telling the agent how to run the skill.
15:21But the great thing is once the skill is in this YAML format, we can port it into all this different tooling, including inside of a graph, cuz it's all sort of this deterministic
15:30format.
15:32And in a graph, it looks kind of like this thing here.
15:36So, let me Actually, let me just run this run this sucker again.
15:40Um This is kind of what it looks like.
15:44So, you have all of these steps out here, and then basically what I just did is I wrapped the small cli around this, so the agent can say, "Hey, run the skill."
15:54And instead of the agent deciding all the routing steps, it's just done programmatically.
15:59We took a lot of that policy down, and we put a lot of it into these execution scripts.
16:05Basically, what's happening here, if if I was just to sum it up, there's an execution that basically looks at, "Hey, do do we meet certain hard rules around policies?"
16:16After that, it's routed if it's not auto approved or denied based on that, it's routed to a decision model like Javelin type safe.
16:25Um, and I can even probably show you a little bit of a clearer view here.
16:31Um, and it's given that that model is given like a risk tier choice.
16:36Um, let me make that a little bigger.
16:38Hold on.
16:41Um, whether or not there's there's some exposure material and then like an outcome of whether or not you're approving or denying.
16:49And then the rest of this um, will basically follow down into writing a report, whether or not you need to escalate to a human review as this router.
16:57And then if not, basically the agent is going to write a report and there's a circulation here to sort of check and lint that report.
17:04Um, so once you do this and you sort of get the model going where okay, well now we have this cli that we can just run this thing in.
17:12Um, well now I can go in here and then maybe I can probably don't have time to run this whole thing again, but I will give it a shot.
17:22Um, I'll see if it goes and picks this up.
17:26It takes about 2 minutes to run.
17:28Uh, but the idea I guess while this is running and it should pick up the AIP scale which which it is here.
17:35Um, once you have it in this format, not only can you have the agent run this very quickly.
17:42Uh, you can track the runs.
17:43So for example, um, if I was to look at a single run, I can run this again.
17:51I can basically persist the runs inside of the graph.
17:55Um, so this is a run of that scale with like individual steps that it ran.
18:00Um, and then uh, I can just persist these over time.
18:03I can examine how well they work.
18:05So from a governance perspective, um, I have a little inspector here that I built really quickly.
18:10You can look at the status of all these.
18:12You can look at how a specific one ran, what was approved, what was denied inside of the process, what branches it took.
18:20Um, and then once you have a graph, you can even do these cool observability things where you can ask, well, like what branches were taken and which branches weren't taken.
18:29Um, which is very interesting from a governance perspective it failed ever [snorts].
18:33Um, all these cool different things, and all of that is stored besides your data.
18:38Um, and so in none of this, we're not like creating a loop, right, inside of our agent.
18:43All we're doing here is basically giving it a cli to run this like programmatic skill that we compiled from documentation or in this case from our memory graph.
18:53Um, and this might take a while to run just because of the the internet.
18:59But what I will show you is I we ran this a few times.
19:04Um, and what we saw is just running I was using Fable so my organization already yells at me for this enough so I'm spending a bunch of money.
19:12Um, but like just using Fable alone on the graph and it was correct you know, pretty much every time but it took 2.7 minutes.
19:20When we had the procedure it almost cut in half just using AIP and then because we had the procedure we can use a simpler model like Haiku and then we can cut the model time down even more and then of course,
19:31you know, the cost just goes way down.
19:33So, cuz you're taking all of that load, right, of having to figure out all the routing, all the stuff you're doing, you're putting into these execution scripts, these Java models and everything,
19:42um which makes everything just much, much more efficient.
19:46Um, and this is going to take too long to run.
19:49Uh, it's mostly internet.
19:52You can see here about half a minute or 2 minutes wall clock on average down from like 7 and 1/2.
19:58All right, so anyway, um, I'll end it here.
20:01So I want to give you some resources to check out if you're interested in this type of stuff.
20:06Aura is our cloud service offering where you can get the database that I was showing you before.
20:11You get $100 of free credits.
20:13We have nodes coming up which are is speakers event.
20:16There's a startup program and then all the code that I was referencing in the other projects that I was using to hook up to your agent memory, all that sort of stuff is down there.
20:25So, I'll leave this up for a second until someone kicks me off, and I guess that's it for me for now.