Everyone Has the Same AI Models. Here's What Sets Agents Apart - Richmond Alake, Oracle

AI Engineer · 25 min · 319 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12I'm going to go through a bunch of keywords and you guys get interactive.
00:16I'm going to ask you guys to put your hands up as I go through a bunch of keywords.
00:20So, hands up if you heard about agent memory.
00:24That makes me unbelievably happy because I did this I did a talk on agent memory about a year ago and no one in the room knew what I was talking about.
00:34So, it was also at AI Engineers San Francisco and once upon a time, maybe around 3 years ago, people thought I was crazy when I was talking about we need to structure memory, we need to model memory in agents.
00:46And now everyone here knows about agent memory and hopefully you're implementing it within your agent.
00:52So, I'm just going to quickly go over agent memory, just very quickly.
00:57So, agent memory is basically how an agent or the memory mechanism an agent uses to retain, reuse, recall, and refine information.
01:04I call it the four Rs, it's easy to remember.
01:07Um but there's also taxonomy of agent memory.
01:10Short-term, long-term is easy.
01:12Then you go into the cognitive function of agent memory, semantic, episodic, procedural, working memory.
01:19Then we go into an implementation that we're going to see a lot today.
01:23And I've done this so many times online, but let's take pictures and you guys are going to get the slides as well.
01:29So, this is all the taxonomy of memory, the what I'm trying to educate people on and I've been educating people on for the past 3 years.
01:36But in this talk, we're going to take it further.
01:38So, another keyword coming up.
01:41Hands up who's heard of the control plane?
01:44Nice, less so.
01:45We're getting We're getting less hands and I have a theory.
01:48As we go through the keywords, you're going to see the number of hands in the presentation reduce.
01:53Eventually, if things go correctly, there will be no hands or maybe one or so.
01:58So, the control plane, we can define it.
02:00Most people know what it is.
02:02It's essentially a centralized management system for how your agent is governed, with the tools they have access to, what they can do, and what they cannot do.
02:10So, there's a lot of different software companies positioning themselves as a control plane.
02:15And yes, we do need these sort of systems.
02:18But in this talk, I'm going to take you a little bit further.
02:22So, this is just an idea of what the control plane is.
02:24So, identity for your agents, the policy and guardrails, the orchestration to permission, and all the auditing.
02:32And these are the components of typical control planes.
02:36All right, another keyword is coming.
02:38You guys ready?
02:42All right, let's go.
02:43Software factory.
02:46Okay, less hands.
02:47We're getting there.
02:48It's It's We're almost in line with what I have on the slides.
02:52So, software factory is actually an interesting term.
02:56So, I did my internship at General Electric like eons ago.
03:00And software factory is a term that someone called Bob Bemer at GE at the time in 1968, he coined the term software factory in one of his paper where he was talking about how we can create
03:12um programs in measurable and repeatable way.
03:15And essentially, he coined the term software factory.
03:18So, it's not a new term within the agentic era.
03:21It's a term that we've used in computer science a very, very long time ago.
03:26I'll wait for people to take pictures.
03:29So, software factory, you can just imagine it as you want to create a system that is able to churn out software reliably, right?
03:37You give it a bunch of um processes and systematic workflow it's meant to follow whenever it's building a software, and you have the agents execute this, usually multiple agents in parallel.
03:47Again, different companies position themselves in this space.
03:51All right, another keyword.
03:54Intelligence plane.
03:57Any hands up?
03:58No, I failed.
03:59I got three hands up.
04:01Um yeah, the intelligence plane is what we're going to be talking about today.
04:04And I'm going to make a promise that I did in my previous talk last year when I was talking about agent memory when people thought I was crazy.
04:12And this promise is the same I'm going to I'm going to make, which is the information I'm going to give you today, everything I'm going to say, is going to put you 6 months ahead
04:22of where most people are today.
04:24So, I made that promise last year talking about agent memory.
04:27I'm making that promise this year talking about the intelligence plane.
04:30So, when I come back next year and I talk about the intelligence plane, everyone's going to put their hands up.
04:36Don't don't quote me.
04:37How about this one?
04:38AF is Midway.
04:39Who's heard about this?
04:42No?
04:42Okay, let's uh do a bit of history lesson.
04:45So, in World War II um in World War II uh Pearl Harbor was getting um attacked by the Japanese Imperial Navy, right?
04:56And there was a bunch of groups that were meant to be intercepting communication from the Japanese army.
05:03And there were three groups.
05:05So, we're only going to talk about two groups here.
05:07One group was based in Washington and one group was based in Pearl Harbor.
05:12And there was a code AF.
05:15Now, AF stood for something and it was very critical to the outcome of the actual um Pacific War itself.
05:23But to cut the long story short, these two groups had a bunch of fragmented information from communication from the Japanese Navy force that they were they were that they were meant to codebreak.
05:37So, they had a lot of fragmented information and basically the one group, which is the group based in Pearl Harbor, a bunch of folks that were basically in a basement
05:48collecting loads of information and intercepted messages on paper and storing it for over months, were able to decode this and they found out that AF is actually short for Midway.
06:00Right?
06:01AF is Midway.
06:03Right?
06:03So, the Japanese were using AF as code word for Midway.
06:06Midway was a little island located in between Asia and North America.
06:12So, this was You can look this up, control of the Pacific.
06:17But, a key thing I wanted to tell you is the difference between control and intelligence.
06:22Because that small group that was in Pearl Harbor, located in a basement, and they were deciphering all the um Japanese communication, and using a lot of papers to record everything for months, which is essentially memory,
06:38what was able to actually decode and position um the American Navy at an advantage.
06:45So, whereas Washington, which was more centralized, had the same information, right?
06:51There was no difference in information.
06:52They had the same information, probably more resources, and they had control, and they had organization, came up with an entirely different um uh interpretation of the data.
07:06And the one that was chosen was the folks of our Station Hypo, um the code breakers there.
07:12And the reason they were able to also do our hypothesis on the fact that AF does mean Mid- Midway.
07:18So, what I'm trying to say is everyone in this room has the same models, probably the same access to the same information, but that doesn't equate to intelligence, and it doesn't equate to you having that unique differentiator.
07:31So, today, I'm going to talk about why we really need to start talking about the intelligence plane.
07:37So, I'm going to go through the agent stack.
07:40I've spoken about the agent stack.
07:41We know what the agent stack is, and then the control plane cuts through a certain level of the agent stack.
07:47Now, what I'm proposing to you is nothing new.
07:49It's just a way of thinking about the software you're building today.
07:53So, this is the intelligence plane.
07:55It cuts through the entire stack, and creates a dimension where you as engineers can start to build agents that are not only orchestrated, but they can actually learn from data and memory as well.
08:09So, we're going to we're going to move from theory into more practical concepts.
08:16So, what is the intelligence plane?
08:19It's essentially a part of your agentic system or your AI infrastructure that allows you to understand or allows your agent to get better over time.
08:28It encapsulate what the agent should know, which is basically knowledge, what it should remember, which is basically memory, what it's allowed to see, which is just governance, data access,
08:40and how it gets better, which is continual learning.
08:43So, there's nothing new here.
08:44Some of you are already doing some of these things already, and some of you are going to start doing it after this talk.
08:51So, that is the difference between the control plane and the intelligence plane.
08:54Are we following?
08:55Right?
08:56Nodding?
08:57Okay, this is good.
08:58Let's go.
08:59Um I'm going to take you through this timeline of where most of you have been and where most of you are going to get to.
09:05First, we were all prompting the models, and we all had like prompt layers or tools where we can see all our prompts.
09:11Then we moved into tools that can allow us to see the context, context layer.
09:15And most of you are now in the control plane error, when you guys are looking for softwares or building softwares that allows you to orchestrate this agent.
09:23So, where we're going to go to is intelligence plane or where you should be heading to is intelligence plane.
09:30And just to get this out there, throughout all the slides, one thing we can understand and agree on, information does not equal intelligence.
09:39So, take that away, and now we can start getting a bit technical.
09:42So, this is the agent stack.
09:43We spoke about the agent plane.
09:46And when we start looking about let's talk about the control plane.
09:48When we start thinking about intelligence in the control plane or intelligence added onto the control plane, we start to think about when you're looking at gateway and our connectivity,
09:58you start to think about end-to-end user identity connected to the data.
10:03Right?
10:04When you start to think about orchestration, you start to think about the context assembly, you start to think about things such as memory and contest construction.
10:13Then all the way to the memory core.
10:15I'm skipping to the memory core because that's my sweet spot, database.
10:19Um we start to talk about database, but we take things further.
10:21We start to talk about ontology and semantic layers and memory as well.
10:26Then infrastructure and platform is the important bit.
10:29Which is where we start to talk about inference, fine-tuning, and continuous learning, right?
10:35When we go beyond the deployment that the control plane gives you.
10:39So, these are all the components of the control plane and the intelligence plane.
10:44There's going to be a reference architecture that you would be able to take a picture of.
10:48And this idea, I wish I came up with new ideas, but this idea I'm actually proposing to you is nothing new.
10:54Um it's been proposed several times.
10:55In fact, in if you If anyone's done networking, this might be familiar to you.
11:00Um some folks have proposed that there needs to be a knowledge plane on top of management and configuration platforms that are meant to actually see um the the routes of different circuits,
11:14which is essentially a a control plane, right?
11:16So, a control plane, a management plane, then folks have um uh proposed a knowledge plane.
11:23And this is what I'm proposing for the agentic systems we're building, an intelligence plane.
11:28We're going to have a demo.
11:29I'm looking at the time to make sure I get to the demo.
11:32And we like equations.
11:33Agent equals model plus harness.
11:35Then you take that into control, that equals governance plus security.
11:40But when you bring intelligence, you add knowledge and memory and learning, it gives you that multiplier effect on your entire on the entire way you're building agents today.
11:50And I'm going to show you what that looks like.
11:52I'm trying to think, should I go to the demo now or later?
11:55Um we'll do a demo version.
11:57Oh, I'll go back.
11:57People were taking pictures.
11:59You are going to get the slides um in a second.
12:03All right.
12:03So, everything I talk about within the intelligence plane, I've tried to capture as just high-level system components.
12:13So, all of these would be different providers today.
12:16Right?
12:16So, we have the agent runtime that your agent is going to be executing on.
12:22Then your agent runtime needs memory and knowledge, probably has a memory manager on top that sits on top of data platform, and then your agent runtime is able to put in traces of memory
12:34into your memory manager, and that could then go into a continual learning loop that allows you to then change models back into your agent runtime.
12:43Right?
12:43And you can put this all on the plane that we've been talking about.
12:47So, we have the data plane, the control plane, the intelligence plane.
12:51I'm not going to go over this cuz we already know what the control plane is.
12:54We know what the intelligence plane is, and that's the key thing that is missing.
12:58So, in the intelligence plane, we have a couple things that are actually quite important.
13:03I'm going a bit too fast.
13:05We have a couple things that are quite important I'm going to show in a demo.
13:08And that is what I'm going to be showing in a second.
13:10So, why is the industry ready for the intelligence plane?
13:13Why am I talking about the intelligence plane now?
13:15So, very quickly, we have frontier AI models.
13:18Um the scaling law allowed us to actually predict the benefits we could get from these models, and then we have the open ways are very like 6 months or 3 months behind.
13:28But we have we had something that happened recently, which was decision models.
13:32Right?
13:32And they changed the trajectory or the way that we started to think of these systems.
13:35The way I thought about the systems, right?
13:37It made me start to think systems that are very fundamental architectural level.
13:42Looking at my system and thinking, okay, what here could I replace as a decision model?
13:47So, this was the traditional way we were seeing things a few years ago, and today you can have models all through your retrieval pipeline.
13:56So, one thing I'm going to do is I'm going to go into a demo right here.
14:01This is the QR for the demo.
14:03A bit going ahead of myself.
14:07Here we go.
14:08So, in this demo, which you're going to get the code to, I'm not going to run it live cuz the internet here isn't great as you guys probably know.
14:16Um this is a simple chat interface.
14:18I have a governed analyst, I have a naive analyst.
14:21My governed analyst is built on what is an intelligence plane.
14:24It gives it the ontology layer, semantic layer, gives it everything we've seen in that reference architecture.
14:30And very quickly, just to show you the effect of having an intelligence plane.
14:36The key thing is having one, saving on tokenomics, right?
14:41Tokenomics is cost, accuracy, and latency.
14:44And we have two agents that are able to execute the same questions, but the one that is powered by an intelligence plane is a bit smarter.
14:54It gets things right 99% of the time.
14:57And it does it with less token consumption.
14:59In 32,000 token compared to the 200 thousand token with the naive agent.
15:05And then we have all of the guard controls in place and the cache as well.
15:09And we have some prompt caching mechanism as well.
15:12So, when you actually start I'm working backwards here.
15:15When you start to implement some of the things I'm going to show you to you in a second, you start to implement systems that are able to actually one, be optimized for production,
15:25and two, learn and grow with the data that you're capturing.
15:29So, over here I just want to talk about the ontology and the semantic layer.
15:33I'm kind of skipping between demos here cuz we're short on time.
15:36So, with the ontology and semantic layer, I asked the question uh to both agents around um what was a counterparty, right?
15:46And the ontology layer adds a level of accuracy that you can't get by just doing simple keyword search.
15:53So, in the demo, this is a keyword search where we're able to actually find some um entities, right?
15:59And the agent was able to answer without the ontology, but with the ontology, the answer actually improves and it gets the right answer using the ontology.
16:09Same with the semantic layer.
16:11The agent the naive agent without any semantic layer, we asked it a question and it got the we asked it what does concentration mean?
16:20And it got it wrong.
16:21The naive agent got it wrong, whereas the semantic the intelligent agent built with the semantic play with the intelligence plane got it right.
16:30And now that margin of error is just five 5%.
16:36But when you're working with the finance sector in any mission critical workload, that becomes very critical.
16:42So getting your semantic layer built out, get your ontology layer built out, it becomes very important.
16:48I'm going to you're going to get access to this demo and I'm going to go back to the slides very quickly.
16:57So there's a bunch of models that we're adding into the intelligence plane and the demo I showed you actually has all of these models and all of this architecture in place.
17:07So what should you your team be doing now?
17:09One, you treat memory as data.
17:11Right?
17:11Don't look at it as different things.
17:14Write down your entities and definition.
17:16You push entitlements into the data layer.
17:19Over Oracle, we have features that actually enable you to do this very easily and capture traces.
17:25So memory is is basically your moat.
17:28You need to own your own memory.
17:30So over Oracle, we have this reference architecture we saw before.
17:35But at Oracle, we have product features that map to it.
17:39So we give you the system components to build your own intelligence plane.
17:44So that means that you have all the tools you need to build something that actually learns and grow as you continue to execute build as you continue to build and execute your agent.
17:55I'm going to touch on a few ones.
17:57Memory, security, deep data security, private inference and ontology and semantics.
18:02So these are the key five areas I want to touch on very quickly.
18:08So agent memory.
18:09We've spoken about it, but now what we did at Oracle is we launched agent memory as a Python package.
18:14We put memory engineering into a box, which allows you to just not worry about all of the complexities of making agents remember, we just convert it into a Python package
18:24and it generates context card for you where you don't have to actually do it yourself.
18:29Then, one thing you have to actually have to start capturing is your memory traces because this become data for fine-tuning.
18:39So, right here we have a slide that just shows where memory engineering and context engineering meets and this is for you to take and go through when I give you the slides.
18:47I'm just going to skip through this very quickly.
18:50So, agent memory and the Oracle AI database is what we just covered in the demo.
18:55I skipped it.
18:55But, the next thing is deep data security.
18:57I spoke to a few folks yesterday and today and they were talking about agent identity and how they were using different solution providers for it.
19:06In agent in Oracle, one thing we did is we made security a first-class citizen of the database.
19:12Which means with a product called deep data security, you can take all of the context of your application and the data access and row access of your user and we propagate that into the data layer.
19:25Which means your agents can take on the identity of the user it's acting on behalf and access the same data that it's only allowed for is only that the user is only allowed to access.
19:35Fumbling my words there.
19:37Okay.
19:38So, all of this is all great including the demo I showed you it uses all of this infrastructure I spoke about and I showed you.
19:46And this is how deep data security works.
19:48That's right.
19:49We have the analyst, we have our identity provider.
19:51The identity provider provides all of the data access that the analyst is meant to have and we can propagate that into the agent and the agent inherits that as well.
20:01And that limits all of the data access for that particular agent.
20:06Okay.
20:06The next couple of slides I'm going to talk about private inference.
20:11Right?
20:11A lot of you work in finance companies where you want to actually have your inference on prem.
20:17So, we have the private AI service container which contains all the infrastructure you need to host your own your own embedding, your own LLM in your own LLM inference,
20:26and re-rankers.
20:28And this is all around owning your own intelligence plane.
20:31So, giving you the components.
20:33But, if you want to go to the cloud, we have Oracle Cloud where we host some of these models that you're hearing about.
20:39So, Grok models and some Open AI models as well.
20:43Either way, air-gapped um infrastructure or cloud infrastructure, you have the components within Oracle itself.
20:52All right.
20:52So, I have a second demo before I go off.
20:58So, this demo is an open-source tool that I built called Memories.
21:03And it's just to show you what how you can actually start to get into the intelligence plane domain.
21:10So, right now there are people working on agent memory, like you guys have mentioned.
21:14And within an agent memory system, there are other ways you can start to bring in models to improve the system itself.
21:21So, decision model is one of it, and this one makes ex- uses example of three different models.
21:28A decision model, an embedding model, and a re-ranker.
21:31And within my agent memory system, I typically have these models.
21:35But, with the intelligence plane or with an understanding that I need to own my own memory, and I own my own memory, I can actually start to fine-tune some of these open-source models.
21:44And that's what we're doing here.
21:46So, over here in memory routing, I'm just going to focus on memory routing for a second.
21:50And what memory routing means is we get some output from the context layer, or we get some output from our LLM, and what we're able to do is route it to the right store.
22:00So, whatever comes out, I can route it into conversation memory or workflow memory.
22:06Normally, you would do this using some code, but with decision models, this becomes very easy and very cheap.
22:13So, right here we have some fine-tuned Oh, skipped it.
22:17Let's go back.
22:19All right.
22:20Right here we have some fine-tuned models for memory routing, and we can see how it was trained.
22:26So we fine tune the equine model and we use it and we fine tune it into a decision model.
22:32The data that we use to fine tune is all here and obtained from the conversation I was have been with the agent and I was able to classify what type of memory it was.
22:44And this allows you to do memory routing very easily.
22:46So once you get any information coming in, the different types of memory you want to route it effectively and cheaply, you can do that with decision models.
22:55So memory routing, embeddings, and re-ranking are some of the lowest hanging fruits you guys can actually start to work with today.
23:04Once you understand your intelligence plane, you're capturing your memory traces, you're formatting them effectively, and you're retraining the models.
23:11And you get the benefit with improvement in quality and in some cases reduction in latency as well.
23:18Okay, so we're going to go back to the slide again.
23:20We're going back and forth a few times, which is good.
23:23Um and we're just going to wrap up.
23:27So there are different ways we're helping developers build agents in in Oracle.
23:31And one of it is private agent factory.
23:33And this is a no code platform we can just drag and drop and build your agents and have the whole control plane at least within the private agent factory.
23:43And the private agent factory also has the same agent memory system that we've we've been building across all our product surfaces.
23:51So again, the private agent factory is built on top of the Oracle AI database, so you get all of the memory core functionalities I've been talking about.
24:00I already showed the ontology and semantic layer the effect it has on the accuracy that you get within agents that have and don't have ontology and semantic layer.
24:10But again, Oracle gives you all the infrastructure and components to build all of this.
24:16So you can ask the same question with and without meaning, right?
24:21So we've already done the demo.
24:22Again, I skipped I'm kind of like going around the slides and the demos, but you saw what we had with ontology layers and without ontology layers and the accuracy improvements
24:32can be very minimal, but in mission-critical workload, they can mean getting a 100 million fine for handling fraudulent transactions or not.
24:41Okay, so we touched on continual learning a bit.
24:44What we saw in the demo was continual learning.
24:47And which is there different ways you can look at continual learning in token space and in weight space and also in in latent space as well if you're going into the prompts caching.
24:56So we've just gone over this.
24:58I'm not going to rehash this again.
25:00But one of the key thing and the key takeaway here is is developers and engineers of these systems, we need to actually own our own memory, engineer the capturing of traces,
25:11and actually build it into a system that allows for continual learning to be possible.
25:17So I'm going to end right here.
25:19Um I think I'm going over time and I'm going to give you guys the QR code to the demo I just showed.
25:28So if you just scan this, but the best thing is you can come meet me at the Oracle booth and you will get this QR code again along with other technical assets and we can have a conversation.
25:38Thank you for your time.