Everyone Has the Same AI Models. Here's What Sets Agents Apart - Richmond Alake, Oracle
https://www.youtube.com/watch?v=1_lX0m64-aI
AI Engineer · 25 min · 319 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] I'm going to go through a bunch of keywords and you guys get interactive.
[00:16] I'm going to ask you guys to put your hands up as I go through a bunch of keywords.
[00:20] So, hands up if you heard about agent memory.
[00:24] That makes me unbelievably happy because I did this I did a talk on agent memory about a year ago and no one in the room knew what I was talking about.
[00:34] So, it was also at AI Engineers San Francisco and once upon a time, maybe around 3 years ago, people thought I was crazy when I was talking about we need to structure memory, we need to model memory in agents.
[00:46] And now everyone here knows about agent memory and hopefully you're implementing it within your agent.
[00:52] So, I'm just going to quickly go over agent memory, just very quickly.
[00:57] So, agent memory is basically how an agent or the memory mechanism an agent uses to retain, reuse, recall, and refine information.
[01:04] I call it the four Rs, it's easy to remember.
[01:07] Um but there's also taxonomy of agent memory.
[01:10] Short-term, long-term is easy.
[01:12] Then you go into the cognitive function of agent memory, semantic, episodic, procedural, working memory.
[01:19] Then we go into an implementation that we're going to see a lot today.
[01:23] And I've done this so many times online, but let's take pictures and you guys are going to get the slides as well.
[01:29] So, this is all the taxonomy of memory, the what I'm trying to educate people on and I've been educating people on for the past 3 years.
[01:36] But in this talk, we're going to take it further.
[01:38] So, another keyword coming up.
[01:41] Hands up who's heard of the control plane?
[01:44] Nice, less so.
[01:45] We're getting We're getting less hands and I have a theory.
[01:48] As we go through the keywords, you're going to see the number of hands in the presentation reduce.
[01:53] Eventually, if things go correctly, there will be no hands or maybe one or so.
[01:58] So, the control plane, we can define it.
[02:00] Most people know what it is.
[02:02] It's essentially a centralized management system for how your agent is governed, with the tools they have access to, what they can do, and what they cannot do.
[02:10] So, there's a lot of different software companies positioning themselves as a control plane.
[02:15] And yes, we do need these sort of systems.
[02:18] But in this talk, I'm going to take you a little bit further.
[02:22] So, this is just an idea of what the control plane is.
[02:24] So, identity for your agents, the policy and guardrails, the orchestration to permission, and all the auditing.
[02:32] And these are the components of typical control planes.
[02:36] All right, another keyword is coming.
[02:38] You guys ready?
[02:42] All right, let's go.
[02:43] Software factory.
[02:46] Okay, less hands.
[02:47] We're getting there.
[02:48] It's It's We're almost in line with what I have on the slides.
[02:52] So, software factory is actually an interesting term.
[02:56] So, I did my internship at General Electric like eons ago.
[03:00] And software factory is a term that someone called Bob Bemer at GE at the time in 1968, he coined the term software factory in one of his paper where he was talking about how we can create
[03:12] um programs in measurable and repeatable way.
[03:15] And essentially, he coined the term software factory.
[03:18] So, it's not a new term within the agentic era.
[03:21] It's a term that we've used in computer science a very, very long time ago.
[03:26] I'll wait for people to take pictures.
[03:29] So, software factory, you can just imagine it as you want to create a system that is able to churn out software reliably, right?
[03:37] You give it a bunch of um processes and systematic workflow it's meant to follow whenever it's building a software, and you have the agents execute this, usually multiple agents in parallel.
[03:47] Again, different companies position themselves in this space.
[03:51] All right, another keyword.
[03:54] Intelligence plane.
[03:57] Any hands up?
[03:58] No, I failed.
[03:59] I got three hands up.
[04:01] Um yeah, the intelligence plane is what we're going to be talking about today.
[04:04] And I'm going to make a promise that I did in my previous talk last year when I was talking about agent memory when people thought I was crazy.
[04:12] And this promise is the same I'm going to I'm going to make, which is the information I'm going to give you today, everything I'm going to say, is going to put you 6 months ahead
[04:22] of where most people are today.
[04:24] So, I made that promise last year talking about agent memory.
[04:27] I'm making that promise this year talking about the intelligence plane.
[04:30] So, when I come back next year and I talk about the intelligence plane, everyone's going to put their hands up.
[04:36] Don't don't quote me.
[04:37] How about this one?
[04:38] AF is Midway.
[04:39] Who's heard about this?
[04:42] No?
[04:42] Okay, let's uh do a bit of history lesson.
[04:45] So, in World War II um in World War II uh Pearl Harbor was getting um attacked by the Japanese Imperial Navy, right?
[04:56] And there was a bunch of groups that were meant to be intercepting communication from the Japanese army.
[05:03] And there were three groups.
[05:05] So, we're only going to talk about two groups here.
[05:07] One group was based in Washington and one group was based in Pearl Harbor.
[05:12] And there was a code AF.
[05:15] Now, AF stood for something and it was very critical to the outcome of the actual um Pacific War itself.
[05:23] But to cut the long story short, these two groups had a bunch of fragmented information from communication from the Japanese Navy force that they were they were that they were meant to codebreak.
[05:37] So, they had a lot of fragmented information and basically the one group, which is the group based in Pearl Harbor, a bunch of folks that were basically in a basement
[05:48] collecting loads of information and intercepted messages on paper and storing it for over months, were able to decode this and they found out that AF is actually short for Midway.
[06:00] Right?
[06:01] AF is Midway.
[06:03] Right?
[06:03] So, the Japanese were using AF as code word for Midway.
[06:06] Midway was a little island located in between Asia and North America.
[06:12] So, this was You can look this up, control of the Pacific.
[06:17] But, a key thing I wanted to tell you is the difference between control and intelligence.
[06:22] Because that small group that was in Pearl Harbor, located in a basement, and they were deciphering all the um Japanese communication, and using a lot of papers to record everything for months, which is essentially memory,
[06:38] what was able to actually decode and position um the American Navy at an advantage.
[06:45] So, whereas Washington, which was more centralized, had the same information, right?
[06:51] There was no difference in information.
[06:52] They had the same information, probably more resources, and they had control, and they had organization, came up with an entirely different um uh interpretation of the data.
[07:06] And the one that was chosen was the folks of our Station Hypo, um the code breakers there.
[07:12] And the reason they were able to also do our hypothesis on the fact that AF does mean Mid- Midway.
[07:18] So, what I'm trying to say is everyone in this room has the same models, probably the same access to the same information, but that doesn't equate to intelligence, and it doesn't equate to you having that unique differentiator.
[07:31] So, today, I'm going to talk about why we really need to start talking about the intelligence plane.
[07:37] So, I'm going to go through the agent stack.
[07:40] I've spoken about the agent stack.
[07:41] We know what the agent stack is, and then the control plane cuts through a certain level of the agent stack.
[07:47] Now, what I'm proposing to you is nothing new.
[07:49] It's just a way of thinking about the software you're building today.
[07:53] So, this is the intelligence plane.
[07:55] It cuts through the entire stack, and creates a dimension where you as engineers can start to build agents that are not only orchestrated, but they can actually learn from data and memory as well.
[08:09] So, we're going to we're going to move from theory into more practical concepts.
[08:16] So, what is the intelligence plane?
[08:19] It's essentially a part of your agentic system or your AI infrastructure that allows you to understand or allows your agent to get better over time.
[08:28] It encapsulate what the agent should know, which is basically knowledge, what it should remember, which is basically memory, what it's allowed to see, which is just governance, data access,
[08:40] and how it gets better, which is continual learning.
[08:43] So, there's nothing new here.
[08:44] Some of you are already doing some of these things already, and some of you are going to start doing it after this talk.
[08:51] So, that is the difference between the control plane and the intelligence plane.
[08:54] Are we following?
[08:55] Right?
[08:56] Nodding?
[08:57] Okay, this is good.
[08:58] Let's go.
[08:59] Um I'm going to take you through this timeline of where most of you have been and where most of you are going to get to.
[09:05] First, we were all prompting the models, and we all had like prompt layers or tools where we can see all our prompts.
[09:11] Then we moved into tools that can allow us to see the context, context layer.
[09:15] And most of you are now in the control plane error, when you guys are looking for softwares or building softwares that allows you to orchestrate this agent.
[09:23] So, where we're going to go to is intelligence plane or where you should be heading to is intelligence plane.
[09:30] And just to get this out there, throughout all the slides, one thing we can understand and agree on, information does not equal intelligence.
[09:39] So, take that away, and now we can start getting a bit technical.
[09:42] So, this is the agent stack.
[09:43] We spoke about the agent plane.
[09:46] And when we start looking about let's talk about the control plane.
[09:48] When we start thinking about intelligence in the control plane or intelligence added onto the control plane, we start to think about when you're looking at gateway and our connectivity,
[09:58] you start to think about end-to-end user identity connected to the data.
[10:03] Right?
[10:04] When you start to think about orchestration, you start to think about the context assembly, you start to think about things such as memory and contest construction.
[10:13] Then all the way to the memory core.
[10:15] I'm skipping to the memory core because that's my sweet spot, database.
[10:19] Um we start to talk about database, but we take things further.
[10:21] We start to talk about ontology and semantic layers and memory as well.
[10:26] Then infrastructure and platform is the important bit.
[10:29] Which is where we start to talk about inference, fine-tuning, and continuous learning, right?
[10:35] When we go beyond the deployment that the control plane gives you.
[10:39] So, these are all the components of the control plane and the intelligence plane.
[10:44] There's going to be a reference architecture that you would be able to take a picture of.
[10:48] And this idea, I wish I came up with new ideas, but this idea I'm actually proposing to you is nothing new.
[10:54] Um it's been proposed several times.
[10:55] In fact, in if you If anyone's done networking, this might be familiar to you.
[11:00] Um some folks have proposed that there needs to be a knowledge plane on top of management and configuration platforms that are meant to actually see um the the routes of different circuits,
[11:14] which is essentially a a control plane, right?
[11:16] So, a control plane, a management plane, then folks have um uh proposed a knowledge plane.
[11:23] And this is what I'm proposing for the agentic systems we're building, an intelligence plane.
[11:28] We're going to have a demo.
[11:29] I'm looking at the time to make sure I get to the demo.
[11:32] And we like equations.
[11:33] Agent equals model plus harness.
[11:35] Then you take that into control, that equals governance plus security.
[11:40] But when you bring intelligence, you add knowledge and memory and learning, it gives you that multiplier effect on your entire on the entire way you're building agents today.
[11:50] And I'm going to show you what that looks like.
[11:52] I'm trying to think, should I go to the demo now or later?
[11:55] Um we'll do a demo version.
[11:57] Oh, I'll go back.
[11:57] People were taking pictures.
[11:59] You are going to get the slides um in a second.
[12:03] All right.
[12:03] So, everything I talk about within the intelligence plane, I've tried to capture as just high-level system components.
[12:13] So, all of these would be different providers today.
[12:16] Right?
[12:16] So, we have the agent runtime that your agent is going to be executing on.
[12:22] Then your agent runtime needs memory and knowledge, probably has a memory manager on top that sits on top of data platform, and then your agent runtime is able to put in traces of memory
[12:34] into your memory manager, and that could then go into a continual learning loop that allows you to then change models back into your agent runtime.
[12:43] Right?
[12:43] And you can put this all on the plane that we've been talking about.
[12:47] So, we have the data plane, the control plane, the intelligence plane.
[12:51] I'm not going to go over this cuz we already know what the control plane is.
[12:54] We know what the intelligence plane is, and that's the key thing that is missing.
[12:58] So, in the intelligence plane, we have a couple things that are actually quite important.
[13:03] I'm going a bit too fast.
[13:05] We have a couple things that are quite important I'm going to show in a demo.
[13:08] And that is what I'm going to be showing in a second.
[13:10] So, why is the industry ready for the intelligence plane?
[13:13] Why am I talking about the intelligence plane now?
[13:15] So, very quickly, we have frontier AI models.
[13:18] Um the scaling law allowed us to actually predict the benefits we could get from these models, and then we have the open ways are very like 6 months or 3 months behind.
[13:28] But we have we had something that happened recently, which was decision models.
[13:32] Right?
[13:32] And they changed the trajectory or the way that we started to think of these systems.
[13:35] The way I thought about the systems, right?
[13:37] It made me start to think systems that are very fundamental architectural level.
[13:42] Looking at my system and thinking, okay, what here could I replace as a decision model?
[13:47] So, this was the traditional way we were seeing things a few years ago, and today you can have models all through your retrieval pipeline.
[13:56] So, one thing I'm going to do is I'm going to go into a demo right here.
[14:01] This is the QR for the demo.
[14:03] A bit going ahead of myself.
[14:07] Here we go.
[14:08] So, in this demo, which you're going to get the code to, I'm not going to run it live cuz the internet here isn't great as you guys probably know.
[14:16] Um this is a simple chat interface.
[14:18] I have a governed analyst, I have a naive analyst.
[14:21] My governed analyst is built on what is an intelligence plane.
[14:24] It gives it the ontology layer, semantic layer, gives it everything we've seen in that reference architecture.
[14:30] And very quickly, just to show you the effect of having an intelligence plane.
[14:36] The key thing is having one, saving on tokenomics, right?
[14:41] Tokenomics is cost, accuracy, and latency.
[14:44] And we have two agents that are able to execute the same questions, but the one that is powered by an intelligence plane is a bit smarter.
[14:54] It gets things right 99% of the time.
[14:57] And it does it with less token consumption.
[14:59] In 32,000 token compared to the 200 thousand token with the naive agent.
[15:05] And then we have all of the guard controls in place and the cache as well.
[15:09] And we have some prompt caching mechanism as well.
[15:12] So, when you actually start I'm working backwards here.
[15:15] When you start to implement some of the things I'm going to show you to you in a second, you start to implement systems that are able to actually one, be optimized for production,
[15:25] and two, learn and grow with the data that you're capturing.
[15:29] So, over here I just want to talk about the ontology and the semantic layer.
[15:33] I'm kind of skipping between demos here cuz we're short on time.
[15:36] So, with the ontology and semantic layer, I asked the question uh to both agents around um what was a counterparty, right?
[15:46] And the ontology layer adds a level of accuracy that you can't get by just doing simple keyword search.
[15:53] So, in the demo, this is a keyword search where we're able to actually find some um entities, right?
[15:59] And the agent was able to answer without the ontology, but with the ontology, the answer actually improves and it gets the right answer using the ontology.
[16:09] Same with the semantic layer.
[16:11] The agent the naive agent without any semantic layer, we asked it a question and it got the we asked it what does concentration mean?
[16:20] And it got it wrong.
[16:21] The naive agent got it wrong, whereas the semantic the intelligent agent built with the semantic play with the intelligence plane got it right.
[16:30] And now that margin of error is just five 5%.
[16:36] But when you're working with the finance sector in any mission critical workload, that becomes very critical.
[16:42] So getting your semantic layer built out, get your ontology layer built out, it becomes very important.
[16:48] I'm going to you're going to get access to this demo and I'm going to go back to the slides very quickly.
[16:57] So there's a bunch of models that we're adding into the intelligence plane and the demo I showed you actually has all of these models and all of this architecture in place.
[17:07] So what should you your team be doing now?
[17:09] One, you treat memory as data.
[17:11] Right?
[17:11] Don't look at it as different things.
[17:14] Write down your entities and definition.
[17:16] You push entitlements into the data layer.
[17:19] Over Oracle, we have features that actually enable you to do this very easily and capture traces.
[17:25] So memory is is basically your moat.
[17:28] You need to own your own memory.
[17:30] So over Oracle, we have this reference architecture we saw before.
[17:35] But at Oracle, we have product features that map to it.
[17:39] So we give you the system components to build your own intelligence plane.
[17:44] So that means that you have all the tools you need to build something that actually learns and grow as you continue to execute build as you continue to build and execute your agent.
[17:55] I'm going to touch on a few ones.
[17:57] Memory, security, deep data security, private inference and ontology and semantics.
[18:02] So these are the key five areas I want to touch on very quickly.
[18:08] So agent memory.
[18:09] We've spoken about it, but now what we did at Oracle is we launched agent memory as a Python package.
[18:14] We put memory engineering into a box, which allows you to just not worry about all of the complexities of making agents remember, we just convert it into a Python package
[18:24] and it generates context card for you where you don't have to actually do it yourself.
[18:29] Then, one thing you have to actually have to start capturing is your memory traces because this become data for fine-tuning.
[18:39] So, right here we have a slide that just shows where memory engineering and context engineering meets and this is for you to take and go through when I give you the slides.
[18:47] I'm just going to skip through this very quickly.
[18:50] So, agent memory and the Oracle AI database is what we just covered in the demo.
[18:55] I skipped it.
[18:55] But, the next thing is deep data security.
[18:57] I spoke to a few folks yesterday and today and they were talking about agent identity and how they were using different solution providers for it.
[19:06] In agent in Oracle, one thing we did is we made security a first-class citizen of the database.
[19:12] Which means with a product called deep data security, you can take all of the context of your application and the data access and row access of your user and we propagate that into the data layer.
[19:25] Which means your agents can take on the identity of the user it's acting on behalf and access the same data that it's only allowed for is only that the user is only allowed to access.
[19:35] Fumbling my words there.
[19:37] Okay.
[19:38] So, all of this is all great including the demo I showed you it uses all of this infrastructure I spoke about and I showed you.
[19:46] And this is how deep data security works.
[19:48] That's right.
[19:49] We have the analyst, we have our identity provider.
[19:51] The identity provider provides all of the data access that the analyst is meant to have and we can propagate that into the agent and the agent inherits that as well.
[20:01] And that limits all of the data access for that particular agent.
[20:06] Okay.
[20:06] The next couple of slides I'm going to talk about private inference.
[20:11] Right?
[20:11] A lot of you work in finance companies where you want to actually have your inference on prem.
[20:17] So, we have the private AI service container which contains all the infrastructure you need to host your own your own embedding, your own LLM in your own LLM inference,
[20:26] and re-rankers.
[20:28] And this is all around owning your own intelligence plane.
[20:31] So, giving you the components.
[20:33] But, if you want to go to the cloud, we have Oracle Cloud where we host some of these models that you're hearing about.
[20:39] So, Grok models and some Open AI models as well.
[20:43] Either way, air-gapped um infrastructure or cloud infrastructure, you have the components within Oracle itself.
[20:52] All right.
[20:52] So, I have a second demo before I go off.
[20:58] So, this demo is an open-source tool that I built called Memories.
[21:03] And it's just to show you what how you can actually start to get into the intelligence plane domain.
[21:10] So, right now there are people working on agent memory, like you guys have mentioned.
[21:14] And within an agent memory system, there are other ways you can start to bring in models to improve the system itself.
[21:21] So, decision model is one of it, and this one makes ex- uses example of three different models.
[21:28] A decision model, an embedding model, and a re-ranker.
[21:31] And within my agent memory system, I typically have these models.
[21:35] But, with the intelligence plane or with an understanding that I need to own my own memory, and I own my own memory, I can actually start to fine-tune some of these open-source models.
[21:44] And that's what we're doing here.
[21:46] So, over here in memory routing, I'm just going to focus on memory routing for a second.
[21:50] And what memory routing means is we get some output from the context layer, or we get some output from our LLM, and what we're able to do is route it to the right store.
[22:00] So, whatever comes out, I can route it into conversation memory or workflow memory.
[22:06] Normally, you would do this using some code, but with decision models, this becomes very easy and very cheap.
[22:13] So, right here we have some fine-tuned Oh, skipped it.
[22:17] Let's go back.
[22:19] All right.
[22:20] Right here we have some fine-tuned models for memory routing, and we can see how it was trained.
[22:26] So we fine tune the equine model and we use it and we fine tune it into a decision model.
[22:32] The data that we use to fine tune is all here and obtained from the conversation I was have been with the agent and I was able to classify what type of memory it was.
[22:44] And this allows you to do memory routing very easily.
[22:46] So once you get any information coming in, the different types of memory you want to route it effectively and cheaply, you can do that with decision models.
[22:55] So memory routing, embeddings, and re-ranking are some of the lowest hanging fruits you guys can actually start to work with today.
[23:04] Once you understand your intelligence plane, you're capturing your memory traces, you're formatting them effectively, and you're retraining the models.
[23:11] And you get the benefit with improvement in quality and in some cases reduction in latency as well.
[23:18] Okay, so we're going to go back to the slide again.
[23:20] We're going back and forth a few times, which is good.
[23:23] Um and we're just going to wrap up.
[23:27] So there are different ways we're helping developers build agents in in Oracle.
[23:31] And one of it is private agent factory.
[23:33] And this is a no code platform we can just drag and drop and build your agents and have the whole control plane at least within the private agent factory.
[23:43] And the private agent factory also has the same agent memory system that we've we've been building across all our product surfaces.
[23:51] So again, the private agent factory is built on top of the Oracle AI database, so you get all of the memory core functionalities I've been talking about.
[24:00] I already showed the ontology and semantic layer the effect it has on the accuracy that you get within agents that have and don't have ontology and semantic layer.
[24:10] But again, Oracle gives you all the infrastructure and components to build all of this.
[24:16] So you can ask the same question with and without meaning, right?
[24:21] So we've already done the demo.
[24:22] Again, I skipped I'm kind of like going around the slides and the demos, but you saw what we had with ontology layers and without ontology layers and the accuracy improvements
[24:32] can be very minimal, but in mission-critical workload, they can mean getting a 100 million fine for handling fraudulent transactions or not.
[24:41] Okay, so we touched on continual learning a bit.
[24:44] What we saw in the demo was continual learning.
[24:47] And which is there different ways you can look at continual learning in token space and in weight space and also in in latent space as well if you're going into the prompts caching.
[24:56] So we've just gone over this.
[24:58] I'm not going to rehash this again.
[25:00] But one of the key thing and the key takeaway here is is developers and engineers of these systems, we need to actually own our own memory, engineer the capturing of traces,
[25:11] and actually build it into a system that allows for continual learning to be possible.
[25:17] So I'm going to end right here.
[25:19] Um I think I'm going over time and I'm going to give you guys the QR code to the demo I just showed.
[25:28] So if you just scan this, but the best thing is you can come meet me at the Oracle booth and you will get this QR code again along with other technical assets and we can have a conversation.
[25:38] Thank you for your time.