Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle

AI Engineer · 51 min · 455 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hi.
00:12Hi everyone.
00:14Uh thank you for coming to this session.
00:16Uh we're it's going to be a session about agent harnesses.
00:19We don't have a lot of time.
00:20It's only one hour session.
00:22So um hopefully we have time to talk about a lot of things that that you guys are interested in and the the session essentially is going to be uh like me talking like 10 15 minutes about the topics that we're going to talk about and
00:37solve in the workshop.
00:38You have the workshop instructions in a QR code or in a website and if you don't want to listen to me, you think I'm boring or something, you can start
00:47uh on your own on the workshop as I'm speaking.
00:51But essentially this is this is what's uh happening on the website.
00:55You just um sign up with your GitHub handle and then you will automatically receive an invitation to the repository.
01:03The workshop is hosted on code spaces.
01:07So you can just like create a code space out of that uh and you will receive an invitation automatically to get access to the workshop itself and you just have to create a a code face over there uh opening GitHub code spaces
01:20and you're good to go and and and begin with the workshop.
01:24Uh so I'll just put the QR code uh one more time for those of you who were uh who who want to take that and I'll I'll get started.
01:39Okay, perfect.
01:41Uh, if you have any questions, also my colleague Richmond is over there.
01:44He's going to go through um, you know, everyone's uh, table.
01:47If you have any questions, just raise your hand and he can help out as well.
01:53Yeah, there you go.
01:56Also, the website, I mean, you can go to the website workshopwaiting room.com.
02:00I paid too much money for that domain.
02:02So, uh, use it.
02:09Okay.
02:10So, who am I?
02:11My name is Nacho Nacho Martinez.
02:13I am an AI developer advocate and a data scientist by heart.
02:19And if you're interested in anything about me, just find me on YouTube.
02:22I have podcasts, I have workshops, I have everything.
02:26Not going to go too much into myself.
02:28Um just one thing though uh we re recently released a course with Andrew Ang and deep learning AI about uh building adaptive AI agents which is one of the subcomponents that we're going to talk about today.
02:42And you know if you're particularly interested in the things that you can do on a model to make it better and adapt over time, learn over time, just check out that course.
02:52is a free course uh a 1hour course that you can check out on on Google.
02:58So what are we going to go uh through today?
03:01Like I said, we don't have a lot of time.
03:03So I'm going to try to be quick.
03:05Um but these are the things that we're going to talk about.
03:07First of all, what is the agent stack?
03:09Uh what are the components of an agent stack?
03:12What are the different shapes that an AI application can take?
03:16What an AI agent is?
03:17And what is the agent harness?
03:19what is the main motivation of building or working around agent harnesses rather than working with AI agents or with an LLM model directly and you know all the subcomponents
03:30of an agent harness like for instance memory context engineering memory engineering and all these things.
03:37So because they told me that there's going to be a lot of people from uh financial uh like a lot of people that work in Wall Street uh I adapted this workshop to be around a bank right so the idea is that the main premise of the workshop is this
03:54we are hearing lots of the uh nowadays like open AAI is launching 10,000 agents or 5,000 agents to solve this problem 10,000 agents to solve this mathematical problem that was solved like in a 100 years 200 years.
04:08The thing is that these agents are autonomous.
04:11They don't have any human in the loop until they arrive to uh to the conclusion or the answers, right?
04:17So the agents themselves don't have the capacity to do this.
04:21Why?
04:21Because I mean the context window is limited.
04:24The memory components are something that a large language model doesn't have inherently.
04:29So all these things we need to implement ourselves to make these agents able to solve all these problems for 10 hours 100 hours autonomously right.
04:38So the idea is that a lot of people are making focus on okay we have these frontier models but my idea is that okay we don't need to focus so much on the frontier models we can use them but let's work on the harness which is the interesting part and the things that you can actually customize
04:54uh more and more.
04:57So on this application, the idea is that it's a medium-sized bank that has lots of uh you know suspicious transactions uh sometimes and they are categorized into five different types.
05:10So for example, if you get like a million dollar transaction that's coming into your bank and automatically and magically uh for three days you're making like $50,000 withdrawals on an ATM.
05:22That's suspicious, right?
05:23because why would you like put $1 million in and then just like distribute uh like that.
05:29So the idea is that we have this medium by uh medium-sized bank that has lots of transactions and these transactions they need to be kind of autonomously decided or at least flagged so that a human that actually works in the bank
05:47needs to spend less time actually looking at these things you know.
05:51So let's go to the theory a little bit.
05:54What is the agent stack?
05:55Right?
05:56When we talk with an AI agent, we typically are talking with the application layer, right?
06:01Uh what we interact with either an an interface, a chatbot, an a mobile application etc. Then we have the model itself which is the reasoning core the the you know we don't get any say in
06:16what the decisions are.
06:18Uh and then we have also the infrastructure that supports it all and then the compute.
06:23But what we are going to focus about today is the data part of it.
06:28And the data is very important because everything that we can manipulate and work around is on this part.
06:35The other parts we get less control over them, right?
06:38Especially if we're using like a third party uh model that isn't ours.
06:44So the data layer is what we focus about uh today, right?
06:47First of all, we have the agent loop and this is arguably one of the most important parts of an AI agent.
06:55Uh the agent loop is in charge of essentially driving the whole thing.
07:00Uh and it's customizable as well.
07:02And inside the data, you also have like a gateway and MCP so that you can connect it to third party external data sources.
07:09You also have the memory layer which we will talk about later and essentially also some context engineering uh parts of it that let you automatically summarize and compact and shrink
07:23the working context window of a model and finally also some tools and skills uh which we can do some interesting and intelligent stuff so that we can essentially keep the context window limited.
07:37This is one of the main ideas of the workshop.
07:41Now, typically the applications that we interact with, they are either a combination of these four, right?
07:48LLM chatbots which are very easy.
07:50Lots of people just create an AI application, create a chatbot so that you can talk with it.
07:55You also have rag applications which is essentially like a glorified uh vector search.
08:01And then you also have LLM driven workflows and AI agents.
08:04And typically what most of us use uh by the way raise of hands who doesn't use AI agents like cloud code etc of course uh so cloud codec etc all these systems are a combination of three and four
08:22which is um you know a combination of LLM driven workflows that provide automation and AI agents that provide autonomy.
08:29So it's a combination of automation and autonomy and these are called agentic systems.
08:35The better systems like cloud code, codeex, spy, uh open claw, hermes agent etc. they are all a combination of these two types and most of the engineers that I talk to and this is important is that we use these systems every day, right?
08:52But we stop or a lot of people not a lot of people think about these problems when building their AI applications.
08:59So you might have an AI application but then you don't consume it like this.
09:03And this is something that we're going to focus on today because I think this is the better way of consuming um AI applications not only through chat bots, not only through rag applications
09:13but also through a combination of automation and autonomy.
09:17So what is an AI agent for those of you uh who don't know?
09:21It's a like a computational entity that is powered by an LLM by reasoning kind of like us humans we have our brain it's augmented by a database for memory
09:35also like our brain it's extended through tools for actions so the equivalent of a human would be like we have our extremities and we can interact with the world just like an AI agent can interact with an operating system through the tools that we give it they provide it
09:51and it's also grounded in inputs.
09:53So for for instance it would be like our senses for a human our eyes etc that let us perceive information and the context like our attention is also limited like our agents
10:04we can't watch seven movies at the same time like we will not understand any of the movies we will not focus on any of them so the context window for humans is similar to AI agents and kind of we try to
10:17assimilate or you know have some of these notions be closer to our human experience.
10:22experience and this is something that we'll we'll work on today.
10:27So the reasoning part of it uh is provided by the frontier labs and we don't get a lot of control over that and then we also have the things that we do have a little bit more control over which are the memory the tool uses and the perception
10:43right and this is the the harness itself.
10:47A harness is essentially all the scaffolding that you need to do to build uh these three components which is the the tool use the memory and the perception and we're not going to focus too much on the model because like I said we don't have a lot of
11:05possibilities on what to do right so the harness is actually one of the most interesting topics in AI I think is the most interesting topic in AI nowadays because these harnesses actually control everything that the AI agent can do.
11:22You give it a different configuration then the agent the same LLM frontier AI model that you're consuming becomes better or worse.
11:31Sometimes you know it's been proven that by just changing the harness the same model performs better or worse for specific tasks.
11:39So it's really really important to make focus on the harnesses and not a lot of people are doing this and like you said in a harness there are you know
11:49seven different or eight different components if you consider continue learning as well but there are lots of decisions that you need to make right you have the model you have the storage like where do you put the data
12:00uh you have some encoding and retrieval components like how do I make the distance between two vectors to actually retrieve the data properly uh what do I store in my memory?
12:11Do I think of it as short-term memory plus long-term memory kind of like humans have?
12:16Do I think of it in different uh compartments or different types of memory?
12:21uh also the semantic layer which is a part that's really um interesting because as humans we are here and we have some context that we have never said to each other right but we are sharing this
12:35for instance I'm a speaker and you are listening you are not singing right now why are you not singing because it's not appropriate in the semantics of the context that we are at right now right
12:47so the model itself also has access to the semantic layer which talks a little bit about all the things that haven't been set to the model but they are assumed.
12:57Also the agent loop which is like all the things that drive and automatically and autonomously make the agent work all over uh again and this uh is like one of the most important parts and also context engineering and we'll focus on each of them but I will not go d uh I will not deep dive into them
13:16because we'll actually do that in the workshop.
13:19So just for reference um there was this big discussion around files versus databases like lot of people were saying we need to use files for everything like you just put everything into a file and then like magically
13:33one month later the agent becomes better.
13:36Some other people are database maximalists and they want to put everything into a database.
13:41Right?
13:42So this dilemma is still going on and I would say that um well let let me give you an example like can can anyone tell me in this example that you see on the screen
13:56can anyone tell me what is the workaround that agents use to solve uh this specific problem that you're seeing on screen multiple agents writing to the same file.
14:08How does that work nowadays in AI agents?
14:11Any ideas?
14:15Work trees.
14:16Exactly.
14:17Good.
14:18So, work trees is the current solution, right?
14:20And this is a a thing or a workaround because files have an inherent problem in themselves, which is that they they like if multiple writes are coming into the same file,
14:32then you have problems with consistency like which right came first?
14:35If someone is reading, what do I do with this person that read the file before I modified it, etc. lots of problems, right?
14:42So for some things I will definitely tell you like files are great.
14:46Files are great at storing lots of unstructured data.
14:50Database on the other hand is also used for some other types of use cases.
14:56But my solution is that for some cases some of these uh solutions work with files, for some other cases they work with databases.
15:04So we actually um need to reconsider some of these things that are going on, right?
15:09Um, and the work tree uh solution or the workaround has been already solved by databases 30 years ago.
15:17And we're trying to reinvent the wheel with some of this.
15:20And we don't actually need to reinvent anything because there is an actual solution to this.
15:25For example, if you're trying to write to a file um at the same time multiple agents, if you had transactional consistency or acid consistency like atomic consistent isolated and durable
15:38um rights into the file, you would not get this issue.
15:41Right?
15:42So files are good for some things and databases are good for for other things.
15:49We don't have to go into only one of them, right?
15:52So in the agent harness just consider this as an as a thought.
15:58Uh another component is the all the memory components right and this comprises all the mechanics and the the systems that are created around an AI agent to make it
16:11use the information um and not only use this information but also be able to refine it and recall it appropriately when it's time right and agent memory you as the engineers
16:24you get to choose like what are the things in your agent memory layer that you're going to consider.
16:30Some people like the short-term versus long-term memory.
16:34And in this case, this is part two.
16:37We have some short-term memory components.
16:40For example, if we are talking with clot or with codeex, we have short-term memory in the sense that the context window itself is short-term memory.
16:50Anything that disappears after the conver like it disappears after the conversation, right?
16:56most things uh in the context window just vanish.
16:59Then you launch another conversation and it starts from zero.
17:03That's short-term memory.
17:04It's very limited.
17:05We cannot put everything into it.
17:08And then we also have long-term memory.
17:09And inside the long-term memory, we also have some categories that we propose.
17:14Uh, for example, episodic memory would be all the things that happened, all the episodes of our conversations, all the tool logs, the previous conversations, um, summaries of previous conversations that you've had with the agent, etc. They all go into this episodic long-term memory
17:30category.
17:31You also have procedures or procedural memory, which are all the things like skills, workflows, which are like a sequence of skills that are being run etc. All of those go into procedural memory.
17:43And also we have semantic memory which is all the unset like I said the agent remembers who you are what are your preferences what's your you know favorite framework
17:54etc all these things go into semantic memory so the problem the main issue here why cannot we put everything into the short-term memory because it's limited and it's not only limited but actually
18:10the bigger the attention matrix is and the attention matrix is essentially the the context window.
18:16The bigger it is, the worse it performs.
18:19And this happens for every model.
18:21There are lots of studies that have shown that if you start at zero tokens on the context window, the problem is being solved much more accurately than if you have 256,000
18:31tokens already and the limit is a million right now.
18:34We're at a million because of the KV cache.
18:37All these things need to be considered and just know that even if the advertised limit stops at uh 256k or 1 million if you keep going the like the performance
18:50goes horribly.
18:51So the smaller the context window is the better it will solve a problem.
18:57And this applies for every type of problem that you can think of uh in humanity right and there are two types of things that tackle this.
19:05One is context summarization which you just ask it hey just generate a summary a short summary for me with this uh with all the things that have been said in the context window and then you get the summary and then you start off from the summary the next conversation
19:19that is fine.
19:20Uh, another one is compaction and that's kind of similar, but in compaction you're just like putting chunks of the conversation into the database or into a file and then you just put it with an ID and if the AI agent needs it later, it has the ID.
19:34It just pulls from the ID with an offset kind of like reading from a file, right?
19:40Some of the these are some of the problems that happen in context uh context engineering.
19:45And then we have the agent loop which is the driver or the the car that that drives all the components that we have talked about.
19:53We have a context assembly part that pulls from the skills pulls from the tools pulls from the previous conversation from the files from the database.
20:02All of these things just brings in uh brings into the context window dynamically.
20:08Then it does some reasoning.
20:10it makes an API call to open AAI or whichever model you're talking to and decides on what needs to be done and this happens you know automatically and infinitely if you want
20:22in my case in this agent loop that that uh we prepared for the workshop it has a maximum patience because I don't want to run out of openi credits or any any credits that I that I have
20:33but uh yeah the the maximum number of iterations in our case will be eight and a maximum of 60 seconds of reasoning for each one of the iterations, right?
20:44Um, but this is completely configurable.
20:46Like I said, as engineers, we have to say in what goes on into the agent loop, right?
20:53And also the toolbox pattern, the skillbox pattern, what we propose for this for storing tools and skills, which are hundreds.
21:01I don't know about you, but I have like a hundred skills already installed.
21:05I'm not putting a 100 skills at the same time into the context window because each skill is like 3,000 5,000 uh tokens.
21:12So that would automatically like go over the advertise limit in no time.
21:17I I would have no space for my actual conversation with the agent.
21:21So how do you fix that?
21:22Well, there are different ways, but what we propose is that you vectorize, you create vector embeddings out of the tools and out of the skills.
21:32And through those vector embeddings, once the agent loop, let me go back.
21:36Once the agent loop wants to retrieve some useful skills or tools, you're going to do vector similarity search against this vector store that you have.
21:45You're going to pull the most relevant skill or tool to solve it.
21:49You solve the problem and then you offload it again.
21:51And that's one of the ways in which you uh keep the context window relevant in this in this sense.
21:57And you know using skills and using tools is a very very uh common pattern like everyone's using tools like a bash command or a change directory all these are specific tools
22:09through bash right so we're using tools and skills constantly this is for some cases more than 50% of the context window that's being used right so I just want you to be mindful
22:20of the vast amount of context window size that is actually being kept by the skills and by the tools.
22:30Um we also have this idea called uh skill distillation where you take previous conversations, runs and you know actual like conversations that you've had with an AI agent and then you try to
22:44create something meaningful out of that.
22:47How do you do that?
22:48Well, you can do it in several ways, but one of the easiest ways that you can do it is just like create a skill MD and put it into the same skill box that you had everything before.
22:58And this is one of the ways like a Hermes agent, one of the innovators in this sense was creating skills that are particularly good to solve specific specific problems.
23:08Let's say that you're working on a front end.
23:11Well, if you're working on front end, you might want to create a skill that has worked very well for you with all your preferences.
23:18Hey, I like this coloring style, etc. And the next time you work on the front end, since you have created vector embeddings out of this, you can just like the agent will automatically
23:29choose the more relevant skill.
23:31So if you just mention, hey, I like this coloring style that I had in this other skill, it's automatically going to know that you're referring to the new skill because you use the word uh coloring style for for instance, right?
23:43So skill like skill distillation is one of the techniques that that you can use to improve the the learning the learning of a model over time.
23:55And it's the same uh this is uh related to what I what I was saying is that you have dynamic skill discovery in the sense that the agent loop
24:04when assembling or building the context will pull the right skills and the right tools when it needs and you know some people ask uh like how does a frozen model get better and what I mean by frozen is that we we don't get any control over the frontier models right
24:22uh but we do get control over all the data the data layer of it right and some people say hey let's modify the weights of the model some other people say let's create an embedding model that works better for this specific problem
24:36and then I'll worry about you know just retrieving the better embeddings there uh etc so there are different ways to do that but the idea is that this is the main
24:47component of continual learning and continual learning is one part of of of the things that you can do to make a mo model better.
24:57And there are three categories, right?
24:58You either work on the parametric memory.
25:01It's called parametric because it modifies the parameters of the weights of the neural network.
25:06You also have representational memory which is what it works with embeddings etc. And you also have nonparametric memory which is everything that we have spoken about today until this point.
25:18But um if you're interested in any of these three, we have focused on the third one.
25:24You also have the other two and I will not go into the uh into those two.
25:28There are lots of techniques like um you know creating low rank adaptation uh fine-tuning a model uh creating embeddings etc. And all these things you can find in that course that I I was referring to before
25:42which focuses on the three categories.
25:44So if you're interested just take out uh take that course if uh or chat chat with us after after the session.
25:50So after the session, we'll be in the event for the for the rest of the event too, right?
25:57And last thing I want to mention is that all of the things that happen during a chat completions API call to any uh OpenAI model or anything that you're using
26:08is that there is an inherent uh instruction hierarchy.
26:11What do I mean by instruction hierarchy?
26:14There is a different weight on the things that are given to the model.
26:17For instance, what we use to interact with is the user, right?
26:21The request.
26:22In this sense, this is the third most important thing that we can pass to a model.
26:28The most important one is the system prompt, right?
26:30Everyone knows about this the system prompt concept.
26:34Um, and the system prompt is fine, but it cannot hold everything because otherwise things would not have a hierarchy in itself.
26:42uh you need to actually put some of the things in the system prompt, some of the things in the user prompt and all the tools and the skills that the things that are made available, they have this hierarchy.
26:54And this is good because it it's it actually um lets us or lets the model focus more on some things than others and we'll also go through that in the workshop.
27:04So just for reference uh this is the the UI of the workshop.
27:09So if you're running it in code spaces already, you'll see that you have uh this deployed.
27:14I will go through through it uh with you.
27:16But this is essentially the way in which I envision someone that works in AML like um a fraud detection system in a in a bank interacts with an an autonomous agent.
27:29The autonomous agent is running in the background 24/7.
27:32It doesn't sleep.
27:33It doesn't need any of the humanly uh necessities that we have, right?
27:38It's connected through a harness to the to a database and it has all the functionalities that we have mentioned but on top of that uh it's autonomously making decisions for the company
27:51even when you know we're offline.
27:54It's not ruining people's lives by just like saying, "Hey, this this guy is suspicious.
27:59Let's just like block the whole bank and everything, you know, because I would ruin ruin the person's life, but rather it's making these decisions and just like printing them so that a human can actually inspect them later on, but it's going to save a lot of time during the,
28:16you know, the discovery process, etc. So by the end of the uh of the workshop, if you complete it, uh you're going to be able to have an agent
28:26that connects to a database that is actually working as if it was uh any any company and it's going to do autonomous analysis and decision making that then a human is going to take and interact with.
28:41So these are all the the components that we have talked about about the harness and without further ado I will uh put the QR code just again or workshopwaiting
28:53room.com um and if you want to check out any of the other workshops that we have uh you can just check out this GitHub repo we're making a lot of uh progress and effort
29:04into all the things that go in there everything from workshops sessions uh notebooks So you can get started on different topics like agent harnesses etc. So just check it out.
29:17Uh and let me let me go to my Firefox and show you how this application works.
29:26So, first of all, let me just uh for those of you who didn't receive an invitation, if you go to workshop waiting room.com, you will get an invitation via the GitHub API
29:36with your uh user to this repository.
29:40So, if you just go here and then open on GitHub code spaces, it's going to ask you uh to create a code space.
29:48And of course, I'm paying for it, so don't worry about that.
29:52uh just be careful with um with the hardware that you deploy because it's expensive.
30:00But uh the idea is that you will get a a deployment like this with a couple of notebooks and I'll show you the workshop itself and then my idea is for me to step down in a couple minutes and then we'll go through the
30:14you know people who have questions etc and then we'll solve it together.
30:18But it's automat automatically also deploys this application which essentially is tracking all the things and this is of course uh fake data.
30:28Don't worry I'm not exposing anyone here but it's essentially coming up with new transactions.
30:34Some of them are fine some of them are uh problematic.
30:38So the autonomous agent which is a grog model running on OCI it's essentially making autonomous decisions and some of them it's saying hey this one is suspicious let's add it to the list
30:51and then afterwards uh you also get like a way to visualize the context window of the agent here um you can also check out all the data that's already in the database
31:02uh here uh like all the customers etc and also an autonomous way in which the agent is actually going to make an analysis of all this um of of all this in real time, right?
31:16So, it's going to run through the tool calls that we give uh that we have available in our tool um in our toolbox.
31:24Uh yeah, it's sorry, it's a little small.
31:27Um you know, so it's going to go through the evidence.
31:30It's going to search for knowledge in the knowledge spaces that we created through vector stores.
31:35It's going to make a lot of tool calls.
31:37And finally, it's going to essentially go through the whole activity uh suspicious activity reports automatically.
31:44And here is just a fancy way of visualizing the 30 most recent um events that happen in the bank.
31:51But the workshop itself is this notebook, the notebook student.
31:57And what I want you to do is first of all just check out the different components that we have there because we are implementing them uh here.
32:06And I'm going to go through the first uh the first to-do.
32:10There are a couple of things that you need to do on the workshop just to confirm that that you know uh what you're doing.
32:15So first of all, let me just go ahead and show you uh the first part of the workshop, right?
32:22Uh you have this rock model 4.3 uh which is running on on cloud Oracle cloud and let's uh connect to the database of course.
32:39And after sorry, let me restart the kernel.
32:45Let me run this cell.
32:53Restart.
32:54But yeah, uh let let us know if you have any questions or issues uh when getting into the code space and then I'll step down and and do this with you and answer any questions.
33:06Any questions by the way so far?
33:10Okay.
33:12Yes.
33:15Uh the the repo.
33:17Yes, you can find it on workshopwingroom.com.
33:21And let me just put the QR code here.
33:25Uh this QR code if you want to uh scan it.
33:29And this will automatically give you access to to it via GitHub API.
33:35Okay.
33:35So let me re uh reframe the question for the people who are on YouTube.
33:40Um so he's asking about what happens with the episodic memory and all these memory components.
33:46Does it get it confused with previous conversations with a with an agent?
33:50Well, it depends on how much uh weight you give it.
33:54And this is why I was mentioning the instruction hierarchy as well.
33:58You might want to put this if it's a really really important thing and you actually wanted to pull from some place or the other place.
34:05Uh you can actually decide your memory layer architecture yourself, right?
34:10So you might want to for instance in some cases where the episodic memory is really important and you actually want to pull from previous conversations you might actually want to vectorize that rather than just save it as raw text
34:23right depends a lot on the on the use case.
34:25I would say in this case it doesn't uh it doesn't there is no confliction there because we have a clear separation on the memory uh components of the architecture.
34:35Yes.
34:35So the the skills can be added to to the system prompt.
34:39uh they typically um oh sorry they typically have a different uh yeah in the instruction hierarchy they have the least weight out of the four uh if you were to think about it it um it's typically here so it has the least weight
34:56I would think of it as a plugandplay place in which you can pull things and we are experimenting with this uh actually Richmond and I because we actually want to create a way in which you can plug
35:08into external internal or third party skill hubs and tool hubs for instance a collection of MCPS that you can give your agent availability by precomputing the embeddings of all of these things
35:21and once you have these precomputed embeddings you can just like say hey do vector searches against this third party uh group of skills or group of tools but rather than thinking of it as being included and by the way this is of course included in every open AAI or every chat completions API request that you're making to
35:41open AAI or any other model.
35:43This is attached at the bottom and this typically is like 80% of it like 80% of the whole request is uh is is tools and skills.
35:55But what what happens or like why don't you get charged like five times more than what you're charged now is because uh OpenAI and all these systems they have a semantic cache and they put uh if
36:0680% of it was the same as the previous request it doesn't recomputee it just goes into the semantic cache and this allows you and this allows all of us to have
36:16um an a way in which to consume the Frontier Labs models in an economical way and of course it's still very expensive.
36:25But what I want you to understand is that it would be four or five times more expensive if we didn't have this semantic cache that just um puts the tools and these skills
36:35into it.
36:36Because in the response like remember that when you're interact with interacting with a frontier AI model, you're essentially making the whole thing through an HTTP request and everything like this would be for example 10K or 5K as a system prompt.
36:54you're an AI assistant and then you need to help me solve this this and that.
36:58You have the developer like the specific app instructions, the user query which is what we type in the chat which is negligible like 100 tokens thousand tokens maybe and then the tool outputs and the the retrieve docs which is 80% of the request like 35 to 100,000 tokens is already this in every request.
37:18Does that answer your question?
37:20One last.
37:21Yes.
37:21Okay.
37:22So his question was like how do you manage the budget and since there are so many things in the skillbox and the toolbox patterns like how do you essentially make this work right
37:33um we have a clear separation the context window we think of it as or think of it as an n-dimensional matrix that starts at zero but it's never zero it's closer to like 10,000
37:46I was reading a paper that was comparing the efficiency of each harness on how much they put into the first system prompt and the first uh completions API request
37:58at the beginning.
38:00Pi, you know, Pi, the harness, that's the most minimalistic and the most efficient uh harness because it includes about 6,000 tokens at the beginning on every request.
38:10But if you go to other harnesses, they have more like 30 40,000 tokens.
38:14Um how do we we have a clear separation the context window?
38:19we just let the everything goes around the agent loop, right?
38:23So, let me go to the agent loop to show you uh one thing.
38:27The agent loop will control everything that it connects to, right?
38:32Um so here, for example, the agent loop.
38:37[snorts]
38:37Oh, I'm at the bottom.
38:38Hold on.
38:42Uh just as a reference um the agent loop.
38:53So here right uh the context assembly when you're working with it uh you will essentially build and you can go through this in the in the notebook.
39:03explain the notebook but um the context assembly is one part that first of all we have a maximum number of defined uh tools or skills to retrieve I think the
39:17defi the default number is three uh so every time a new request is coming it's determining and this is done by the model itself it's determining what is the thing to do
39:28then it's going to as try to assemble the context with the things that it needs to perform Right?
39:33And one of these things is retrieving the three most relevant tools, the three most relevant skills.
39:39And we are not using a reranker.
39:41But what you could also do is like at the at the end of uh the similarity search, you can also put an embedding model plus a reranker that will take those three and tell you out of these three,
39:56this is the most important one out of the three.
39:59And you could actually have a better refinement out of that because you you might not want to have a total of three skills loaded for every task.
40:06The task might be solvable by only one.
40:08Right?
40:09So the idea is that you can put uh whichever strategy you want just uh stick with it and uh stay consistent with it until the end.
40:19Uh so the idea in this case in the particular workshop is that we have a total number of uh vector searches that we're doing against one vector store which holds the tools
40:29another vector store that holds the skills k equals three and then out of the three you can retrieve the most relevant one with a reranker and then it's going to dispatch the tools of course and build a context window
40:43but this is entirely the decision and the job of the agent loop.
40:48Yep.
40:49Okay.
40:50So what is the most uh important or interesting way in which you you can interact with the memory layer of
40:57I mean the efficiency comes just from the number of entities that you create right so my first recommendation would be just like keep it simple uh so don't create too many abstractions or too many components of the memory layer so for instance
41:12try to experiment first of all with only short-term memory and long-term memory and just have two structures Right?
41:20If you think or if you have an agent that considers that something that was in the short-term memory or in the context window is kept is is worth keeping,
41:31you promote it to the long-term memory and you just put it somewhere as raw text.
41:37That's the easiest thing that you can do, right?
41:39Once you have it in text somewhere else in a long-term memory component or object, then you can take that vectorize it and then let it be explored by the agent loop in some way.
41:52So for instance, here you would just have like an extra thingy in the middle of observing and reasoning that lets the agent recall memories or something.
42:03You can call it like recall recall long-term memory.
42:06And this is going to make it uh be introspective before solving the problem once more.
42:13Uh, so it's going to go to your vector store or whatever you have this long-term memory uh, episode that you recorded in a previous conversation and from that you can let it see okay
42:25I'm going to let it pull from it if it considers it to be interesting or if the vector similar like if the distance between the vector of the query and what I have in my long-term memory
42:37is relevant enough and the distance is small enough then I will consider it into the context window if it's too big or it's not relevant enough then I will just discard it so it will be like a different an additional abstraction
42:51between the observing and the reasoning part there are ways we are not doing MCPS right here but one of the benefits of of using MCPS would be to solve
43:03um I would say that MCP is not really that that relevant in this case um but it depends on what you have in the database so for example If you have a database, you're interacting with a database like we are doing right here.
43:16Having an MCP is interesting, but one of the main problems that I like to say to people like what does an MCP solve, especially in the database, because I'm a database guy because I work in Oracle.
43:26Um, so one of the things that you can solve is letting AI agents make SQL queries, but by not hallucinating or trying to come up with a new SQL query every time.
43:38This is one of the bigger problems that that AI agents have with SQL.
43:42SQL has like 15 times like 15 engines like posgres, you have Oracle database, you have VSQL, you have like lots of different queries that work on SQL, but they have specific
43:56hints or subtleties in the query that makes it not not work well.
44:01Right?
44:01So the main problem that MCP solves in this case would be okay, I'm going to pre-create some SQL queries and then let the agents access those.
44:12So those queries are proven to work um rather than having an agent try to come up with a SQL query dynamically every time, right?
44:21Yep.
44:21Good question.
44:22Uh let me give you an example.
44:24So the most important thing would be the rapidness and that the meaning of the sentences and what the skill means itself.
44:34Uh rather than having it as text and if you were to store this just as text in a SQL file it would be able sorry in a in a file in the file system right in the operating system
44:46then you would have uh lots of ways in which an agent can do regular expression matching.
44:52So for instance, all the search would basically be either keyword search or like word search based uh into that skill which is fine.
45:03Um but the bigger that gets the less in like the the harder it gets for an agent.
45:10First of all, regular expression matching like GP is inefficient for huge files and sometimes if uh like the the file system grows and grows linearly, the performance of the regular expression matching decreases exponentially.
45:26Well, not exponentially, quadratically, let's say just quadratically.
45:29But this is one of the problems um that it doesn't scale as well as the size of the files.
45:36Uh, another one would be that if you premp compute the embeddings of a skill, you're actually also capturing the meaning in these embeddings, the hidden meaning that we as humans don't
45:47cannot comprehend.
45:48But rather than just describing it in text, which is fine for us to read, uh, it also has a premputation of 384 minimum to a,000 plus dimensions.
46:02And this embedding will actually hold uh things between each one of the tokens in the in the in the embedding matrix to be much more discoverable for instance by semantic search.
46:14And then this is what leads you to try hybrid search which is a combination of semantic searchs uh which is embeddings plus keyword search and and this is called RRF
46:27uh which is a type of hybrid retrieval.
46:29it performs much better because it will essentially just have both of them.
46:34Uh and it will essentially and this is uh we can I I can show you uh the example in the in the workshop because I I created this visual
46:43visualization here that explains exactly what I mean with hybrid search, right?
46:48Uh you see yeah so depending on the on the this essentially is a plot that uh combines the search capability of keyword search plus semantic search and this gap that you see at the middle
47:04is anything that's that's plotted here when doing hybrid search it will be more relevant than any of the other two.
47:12So uh as a summary uh the more like the more you can have two different types of of search.
47:20Each one has its advantages but if you combine them both you will uh effectively do a little bit more compute but then the quality of the retrieval will be better over time.
47:30Um these are like most of these I'm thinking of it from the AI agent perspective.
47:36So I'm not expecting anyone uh from this uh room to just like start building their own custom harnesses because we already have great uh solutions in the in the market right we have cloud code we have codeex
47:49each one has their own um preference by the way a show of hands for who likes pi okay less than I expected by but uh good so um I would say that I'm thinking of it as from the AI agent perspective.
48:07This what I want you to understand is that you can build custom harnesses for specific problems.
48:13And building custom harnesses doesn't to be doesn't have to be as sophisticated as creating the new clo the new clo code and then you're running all of your work in that custom harness.
48:24Rather, let's simplify the problems that you're trying to solve and then build custom harnesses that attach on to onto that.
48:32So in this case I'm thinking of um you know if you're a consultant you are my perfect uh use case or my perfect user for this workshop because you could be creating a custom harness that
48:45is working fine for a financial agent and has all the things that it needs to a financial uh use case.
48:51But then if you have another type of customer or with different types of data or with different types of constraints if you want to solve the next navir stocks problem or something
48:59uh you know those have different constraints for for instance the open AI problems the mathematical problems they need an infinite an automatic compaction or automatic context summarization because these agents don't have any human in the loop in the middle.
49:15So the like everything that goes around solving these problems have to have by default automatic uh automatic context uh summarization.
49:26It also has to have infinite patience or near infinite patience.
49:30In this case this agent loop is simplified to only eight steps maximum.
49:34All these considerations right depends on the on the problem.
49:38Yeah, I would start from scratch because that way you can you get to uh strip 90% of what you don't need.
49:45Um so I would definitely yeah I mean everything is built around the chat completions API and both Anthropic and OpenAI have created their own standards to fight off against this but everything goes around that API.
49:58So if you just think of uh the way to communicate with a model just as an additional plugin to the to the harness, it becomes much clearer that the smaller the everything is, the better it is at the end.
50:11Does that answer the question?
50:14All right.
50:15So let's do uh this.
50:17You guys uh start with the code base um with a code space.
50:22just set that uh notebook uh by yourselves and then we're gonna go I'm gonna go through and do a round and if anyone has any questions we can chat in a less uh
50:34aggressive way because I feel observed and you probably feel no pressure.
50:39So I'm going to put the pressure on you by stepping off and if you Yeah, let's uh keep going.
50:45I mean we don't have a lot of time.
50:47Oh my god.
50:48Um I time flew.
50:51Um but yeah.
50:53Yeah.
50:53Thank you.
50:54Thank you for the clap.
50:56Um thank you.
50:58Thank you.
50:58[applause]
51:01Uh so I thought like 15 minutes or 20 minutes went by, but one hour went by.
51:07So uh one thing uh so I'm going to leave I set this up so that you can create the code space in the hotel at home.
51:18You can I will pay for it myself.
51:20So you can, you know, use this workshop and and play with it as as you wish and I will pay for it.
51:26So don't worry about it.
51:27Just um if you want to chat about some of the agent harness concept, just come to the Oracle booth.
51:32We're going to be here until Wednesday evening.
51:36And thank you for coming to the to the session.
51:39Yeah.
51:40Okay.
51:40[applause]