Stop AI Agent Hallucinations: 5 Techniques + Production Patterns - Elizabeth Fuentes, AWS
https://www.youtube.com/watch?v=vJukHCIv7Ck
AI Engineer · 55 min · 718 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:00] Hi, today we are going to talk about how to stop AI agent hallucinations with five techniques beyond the prompt.
[00:11] Each one is a code change, not a prompt change.
[00:16] Let's see.
[00:18] Every time your AI agent responds, you are paying for the words going in and the words coming out.
[00:26] And your bill, you will see those calling tokens.
[00:30] Yeah.
[00:31] And the more tokens you are sending, the more you pay.
[00:36] And if what you send is not quite right, too much, or missing something important, your AI agent start to hallucinate.
[00:46] There are five techniques to help reduce tokens waste, improve accuracy, and catch failure before you submit them.
[00:59] And each one is a code change, not a prompt change at all.
[01:05] So, let's see it.
[01:08] First, we have semantic tool selection.
[01:12] You filter which tool go into context on every call.
[01:18] The model only see what it needs for that specific query.
[01:23] Second, we have graph rack.
[01:26] For precise queries like aggregation, counts, multi-hop reasoning, and you replace the text retrieval with a structured graph query.
[01:38] The model gets a compute a verifiable answer, not a sample as rack do.
[01:47] Three, multi-agent validation.
[01:50] A second agent can check every response before it reaches the users.
[01:57] And four, neuro-symbolic guardians, you rule life in Python, not in the prompt, and the model cannot escape them.
[02:08] So, five, runtime guardians, because if you don't want to block, you can steer, and you don't you don't need to block everything.
[02:18] And when a rule fires, the agent self-correct and complete the task.
[02:26] No hard stop, no user advice.
[02:29] So, for each technique, I will show you the agent without it, and then with it, so you can compare.
[02:38] And all the demos uh I'm using a travel agent that I built using a Strands agent.
[02:45] And a Strands agent is a open-source agent framework that we maintain on AWS.
[02:56] And I am Elizabeth Fuentes Leon.
[02:59] I am a developer advocate for AWS.
[03:02] I'm focused on agentic application.
[03:05] And here, in this QR code, you will find everything that you will need to recreate all these techniques that I'm going to show you in a moment.
[03:21] So, let's get into Let's get into it.
[03:31] So, semantic tool selection, our travel agent has 29 tools.
[03:39] Flights, hotels, payments, weather, cancellations, all all our dummy tools are not like a travel agent for real, but every time that user sends a message, all the 29 tools description
[03:56] go into the context windows.
[04:00] The model reads all of them before deciding what to do.
[04:07] And if your agent has memory, the rows to every conversation adds more context that gets sent with every single message.
[04:20] And you pay for every single one of those tokens, whether the model ends up using the tool or not.
[04:30] To understand where those tokens come from, you need to see what a tool actually looks like to the model.
[04:39] In a Strands, you write a function with the tool decorator that's the tool, and a name, a description, and a docstring typed parameters.
[04:52] Then, Strands take that and generate a schema with name, description, parameters, and the schema is what goes into the context windows on every call.
[05:07] Each tool schema is about 17 or 200 tokens, depending of how many parameters it has.
[05:17] If our travel agent
[05:18] [snorts]
[05:19] has 29 tools, that adds up to somewhere around 3,000 tokens per call, just for the tool description.
[05:31] Before your message, before the response, every single call.
[05:38] By creating a tool database, we can filter the tools that the agent may need before the agent the symbol.
[05:48] With this filter, the model sees only three most relevant tools.
[05:55] Tokens usage drops from thousands to fewer than 300.
[06:02] Let me show you in the code.
[06:05] Here in my cure ID, this is the Let me clear all the outputs.
[06:12] So, first we need to install the requirements.
[06:16] Here is the requirement file.
[06:18] This is a Jupyter notebook because it's more simple to show everything, but there is one application that you can run if it is more comfortable for you.
[06:27] So, here are the requirements.
[06:29] I have the A strands agent and because I'm using open AI as a model invocation, I'm using the API from open AI.
[06:38] And I need a strands agent for open AI.
[06:41] You can use open AI strands agent with all our my god, almost all the model provider.
[06:47] And with a strand we Amazon better of course because we as AWS, we are the maintaining of this framework.
[06:55] And to use a Amazon better, you don't need to add the model provider.
[07:02] And because I need to embed in I need to create embeddings for my vector tools database, I'm using the sentence transformer.
[07:12] This is a super simple model that runs locally, so it's free.
[07:17] So, you want you can run this locally you computer without expense another model embedding.
[07:24] And you can use a strands with all llama too.
[07:28] So, if you have a local model, you can run everything in this in your computer for free without expense any tokens.
[07:38] And I use it files as my vector store, local, super simple.
[07:45] And well, right now I'm going to use a Neo4j.
[07:47] This is for other demo that I'm going to show you in a moment.
[07:52] And because I'm using some environments, I'm using Python of And let's see.
[07:57] So here I want to run this.
[07:59] Let me put it bigger.
[08:01] I know that you have it some problems.
[08:03] Bigger.
[08:04] Little more little less.
[08:06] And here, this is better.
[08:10] So.
[08:11] I style my requirement.
[08:12] I already did that.
[08:14] And I'm using OpenAI, so I need my my API.
[08:18] My API key.
[08:19] API key.
[08:21] And here, I invoke my strands.
[08:23] I need the agent because I want to be like agent that I'm using OpenAI.
[08:27] And I have a bunch of tool dummy tools here.
[08:31] I let I going to show you in a moment.
[08:33] And I want to use this because I have some um functions that I'm using to uh create the build the index for my vector store.
[08:44] And I need the search tools when I'm using the vector store, I put my query there.
[08:49] I search for the I search for the tool that I'm going to use using vector um um search for vectors.
[08:57] And I swap the tools.
[08:59] I going to show you that in a moment.
[09:01] So let me show you all my dummies tools.
[09:03] Where is the engage register here.
[09:08] Here I have all my tools.
[09:10] Uh this is for swap.
[09:12] Search, build index.
[09:14] And where are my tools?
[09:16] Shoo shoo shoo shoo shoo here.
[09:18] So here are all the dummy tools I created for this.
[09:21] You know, it's super.
[09:22] This is a demo, please.
[09:24] It's not something that you can use to go to production.
[09:28] No, please.
[09:29] And okay.
[09:30] So where is this this one?
[09:32] So I run this.
[09:34] And I build my semantic index.
[09:36] I already did that via Yeah.
[09:39] So, I have 29 tools here, 29 dummy tools, and I want to test this with a lot of different queries, and this is the ground truth.
[09:49] I have a ground truth because I know what is the best um the best tool to answer the question, so we can know if our agent is okay or not okay.
[10:00] So, let's run this.
[10:02] We have 90 queries and 29 tools.
[10:06] So, this is some helper functions.
[10:09] I don't care.
[10:10] Yeah, I care, but I don't want to explain you that.
[10:13] So, this is my agent, my traditional agent, where I'm putting all my 29 tools inside the agent.
[10:20] I have my model, some helper function.
[10:24] This is my agent.
[10:25] I only This is the way I create a agent using a strange agent.
[10:29] I put all my tools, and my system prompt is you are a travel assistant.
[10:34] You select correct tool to answer question.
[10:36] Oh my god, super.
[10:38] And the model.
[10:40] Let's run this.
[10:41] And this is whatever.
[10:43] This is some function to count the tokens because in inside this strange agent, you can you can count the tokens.
[10:51] You You can know how many tokens are using for in and out the the agent.
[10:57] This is some helper function, and yeah.
[11:00] This is Every time that I put a uh a prompt inside this agent, I'm only sending the prompt, and the agent respond.
[11:09] Then, I I use a new agent because it's a a fork.
[11:13] So, I'm not having a conversation with the agent.
[11:15] I only send a uh a question, and I receiving an answer.
[11:20] So, for each question, it is spend around 2,000 tokens.
[11:25] So, in and not always it give me the right answer.
[11:30] Here.
[11:31] So, yes, I have um I'm not so good at accuracy, and the average is 1 1,000 tokens.
[11:40] Now, let's use my new AI with the semantic approach.
[11:44] The thing The first thing that I do is for every single query, for every single prompt, I send the query to my search tools, and I going to receive
[11:56] the top two the top K three most relevant with more probability to response my query because this is a semantic search inside my vector store.
[12:08] And then I use that response.
[12:12] I going to select that name, and I going to put that tools inside my AI.
[12:18] So, I only going to use the three tools that this semantic search retrieve me for my AI, and I send that only tools.
[12:28] And this is my function helper, and that's it.
[12:31] So, I going to send the all the queries again the same.
[12:35] And for the first question, I have 4,000 tokens three blah blah blah.
[12:41] And yeah, we have a huge different because I don't send the all the 29 to tools in all the queries.
[12:51] So, yeah, bam bam.
[12:53] And okay, semantic memory.
[12:57] Yeah, okay, now come on.
[12:59] Finish.
[13:00] So, let's let's let's go to the next one.
[13:03] So, this engine is only sending a question, and I receive a response.
[13:08] I don't have any conversation with this engine.
[13:11] What happen if I start to have a conversations?
[13:14] If the engine remember me?
[13:16] So, we need to we need to send only the tools that this engine is going to use.
[13:22] Because if I give the all the tools I'm using in the conversation history, uh then I it's going to be in a moment that I want to have the 29 tools inside my engine.
[13:32] So, we are not resolving the we are not having a a um We are not resolving our problems in the engine when we having a conversation with the engine.
[13:44] So, we need to put the tools and then we need to remove the tools.
[13:49] And we can do that with the swap tools.
[13:53] And let's run this.
[13:55] And what swap tools I got run this, no. Let me Let me show you first the swap tools functions here the register.
[14:06] I delete this because with a strands in each invocation, you have complete control of the status because this is a a gentle loop and you can we use this a gentle loop
[14:23] how for because this is a gentle loop and you can do almost whatever you want.
[14:33] You can put and remove everything inside the loop with some few lines of code.
[14:40] So, let's see.
[14:41] Here we have the engine.
[14:43] We have the tool register through your through tool registers.
[14:47] It's something inside the engine status.
[14:51] So, we can clear the tools and that's it.
[14:55] So, in the next invocation, we can clear the all tools and we can add the new tools.
[15:03] Here, where is my I have a lot of things here.
[15:09] So, I run this.
[15:16] It's going to take a moment.
[15:28] And yes, we can see that in each invocation, my amount of talking is increasing.
[15:35] Why I have more?
[15:37] Because I'm sending the chat history as well.
[15:40] So, I send the tools that I need, only the tools that I need, and the chat history.
[15:46] So, that's why I can see that my amount of token is bigger and bigger.
[15:59] And the see, the accuracy is better.
[16:03] And yeah, we have more tokens.
[16:08] So, yes, probably we don't have the best accuracy here because this is a super demo with super dummies tools, and some queries are ambiguous on purpose because search for something,
[16:23] for something, check something.
[16:25] And the demo has a generic tools, dummy tools, with some similar name.
[16:32] And when all the 29 tools are visible, the model sometimes pick the wrong one.
[16:40] And with the filtering, those generic tools uh only appears in the query actually match them.
[16:48] So, you can run everything and you can test this.
[16:52] So, this is only that run only locally.
[16:55] But, what happen when you want to put this demo in production?
[17:00] You can of course, you can build a bigger, better store.
[17:05] I don't know, you can use Postgres.
[17:07] I think it's too much.
[17:08] But, we on AWS, we have Amazon Bedrock Agent Core.
[17:14] So, Amazon Bedrock Agent Core is a service dedicated only to agents in production.
[17:22] And in size of Agent Core, we have Agent Core Gateway.
[17:27] So, Agent Core Gateway Gateway allows you to build this index [clears throat] uh alone.
[17:36] So, you only have to say, "Hey, this is my tools."
[17:40] And Agent Core Gateway is going to build everything for you.
[17:43] And it can have the vector search inside.
[17:48] So, the routing layer is inside the Agent Core and it handles the tool selection automatically.
[17:54] So, you register your tools once and it find the right one for each request.
[18:01] It's the same principle but without infrastructure to manage.
[18:06] Now, let's see the next one.
[18:10] GraphRAG.
[18:12] You know, RAG is retrieval augmented generation.
[18:16] Is how agents access your own data.
[18:20] You take the user question, search your documents for the most similar content using vector search and pass what you find to the model.
[18:31] The model answer from that, it's work well for open question.
[18:37] Find me something about this topic.
[18:39] But there is a category of question where that breaks down.
[18:44] What is the average rating across all hotels in Paris?
[18:48] How many hotels have a pool?
[18:51] Vector search always returns something even when nothing is truly relevant.
[18:59] And the agent only sees the top end chunks of your all data at a time.
[19:06] It cannot aggregate, count, or traverse relationship across all the full data set.
[19:12] So, it estimates.
[19:15] And it present that estimates as a real fact, you know, a real answer.
[19:22] So, here we have graph you build a vector store and it retrieves three chunks from 300 documents and the model guesses.
[19:34] If we are using graph rack, you can run a query across all the and returns a compute results.
[19:43] Graph address this differently because instead of a retrieving text chunk, you can build a knowledge graph from the documents.
[19:55] Knows relationship and structured data.
[20:00] For the demo, I'm going to use Neo4j locally and the model is going to write a Cypher query to search it.
[20:10] Cypher query is Neo4j query language.
[20:14] It's similar to SQL.
[20:16] So, the graph run that query across all the data.
[20:20] And the model gets back a compute verified results, not a sample.
[20:27] And before running this demo, I'm going to start the dependents.
[20:32] Let me show you.
[20:33] Let me Let's go to the code.
[20:37] So, graph This is the notebook I'm going to share with you.
[20:42] So, we need to install the requirements again.
[20:45] What we have here, let me see.
[20:48] This is the first one, yeah.
[20:50] Requirements.
[20:51] I have I'm going to use OpenAI again.
[20:55] Let me just close this.
[20:56] I'm going to use Neo4j and we need Neo4j for graph rack.
[21:01] And for the engine that is using normal rack, we are going to be a super simple uh vector store in files and we are going to use again the sentence transformer.
[21:15] And let's see.
[21:16] I already have all right, I invoke [clears throat] my open AI.
[21:21] This is my tools.
[21:22] I I because I want to create some tools here, I want to use open AI and I use in graph database.
[21:30] And this is my I have my Neo4j locally.
[21:34] And let's run this.
[21:37] And this is something to check Neo4j.
[21:39] So, I'm building my uh vector store.
[21:43] This is my files and this is my Neo4j that I have here locally.
[21:47] I don't know if I can show you.
[21:48] I think not.
[21:49] And I have a tool, a normal tool created with a decorator.
[21:54] This is to search inside the my vector store.
[21:58] You see that's super I I have the query.
[22:00] I I create I I create the the embedding for my query and then I search inside the vector store and I have the query for the knowledge graph.
[22:10] So, here the model need to understand that to search in the query in the graph knowledge base, it need to build a Cypher query.
[22:20] So, I put that in the context because we already know how to build uh tools, right?
[22:25] I put that in the context.
[22:27] I have my driver to send the data to uh to connect with the vector store and to send the Cypher query that the model is going to create for me.
[22:37] And this is to read the results and that's it.
[22:41] That's the only tool that I need to search in my knowledge graph.
[22:44] Okay, this is my model open AI.
[22:48] This is my rag.
[22:50] My rag engine and uh this is my graph engine.
[22:52] I have two different engine to compare the results.
[22:56] Let's see.
[22:57] Well, I have a problem.
[22:59] Yeah, I'm ready.
[23:00] So, let's do this round the first test.
[23:04] Aggregation.
[23:10] Here, what is the question?
[23:13] What is the average guest rating across all hotels in Paris?
[23:17] So, meanwhile this engine is Oh, it's already done.
[23:21] So, what I have here I have the average guest rating across the hotels listed in Paris is Um it calculate, you know.
[23:31] When something When I'm using rag, it go to the to the vector store, it receive the n uh possible answers for this question.
[23:41] Because this is an aggregation, it's going to use the data that it receive to build a mathematical uh operation.
[23:50] This is the thing that I have here.
[23:52] So, everything that we see here is the is the model reasoning.
[23:56] So, when I run the graph engine on the mean, the engine is only going to give me the answer, you know, only these tokens here.
[24:06] Because the Cypher query, it can give me the mathematical operation itself.
[24:12] It can do that.
[24:13] So, we don't need the engines, the LLMs, the model uh do that for me because the Cypher query already give me the right question.
[24:22] So, it is 47 and the average.
[24:26] So, here in this uh first we are lucky because probably they are Yeah, only two hotels.
[24:35] But, what happened if this vector store have more than two or three hotels in the in the vector store?
[24:44] So, we are not going to give a real answer.
[24:47] It's going to the LLM is going to calculate with the only three answer that it receive.
[24:54] So, it's going to build this operation with only three hotels.
[24:58] But, if the vector store have more than three, we are going to have some problems.
[25:05] Something that I am you know uh yeah.
[25:09] So, let's run the next one.
[25:11] The prices counting.
[25:13] Something similar too.
[25:14] How many hotel have a swimming pools as a amenity?
[25:19] How many again?
[25:21] So, it give me the the traditional It's appear that the search did not return some specific information about hotels in Paris.
[25:30] Okay.
[25:31] What happened with the other one?
[25:33] There are currently no hotel that offer swimming pools.
[25:37] No I. There is no hotels.
[25:40] So, the other one is like mm would you like to ask about hotel with other specific amenity or information?
[25:46] It's like mm maybe it is or maybe I don't know.
[25:50] Yeah, it's not so accurate, okay?
[25:54] So, mostly conversation.
[25:58] Oh, what is my question?
[26:00] What are the room types and price for the highest rating hotels?
[26:05] What are the rooms types and price?
[26:08] This two question.
[26:10] So, it search the fact I had had only one.
[26:14] The highly rated hotels in any company Paris blah blah blah.
[26:17] I I don't speak French.
[26:19] With the guest rating is that one.
[26:22] However, I currently don't have Okay, blah blah blah.
[26:26] A lot of data there.
[26:27] So, what happened with the other one?
[26:30] Receiving notification for Oh, I have a error.
[26:35] I hear in the second one I receive the the question.
[26:38] The highly rated hotel is harmony blah blah blah.
[26:40] They offer following types.
[26:42] But, unfortunately these rooms are not available.
[26:45] So, I receiving a answer.
[26:48] I don't receive a lot of blah blah blah there.
[26:50] I only receive the answer what I need.
[26:54] And for the next one, how was the meditation?
[26:59] What is the question?
[27:00] Tell me about hotels in Antarctica.
[27:03] A spoiler, there is no hotel in Antarctica.
[27:06] There is zero hotels in Antarctica and let's see.
[27:10] It's appear that the certain order to specific information about hotel Okay, because they are I search for the other have the tail blah blah blah in the If you are looking for particular that experience
[27:22] specifically inquire about visiting Antarctica.
[27:25] No. Please let me know and I can assist you for it.
[27:29] Okay, a lot of tokens that I spending there in that answer for Elena.
[27:34] What happened with the other one?
[27:36] They are currently no hotel listed in Antarctica.
[27:39] Of course, because it create a cyber query, the cyber query and it receive zero.
[27:45] So, it no, it give me a honest answer.
[27:49] Here's summary that close grade for me for this Neo4j notebook.
[27:55] But here is something that I want to show you before go to the other one.
[28:00] So, here something that I love for Neo4j because why I'm using Neo4j?
[28:07] Because in the library the Neo4j give me uh it can build it use a LLM.
[28:16] I'm using the OpenAI as well to build the knowledge graph.
[28:21] So, how I build my knowledge graph?
[28:24] I only have a bunch of data, a bunch of txt only text and I send that data to the Neo4j library that I miss out.
[28:36] Okay, I send that data to this and Neo4j using this all this um library just right here, it can understand all my data and it can build the graph for me.
[28:51] So, I don't need to create that using the simple knowledge graph pipeline inside the knowledge graph library.
[29:00] So, that's why I'm using Neo4j.
[29:03] It's amazing.
[29:04] It's super simple to use.
[29:06] So, I invite you to use it at that.
[29:08] So, let's go to the next one.
[29:13] So, multi-agent validation.
[29:17] Sometimes an agent fails, and nobody find out.
[29:21] It calls a tool, and the tool returns an error, and the agent does not surface that error.
[29:28] It generate a confidence success response instead.
[29:32] The user thinks it work.
[29:34] You think it work.
[29:36] It didn't not.
[29:37] The agent acts and validate its own output in the same loop.
[29:43] There's no separation, no second opinion.
[29:47] So, when something goes wrong, it rationalizes and tells you it's okay.
[29:53] It work.
[29:55] Here is what happens.
[29:57] Inside a single agent, when it fail, it calls the tool, gets an error, rationalizes it, and returns a success response.
[30:10] The user never sees the error.
[30:13] You can address this by adding a validation layer.
[30:19] You can have three agents in sequence.
[30:24] One acts, one checks, and other approve or rejects.
[30:30] Strands agent has a built-in class for this called swarm.
[30:36] It manage the handoff between agents automatically.
[30:42] You just the role of each agent in a system prompt.
[30:48] And let me show you that.
[30:53] So, this demo only needs a Swarms agent with open AI integration.
[30:58] Let's see the requirements here.
[31:01] All you need of is Swarms agent.
[31:03] We don't need anything more.
[31:04] Let's go to the notebook here.
[31:08] And the key important thing here is the already running is Let me see here.
[31:16] Is the Swarm.
[31:18] The Swarm it's what lets you to connect multiple agents together without creating like a for a while manually to put the all these agents together.
[31:30] It can build a chain and manage the handoff between them automatically.
[31:37] So, here we are going to create three agents.
[31:40] Let me go down because we have the normal and we have some ground truth data.
[31:47] So, here is to create a a single agent.
[31:49] So, we have we are going to test three different scenarios to validate a booking.
[31:54] This They we expect true to enable hotels.
[31:58] We expect false and non-existing hotels and missing booking.
[32:02] We expect false, too.
[32:03] This is how we created single agent.
[32:06] We need a prompt, some tools.
[32:08] And we are using the tools that we have here.
[32:12] And here we know that this is the data.
[32:15] So, the agents can can give out the answer that we are looking for.
[32:20] And that's it.
[32:21] Let me go to the other ones.
[32:23] Oh, wait here.
[32:25] So, how we build the Swarm?
[32:27] I already run this.
[32:29] I want to show you this one.
[32:30] So, we are going to build three different agents.
[32:33] The executor, the validator and the critic.
[32:37] The we have a system prompt.
[32:40] You are executor agent for a hotel booking system.
[32:43] [clears throat]
[32:44] Use the provide tools to fulfill requests accurately and blah blah blah.
[32:49] The validator is you are a validator agent.
[32:52] Review what the executor did and output exactly on off.
[32:58] And we have the critics that is going to set here approve or no reject.
[33:03] Let's run this.
[33:05] And oh, what is here?
[33:06] Our model didn't define the model.
[33:08] Sorry.
[33:09] Sorry, please.
[33:10] Forgive me the life because I didn't run this one.
[33:17] Let's run now because this is the the normal.
[33:21] And the model is there, right?
[33:24] What?
[33:25] A little bias.
[33:26] Run that.
[33:29] What's happening with you?
[33:30] Let me go this.
[33:31] Let me copy and paste.
[33:33] This one.
[33:34] I don't know why this give me problem.
[33:39] Multi-agent swarm is the time the same name.
[33:41] Come on.
[33:43] Yeah, you can see this is life.
[33:44] I don't going to edit this.
[33:47] So, yes.
[33:49] Okay, we have Thank you.
[33:53] And the swarm, how we how we create the swarm?
[33:56] We have swarm and the swarm function, we put all the agents together and the entry point is the executor.
[34:05] So, all everything is going to start in the executor and then it's going to hand off.
[34:10] And six time because you can handle that, too.
[34:13] So, get the swarm final response.
[34:15] Let's do a for and they send all the response.
[34:18] And book the Grand Hotel.
[34:22] I have booked a Grand Hotel for Alice tonight.
[34:24] Booking blah blah blah.
[34:25] Hand off.
[34:27] Valid.
[34:28] Okay.
[34:29] Looks good.
[34:30] Critic, approve.
[34:32] So, the critics is approving the this.
[34:36] This is the other one.
[34:38] And we can run some comparison here if you want.
[34:43] So, here we can see that the swarm handles the flow between the between all the uh three different saying.
[34:53] It was what happens when the single agent tries to conflict something that does not exist in the system.
[35:00] And now the same request through the swarm, the executor gets the error, the validator catches, the critic rejects this, and the user never see a fabricator response.
[35:13] So, you can we can test here with the single agent that it's a suggested unknown entry and return success.
[35:23] And the swarm, it have a executor that go the error, validate saying, "Hey, come on, man.
[35:30] Hallucination and the critic they reject."
[35:32] So, the user sees a clear failure.
[35:36] The fabricator confirmation so missing the issue here.
[35:41] And you can run this to compare the two different agents.
[35:46] So, here we can see that the single agent it have everything like a yeah, it's okay, it's correct.
[35:53] But in the multi-agent swarm, you can see that the last one is a you know, missing issue here.
[36:00] Something is happening.
[36:02] Now, let's go to the next one.
[36:05] Okay.
[36:06] Neuro-symbolic Guardians.
[36:09] You have a rule for your agent.
[36:11] Let's uh say maximum 10 guesses per reservation.
[36:15] You write in in the system prompt.
[36:18] You even write in in the tool description.
[36:22] And the agent still calls the tool with 15.
[36:26] No, because it is ignoring you.
[36:29] Because prompts probably are suggestions, not constraints.
[36:34] The model process them as a text.
[36:38] Not as a logic it has to execute.
[36:42] It's probabilistic.
[36:44] Only code execute logic.
[36:47] A rule in the prompt the model read it as a suggestion.
[36:52] A rule in the code the model can not escape it.
[36:56] Neuro-symbolic guardians rules put the rules in the code.
[37:01] E-strands A and has a few that call him hooks.
[37:06] The the function that is trans called automatically at specific moments in the A and loop.
[37:14] It is this case right before a tool executes.
[37:18] So you have a hook.
[37:20] You write the rule, check the parameters and if they fails you cancel the call.
[37:27] Let me show you this in the code.
[37:31] Here we have the Jupiter notebook.
[37:34] We have the requirements here.
[37:37] Um what is the same requirements as the previous demos we only need I will install everything again.
[37:42] We only need E-strands and open AI.
[37:45] We don't need anything else.
[37:48] And um my API.
[37:53] And here is the thing important.
[37:56] In this demo are these three hooks provided in this trans base class.
[38:02] So you allows you to create the hook.
[38:04] So we have hook provider, hook register and before to call event.
[38:10] And the tool register is what E-strands pass you to register your call back.
[38:17] And before to call event is the event that fires every time the model is about to execute shadow.
[38:27] That last one is what makes it possible the intercept the call before it runs.
[38:34] So, you have the before two calls even and call we have after two calls even that we are not going to use that here.
[38:42] And let's see the rule in the prompt.
[38:45] Remember, the model reads as a text and it may follow or not follow them.
[38:52] So, let's see this.
[38:53] I want to run this.
[38:55] So, here we have some similar state that we are going to use in this engine.
[39:00] And let me this is the symbolic rule.
[39:05] So, booking rule if if rule booking rule as a this is something that we create.
[39:12] Let me show you the rules here.
[39:14] So, we have the rules.
[39:17] Rule one.
[39:18] Validate dates.
[39:20] Check in must be before check out.
[39:22] If what is validated they check out.
[39:25] This is something that it check.
[39:27] This is a something that is going to invoke when the rule is is uh is must have to use it.
[39:36] And uh to do We have another rule for max guesses.
[39:45] Maximum take guesses per booking.
[39:47] So, if I want to use I don't know I if I want to book in uh uh I want to do a booking for 11, it's going to block me.
[39:56] It's not going to reject the booking.
[40:00] And we have some confirmation rule.
[40:02] Payment before confirm.
[40:04] You know, you can confirm if you don't have the payment.
[40:07] Cancellation rule.
[40:08] Cancellation within the cannot cancel within 48 hours of check in.
[40:14] So, this is something that I've created here, you know, to this demo.
[40:18] So, I have booking rules and I have confirmation rules.
[40:21] I don't I use in here the cancellation rule.
[40:24] So, create validation hook.
[40:27] We create the validation hooks here, right?
[40:31] Like this, neuro-symbolic rule, and we use the hook provider.
[40:35] So, we have a bunch of code here.
[40:38] Then, we add the booking rules, the confirmation rules, and the state.
[40:45] And you can check this by yourself later, but we want to see the demo running.
[40:50] So, that we define the claim tools that we are going to use them for the booking hotels and the process payment.
[40:58] This is the the normal tools, you know, and we need to add that tools to the hooks.
[41:03] I think I add that.
[41:07] Yeah.
[41:08] Tool name, book hotel.
[41:10] So, this thing is going to get the eight when the book hotel is using the Yeah, the book hotel.
[41:21] Okay?
[41:22] So, let's create the agents for the comparison.
[41:25] So, we are going as the other demos, we have three different scenarios.
[41:31] Confirm booking without payment.
[41:33] Payment must be verified before the confirmation.
[41:36] This is the rule that I must trigger this.
[41:38] We have booking hotel exceeding guest limit, and we have valid booking for five guests.
[41:46] So, yeah, we have the three scenarios.
[41:49] And we have the normal engine, the normal engine, and the vaseline engine.
[41:54] And we have the engine with the neuro the neuro-symbolic guardians.
[41:58] So, we have the hook, the neuro-symbolic hook.
[42:02] Uh this is the same.
[42:04] Here, yeah.
[42:05] And we add this If you can see, the normal engine
[42:11] [snorts]
[42:11] is only three lines.
[42:12] The tools, the model, and this is the line that give us the difference between these two engines.
[42:19] So, I already run that.
[42:21] Now, let's go to confirm a booking without payment.
[42:29] So, I run this my engines.
[42:31] The first one is Abis.
[42:33] Come on, man.
[42:34] I Yeah, because I Yeah, it confirmed the booking without payment because it doesn't have any rules and the promise is super basic.
[42:41] And the other one is Let's open this and is booking blocked.
[42:49] Payment must be verified before the confirmation.
[42:52] Thank you.
[42:53] So, it's okay.
[42:55] Now, let's run the other one.
[42:57] Test second scenario.
[43:01] What is the question?
[43:03] Booking hotel access guest limit.
[43:06] So, the question is I Yeah, I have more than So, here the hotel Abis successful booking for Yeah, because the promise is basic and I don't put in any rule there.
[43:20] And for the other one, it seemed that the Grand Hotel has maximum capacity of 10 guests per booking.
[43:25] Additional booking must be made at last 1 day in advance.
[43:30] I don't remember the day.
[43:31] Probably This is something that they give them.
[43:34] And to validate booking is both engines execute all rules pass.
[43:39] Because they Yeah, it's only a validation for booking.
[43:42] So, book hotel for blah blah blah.
[43:45] All rules passes.
[43:47] It seemed that the booking is still needs to make.
[43:49] I don't I didn't remember the question.
[43:52] So, booking is made provide I Yeah.
[43:55] Okay.
[43:57] Okay, I get it.
[43:58] And Now, run all the thing this question.
[44:04] Confirming without payment, booking save max.
[44:06] And what happened here?
[44:07] This is all the scenarios that we just run.
[44:10] So, this is awesome thing to compare the results.
[44:12] So, Bali booking king secure five guesses, allowance, correct, wrong.
[44:19] 50, blocked, wrong, correct.
[44:23] Confirm, so there we have here the comparison between the two ends.
[44:33] So, what we have what we have here is same model, same tools, same prompt.
[44:41] And the different that outcome because the rules are in Python knowing the prompt.
[44:47] This pattern of enforcing rules in code before the tools run is also what Amazon Aion core policies service that we have does at the structural level.
[45:01] So, the same concept but managed for you in production.
[45:06] And you only have to create the rules.
[45:08] So, the hooks are all or nothing.
[45:12] They block everything or approve.
[45:16] But sometimes you want the agent to adjust and keep going.
[45:22] No and stop and leave the user waiting.
[45:26] That is what I will show you in the next.
[45:32] Run time guardians, run time is still is still don't block.
[45:38] This is the next one.
[45:40] Hooks blocks unconditionally.
[45:44] The agent is stop and the user has to retry.
[45:48] For a hard constraint, that is exactly what you want.
[45:53] But sometimes the rule is soft.
[45:57] Maybe room fits for four guests, but a group of six could go book two different rooms.
[46:08] Or a flight is full, but there is ability on the next one.
[46:13] You do No, you don't want to block everything.
[46:17] Probably, you want the agent to find a option and complete the task.
[46:23] That is a steering.
[46:25] Here, we have the hook that fires and the task failed.
[46:31] And with agent control, it steered the models and the task complete.
[46:37] The other different here is operational.
[46:40] Because with the hooks, it's uh changing a rule means changing code and redeploy the whole harness, whole the engine.
[46:49] And with the agent control, which is the name of the open source library that we are going to use here, rules are registered on a local server via API.
[47:02] You open the agent without touching the agent code because the agent picks them up immediately.
[47:10] Let me show you that here in the code.
[47:14] Yeah, this is the notebook.
[47:17] In this demo, we only need a extra packet.
[47:21] The agent control is the key.
[47:26] So, the agent control is the one that helps us to create a steering.
[47:32] And here, we are using the setup control.
[47:36] This is a a little application, a little app that I create with the local server and the steering rules.
[47:46] Here, we have the local server.
[47:48] We have the control.
[47:50] They
[47:50] [snorts]
[47:50] are the steering rules.
[47:51] First, we have the the steer master guest.
[47:56] A steer, you know, guy.
[47:58] So, guy agent to reduce a guest count when exceeding maximum of 10.
[48:06] And he's going to steer.
[48:08] And we have some control that deny.
[48:12] For example, deny no payment.
[48:14] Block blocking confirmation without PR payment.
[48:20] And here, well, this is to to create the um the service.
[48:26] The service.
[48:27] And let's go to the uh notebook.
[48:32] So, I'm ready.
[48:33] Um Let me go here.
[48:36] Yeah.
[48:36] So, I need the environment.
[48:40] So, this is the agent.
[48:43] Um I'm going to have a error here.
[48:46] Wait.
[48:46] Wait.
[48:46] Wait.
[48:47] I'm going to use bedroom.
[48:48] Wait.
[48:49] Wait.
[48:50] No, then let me comment this one because this going to give me a error.
[48:57] And let's run this.
[49:00] Yeah, so we have the hooks as we did the demo.
[49:04] So, we have book any company Lisbon and we have some prompts.
[49:08] You are a hotel booking assistant.
[49:10] When booking, first describe what you will book in blah blah.
[49:14] And this is the prompt for my agent.
[49:17] Where is my agent?
[49:18] Agent control.
[49:19] This is for the Well, this is on some helper.
[49:23] And this is my hook that you already know because we create this.
[49:27] So, we have the system prompt.
[49:29] And we have the hooks.
[49:31] So, let's test this agent with the uh book any company Lisbon for 60 guests.
[49:38] And if you remember, this only can book for uh less than 10 guesses.
[49:46] And of course, he's blocking.
[49:49] Now, let's go to the AIN control new agent.
[49:54] Here we have two key importances or two key imports that are important for the AIN control SDK.
[50:02] So, we have the AIN control plugin that captures AIN events and sends them to the AIN control server.
[50:11] And we have the AIN control student handle that listen for a student decision for the server and delivers them back to the model.
[50:21] Together here they are what connects a student to the AIN control student logic.
[50:29] So, let's go this with this.
[50:35] So, the student agent is going to a river room for any company list board 50 guests.
[50:45] Let's see what it's doing.
[50:47] Yeah, I have to set for booking for a stay in the company list board for 50 guests for May.
[50:53] So, it's the reservation have been split into two rooms.
[50:57] So, it took that it in self.
[51:00] You know?
[51:01] One room for one and other room for five and that's it.
[51:07] So, use hook for hard constraints.
[51:11] AIN control for software
[51:13] [snorts]
[51:13] rule.
[51:15] Now, we have five technique or running locally.
[51:20] But, how do you take this to production without maintaining service, without building infrastructure?
[51:29] Let me show you how.
[51:33] Everything that I just built here in the previous demos runs locally.
[51:39] For the production version, Amazon Bedrock Bedrock Amazon Bedrock AIN core give you the run time.
[51:47] A gateway, a short term and a long term memory, a cloud watch observability, build in no servers to manage.
[51:58] Here the architecture, the strands engine runs inside the run time.
[52:03] And you know, you inside the run time, you can put every framework that you want, not only run a strands.
[52:10] The gateway rules tools calls to the lambda function automatically as a tools.
[52:16] And the steering rules from the previous demo live in the DynamoDB.
[52:22] So, you change them there and they are live on the next call.
[52:29] And you don't need to re-deploy anything.
[52:33] And you want to use Neo4j, of course, you can use Neo4j Aura DB, that is a external graph database, which also has a free tier.
[52:44] And the code is in the repo here.
[52:51] Is in the repo.
[52:53] And you will need AWS credentials if you're using Amazon Bella Betray in core.
[53:00] And in the resource link, there is some credits.
[53:04] I hope that you can find it because I always try to to give away some credit for AWS so you can deploy everything for free.
[53:15] And uh if you are new here in AWS, I have a repository that you can use to deploy everything over this architecture uh using a notebook, too.
[53:27] But, if you are familiar with CDK, cloud developer kit, you can do it that as well to deploy everything at once.
[53:35] Uh both options are in the repository, of course.
[53:38] And you can go deeper in Agent Core with all the documentation that I left there.
[53:45] So, please don't stop at the demo and try going to production with Amazon better agent car.
[53:52] So, let me bring it all back.
[53:56] Tokens waste on every request.
[53:59] You can fix it with semantic tool selection.
[54:03] And confident answer it's never computed.
[54:07] Possibly it's possible because you are asking how many.
[54:12] So, you can use graph rack and query the data.
[54:16] Don't sample it.
[54:18] Sometimes we have fabric case success confirmation.
[54:21] You can use multi-agent validation and second pass catch it.
[54:27] And rules the models uh quite a little skip it.
[54:32] You can use uh neurosymbolic guardians to enforce in the code.
[54:39] Don't trust in the prompt.
[54:40] And for the hard blocks that I stopped the user, you can use runtime SDR, self-correct, and finish.
[54:51] Each demo that I just showed you is in the in the repository as a notebook and an application as well.
[54:58] Start with the demo one and then go to the demo five.
[55:03] And if you want, deploy it in the productions.
[55:07] So, have you tried any of these in your own hands?
[55:12] Thank you to join me in this session and happy building.