Why Bigger Context Windows Won't Save Your Agent — Elizabeth Fuentes Leone, AWS

AI Engineer · 15 min · 135 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Hi.
00:13Well, first I have to say I am Elizabeth.
00:16I am not Morgan.
00:18But the guy for the agenda they never change agenda.
00:22So we are almost the same but with different colors hair.
00:27Okay.
00:28So it's it's going to be similar.
00:30So today I going to talk about the infinite content window is a meet.
00:37Yes.
00:38Yes is because right now edge system breaks when tools returns a large amount of data.
00:47Context window overflow and the engine start to hallucinate and start the recar is degrad.
00:56So right now context engineering help us to uh give us a better architecture for this agent application.
01:05So we have some techniques that allows us to improve the context the context window for this agentics application.
01:14So imagine that you have a agent that is responsible to uh supervise some application.
01:22So this agent they have to retrieve log logs for that application.
01:28So every time this agent is doing the retrieve for the logs is putting data for the content window and every time this content window is going to grow.
01:38It's going to grow every time because when we have a one session and each session is going to retrieve the last session because it's going to remember everything.
01:48That's the way that agents remember and they use the content window for that.
01:54In the past about probably one year ago before this agentic era we saw that just add more token that will fix it.
02:04Yeah, there's more bigger content window that is going to fix it.
02:08And today we know that is not true because we have the uh attention you curve.
02:17What is this?
02:19We have a lot of data, but if that data is super super big, the engine is going to forget the meter in the in the in the curve.
02:30It's only going to remember the first part and the last part that we sent to that content window.
02:37And I'm talking here about context engineering.
02:41But I can move.
02:43But content engineer is giving the model the information it needs when it needed.
02:48Sometimes the agent doesn't need all the information inside that content window.
02:54Sometimes the content window is poison the engine because you can have a problem injection inside your content window and the content engineer help us to incorout the context window for optimal outcomes and token consumption.
03:11So today I going to show you some techniques that can help you to improve the content window.
03:18And these techniques uh I manage for a strategy.
03:22We have we have the externalize because move this that this big data to a persistent storage.
03:30You can create a memory pointer to storage that data.
03:36I going to share the deck at the end.
03:38So you need to you don't need to have to.
03:40Please pay me attention and because I going to share you all the information that I going to talk share.
03:46And the second one is select you can retrieve only what is relevant.
03:52We have compress that reduce token because you can summarize and you can compact that information.
04:00And the last one is isolate because you can separate context across agents.
04:07and let's start.
04:09So the starting point conversation manager this strategy yeah we have some strategy that are a little complicated but we are going to start for the easy one that we can use inside the framework.
04:24So here in this strategy we are going to use a strand engine is a framework like we are a we maintain is completely open source yes it's free and is a
04:37yeah it's open source free hence it's completely model and so here with strands you can use three different strategy inside the framework so we have sliding window that allows you to keep only the most recent messenger
04:53we have summarization inside the framework that allows you to summarize all their messenger and keep present one and the last one is a combination of two we have that you can combine summarize of older messenger with resent one so first let's see what is a strand
05:13a strand is this so you only need one line of code to create the agent loop so this line here create all the agent loop behind the scene.
05:26You need to create the nose, the graph, nothing because the strands can do that for you and you have the system prom and the tools.
05:36And yes, this is simple because it's using Amazon better behind the scene.
05:40But if you want to use Lamba uh Gemini and tropics, you only need to add one line because you can do that.
05:48You import the the library for OpenAI for example and then you add the model line and that's it.
05:55So let's return to our strategy.
05:59If you want to use summarization conversation manager that's the only line that you that you can add for this agent loop.
06:07So you have the summary radio that is in this case is 50% and you want to preserve preserve the last four messages.
06:17So every time that you invoke this agent in this session, this session is going to manage your content window in this way.
06:25So it's going to keep the four last messages and it's going to summarize the other 50%.
06:33And yes, this is amazing.
06:35But what happen when you have a lot of memory and you don't want to lose you don't want to lose all the memory inside that content window.
06:42So you can use well always we have shortened memory because shortterm memory is when you have the recent conversation history.
06:51So you can use this uh application in the shortterm memory.
06:59But what happen when you want to return and uh you for you uh you have a new session for this agent and you don't have short-term memory.
07:07You can have long-term memory and you can create a vector database for that long-term memory.
07:13You can have different types of memory inside that vector database with different prompts that can understand your conversation and say whatever you want.
07:23And the third one is relation memory.
07:26You can use entity graph.
07:27You can go to the uh to the guy for neo forj that they can explain you more about that.
07:32You can use traversal baset or how things connect.
07:36So you can understand you can use a combination of three of those.
07:40You can have long short-term memory for the me the session that you have.
07:45Then you can retrieve the memory that is in the past the long longterm memory and if you have some relation thing that you want to keep in a different way you can add for that a graph memory as well.
07:59Let's continue with the other one.
08:02So what happened with this application where a lot of logs that I told you you don't want to put that locks inside your content window every time that you invoke the agentic application that agent
08:13you can save that inside a memory pointer.
08:17So what how this work?
08:19You have the the tools.
08:20The tool returns you and a bunch of uh logs.
08:25You can use another tools that is going to capable to took that logs and save that logs in a storage and you are going to retrieve an ID for that storage and for that using a strands you can have that using the agent state.
08:42In the agent state you are going to have all the information for the tools.
08:46Then you can use another tool that is going to capable to understand that logs and it's going to send um it's going to send to the agent hey the logs okay you don't need the logs to understand that but what happen is you need the logs
09:01you can go say hey this is my ID come on retrieve the logs because I needed to know so this is something that you can build in your architecture
09:11and it's something like this you can have two different tools that's the way that we create tools inside the trans only with the tool decorator.
09:20So we have fetch application logs and if you can see there you have the tool context agent step state.
09:28So we retrieve that information and we save that in a memory pointer.
09:33We create a memory pointer ID that is going to preserve inside our content window.
09:39But what happened is your application is super complicated and the tools is not enough.
09:47You can use ah if I forget that you can use a multi- aenting application.
09:53So you can create a a trans swam with multi- aents but you share the context window between agents is like add garbage to the other agents because no all the agents
10:09need the information between them.
10:11So you can share the information between the invocation state pointer.
10:18So you can create an ID between that swam agent swam and you can share the memory pointer between the the invocation state between that aents and that's the way that we create the swamp in a strand agent
10:35and oh my god 9 minutes I going to be fast and the next one is where the next one is what happened when the engine uh want to stop never in sometimes
10:51we have this engine that stay in the loop like forever invoking the same tool the same tool again and it's never ending so you can stop that with a minimum amount on invocation inside the loop
11:05so what is the problem I said the problem to sell more soul may be available so what does happen when you not when you you don't have a clear response
11:16when you evolve the tool So you can avoid that with the tool way with the clear response and with amounts of invocation for the tool and you do like this.
11:26So you have a limited tool count.
11:29So you set max tool counts for this example is my tools search fly.
11:35You can you only can invoke that tool three times and the other tool.
11:41Yeah.
11:41Three time as well.
11:42So the agent never is going to do to do this uh internal loop.
11:48It's only going to invoke that tools three times.
11:53And the last one, yes, I almost there.
11:57The last one is uh is not something that help you to the context window.
12:02is something that you can use to help your content window because sometimes we are invoking IMCP tools or IMCP or we have external API that never answer you at the time that you need it.
12:17Sometimes for example we have EP gateway AWS EP gateway it have a a maximum time that is going to receive the the answer.
12:28So if you don't have that time, it's going to uh like stop your application because the delay.
12:33So you can avoid that using a a synchron.
12:38So when you use synchronous, your engine is going to wait forever for the answer for your NCP.
12:45But if you have an a sync handle, you can invoke your NCP, your external API.
12:51So then you can save that and you can retrieve the answer in the next invocation or when you need it.
12:57Yes, this is going to depend of the case that you are building and you can do that like this create MCP this is a function that use fast API
13:10and with fast API you only need to create the tools.
13:13So I going to have two different tools.
13:15One is the star lob and another one is check job status and I did a lot of test using this uh a sync handle and the different is a lot.
13:28For example, if you have the different um the well the agent never stop to answer me.
13:37So because I wait then he can retrieve the information and to close this session six minutes we have the antipartis so why uh you have to avoid you have to avoid
13:52contact stuffing so you need you don't need to include everything and the kitchen you know you don't have to include everything the content window because the engine doesn't need all the information there
14:04you can only share what a need to create the answer that your user or you're asking for.
14:11You can have uh native summarization.
14:14Sometimes when you do native summarization, the agent uh can uh you can uh forget what you need.
14:23If you're doing whatever superization what you may for probably in that summarization, you are not going to have everything that the engine is needed to answer you.
14:32and context population.
14:34Again, don't pass everything to the agent.
14:37The agent doesn't need all that information.
14:39And what is a good scene here?
14:42So tool returns large data, logs, database, alert information, externalize, select and use a memory pointer.
14:54Many engines need the same large data.
14:56You can share invocation between stage and agent loops or tool hands.
15:02You can have clearest st you know give a clear answer for the tool and limited to counts and a thin handle.
15:13Thank you.
15:16And of course if you want to learn more about stringent we have a huge boot there.
15:21We have a lot of swag.
15:23We have a little Legos for Amazon that you can have and see you there.
15:29We have the specialist and I am Elizabeth No Morgan remember.
15:34[laughter]
15:35Thank you so much.
15:49[music]