WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=DrfyORO8RqA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/DrfyORO8RqA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.559 --> 00:00:13.840
Hi.

s2
00:00:13.840 --> 00:00:16.800
Well, first I have to say I am Elizabeth.

s3
00:00:16.800 --> 00:00:18.640
I am not Morgan.

s4
00:00:18.640 --> 00:00:22.240
But the guy for the agenda they never change agenda.

s5
00:00:22.240 --> 00:00:27.279
So we are almost the same but with different colors hair.

s6
00:00:27.279 --> 00:00:28.240
Okay.

s7
00:00:28.240 --> 00:00:30.720
So it's it's going to be similar.

s8
00:00:30.720 --> 00:00:37.280
So today I going to talk about the infinite content window is a meet.

s9
00:00:37.280 --> 00:00:38.559
Yes.

s10
00:00:38.559 --> 00:00:47.680
Yes is because right now edge system breaks when tools returns a large amount of data.

s11
00:00:47.680 --> 00:00:56.320
Context window overflow and the engine start to hallucinate and start the recar is degrad.

s12
00:00:56.320 --> 00:01:05.680
So right now context engineering help us to uh give us a better architecture for this agent application.

s13
00:01:05.680 --> 00:01:14.240
So we have some techniques that allows us to improve the context the context window for this agentics application.

s14
00:01:14.240 --> 00:01:22.640
So imagine that you have a agent that is responsible to uh supervise some application.

s15
00:01:22.640 --> 00:01:28.080
So this agent they have to retrieve log logs for that application.

s16
00:01:28.080 --> 00:01:38.799
So every time this agent is doing the retrieve for the logs is putting data for the content window and every time this content window is going to grow.

s17
00:01:38.799 --> 00:01:48.399
It's going to grow every time because when we have a one session and each session is going to retrieve the last session because it's going to remember everything.

s18
00:01:48.399 --> 00:01:54.000
That's the way that agents remember and they use the content window for that.

s19
00:01:54.000 --> 00:02:04.479
In the past about probably one year ago before this agentic era we saw that just add more token that will fix it.

s20
00:02:04.479 --> 00:02:08.720
Yeah, there's more bigger content window that is going to fix it.

s21
00:02:08.720 --> 00:02:17.680
And today we know that is not true because we have the uh attention you curve.

s22
00:02:17.680 --> 00:02:19.120
What is this?

s23
00:02:19.120 --> 00:02:30.000
We have a lot of data, but if that data is super super big, the engine is going to forget the meter in the in the in the curve.

s24
00:02:30.000 --> 00:02:37.200
It's only going to remember the first part and the last part that we sent to that content window.

s25
00:02:37.200 --> 00:02:41.040
And I'm talking here about context engineering.

s26
00:02:41.040 --> 00:02:43.040
But I can move.

s27
00:02:43.040 --> 00:02:48.800
But content engineer is giving the model the information it needs when it needed.

s28
00:02:48.800 --> 00:02:54.080
Sometimes the agent doesn't need all the information inside that content window.

s29
00:02:54.080 --> 00:03:11.440
Sometimes the content window is poison the engine because you can have a problem injection inside your content window and the content engineer help us to incorout the context window for optimal outcomes and token consumption.

s30
00:03:11.440 --> 00:03:18.239
So today I going to show you some techniques that can help you to improve the content window.

s31
00:03:18.239 --> 00:03:22.720
And these techniques uh I manage for a strategy.

s32
00:03:22.720 --> 00:03:30.879
We have we have the externalize because move this that this big data to a persistent storage.

s33
00:03:30.879 --> 00:03:36.159
You can create a memory pointer to storage that data.

s34
00:03:36.159 --> 00:03:38.080
I going to share the deck at the end.

s35
00:03:38.080 --> 00:03:40.159
So you need to you don't need to have to.

s36
00:03:40.159 --> 00:03:46.239
Please pay me attention and because I going to share you all the information that I going to talk share.

s37
00:03:46.239 --> 00:03:52.640
And the second one is select you can retrieve only what is relevant.

s38
00:03:52.640 --> 00:04:00.480
We have compress that reduce token because you can summarize and you can compact that information.

s39
00:04:00.480 --> 00:04:07.120
And the last one is isolate because you can separate context across agents.

s40
00:04:07.120 --> 00:04:09.840
and let's start.

s41
00:04:09.840 --> 00:04:24.160
So the starting point conversation manager this strategy yeah we have some strategy that are a little complicated but we are going to start for the easy one that we can use inside the framework.

s42
00:04:24.160 --> 00:04:37.759
So here in this strategy we are going to use a strand engine is a framework like we are a we maintain is completely open source yes it's free and is a

s43
00:04:37.759 --> 00:04:53.360
yeah it's open source free hence it's completely model and so here with strands you can use three different strategy inside the framework so we have sliding window that allows you to keep only the most recent messenger

s44
00:04:53.360 --> 00:05:13.600
we have summarization inside the framework that allows you to summarize all their messenger and keep present one and the last one is a combination of two we have that you can combine summarize of older messenger with resent one so first let's see what is a strand

s45
00:05:13.600 --> 00:05:26.240
a strand is this so you only need one line of code to create the agent loop so this line here create all the agent loop behind the scene.

s46
00:05:26.240 --> 00:05:36.080
You need to create the nose, the graph, nothing because the strands can do that for you and you have the system prom and the tools.

s47
00:05:36.080 --> 00:05:40.639
And yes, this is simple because it's using Amazon better behind the scene.

s48
00:05:40.639 --> 00:05:48.400
But if you want to use Lamba uh Gemini and tropics, you only need to add one line because you can do that.

s49
00:05:48.400 --> 00:05:55.759
You import the the library for OpenAI for example and then you add the model line and that's it.

s50
00:05:55.759 --> 00:05:59.039
So let's return to our strategy.

s51
00:05:59.039 --> 00:06:07.840
If you want to use summarization conversation manager that's the only line that you that you can add for this agent loop.

s52
00:06:07.840 --> 00:06:17.120
So you have the summary radio that is in this case is 50% and you want to preserve preserve the last four messages.

s53
00:06:17.120 --> 00:06:25.840
So every time that you invoke this agent in this session, this session is going to manage your content window in this way.

s54
00:06:25.840 --> 00:06:33.360
So it's going to keep the four last messages and it's going to summarize the other 50%.

s55
00:06:33.360 --> 00:06:35.600
And yes, this is amazing.

s56
00:06:35.600 --> 00:06:42.800
But what happen when you have a lot of memory and you don't want to lose you don't want to lose all the memory inside that content window.

s57
00:06:42.800 --> 00:06:51.840
So you can use well always we have shortened memory because shortterm memory is when you have the recent conversation history.

s58
00:06:51.840 --> 00:06:59.120
So you can use this uh application in the shortterm memory.

s59
00:06:59.120 --> 00:07:07.360
But what happen when you want to return and uh you for you uh you have a new session for this agent and you don't have short-term memory.

s60
00:07:07.360 --> 00:07:13.360
You can have long-term memory and you can create a vector database for that long-term memory.

s61
00:07:13.360 --> 00:07:23.039
You can have different types of memory inside that vector database with different prompts that can understand your conversation and say whatever you want.

s62
00:07:23.039 --> 00:07:26.000
And the third one is relation memory.

s63
00:07:26.000 --> 00:07:27.840
You can use entity graph.

s64
00:07:27.840 --> 00:07:32.639
You can go to the uh to the guy for neo forj that they can explain you more about that.

s65
00:07:32.639 --> 00:07:36.400
You can use traversal baset or how things connect.

s66
00:07:36.400 --> 00:07:40.479
So you can understand you can use a combination of three of those.

s67
00:07:40.479 --> 00:07:45.120
You can have long short-term memory for the me the session that you have.

s68
00:07:45.120 --> 00:07:59.520
Then you can retrieve the memory that is in the past the long longterm memory and if you have some relation thing that you want to keep in a different way you can add for that a graph memory as well.

s69
00:07:59.520 --> 00:08:02.080
Let's continue with the other one.

s70
00:08:02.080 --> 00:08:13.599
So what happened with this application where a lot of logs that I told you you don't want to put that locks inside your content window every time that you invoke the agentic application that agent

s71
00:08:13.599 --> 00:08:17.199
you can save that inside a memory pointer.

s72
00:08:17.199 --> 00:08:19.440
So what how this work?

s73
00:08:19.440 --> 00:08:20.879
You have the the tools.

s74
00:08:20.879 --> 00:08:25.840
The tool returns you and a bunch of uh logs.

s75
00:08:25.840 --> 00:08:42.320
You can use another tools that is going to capable to took that logs and save that logs in a storage and you are going to retrieve an ID for that storage and for that using a strands you can have that using the agent state.

s76
00:08:42.320 --> 00:08:46.399
In the agent state you are going to have all the information for the tools.

s77
00:08:46.399 --> 00:09:01.920
Then you can use another tool that is going to capable to understand that logs and it's going to send um it's going to send to the agent hey the logs okay you don't need the logs to understand that but what happen is you need the logs

s78
00:09:01.920 --> 00:09:11.600
you can go say hey this is my ID come on retrieve the logs because I needed to know so this is something that you can build in your architecture

s79
00:09:11.600 --> 00:09:20.720
and it's something like this you can have two different tools that's the way that we create tools inside the trans only with the tool decorator.

s80
00:09:20.720 --> 00:09:28.959
So we have fetch application logs and if you can see there you have the tool context agent step state.

s81
00:09:28.959 --> 00:09:33.600
So we retrieve that information and we save that in a memory pointer.

s82
00:09:33.600 --> 00:09:39.519
We create a memory pointer ID that is going to preserve inside our content window.

s83
00:09:39.519 --> 00:09:47.040
But what happened is your application is super complicated and the tools is not enough.

s84
00:09:47.040 --> 00:09:53.279
You can use ah if I forget that you can use a multi- aenting application.

s85
00:09:53.279 --> 00:10:09.040
So you can create a a trans swam with multi- aents but you share the context window between agents is like add garbage to the other agents because no all the agents

s86
00:10:09.040 --> 00:10:11.440
need the information between them.

s87
00:10:11.440 --> 00:10:18.480
So you can share the information between the invocation state pointer.

s88
00:10:18.480 --> 00:10:35.920
So you can create an ID between that swam agent swam and you can share the memory pointer between the the invocation state between that aents and that's the way that we create the swamp in a strand agent

s89
00:10:35.920 --> 00:10:51.120
and oh my god 9 minutes I going to be fast and the next one is where the next one is what happened when the engine uh want to stop never in sometimes

s90
00:10:51.120 --> 00:11:05.440
we have this engine that stay in the loop like forever invoking the same tool the same tool again and it's never ending so you can stop that with a minimum amount on invocation inside the loop

s91
00:11:05.440 --> 00:11:16.640
so what is the problem I said the problem to sell more soul may be available so what does happen when you not when you you don't have a clear response

s92
00:11:16.640 --> 00:11:26.720
when you evolve the tool So you can avoid that with the tool way with the clear response and with amounts of invocation for the tool and you do like this.

s93
00:11:26.720 --> 00:11:29.760
So you have a limited tool count.

s94
00:11:29.760 --> 00:11:35.760
So you set max tool counts for this example is my tools search fly.

s95
00:11:35.760 --> 00:11:41.040
You can you only can invoke that tool three times and the other tool.

s96
00:11:41.040 --> 00:11:41.360
Yeah.

s97
00:11:41.360 --> 00:11:42.800
Three time as well.

s98
00:11:42.800 --> 00:11:48.720
So the agent never is going to do to do this uh internal loop.

s99
00:11:48.720 --> 00:11:53.279
It's only going to invoke that tools three times.

s100
00:11:53.279 --> 00:11:57.040
And the last one, yes, I almost there.

s101
00:11:57.040 --> 00:12:02.399
The last one is uh is not something that help you to the context window.

s102
00:12:02.399 --> 00:12:17.920
is something that you can use to help your content window because sometimes we are invoking IMCP tools or IMCP or we have external API that never answer you at the time that you need it.

s103
00:12:17.920 --> 00:12:28.000
Sometimes for example we have EP gateway AWS EP gateway it have a a maximum time that is going to receive the the answer.

s104
00:12:28.000 --> 00:12:33.680
So if you don't have that time, it's going to uh like stop your application because the delay.

s105
00:12:33.680 --> 00:12:38.480
So you can avoid that using a a synchron.

s106
00:12:38.480 --> 00:12:45.200
So when you use synchronous, your engine is going to wait forever for the answer for your NCP.

s107
00:12:45.200 --> 00:12:51.519
But if you have an a sync handle, you can invoke your NCP, your external API.

s108
00:12:51.519 --> 00:12:57.680
So then you can save that and you can retrieve the answer in the next invocation or when you need it.

s109
00:12:57.680 --> 00:13:10.079
Yes, this is going to depend of the case that you are building and you can do that like this create MCP this is a function that use fast API

s110
00:13:10.079 --> 00:13:13.760
and with fast API you only need to create the tools.

s111
00:13:13.760 --> 00:13:15.920
So I going to have two different tools.

s112
00:13:15.920 --> 00:13:28.959
One is the star lob and another one is check job status and I did a lot of test using this uh a sync handle and the different is a lot.

s113
00:13:28.959 --> 00:13:37.360
For example, if you have the different um the well the agent never stop to answer me.

s114
00:13:37.360 --> 00:13:52.639
So because I wait then he can retrieve the information and to close this session six minutes we have the antipartis so why uh you have to avoid you have to avoid

s115
00:13:52.639 --> 00:14:04.560
contact stuffing so you need you don't need to include everything and the kitchen you know you don't have to include everything the content window because the engine doesn't need all the information there

s116
00:14:04.560 --> 00:14:11.760
you can only share what a need to create the answer that your user or you're asking for.

s117
00:14:11.760 --> 00:14:14.959
You can have uh native summarization.

s118
00:14:14.959 --> 00:14:23.199
Sometimes when you do native summarization, the agent uh can uh you can uh forget what you need.

s119
00:14:23.199 --> 00:14:32.000
If you're doing whatever superization what you may for probably in that summarization, you are not going to have everything that the engine is needed to answer you.

s120
00:14:32.000 --> 00:14:34.240
and context population.

s121
00:14:34.240 --> 00:14:37.279
Again, don't pass everything to the agent.

s122
00:14:37.279 --> 00:14:39.920
The agent doesn't need all that information.

s123
00:14:39.920 --> 00:14:42.480
And what is a good scene here?

s124
00:14:42.480 --> 00:14:54.160
So tool returns large data, logs, database, alert information, externalize, select and use a memory pointer.

s125
00:14:54.160 --> 00:14:56.880
Many engines need the same large data.

s126
00:14:56.880 --> 00:15:02.240
You can share invocation between stage and agent loops or tool hands.

s127
00:15:02.240 --> 00:15:13.600
You can have clearest st you know give a clear answer for the tool and limited to counts and a thin handle.

s128
00:15:13.600 --> 00:15:15.839
Thank you.

s129
00:15:16.399 --> 00:15:21.680
And of course if you want to learn more about stringent we have a huge boot there.

s130
00:15:21.680 --> 00:15:23.199
We have a lot of swag.

s131
00:15:23.199 --> 00:15:29.600
We have a little Legos for Amazon that you can have and see you there.

s132
00:15:29.600 --> 00:15:34.040
We have the specialist and I am Elizabeth No Morgan remember.

s133
00:15:34.040 --> 00:15:35.120
[laughter]

s134
00:15:35.120 --> 00:15:37.680
Thank you so much.

s135
00:15:49.760 --> 00:15:51.760
[music]