WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=uiP88SpCi1Q

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/uiP88SpCi1Q.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.000 --> 00:00:02.480
Hey everyone, my name is Eric Hanchett.

s2
00:00:02.480 --> 00:00:09.400
I am a senior developer advocate at AWS and I'm going to talk to you about how you can save on token costs.

s3
00:00:09.400 --> 00:00:18.760
Now, I'm going to show you five ways that you can reduce your token costs while using and creating agents.

s4
00:00:18.760 --> 00:00:24.240
So, the first way you can do that is to cache your system prompt.

s5
00:00:24.240 --> 00:00:25.920
Let me show you some code.

s6
00:00:25.920 --> 00:00:29.840
Now, I'm using AWS's Strands agents.

s7
00:00:29.840 --> 00:00:32.960
This works with all different providers.

s8
00:00:32.960 --> 00:00:39.320
This is a little bit of pseudo code, but the idea is that you can add cache prompt equals default.

s9
00:00:39.320 --> 00:00:52.200
And what that'll do is on the first call of your agent, it will send the full system prompt over and then on every subsequent call, it will have a much reduced

s10
00:00:52.200 --> 00:00:54.360
system prompt being sent over.

s11
00:00:54.360 --> 00:00:55.800
So, it'll be cached.

s12
00:00:55.800 --> 00:01:01.080
You can also cache the tool prompts and messages as well.

s13
00:01:01.080 --> 00:01:09.200
This may sound obvious, but you want to look into routing your different messages based on the difficulty.

s14
00:01:09.200 --> 00:01:11.440
So, here's a code example.

s15
00:01:11.440 --> 00:01:15.360
Let's imagine that you have a task that's very difficult.

s16
00:01:15.360 --> 00:01:18.680
You may want to use one of the newer frontier models.

s17
00:01:18.680 --> 00:01:22.920
However, if it's something simpler, you want to use a cheaper model.

s18
00:01:22.920 --> 00:01:30.120
In this case, maybe we use Claude Haiku for a cheap something cheap and then use Claude Sonnet for something a little bit more difficult.

s19
00:01:30.120 --> 00:01:32.600
And then you can use an if statement.

s20
00:01:32.600 --> 00:01:37.560
You can even have another model that's very cheap decide which model to use.

s21
00:01:37.560 --> 00:01:43.280
So, you can play around with this, but I highly recommend don't use the most expensive model for everything you're doing.

s22
00:01:43.280 --> 00:01:47.160
You want to use multiple different models based on the use case.

s23
00:01:47.160 --> 00:01:51.120
And then try to route to it inside your agent.

s24
00:01:51.120 --> 00:01:55.040
Another good tip is to offload the tool result.

s25
00:01:55.040 --> 00:01:56.720
Let me show you some code on here.

s26
00:01:56.720 --> 00:01:58.760
Once again, I'm using strands agents.

s27
00:01:58.760 --> 00:02:00.320
This is a a manual way to do it.

s28
00:02:00.320 --> 00:02:06.600
There is some additional APIs that the strands agents offers to do this.

s29
00:02:06.600 --> 00:02:18.560
If you have a large tool result that's coming back, you can store it locally or in the cloud and then use some kind of summarization that saves on tokens.

s30
00:02:18.560 --> 00:02:30.480
So that way when it's being called over and over again, the tool result isn't added into the context every time every time the tool loops or every time the agent loops.

s31
00:02:30.480 --> 00:02:41.400
So if you can find any way that where you have this tool result that you don't necessarily send it on every single call back to the large language model, that will save a lot of tokens for you.

s32
00:02:41.400 --> 00:02:47.320
And like I said, there's a few APIs to do this, but it essentially you can do the summarization technique.

s33
00:02:47.320 --> 00:02:50.400
You can also cap your tool loops.

s34
00:02:50.400 --> 00:03:00.200
So when you're dealing with the agent loop and it decides to do a tool call, I've had this happen often where it calls the tool over and over and over again.

s35
00:03:00.200 --> 00:03:06.440
And if you don't cap that tool call, then it might run 10, 20 times.

s36
00:03:06.440 --> 00:03:10.840
It might get into an infinite loop, which would be very bad for your token usage.

s37
00:03:10.840 --> 00:03:16.000
So always set a max iterations of how many times it will loop.

s38
00:03:16.000 --> 00:03:26.600
A good thing you can do before you deploy your agent is to run some observability tools and take a look at the tool call use for every single tool

s39
00:03:26.600 --> 00:03:31.480
and then see how long each one of them is running and how many times they're looping.

s40
00:03:31.480 --> 00:03:37.239
And that way you can get an idea of how efficient the tool call is.

s41
00:03:37.239 --> 00:03:42.160
Last but not least, we can trim the history.

s42
00:03:42.400 --> 00:03:54.600
So if we're using a multi-turn agent and we are talking back and forth, you will find at times that the conversation history will get very large on every single call,

s43
00:03:54.600 --> 00:03:59.120
that whole conversation history will be sent back to the large language model.

s44
00:03:59.120 --> 00:04:03.440
And this can eat through hundreds, if not thousands, of tokens.

s45
00:04:03.440 --> 00:04:14.200
In Strand's agents, we have something called sliding window conversation manager, which which this does is it looks back at the last 10 messages and only sends those back.

s46
00:04:14.200 --> 00:04:16.720
And you can set this to whatever you want.

s47
00:04:16.720 --> 00:04:25.480
And that way you're not sending these huge message histories back to the agent every single time a new message comes in.

s48
00:04:25.480 --> 00:04:32.800
The downfall of this, or the trade-off of this, I should say, is that you will lose the message history from the beginning.

s49
00:04:32.800 --> 00:04:40.800
The way you want to deal with that is you can use uh some sort of summarization of the history and then put that into the context window.

s50
00:04:40.800 --> 00:04:46.720
So, rather than sending all of it, you may send a small amount once you hit this sliding window.

s51
00:04:46.720 --> 00:04:50.440
Uh this will save you a lot of tokens.

s52
00:04:50.640 --> 00:04:53.880
So, in conclusion, we have five things.

s53
00:04:53.880 --> 00:04:56.000
Cache the system prompt.

s54
00:04:56.000 --> 00:04:59.200
And if you can, maybe the tool prompt and messages.

s55
00:04:59.200 --> 00:05:00.480
Route by difficulty.

s56
00:05:00.480 --> 00:05:05.120
Don't use the same expensive model for everything you're doing, for every single task.

s57
00:05:05.120 --> 00:05:08.120
Offload these big tool results.

s58
00:05:08.120 --> 00:05:16.320
So, if you have a large tool result, you can summarize it and not have it being sent in the agent loop and overloading your context.

s59
00:05:16.320 --> 00:05:17.720
You can cap those tool loops.

s60
00:05:17.720 --> 00:05:27.000
Make sure you use observability tools to see how long tool calls are taking and how many loops they're they're doing and and then try to iterate over that.

s61
00:05:27.000 --> 00:05:37.680
And then, of course, trim the history if you're using a multi-turn conversation with your agent, so that way the whole conversation isn't being sent over and over again for very long conversations.

s62
00:05:37.680 --> 00:05:42.320
Thank you for listening to my very quick, almost lightning talk today.

s63
00:05:42.320 --> 00:05:45.520
If you like to go deeper, find me on LinkedIn at Eric Hanchett.

s64
00:05:45.520 --> 00:05:49.120
I also blog at programwitheric.com.

s65
00:05:49.760 --> 00:05:52.760
And then you can find me on social media at EricCH.

s66
00:05:52.760 --> 00:05:54.040
Love to talk to you guys more.

s67
00:05:54.040 --> 00:05:56.040
Thanks.