Your Agent Is Wasting Tokens and You Don't Know It - Erik Hanchett, AWS

AI Engineer · 5 min · 67 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:00Hey everyone, my name is Eric Hanchett.
  2. 00:02I am a senior developer advocate at AWS and I'm going to talk to you about how you can save on token costs.
  3. 00:09Now, I'm going to show you five ways that you can reduce your token costs while using and creating agents.
  4. 00:18So, the first way you can do that is to cache your system prompt.
  5. 00:24Let me show you some code.
  6. 00:25Now, I'm using AWS's Strands agents.
  7. 00:29This works with all different providers.
  8. 00:32This is a little bit of pseudo code, but the idea is that you can add cache prompt equals default.
  9. 00:39And what that'll do is on the first call of your agent, it will send the full system prompt over and then on every subsequent call, it will have a much reduced
  10. 00:52system prompt being sent over.
  11. 00:54So, it'll be cached.
  12. 00:55You can also cache the tool prompts and messages as well.
  13. 01:01This may sound obvious, but you want to look into routing your different messages based on the difficulty.
  14. 01:09So, here's a code example.
  15. 01:11Let's imagine that you have a task that's very difficult.
  16. 01:15You may want to use one of the newer frontier models.
  17. 01:18However, if it's something simpler, you want to use a cheaper model.
  18. 01:22In this case, maybe we use Claude Haiku for a cheap something cheap and then use Claude Sonnet for something a little bit more difficult.
  19. 01:30And then you can use an if statement.
  20. 01:32You can even have another model that's very cheap decide which model to use.
  21. 01:37So, you can play around with this, but I highly recommend don't use the most expensive model for everything you're doing.
  22. 01:43You want to use multiple different models based on the use case.
  23. 01:47And then try to route to it inside your agent.
  24. 01:51Another good tip is to offload the tool result.
  25. 01:55Let me show you some code on here.
  26. 01:56Once again, I'm using strands agents.
  27. 01:58This is a a manual way to do it.
  28. 02:00There is some additional APIs that the strands agents offers to do this.
  29. 02:06If you have a large tool result that's coming back, you can store it locally or in the cloud and then use some kind of summarization that saves on tokens.
  30. 02:18So that way when it's being called over and over again, the tool result isn't added into the context every time every time the tool loops or every time the agent loops.
  31. 02:30So if you can find any way that where you have this tool result that you don't necessarily send it on every single call back to the large language model, that will save a lot of tokens for you.
  32. 02:41And like I said, there's a few APIs to do this, but it essentially you can do the summarization technique.
  33. 02:47You can also cap your tool loops.
  34. 02:50So when you're dealing with the agent loop and it decides to do a tool call, I've had this happen often where it calls the tool over and over and over again.
  35. 03:00And if you don't cap that tool call, then it might run 10, 20 times.
  36. 03:06It might get into an infinite loop, which would be very bad for your token usage.
  37. 03:10So always set a max iterations of how many times it will loop.
  38. 03:16A good thing you can do before you deploy your agent is to run some observability tools and take a look at the tool call use for every single tool
  39. 03:26and then see how long each one of them is running and how many times they're looping.
  40. 03:31And that way you can get an idea of how efficient the tool call is.
  41. 03:37Last but not least, we can trim the history.
  42. 03:42So if we're using a multi-turn agent and we are talking back and forth, you will find at times that the conversation history will get very large on every single call,
  43. 03:54that whole conversation history will be sent back to the large language model.
  44. 03:59And this can eat through hundreds, if not thousands, of tokens.
  45. 04:03In Strand's agents, we have something called sliding window conversation manager, which which this does is it looks back at the last 10 messages and only sends those back.
  46. 04:14And you can set this to whatever you want.
  47. 04:16And that way you're not sending these huge message histories back to the agent every single time a new message comes in.
  48. 04:25The downfall of this, or the trade-off of this, I should say, is that you will lose the message history from the beginning.
  49. 04:32The way you want to deal with that is you can use uh some sort of summarization of the history and then put that into the context window.
  50. 04:40So, rather than sending all of it, you may send a small amount once you hit this sliding window.
  51. 04:46Uh this will save you a lot of tokens.
  52. 04:50So, in conclusion, we have five things.
  53. 04:53Cache the system prompt.
  54. 04:56And if you can, maybe the tool prompt and messages.
  55. 04:59Route by difficulty.
  56. 05:00Don't use the same expensive model for everything you're doing, for every single task.
  57. 05:05Offload these big tool results.
  58. 05:08So, if you have a large tool result, you can summarize it and not have it being sent in the agent loop and overloading your context.
  59. 05:16You can cap those tool loops.
  60. 05:17Make sure you use observability tools to see how long tool calls are taking and how many loops they're they're doing and and then try to iterate over that.
  61. 05:27And then, of course, trim the history if you're using a multi-turn conversation with your agent, so that way the whole conversation isn't being sent over and over again for very long conversations.
  62. 05:37Thank you for listening to my very quick, almost lightning talk today.
  63. 05:42If you like to go deeper, find me on LinkedIn at Eric Hanchett.
  64. 05:45I also blog at programwitheric.com.
  65. 05:49And then you can find me on social media at EricCH.
  66. 05:52Love to talk to you guys more.
  67. 05:54Thanks.