The 100-Tool Agent Is a Trap - Sohail Shaikh & Ankush Rastogi, Prosodica

AI Engineer · 28 min · 271 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01Hi everyone.
2. 00:02Thanks for being here.
3. 00:04I am Sohaib here and along with me is Ankush.
4. 00:07So, today we'll be talking about a mistake that looks harmless at first.
5. 00:13Giving which is basically giving an AI agent every tool access it might ever need all at once.
6. 00:20So, basically that approach works well in a demo.
7. 00:24It might even work with a small number of tools like like say for example 10 tools.
8. 00:30But once the catalog grows, the agent gets slower.
9. 00:36It might become more expensive and less accurate as well.
10. 00:40That is why we are calling it the 100 tool agent trap.
11. 00:43In the next half an hour or so, we'll show why it breaks, what the numbers look like, and how semantic routing with just-in-time context can help us fix this problem.
12. 00:57So, a quick introduction about myself.
13. 01:00I am Sohaib Shaikh.
14. 01:02I'm currently working as a data scientist with Presodica.
15. 01:06My background spans across AI, NLP, marketing, analytics, and even engineering.
16. 01:13My current focus is on applied AI, NLP, and conversational intelligence along with rack systems.
17. 01:20I'm especially interested in making AI systems more reliable, measurable, and even scalable beyond the demos in production.
18. 01:31And I'm Ankush Astogi.
19. 01:32I work as a senior data solutions engineer at Presodica.
20. 01:38I have spent more than a decade in AI, data engineering, and production systems.
21. 01:43My focus is the engineering side.
22. 01:47So, it's not about what's going to work in notebook, but whether it's going to survive in with real load, real user, and real failures.
23. 01:59So, that is the angle we are taking today.
24. 02:04Suhail will focus more on the model and routing behavior, and I will focus more on system design, implementation, and production trade-offs.
25. 02:15Awesome.
26. 02:15Thank you, Ankush.
27. 02:17Um so, let's get into it.
28. 02:20So, let's imagine a common design.
29. 02:22Um you tend to build a system, and it can do many things.
30. 02:28Say, for example, querying a database, sending an email, um even checking a calendar, or looking up an order, calling an API, and so on and so forth.
31. 02:38The simplest approach over here would be to give a model every tool definition on every request.
32. 02:46Every function name, um every description, and even every JSON schema will go into the prompt, whether the user might need it or not.
33. 02:55So, that We are calling that as a fat agent.
34. 02:57At small scale, it feels fine.
35. 03:00With 10 tools as well, the model might usually pick the right one.
36. 03:04The demo looks good.
37. 03:07Then, the product grows.
38. 03:08Um 10 tools might become 30, or it will keep on increasing, and eventually the model starts calling the wrong function, and starts confusing similar tools, may invent tool names, and even take longer to respond.
39. 03:28The important point is, basically, the design does not fail because one tool is badly written.
40. 03:35It fails because every request is forced to carry the entire catalog.
41. 03:40So, let's look here.
42. 03:42There are, say for example, uh 741 tools in your uh in your entire schema, but and it will look, basically, take up to 127,000 tokens just to have all those tool descriptions in it.
43. 04:00And this is even before the user user's actual question is even considered.
44. 04:06So, basically, this will lead to context overload and uh we need to manage that properly.
45. 04:15So, on this slide we see why it is failing and why the accuracy collapses beyond a point.
46. 04:22So, when you look at the accuracy curve with the 10 tools, fat agent will get the tools right almost 78% of the times.
47. 04:31That is not perfect, but it's usable.
48. 04:35At almost 100 tools, the accuracy drops to around 40%.
49. 04:39Less than half of the tools that are called are the correct tools.
50. 04:45And if it grows beyond that, like say for example, in over here, at 741 tools, the accuracy will be a mere 13.6%.
51. 04:55So, in short, it's roughly one correct tool out of eight tools.
52. 05:00So, when we compare it with the semantic router, semantic router behaves very differently.
53. 05:05It stays above 83% across the same catalog sizes.
54. 05:11That is because the model is not choosing from hundreds of tools, it's choosing from a small and relevant set.
55. 05:18One reason the fat agent fails is because it's lost in the middle problem.
56. 05:24Model pays stronger attention to the beginning and end of the long context.
57. 05:29When hundreds of tool schemas are packed into the middle, the model does not reliably use them.
58. 05:35So, we end up paying paying for a huge prompt, and that prompts makes the decisions even harder.
59. 05:45So, two more reasons over here.
60. 05:48First is the latency and the second is the cost.
61. 05:51So, we saw in the earlier slide that accuracy was a big problem.
62. 05:56Another issue is with latency and cost over here as we can see.
63. 06:03So, like say for example, if we have 741 tools, we saw that it requires almost 127,000 tokens.
64. 06:11That will include the tool description and the schema text.
65. 06:15So, that cost is basically being paid on every request.
66. 06:20At 100,000 requests a day, if you push this into production, you are sending in billions of tokens and just to describe those tools.
67. 06:30And with just-in-time routing, the prompt may include only three to five relevant schemas, closer to about 1,000 tokens.
68. 06:38That is roughly a 99% reduction in tool context tokens.
69. 06:44Another issue is latency.
70. 06:46With fat agent approach, time to first token grows as the tool catalog grows because the model has to process a larger prompt before it can answer the user question.
71. 06:57So, say for example, if you have 500 tools in your agent, the fat agent path can push first token latency past 5 seconds.
72. 07:08So, this is specifically important if your product is real-time because this will take longer time to respond.
73. 07:18So, which will make the design feel slow and unpredictable.
74. 07:21So, and with semantic routing, it might feel more responsive and it will be more real-time in a way.
75. 07:33Here is the clean comparison.
76. 07:35So, on the left side, we have a fat agent design and on the right side, we have our semantic router design.
77. 07:43So, in the fat agent design, every schema is loaded for every request, no matter what the user asked.
78. 07:54And because of that, the catalog will grow, the prompt will grow, due to which the latency will rise and accuracy will drop.
79. 08:04The agent also becomes one large monolithic, which makes it hard to test, risky to update, and even painful to debug.
80. 08:16But, in the case of semantic routing design, the agent does not start with every tool.
81. 08:23So, in this case, the router first look at the user's query.
82. 08:29It retrieves the three to five most relevant tool, and then it injects only those into the model call.
83. 08:38So, the context stays small, latency stays stable, and the accuracy holds because the model is choosing from a focused list instead of giant catalog.
84. 08:51This is Okay, so there is a one important caveat here.
85. 08:56If you have fewer than 20 tools, a router may be unnecessary.
86. 09:02Just load the tools directly.
87. 09:04But, once you passed 50 tools in the production system, then justifying router-based schema make more sense.
88. 09:19Thanks, Ankush.
89. 09:20So, let's now look into how semantic routing works.
90. 09:25If you have built a rack system previously, this will feel pretty similar to that.
91. 09:30The difference is that we will retrieve tools instead of the documents.
92. 09:35So, at the first stage, say for example, over here, each tool will have a clear description.
93. 09:42For example, a search flights tool, uh a check calendar availability tool, or a retrieve customer order status.
94. 09:50Say for example, the second step would be those descriptions needs to get embedded and stored in a vector index.
95. 09:59This is usually done offline when the tool catalog is created or updated.
96. 10:06At the runtime, the user will ask a question, we embed the user query with the same embedding model.
97. 10:15Then, we search that vector index for the tools with the description that are closest to the query.
98. 10:23So, the router will then return the top K tools.
99. 10:27Often, K would be three or five tools that matches the user query.
100. 10:32And only those selected schemas will get injected into the model call.
101. 10:38So, this is the whole pattern.
102. 10:40At the first step, you need to index tool descriptions offline.
103. 10:45Then, retrieve relevant tools at runtime and keep the model's context focused.
104. 10:51So, in short, the idea is pretty simple.
105. 10:55Semantic routing is basically a rack for tools.
106. 10:58Um if you already have an embedding model and vector database in your stack, most of the infrastructure is already familiar to you.
107. 11:10Thank you, Sohil.
108. 11:11Now, we will deep dive into just-in-time context injection.
109. 11:16So, semantic routing is the retrieval layer.
110. 11:21And just-in-time context injection is the context management strategy.
111. 11:27So, in the case of fat agent approach, it loads everything before the query is understood.
112. 11:35Due to which the model receives a huge list of tools first, and then it tries to reason through it.
113. 11:45Just-in-time context does the reverse.
114. 11:48It will wait until the query is known, then it will inject only the context needed for that request.
115. 11:58I mean, this is not a new software idea.
116. 12:01We have used lazy loading, just-in-time compilation, and on-demand resource loading from years.
117. 12:10We are just applying the simple and same principle to the LLM context.
118. 12:16Anthropic has written about this pattern with on-demand tooling loading through MCP.
119. 12:24Their report token usage went from 150k tokens down to 2,000, which is 98.7% token reduction.
120. 12:36That is the signal.
121. 12:37Large tool catalog should not be dumped into the into every prompt.
122. 12:43They should be retrieved when needed.
123. 12:47So, moving to the next slide, let's talk about how to evaluate this fairly.
124. 12:56We measure four things here: tool selection accuracy, time to first token, input tokens per request, and the estimated cost per thousand call.
125. 13:09So, here for data sets, we used Berkeley function calling leaderboard, Skills bench style scenarios, and synthetic tool pools that allow us to scale the number of tools, so we can test
126. 13:24at 10, 50, 100, 200, or even 1041 tools.
127. 13:30So, we ran the same queries in two modes, the fat agent one and the semantic routing.
128. 13:37Same model, same answer key, same tool catalog.
129. 13:42The only difference was whether the model saw every tool or only the routed tool.
130. 13:49We also swept K at 3, 5, 10 to understand the trade-off.
131. 13:56So, the smaller K is the faster and cheaper.
132. 13:59Larger K can be the can recover more edge cases.
133. 14:05In practice, K equals 5 is a strong default starting point.
134. 14:09So, if you are want to try this, you can default it to five.
135. 14:15Here, K means the number of tools the semantic router retrieves and gives to the LLM for each user queries.
136. 14:26Here are the benchmark result that we got.
137. 14:30So, here is the result on the left side.
138. 14:35We see the fat agent accuracy line drop sharply as the tool count grows.
139. 14:42It started around 78% at 10 tool and then falls to around 13% at 1041 tools.
140. 14:51But, in the case of router, the router line stays above 83% because the model stays stable.
141. 15:00From the model points of view, it always choosing the handful of tool, even if the real catalog contains 100.
142. 15:13On the right side, time to first token shows the same story.
143. 15:19The fat agent path gets slower as more tool schemas are added.
144. 15:26At large catalog sizes, the model spend noticeable time just processing the prompt.
145. 15:34But in case of router, it stays almost flat because of the prompt size is under controlled.
146. 15:43This is the core lesson from the benchmark.
147. 15:47The catalog can grow, but the model's working set should stay small.
148. 15:54Now, on the next slide, we'll be discussing about the implementation pattern.
149. 16:00There are three steps in it.
150. 16:03Step one is offline.
151. 16:05So, build a catalog of tool.
152. 16:08For each tool, store the name, description, schema.
153. 16:13Then, embed the description and store it into a vector database.
154. 16:19You can use Chroma DB, you can use Pinecone, you can use Quadrant.
155. 16:25Whichever vector database you are using.
156. 16:28Then, step two will happen on every request.
157. 16:33Embed the user's query.
158. 16:35Then, run the nearest neighbor search.
159. 16:38And return the top K tools.
160. 16:42In the step three, it will also happen on every request.
161. 16:48Fetch the schema for those selected tools.
162. 16:52And then, pass only those schemas into the model call and log which tool was selected.
163. 16:59So, here, logging really matters.
164. 17:02So, let's say if you miss something and if you want to improve the description or you want to tune the K, you need a good logging system that you can use it.
165. 17:16So, the runtime overhead is small, one embedding call, and one vector search.
166. 17:24The payoff is much smaller prompt and more stable tool selection problem.
167. 17:33Awesome.
168. 17:34So, now that you have learned about the implementation pattern, I'll take you through the code version of it, which is pretty straightforward.
169. 17:45So, initially you need to loop through the tool catalog, and for each tool, as Ankur mentioned, embed the description, store that in the vector database with the tool name and other details.
170. 17:57At run time, embed that user query, compare that query vector against the stored tool vectors using cosine similarity or vector search, and then retrieve the top most relevant tools,
171. 18:13and then call the model only with those schemas and the function or tool parameter.
172. 18:19That final line is the most important part because the model does not receive all the tools, it only receive the routed tools.
173. 18:26This works with any embedding model and at and with any vector database.
174. 18:31You can start, as Ankur mentioned, you can start with Quadrant or any other database vector database locally, and then move to a more managed vector store later if needed.
175. 18:44If you already have an existing rag infrastructure, this is not a new infrastructure as we mentioned earlier, it is the same retrieval pattern applied to the tool selection.
176. 18:56Thank you, Suhail.
177. 18:58Uh Now, we will discuss semantic routing in action.
178. 19:03So, imagine you have a 200 tool agent.
179. 19:07A user asks, "Find me a flight to New York next Wednesday.
180. 19:13The router embeds that query.
181. 19:16Returns tool like search flights, book flight, calendar check.
182. 19:21The model sees only those relevant schema, so it is much more likely to call the correct tool.
183. 19:29But, if we load all 200 tools, hotels tool, weather tool, emails tool, sequels tool, or any unrelated workflow tool, it would all be competing for attention.
184. 19:44That is when model starts choosing the wrong function.
185. 19:49Now, let's take a second query.
186. 19:52What is the weather in Paris right now?
187. 19:55The router returns get weather and get forecast.
188. 20:00Flights tools are not injected.
189. 20:03Hotels tools are not injected.
190. 20:06They are simply not available at this request.
191. 20:10This is an underrated benefit.
192. 20:13But, the router does not only add the right tool, it also removes the wrong tool from the model's choice set.
193. 20:22Now, in the next slide, we will discuss implementation checklist.
194. 20:28So, here is the production checklist if somebody wants to uh follow this along.
195. 20:34So, first, get locked the tool in one place.
196. 20:39So, add name, description, schema, owner, and the version.
197. 20:45Second is to build the index.
198. 20:48Embed each description and store the vectors.
199. 20:53The third one is write the router.
200. 20:57So, in this, embed the query, then search the index, and take top K and fetch schema.
201. 21:06The next one is the fourth one in which a user need to wire it up into the agent loop.
202. 21:13So, the model tool list should come from the router, not from a hard-coded full catalog.
203. 21:21Fifth is to evaluate.
204. 21:23Run your test set at K equals to three, five, and 10, and then pick the smallest K that meets your accuracy target.
205. 21:36Sixth is to monitor production.
206. 21:39Log selected tools, final tool calls, failure, and fallback usage.
207. 21:47Then re-embed tools when descriptions or schema change.
208. 21:52This is not the 6-month platform rewrite.
209. 21:56For most team, it is a focused sprint.
210. 22:00So, in the next slide, we will discuss what community already has confirmed.
211. 22:09This is not just the observation.
212. 22:11So, Anthropic has published on-demand tool loading with MCP, and reported a drop of 150k tokens to 2k, which is a huge reduction in tokens.
213. 22:24Developers have raised similar issues in mainstream agent and SDK projects, pointing out that sending all tools definition on every request can increase latency, token usage, and tool confusion.
214. 22:44Open proj open-source projects like MCP zero have also explored routing at very large scale, including thousands of tool across many servers.
215. 22:59And you can find form forums post from builder who hit tool confusion with only few tools.
216. 23:08Long before they reach 100.
217. 23:10So, if you if your agent start failing as tool are added, it doesn't automatically means your prompt are bad.
218. 23:20It may means the architecture is asking the model to solve the wrong problem.
219. 23:30So, here we are just trying to say about the We are going to talk about the open-source projects that we came across.
220. 23:39So, one of them is of from Areilo Labs.
221. 23:43They have an open-source semantic router project that you can easily use to test locally for benchmarking, Toolbench, and the Berkeley function calling loader leaderboard are also useful starting points.
222. 23:57And for practical guidance, you could read the Anthropic MCP write-up.
223. 24:02It explains why on-demand tool loading matters and gives a strong real-world signal for this architecture.
224. 24:10The point is that you don't need to invent the ecosystem from scratch.
225. 24:14Those pieces already exist.
226. 24:18Next slide.
227. 24:24Sorry.
228. 24:25Here it is.
229. 24:27Thank you.
230. 24:27So, there are a few trade-offs with this.
231. 24:30The first risk could be a router miss.
232. 24:34So, if the So, the router may fail to retrieve the tool the models need and handle that with you could handle that with a feed fallback.
233. 24:45If the model cannot complete the task, you could widen K, run a second retrieval pass, or even route it to a broader tool group.
234. 24:55The second risk would be weak tool descriptions.
235. 24:59If your descriptions are weak, embeddings will end up being weak.
236. 25:04Write descriptions in the words users actually use and include intent, action, and key entities along with it.
237. 25:13The third risk would over here would be rare tools.
238. 25:17So, some tools may never score high unless their descriptions include the right language.
239. 25:23So, you need to monitor data misses and rewrite those descriptions accordingly.
240. 25:28And finally, just to conclude, do not over-engineer small systems.
241. 25:33If you have 10 or 15 tools, static loading may still be fine.
242. 25:38Routing generally pays off when the catalog is large enough that the prompt size, latency, or tool confusion becomes a real production problem.
243. 25:51So, a few takeaways.
244. 25:55First thing is, as we discussed, tool overload hurts accuracy.
245. 25:59Loading every schema into every prompt makes the model's decision harder as the catalog grows on.
246. 26:07The second one would be to about tokens.
247. 26:11It's like tokens are both cost and latency.
248. 26:16Larger tool catalogs can add add tens of thousands of tokens before the user's actual request is handled.
249. 26:24Semantic routing can help fix that problem.
250. 26:28The When the catalog can be is large, the model only sees the tools relevant to the current query.
251. 26:37This is basically a rag for tools.
252. 26:40Index those descriptions, properly retrieve at run time, and inject only what is needed.
253. 26:46And finally, start simple.
254. 26:49Use K as five as a starting point, log every decision, and evaluate against a real test set.
255. 26:58And improve their tools descriptions over time in case you get incorrect tools being picked up.
256. 27:07And the goal is not to make these agents more complicated.
257. 27:11It's It is basically to stop forcing the model to reason over irrelevant tools.
258. 27:20So, there are some references on this slide.
259. 27:24So, if you're building something, you can use these references.
260. 27:29Like semantic routing libraries, tool bench, Berkeley function calling leaderboard, Anthropic MCP write-up, and vector stores like Pinecone, Quadrant.
261. 27:43Thank you so much for watching this.
262. 27:46If this was useful, please share it with someone building agents.
263. 27:50They have either already hit this wall or they're about to.
264. 27:55You can scan our LinkedIn QR codes on this slide to connect with us.
265. 28:01We are happy to continue the conversation, share resources, or hear how others are approaching this problem in their own system.
266. 28:11Thank you so much for watching this.
267. 28:14I'm Ankush.
268. 28:16This is Suheel, here.
269. 28:17Thank you so much.
270. 28:18Yeah.
271. 28:19Thanks, everyone.