WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=vh2VGuQ3zhY

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/vh2VGuQ3zhY.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.360 --> 00:00:02.280
Hi everyone.

s2
00:00:02.280 --> 00:00:04.160
Thanks for being here.

s3
00:00:04.160 --> 00:00:07.600
I am Sohaib here and along with me is Ankush.

s4
00:00:07.600 --> 00:00:13.320
So, today we'll be talking about a mistake that looks harmless at first.

s5
00:00:13.320 --> 00:00:20.640
Giving which is basically giving an AI agent every tool access it might ever need all at once.

s6
00:00:20.640 --> 00:00:24.560
So, basically that approach works well in a demo.

s7
00:00:24.560 --> 00:00:30.440
It might even work with a small number of tools like like say for example 10 tools.

s8
00:00:30.440 --> 00:00:36.280
But once the catalog grows, the agent gets slower.

s9
00:00:36.280 --> 00:00:40.160
It might become more expensive and less accurate as well.

s10
00:00:40.160 --> 00:00:43.640
That is why we are calling it the 100 tool agent trap.

s11
00:00:43.640 --> 00:00:56.000
In the next half an hour or so, we'll show why it breaks, what the numbers look like, and how semantic routing with just-in-time context can help us fix this problem.

s12
00:00:57.680 --> 00:01:00.720
So, a quick introduction about myself.

s13
00:01:00.720 --> 00:01:02.520
I am Sohaib Shaikh.

s14
00:01:02.520 --> 00:01:06.040
I'm currently working as a data scientist with Presodica.

s15
00:01:06.040 --> 00:01:13.240
My background spans across AI, NLP, marketing, analytics, and even engineering.

s16
00:01:13.240 --> 00:01:20.600
My current focus is on applied AI, NLP, and conversational intelligence along with rack systems.

s17
00:01:20.600 --> 00:01:30.800
I'm especially interested in making AI systems more reliable, measurable, and even scalable beyond the demos in production.

s18
00:01:31.280 --> 00:01:32.920
And I'm Ankush Astogi.

s19
00:01:32.920 --> 00:01:38.000
I work as a senior data solutions engineer at Presodica.

s20
00:01:38.000 --> 00:01:43.480
I have spent more than a decade in AI, data engineering, and production systems.

s21
00:01:43.480 --> 00:01:47.000
My focus is the engineering side.

s22
00:01:47.000 --> 00:01:59.760
So, it's not about what's going to work in notebook, but whether it's going to survive in with real load, real user, and real failures.

s23
00:01:59.760 --> 00:02:04.080
So, that is the angle we are taking today.

s24
00:02:04.080 --> 00:02:14.920
Suhail will focus more on the model and routing behavior, and I will focus more on system design, implementation, and production trade-offs.

s25
00:02:15.360 --> 00:02:15.840
Awesome.

s26
00:02:15.840 --> 00:02:17.160
Thank you, Ankush.

s27
00:02:17.160 --> 00:02:20.000
Um so, let's get into it.

s28
00:02:20.000 --> 00:02:22.520
So, let's imagine a common design.

s29
00:02:22.520 --> 00:02:28.440
Um you tend to build a system, and it can do many things.

s30
00:02:28.440 --> 00:02:38.920
Say, for example, querying a database, sending an email, um even checking a calendar, or looking up an order, calling an API, and so on and so forth.

s31
00:02:38.920 --> 00:02:46.360
The simplest approach over here would be to give a model every tool definition on every request.

s32
00:02:46.360 --> 00:02:55.120
Every function name, um every description, and even every JSON schema will go into the prompt, whether the user might need it or not.

s33
00:02:55.120 --> 00:02:57.840
So, that We are calling that as a fat agent.

s34
00:02:57.840 --> 00:03:00.680
At small scale, it feels fine.

s35
00:03:00.680 --> 00:03:04.920
With 10 tools as well, the model might usually pick the right one.

s36
00:03:04.920 --> 00:03:07.239
The demo looks good.

s37
00:03:07.239 --> 00:03:08.880
Then, the product grows.

s38
00:03:08.880 --> 00:03:28.600
Um 10 tools might become 30, or it will keep on increasing, and eventually the model starts calling the wrong function, and starts confusing similar tools, may invent tool names, and even take longer to respond.

s39
00:03:28.600 --> 00:03:35.400
The important point is, basically, the design does not fail because one tool is badly written.

s40
00:03:35.400 --> 00:03:40.800
It fails because every request is forced to carry the entire catalog.

s41
00:03:40.800 --> 00:03:42.640
So, let's look here.

s42
00:03:42.640 --> 00:04:00.360
There are, say for example, uh 741 tools in your uh in your entire schema, but and it will look, basically, take up to 127,000 tokens just to have all those tool descriptions in it.

s43
00:04:00.360 --> 00:04:06.400
And this is even before the user user's actual question is even considered.

s44
00:04:06.400 --> 00:04:13.600
So, basically, this will lead to context overload and uh we need to manage that properly.

s45
00:04:15.680 --> 00:04:22.720
So, on this slide we see why it is failing and why the accuracy collapses beyond a point.

s46
00:04:22.720 --> 00:04:31.880
So, when you look at the accuracy curve with the 10 tools, fat agent will get the tools right almost 78% of the times.

s47
00:04:31.880 --> 00:04:35.400
That is not perfect, but it's usable.

s48
00:04:35.400 --> 00:04:39.960
At almost 100 tools, the accuracy drops to around 40%.

s49
00:04:39.960 --> 00:04:45.160
Less than half of the tools that are called are the correct tools.

s50
00:04:45.160 --> 00:04:55.240
And if it grows beyond that, like say for example, in over here, at 741 tools, the accuracy will be a mere 13.6%.

s51
00:04:55.240 --> 00:05:00.360
So, in short, it's roughly one correct tool out of eight tools.

s52
00:05:00.360 --> 00:05:05.960
So, when we compare it with the semantic router, semantic router behaves very differently.

s53
00:05:05.960 --> 00:05:11.360
It stays above 83% across the same catalog sizes.

s54
00:05:11.360 --> 00:05:18.640
That is because the model is not choosing from hundreds of tools, it's choosing from a small and relevant set.

s55
00:05:18.640 --> 00:05:24.440
One reason the fat agent fails is because it's lost in the middle problem.

s56
00:05:24.440 --> 00:05:29.200
Model pays stronger attention to the beginning and end of the long context.

s57
00:05:29.200 --> 00:05:35.120
When hundreds of tool schemas are packed into the middle, the model does not reliably use them.

s58
00:05:35.120 --> 00:05:43.200
So, we end up paying paying for a huge prompt, and that prompts makes the decisions even harder.

s59
00:05:45.200 --> 00:05:48.440
So, two more reasons over here.

s60
00:05:48.440 --> 00:05:51.840
First is the latency and the second is the cost.

s61
00:05:51.840 --> 00:05:56.800
So, we saw in the earlier slide that accuracy was a big problem.

s62
00:05:56.800 --> 00:06:03.000
Another issue is with latency and cost over here as we can see.

s63
00:06:03.000 --> 00:06:11.800
So, like say for example, if we have 741 tools, we saw that it requires almost 127,000 tokens.

s64
00:06:11.800 --> 00:06:15.760
That will include the tool description and the schema text.

s65
00:06:15.760 --> 00:06:20.760
So, that cost is basically being paid on every request.

s66
00:06:20.760 --> 00:06:30.520
At 100,000 requests a day, if you push this into production, you are sending in billions of tokens and just to describe those tools.

s67
00:06:30.520 --> 00:06:38.880
And with just-in-time routing, the prompt may include only three to five relevant schemas, closer to about 1,000 tokens.

s68
00:06:38.880 --> 00:06:44.040
That is roughly a 99% reduction in tool context tokens.

s69
00:06:44.040 --> 00:06:46.360
Another issue is latency.

s70
00:06:46.360 --> 00:06:57.640
With fat agent approach, time to first token grows as the tool catalog grows because the model has to process a larger prompt before it can answer the user question.

s71
00:06:57.640 --> 00:07:08.440
So, say for example, if you have 500 tools in your agent, the fat agent path can push first token latency past 5 seconds.

s72
00:07:08.440 --> 00:07:18.080
So, this is specifically important if your product is real-time because this will take longer time to respond.

s73
00:07:18.080 --> 00:07:21.919
So, which will make the design feel slow and unpredictable.

s74
00:07:21.919 --> 00:07:31.120
So, and with semantic routing, it might feel more responsive and it will be more real-time in a way.

s75
00:07:33.000 --> 00:07:35.160
Here is the clean comparison.

s76
00:07:35.160 --> 00:07:43.200
So, on the left side, we have a fat agent design and on the right side, we have our semantic router design.

s77
00:07:43.200 --> 00:07:54.160
So, in the fat agent design, every schema is loaded for every request, no matter what the user asked.

s78
00:07:54.160 --> 00:08:04.760
And because of that, the catalog will grow, the prompt will grow, due to which the latency will rise and accuracy will drop.

s79
00:08:04.760 --> 00:08:16.120
The agent also becomes one large monolithic, which makes it hard to test, risky to update, and even painful to debug.

s80
00:08:16.120 --> 00:08:23.960
But, in the case of semantic routing design, the agent does not start with every tool.

s81
00:08:23.960 --> 00:08:29.080
So, in this case, the router first look at the user's query.

s82
00:08:29.080 --> 00:08:38.960
It retrieves the three to five most relevant tool, and then it injects only those into the model call.

s83
00:08:38.960 --> 00:08:51.960
So, the context stays small, latency stays stable, and the accuracy holds because the model is choosing from a focused list instead of giant catalog.

s84
00:08:51.960 --> 00:08:56.560
This is Okay, so there is a one important caveat here.

s85
00:08:56.560 --> 00:09:02.920
If you have fewer than 20 tools, a router may be unnecessary.

s86
00:09:02.920 --> 00:09:04.880
Just load the tools directly.

s87
00:09:04.880 --> 00:09:17.360
But, once you passed 50 tools in the production system, then justifying router-based schema make more sense.

s88
00:09:19.720 --> 00:09:20.560
Thanks, Ankush.

s89
00:09:20.560 --> 00:09:25.080
So, let's now look into how semantic routing works.

s90
00:09:25.080 --> 00:09:30.920
If you have built a rack system previously, this will feel pretty similar to that.

s91
00:09:30.920 --> 00:09:35.920
The difference is that we will retrieve tools instead of the documents.

s92
00:09:35.920 --> 00:09:42.120
So, at the first stage, say for example, over here, each tool will have a clear description.

s93
00:09:42.120 --> 00:09:50.680
For example, a search flights tool, uh a check calendar availability tool, or a retrieve customer order status.

s94
00:09:50.680 --> 00:09:59.040
Say for example, the second step would be those descriptions needs to get embedded and stored in a vector index.

s95
00:09:59.040 --> 00:10:05.560
This is usually done offline when the tool catalog is created or updated.

s96
00:10:06.000 --> 00:10:15.160
At the runtime, the user will ask a question, we embed the user query with the same embedding model.

s97
00:10:15.160 --> 00:10:23.320
Then, we search that vector index for the tools with the description that are closest to the query.

s98
00:10:23.320 --> 00:10:27.360
So, the router will then return the top K tools.

s99
00:10:27.360 --> 00:10:32.720
Often, K would be three or five tools that matches the user query.

s100
00:10:32.720 --> 00:10:38.400
And only those selected schemas will get injected into the model call.

s101
00:10:38.400 --> 00:10:40.600
So, this is the whole pattern.

s102
00:10:40.600 --> 00:10:45.640
At the first step, you need to index tool descriptions offline.

s103
00:10:45.760 --> 00:10:51.680
Then, retrieve relevant tools at runtime and keep the model's context focused.

s104
00:10:51.680 --> 00:10:55.200
So, in short, the idea is pretty simple.

s105
00:10:55.200 --> 00:10:58.920
Semantic routing is basically a rack for tools.

s106
00:10:58.920 --> 00:11:08.800
Um if you already have an embedding model and vector database in your stack, most of the infrastructure is already familiar to you.

s107
00:11:10.160 --> 00:11:11.240
Thank you, Sohil.

s108
00:11:11.240 --> 00:11:16.760
Now, we will deep dive into just-in-time context injection.

s109
00:11:16.760 --> 00:11:21.880
So, semantic routing is the retrieval layer.

s110
00:11:21.880 --> 00:11:27.080
And just-in-time context injection is the context management strategy.

s111
00:11:27.080 --> 00:11:35.800
So, in the case of fat agent approach, it loads everything before the query is understood.

s112
00:11:35.800 --> 00:11:44.800
Due to which the model receives a huge list of tools first, and then it tries to reason through it.

s113
00:11:45.480 --> 00:11:48.480
Just-in-time context does the reverse.

s114
00:11:48.480 --> 00:11:58.280
It will wait until the query is known, then it will inject only the context needed for that request.

s115
00:11:58.280 --> 00:12:01.240
I mean, this is not a new software idea.

s116
00:12:01.240 --> 00:12:10.680
We have used lazy loading, just-in-time compilation, and on-demand resource loading from years.

s117
00:12:10.680 --> 00:12:16.800
We are just applying the simple and same principle to the LLM context.

s118
00:12:16.800 --> 00:12:24.520
Anthropic has written about this pattern with on-demand tooling loading through MCP.

s119
00:12:24.520 --> 00:12:35.400
Their report token usage went from 150k tokens down to 2,000, which is 98.7% token reduction.

s120
00:12:36.000 --> 00:12:37.280
That is the signal.

s121
00:12:37.280 --> 00:12:43.880
Large tool catalog should not be dumped into the into every prompt.

s122
00:12:43.880 --> 00:12:47.200
They should be retrieved when needed.

s123
00:12:47.200 --> 00:12:56.120
So, moving to the next slide, let's talk about how to evaluate this fairly.

s124
00:12:56.800 --> 00:13:09.520
We measure four things here: tool selection accuracy, time to first token, input tokens per request, and the estimated cost per thousand call.

s125
00:13:09.520 --> 00:13:24.880
So, here for data sets, we used Berkeley function calling leaderboard, Skills bench style scenarios, and synthetic tool pools that allow us to scale the number of tools, so we can test

s126
00:13:24.880 --> 00:13:30.960
at 10, 50, 100, 200, or even 1041 tools.

s127
00:13:30.960 --> 00:13:37.920
So, we ran the same queries in two modes, the fat agent one and the semantic routing.

s128
00:13:37.920 --> 00:13:42.240
Same model, same answer key, same tool catalog.

s129
00:13:42.240 --> 00:13:49.760
The only difference was whether the model saw every tool or only the routed tool.

s130
00:13:49.760 --> 00:13:56.160
We also swept K at 3, 5, 10 to understand the trade-off.

s131
00:13:56.160 --> 00:13:59.800
So, the smaller K is the faster and cheaper.

s132
00:13:59.800 --> 00:14:05.400
Larger K can be the can recover more edge cases.

s133
00:14:05.400 --> 00:14:09.440
In practice, K equals 5 is a strong default starting point.

s134
00:14:09.440 --> 00:14:15.360
So, if you are want to try this, you can default it to five.

s135
00:14:15.360 --> 00:14:25.560
Here, K means the number of tools the semantic router retrieves and gives to the LLM for each user queries.

s136
00:14:26.640 --> 00:14:30.400
Here are the benchmark result that we got.

s137
00:14:30.400 --> 00:14:34.880
So, here is the result on the left side.

s138
00:14:35.000 --> 00:14:42.120
We see the fat agent accuracy line drop sharply as the tool count grows.

s139
00:14:42.120 --> 00:14:51.160
It started around 78% at 10 tool and then falls to around 13% at 1041 tools.

s140
00:14:51.160 --> 00:15:00.680
But, in the case of router, the router line stays above 83% because the model stays stable.

s141
00:15:00.680 --> 00:15:13.200
From the model points of view, it always choosing the handful of tool, even if the real catalog contains 100.

s142
00:15:13.200 --> 00:15:19.640
On the right side, time to first token shows the same story.

s143
00:15:19.640 --> 00:15:26.320
The fat agent path gets slower as more tool schemas are added.

s144
00:15:26.320 --> 00:15:34.520
At large catalog sizes, the model spend noticeable time just processing the prompt.

s145
00:15:34.520 --> 00:15:43.000
But in case of router, it stays almost flat because of the prompt size is under controlled.

s146
00:15:43.720 --> 00:15:47.120
This is the core lesson from the benchmark.

s147
00:15:47.120 --> 00:15:53.840
The catalog can grow, but the model's working set should stay small.

s148
00:15:54.080 --> 00:16:00.320
Now, on the next slide, we'll be discussing about the implementation pattern.

s149
00:16:00.320 --> 00:16:03.120
There are three steps in it.

s150
00:16:03.120 --> 00:16:05.480
Step one is offline.

s151
00:16:05.480 --> 00:16:08.280
So, build a catalog of tool.

s152
00:16:08.280 --> 00:16:13.880
For each tool, store the name, description, schema.

s153
00:16:13.880 --> 00:16:19.840
Then, embed the description and store it into a vector database.

s154
00:16:19.840 --> 00:16:25.000
You can use Chroma DB, you can use Pinecone, you can use Quadrant.

s155
00:16:25.000 --> 00:16:28.040
Whichever vector database you are using.

s156
00:16:28.040 --> 00:16:33.280
Then, step two will happen on every request.

s157
00:16:33.280 --> 00:16:35.800
Embed the user's query.

s158
00:16:35.800 --> 00:16:38.960
Then, run the nearest neighbor search.

s159
00:16:38.960 --> 00:16:42.000
And return the top K tools.

s160
00:16:42.000 --> 00:16:48.600
In the step three, it will also happen on every request.

s161
00:16:48.600 --> 00:16:52.240
Fetch the schema for those selected tools.

s162
00:16:52.240 --> 00:16:59.120
And then, pass only those schemas into the model call and log which tool was selected.

s163
00:16:59.120 --> 00:17:02.400
So, here, logging really matters.

s164
00:17:02.400 --> 00:17:16.040
So, let's say if you miss something and if you want to improve the description or you want to tune the K, you need a good logging system that you can use it.

s165
00:17:16.040 --> 00:17:24.240
So, the runtime overhead is small, one embedding call, and one vector search.

s166
00:17:24.240 --> 00:17:30.520
The payoff is much smaller prompt and more stable tool selection problem.

s167
00:17:33.880 --> 00:17:34.240
Awesome.

s168
00:17:34.240 --> 00:17:45.240
So, now that you have learned about the implementation pattern, I'll take you through the code version of it, which is pretty straightforward.

s169
00:17:45.240 --> 00:17:57.840
So, initially you need to loop through the tool catalog, and for each tool, as Ankur mentioned, embed the description, store that in the vector database with the tool name and other details.

s170
00:17:57.840 --> 00:18:13.200
At run time, embed that user query, compare that query vector against the stored tool vectors using cosine similarity or vector search, and then retrieve the top most relevant tools,

s171
00:18:13.200 --> 00:18:19.080
and then call the model only with those schemas and the function or tool parameter.

s172
00:18:19.080 --> 00:18:26.240
That final line is the most important part because the model does not receive all the tools, it only receive the routed tools.

s173
00:18:26.240 --> 00:18:31.400
This works with any embedding model and at and with any vector database.

s174
00:18:31.400 --> 00:18:44.120
You can start, as Ankur mentioned, you can start with Quadrant or any other database vector database locally, and then move to a more managed vector store later if needed.

s175
00:18:44.120 --> 00:18:55.520
If you already have an existing rag infrastructure, this is not a new infrastructure as we mentioned earlier, it is the same retrieval pattern applied to the tool selection.

s176
00:18:56.440 --> 00:18:58.040
Thank you, Suhail.

s177
00:18:58.040 --> 00:19:03.200
Uh Now, we will discuss semantic routing in action.

s178
00:19:03.200 --> 00:19:07.320
So, imagine you have a 200 tool agent.

s179
00:19:07.320 --> 00:19:13.920
A user asks, "Find me a flight to New York next Wednesday.

s180
00:19:13.920 --> 00:19:16.400
The router embeds that query.

s181
00:19:16.400 --> 00:19:21.440
Returns tool like search flights, book flight, calendar check.

s182
00:19:21.440 --> 00:19:29.160
The model sees only those relevant schema, so it is much more likely to call the correct tool.

s183
00:19:29.160 --> 00:19:44.240
But, if we load all 200 tools, hotels tool, weather tool, emails tool, sequels tool, or any unrelated workflow tool, it would all be competing for attention.

s184
00:19:44.440 --> 00:19:49.360
That is when model starts choosing the wrong function.

s185
00:19:49.600 --> 00:19:52.200
Now, let's take a second query.

s186
00:19:52.200 --> 00:19:55.200
What is the weather in Paris right now?

s187
00:19:55.200 --> 00:20:00.880
The router returns get weather and get forecast.

s188
00:20:00.880 --> 00:20:03.520
Flights tools are not injected.

s189
00:20:03.520 --> 00:20:06.240
Hotels tools are not injected.

s190
00:20:06.240 --> 00:20:10.480
They are simply not available at this request.

s191
00:20:10.480 --> 00:20:13.000
This is an underrated benefit.

s192
00:20:13.000 --> 00:20:22.280
But, the router does not only add the right tool, it also removes the wrong tool from the model's choice set.

s193
00:20:22.280 --> 00:20:28.000
Now, in the next slide, we will discuss implementation checklist.

s194
00:20:28.000 --> 00:20:34.880
So, here is the production checklist if somebody wants to uh follow this along.

s195
00:20:34.880 --> 00:20:39.600
So, first, get locked the tool in one place.

s196
00:20:39.600 --> 00:20:45.760
So, add name, description, schema, owner, and the version.

s197
00:20:45.760 --> 00:20:48.680
Second is to build the index.

s198
00:20:48.680 --> 00:20:53.520
Embed each description and store the vectors.

s199
00:20:53.520 --> 00:20:57.280
The third one is write the router.

s200
00:20:57.280 --> 00:21:06.280
So, in this, embed the query, then search the index, and take top K and fetch schema.

s201
00:21:06.280 --> 00:21:13.800
The next one is the fourth one in which a user need to wire it up into the agent loop.

s202
00:21:13.800 --> 00:21:21.880
So, the model tool list should come from the router, not from a hard-coded full catalog.

s203
00:21:21.960 --> 00:21:23.600
Fifth is to evaluate.

s204
00:21:23.600 --> 00:21:36.280
Run your test set at K equals to three, five, and 10, and then pick the smallest K that meets your accuracy target.

s205
00:21:36.280 --> 00:21:39.520
Sixth is to monitor production.

s206
00:21:39.520 --> 00:21:47.600
Log selected tools, final tool calls, failure, and fallback usage.

s207
00:21:47.600 --> 00:21:52.600
Then re-embed tools when descriptions or schema change.

s208
00:21:52.600 --> 00:21:56.520
This is not the 6-month platform rewrite.

s209
00:21:56.520 --> 00:22:00.600
For most team, it is a focused sprint.

s210
00:22:00.600 --> 00:22:09.840
So, in the next slide, we will discuss what community already has confirmed.

s211
00:22:09.840 --> 00:22:11.600
This is not just the observation.

s212
00:22:11.600 --> 00:22:24.760
So, Anthropic has published on-demand tool loading with MCP, and reported a drop of 150k tokens to 2k, which is a huge reduction in tokens.

s213
00:22:24.760 --> 00:22:44.160
Developers have raised similar issues in mainstream agent and SDK projects, pointing out that sending all tools definition on every request can increase latency, token usage, and tool confusion.

s214
00:22:44.440 --> 00:22:58.880
Open proj open-source projects like MCP zero have also explored routing at very large scale, including thousands of tool across many servers.

s215
00:22:59.160 --> 00:23:08.280
And you can find form forums post from builder who hit tool confusion with only few tools.

s216
00:23:08.280 --> 00:23:10.120
Long before they reach 100.

s217
00:23:10.120 --> 00:23:20.400
So, if you if your agent start failing as tool are added, it doesn't automatically means your prompt are bad.

s218
00:23:20.400 --> 00:23:26.680
It may means the architecture is asking the model to solve the wrong problem.

s219
00:23:30.000 --> 00:23:39.680
So, here we are just trying to say about the We are going to talk about the open-source projects that we came across.

s220
00:23:39.680 --> 00:23:43.320
So, one of them is of from Areilo Labs.

s221
00:23:43.320 --> 00:23:57.640
They have an open-source semantic router project that you can easily use to test locally for benchmarking, Toolbench, and the Berkeley function calling loader leaderboard are also useful starting points.

s222
00:23:57.640 --> 00:24:02.960
And for practical guidance, you could read the Anthropic MCP write-up.

s223
00:24:02.960 --> 00:24:10.200
It explains why on-demand tool loading matters and gives a strong real-world signal for this architecture.

s224
00:24:10.200 --> 00:24:14.080
The point is that you don't need to invent the ecosystem from scratch.

s225
00:24:14.080 --> 00:24:17.280
Those pieces already exist.

s226
00:24:18.680 --> 00:24:20.920
Next slide.

s227
00:24:24.640 --> 00:24:25.480
Sorry.

s228
00:24:25.480 --> 00:24:27.040
Here it is.

s229
00:24:27.040 --> 00:24:27.960
Thank you.

s230
00:24:27.960 --> 00:24:30.200
So, there are a few trade-offs with this.

s231
00:24:30.200 --> 00:24:34.000
The first risk could be a router miss.

s232
00:24:34.000 --> 00:24:45.760
So, if the So, the router may fail to retrieve the tool the models need and handle that with you could handle that with a feed fallback.

s233
00:24:45.760 --> 00:24:55.600
If the model cannot complete the task, you could widen K, run a second retrieval pass, or even route it to a broader tool group.

s234
00:24:55.600 --> 00:24:59.680
The second risk would be weak tool descriptions.

s235
00:24:59.680 --> 00:25:04.080
If your descriptions are weak, embeddings will end up being weak.

s236
00:25:04.080 --> 00:25:13.000
Write descriptions in the words users actually use and include intent, action, and key entities along with it.

s237
00:25:13.000 --> 00:25:17.600
The third risk would over here would be rare tools.

s238
00:25:17.600 --> 00:25:23.560
So, some tools may never score high unless their descriptions include the right language.

s239
00:25:23.560 --> 00:25:28.680
So, you need to monitor data misses and rewrite those descriptions accordingly.

s240
00:25:28.680 --> 00:25:33.480
And finally, just to conclude, do not over-engineer small systems.

s241
00:25:33.480 --> 00:25:38.040
If you have 10 or 15 tools, static loading may still be fine.

s242
00:25:38.040 --> 00:25:49.800
Routing generally pays off when the catalog is large enough that the prompt size, latency, or tool confusion becomes a real production problem.

s243
00:25:51.440 --> 00:25:55.440
So, a few takeaways.

s244
00:25:55.440 --> 00:25:59.960
First thing is, as we discussed, tool overload hurts accuracy.

s245
00:25:59.960 --> 00:26:07.160
Loading every schema into every prompt makes the model's decision harder as the catalog grows on.

s246
00:26:07.160 --> 00:26:11.480
The second one would be to about tokens.

s247
00:26:11.480 --> 00:26:16.000
It's like tokens are both cost and latency.

s248
00:26:16.000 --> 00:26:24.400
Larger tool catalogs can add add tens of thousands of tokens before the user's actual request is handled.

s249
00:26:24.400 --> 00:26:28.120
Semantic routing can help fix that problem.

s250
00:26:28.120 --> 00:26:36.800
The When the catalog can be is large, the model only sees the tools relevant to the current query.

s251
00:26:37.040 --> 00:26:40.880
This is basically a rag for tools.

s252
00:26:40.880 --> 00:26:46.800
Index those descriptions, properly retrieve at run time, and inject only what is needed.

s253
00:26:46.800 --> 00:26:49.120
And finally, start simple.

s254
00:26:49.120 --> 00:26:58.960
Use K as five as a starting point, log every decision, and evaluate against a real test set.

s255
00:26:58.960 --> 00:27:07.160
And improve their tools descriptions over time in case you get incorrect tools being picked up.

s256
00:27:07.160 --> 00:27:11.120
And the goal is not to make these agents more complicated.

s257
00:27:11.120 --> 00:27:18.560
It's It is basically to stop forcing the model to reason over irrelevant tools.

s258
00:27:20.680 --> 00:27:24.240
So, there are some references on this slide.

s259
00:27:24.240 --> 00:27:29.440
So, if you're building something, you can use these references.

s260
00:27:29.480 --> 00:27:42.720
Like semantic routing libraries, tool bench, Berkeley function calling leaderboard, Anthropic MCP write-up, and vector stores like Pinecone, Quadrant.

s261
00:27:43.160 --> 00:27:46.320
Thank you so much for watching this.

s262
00:27:46.320 --> 00:27:50.360
If this was useful, please share it with someone building agents.

s263
00:27:50.360 --> 00:27:55.360
They have either already hit this wall or they're about to.

s264
00:27:55.360 --> 00:28:01.880
You can scan our LinkedIn QR codes on this slide to connect with us.

s265
00:28:01.880 --> 00:28:11.840
We are happy to continue the conversation, share resources, or hear how others are approaching this problem in their own system.

s266
00:28:11.840 --> 00:28:14.280
Thank you so much for watching this.

s267
00:28:14.280 --> 00:28:16.240
I'm Ankush.

s268
00:28:16.240 --> 00:28:17.160
This is Suheel, here.

s269
00:28:17.160 --> 00:28:18.600
Thank you so much.

s270
00:28:18.600 --> 00:28:19.040
Yeah.

s271
00:28:19.040 --> 00:28:21.360
Thanks, everyone.