{ "video": { "id": "vh2VGuQ3zhY", "title": "The 100-Tool Agent Is a Trap - Sohail Shaikh & Ankush Rastogi, Prosodica", "duration": 1707, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "Online" }, "analysis": { "video_id": "vh2VGuQ3zhY", "title": "The 100-Tool Agent Is a Trap - Sohail Shaikh & Ankush Rastogi, Prosodica", "one_liner": "Loading every tool schema into every prompt is what breaks agents at scale — the speakers' benchmarks show tool-selection accuracy falling from 78% at 10 tools to 13.6% at 741 tools, while a semantic router that retrieves only the top 3–5 relevant tools holds above 83%.", "summary": "Sohail Shaikh and Ankush Rastogi of Prosodica argue that the common 'fat agent' design — sending every function name, description and JSON schema on every request — fails not because any one tool is badly written but because each request carries the whole catalog. They present benchmark numbers on accuracy, time-to-first-token, input tokens and cost across catalogs of 10 to 1041 tools, then propose semantic routing (RAG for tools: embed tool descriptions offline, embed the user query at runtime, inject only the top-K schemas) plus just-in-time context injection. They give a six-step production checklist, K=5 as a default starting point, and honest trade-offs: router misses, weak descriptions, rare tools, and the fact that under ~20 tools a router is unnecessary.", "key_points": [ "A 741-tool catalog costs about 127,000 tokens of tool descriptions and schemas on every request, before the user's actual question is even considered.", "Fat-agent tool-selection accuracy: ~78% at 10 tools, ~40% at 100 tools, 13.6% at 741 tools — 'roughly one correct tool out of eight'. A semantic router stays above 83% across the same catalog sizes.", "The failure mechanism is 'lost in the middle': models attend more strongly to the beginning and end of long context, so hundreds of schemas packed into the middle are not reliably used — you pay for a huge prompt that makes the decision harder.", "Latency and cost scale with the catalog too: at ~500 tools the fat-agent path can push time-to-first-token past 5 seconds, and at 100,000 requests/day you send billions of tokens just describing tools. Just-in-time routing injects 3–5 schemas (~1,000 tokens), about a 99% reduction in tool-context tokens.", "Semantic routing is RAG for tools: embed tool descriptions into a vector index offline (Chroma, Pinecone, Qdrant), embed the user query at runtime with the same model, nearest-neighbour search, return top-K, fetch and inject only those schemas. Runtime overhead is one embedding call plus one vector search.", "Evaluation measured four things — tool-selection accuracy, time to first token, input tokens per request, estimated cost per 1,000 calls — over the Berkeley Function Calling Leaderboard, Skills-bench-style scenarios, and synthetic tool pools scaled to 10/50/100/200/1041 tools, running identical queries in fat-agent and routed modes with K swept at 3, 5 and 10.", "The router also subtracts: for 'What is the weather in Paris right now?' it returns get_weather and get_forecast, and the flight and hotel tools are simply not available to be mis-called at that request.", "Corroboration cited: Anthropic's write-up on on-demand tool loading via MCP reports 150k tokens down to 2k (98.7% reduction), and open-source projects like MCP Zero have explored routing across thousands of tools on many servers." ], "takeaways": [ "Don't route under ~20 tools — static loading is fine at 10–15; routing starts paying off past ~50 tools in production, or whenever prompt size, latency or tool confusion becomes a real problem.", "Build the router as a six-step sprint, not a platform rewrite: catalog each tool (name, description, schema, owner, version), index the embedded descriptions, write the router (embed query → search → top-K → fetch schema), wire the agent's tool list to come from the router rather than a hard-coded catalog, evaluate at K=3/5/10 and pick the smallest K that hits your accuracy target, then monitor.", "Start at K=5, log every decision — selected tools, final tool calls, failures, fallback usage — because tuning K and rewriting descriptions depends on that log.", "Treat tool descriptions as retrieval surface: write them in the words users actually use, include intent, action and key entities, re-embed when descriptions or schemas change, and watch for rare tools that never score high.", "Plan the router-miss fallback up front: widen K, run a second retrieval pass, or route to a broader tool group when the model can't complete the task.", "If your agent starts failing as tools are added, stop assuming the prompt is bad — it may be an architecture problem you're asking the model to solve." ], "topics": [ "agents", "tool-calling", "semantic-routing", "rag", "context-engineering", "benchmarks", "latency", "mcp" ], "tools": [ "Prosodica", "Anthropic", "MCP", "MCP Zero", "Berkeley Function Calling Leaderboard", "ToolBench", "Chroma DB", "Pinecone", "Qdrant", "Aurelio Labs semantic-router" ], "quotes": [ { "text": "the design does not fail because one tool is badly written.", "at": "03:31", "url": "https://www.youtube.com/watch?v=vh2VGuQ3zhY&t=211s" }, { "text": "So, in short, it's roughly one correct tool out of eight tools.", "at": "04:55", "url": "https://www.youtube.com/watch?v=vh2VGuQ3zhY&t=295s" }, { "text": "The catalog can grow, but the model's working set should stay small.", "at": "15:47", "url": "https://www.youtube.com/watch?v=vh2VGuQ3zhY&t=947s" }, { "text": "the router does not only add the right tool, it also removes the wrong tool from the model's choice set.", "at": "20:13", "url": "https://www.youtube.com/watch?v=vh2VGuQ3zhY&t=1213s" } ], "words": 4106 }, "summary_url": "/#vh2VGuQ3zhY", "transcript": { "html": "/transcripts/vh2VGuQ3zhY.html", "txt": "/transcripts/vh2VGuQ3zhY.txt", "vtt": "/transcripts/vh2VGuQ3zhY.vtt" } }