Progressive MCP Tool Routing: Stop Drowning Your Agents in 50K Tokens
The 50,000-Token Problem in Tool-Rich Environments
Modern AI agents are powerful, but their potential is crippled by a fundamental inefficiency in standard Model Context Protocol (MCP) implementations. When an agent needs access to a broad suite of tools—think a full DevOps platform with 47 distinct tools for Git, CI/CD, infrastructure, monitoring, and ticketing—naive implementations force the entire tool schema into the context window. The result? An immediate context overload.
In one production system we analyzed, an agent designed to handle cross-functional engineering tasks was initialized with a flat list of 47 tools. The combined JSON schema, including descriptions, parameters, and examples, consumed over 50,000 tokens before the user even typed a single word. This left a minuscule fraction of the context for the actual conversation, history, and reasoning, leading to frequent hallucinations as the model struggled to parse the irrelevant tool options for each simple query.
Case Study: From 40% Hallucination to Production-Ready Routing
The case study involved a SaaS company's internal "Engineering Assistant" agent. Its mandate was broad: "Manage development workflows." Initially, it was given direct, simultaneous access to all 47 MCP tools. The performance was poor. For a query like "Deploy the latest frontend build to staging," the model often hallucinated parameters for tools like `database_backup` or `log_analyzer` because their schemas were competing for attention in the crowded context.
By implementing a two-tier progressive disclosure architecture, we achieved these metrics within one sprint:
- Context Size Reduction: Average prompt size dropped from 50,120 tokens to 8,200 tokens (a **83% reduction**).
- Hallucination Rate: Incorrect tool invocations or parameter generation fell by **40%** (measured via a regression test suite of 300 common queries).
- First-Token Latency: Improved by 15% due to smaller input processing overhead.
- Tool Retrieval Accuracy: Rose to 97% when the correct tool was in the top-5 suggested.
Architecture Deep Dive: The Two-Tier Progressive Disclosure Model
The core idea is simple: don't show the agent everything upfront. Implement a routing layer that uses semantic tool search to fetch only the relevant tool schemas based on the initial intent.
The system works in two steps:
Tier 1: Intent Classification & Tool Grouping. All 47 tools are categorized into logical groups (e.g., `source_control`, `deployment`, `monitoring`, `project_management`). A lightweight classifier determines the user's probable intent.
Tier 2: Dynamic Schema Injection. Based on the classification, only the schemas for the relevant tool group (and perhaps a few high-probability tools from adjacent groups) are injected into the live context for the LLM to use. This leverages progressive disclosure to present a focused, manageable set of tools.
# Pseudocode for the Routing Logic
def route_tools(user_query):
# Step 1: Quick classification against tool groups
probable_intent_groups = classifier.predict(user_query, top_k=2)
# e.g., ['deployment', 'source_control']
# Step 2: Fetch detailed schemas only for these groups
tool_schemas = []
for group in probable_intent_groups:
tool_schemas.extend(get_schemas_for_group(group))
# Step 3: Inject ONLY these schemas into the system prompt
system_prompt_with_tools = build_prompt(tool_schemas)
return call_llm(system_prompt_with_tools, user_query)
Implementing Semantic Tool Search for Precise Routing
The classifier in Tier 1 isn't a simple keyword matcher. It's a small embedding model fine-tuned on your toolset's natural language descriptions. This enables true semantic tool search. When a user says "ship the feature," the classifier understands this semantically aligns with the `deployment` and `source_control` groups, not the `monitoring` group, even without explicit keywords like "deploy."
We built a search index using the `description` field of each MCP tool. For the 47-tool setup, this index was under 1MB and could perform sub-50ms retrieval. This index is the cornerstone of effective MCP tool routing.
Agent Context Optimization: Beyond Tool Filtering
Reducing tool count is only part of the solution for agent context optimization. The progressive system also prunes unnecessary details from the tool schemas it does inject. For example, for a simple query, it might only include the `name`, `description`, and `required_parameters` of a tool, deferring complex `optional_parameters` and verbose `examples` unless the conversation requires them.
Furthermore, the routing layer maintains a lightweight conversation history. If the last three turns were about GitHub PRs, it will dynamically bias the tool routing toward `source_control` tools, even if the new query is ambiguous. This creates a context-aware experience that feels more natural and reduces the cognitive load on the LLM.
Overwhelmed by your toolset? Stop stuffing your agent's context. Learn how TormentNexus implements intelligent, progressive MCP tool routing for production agents. Explore the TormentNexus Platform →