Tool Forge: A Validation-Carrying Toolchain for Governed Agentic Execution Swanand Rao Next Moca Global, Inc. [email protected] https://github.com/nextmoca/tool-forge
arXiv:2605.28000v1 [cs.SE] 27 May 2026
May 2026
Abstract
an open-source toolchain, not as a state-of-the-art claim against other generators or agent frameworks. We argue Large language model agents are increasingly expected that validation-carrying tools and token-efficient routing to perform operational work: to call APIs, manipulate are necessary systems mechanisms for moving agentic sysfiles, assemble workflows, and act inside enterprise systems. tems from demos toward governed production execution. Yet the tool layer on which this execution depends is still commonly treated as either a hand-written integration artifact or a static list of schemas exposed to a model. 1 Introduction This paper introduces Tool Forge, a validation-carrying toolchain for converting natural-language capability intent Artificial intelligence systems are moving from a world of into governed, sandbox-verified, cataloged tool artifacts isolated inference requests toward a world of persistent, and for exposing those artifacts to agents through a token- tool-using, operational execution entities. In the earlier efficient routing layer. phase of large language model adoption, the dominant unit The central idea is that a tool should not be interpreted of interaction was the prompt. A user supplied context, merely as executable code. A production agent tool should a model returned text, and the surrounding application be a capsule: a composition of intent, capability contract, interpreted the answer. The emerging agentic pattern implementation, dependency policy, tests, documentation, changes that assumption. The software layer above the runtime validation evidence, lifecycle state, credential bind- model is now expected to select tools, call APIs, transform ings, and routing metadata. By making validation evidence files, update systems of record, coordinate workflows, and part of the artifact, Tool Forge shifts tool generation from preserve continuity across tasks [1, 2, 5, 7, 12]. model completion to software compilation. By routing over This shift exposes a substrate problem. Agents can reacompact catalog records rather than listing all schemas, son over a task, but they can only act through tools. The it shifts agent tool loading from static exposure to intent- quality, safety, and cost profile of an agentic system therescoped resolution. fore depends heavily on how tools are produced, validated, We describe the conceptual model, architecture, vali- cataloged, selected, and governed. A tool that is generdation pipeline, MCP-facing Router, governance model, ated as a plausible script is not necessarily an operational and initial reproducible benchmark results from the cur- capability. It may have missing input validation, stale rent open-source implementation, available at https: dependencies, incomplete error handling, undocumented //github.com/nextmoca/tool-forge. Across 83 router credential requirements, unmocked network calls in tests, benchmark cases spanning lite, realistic, and adversarial or no runtime evidence that it performs the requested acsuites, Tool Forge Router achieves an aggregate micro-F1 tion. Conversely, a large tool catalog may be semantically of 0.908 while reducing estimated task-flow tool context by rich but operationally expensive if every schema is placed 99.49naive full-catalog schema exposure. In a 25-case end- into the model context for every task. to-end generation probe spanning L1 smoke, L2 realistic, The common architecture still resembles explicit materiand L3 adversarial local-tool tasks, the system generates alization. A tool is constructed as code, then exposed as a 25 of 25 tool bundles, reaches micro-F1 0.940 against de- full schema to the agent. When the catalog grows, this apterministic acceptance patterns, and passes 23 of 25 live proach couples capability growth to context growth. When sandbox validations. The adversarial suites expose impor- tools are generated, it couples model output to operational tant remaining failure modes around negation, semantically trust. Both couplings are problematic. Context windows confusable tools, and edge-case validation, which we an- may grow, but token traffic remains expensive, latencyalyze as open research and engineering problems. These sensitive, and cognitively noisy for the model. Code generresults should be read as an early systems benchmark for ation may improve, but generated code remains software 1
and must be treated as untrusted until checked. Tool Forge begins with a different interpretation. It treats a tool as a validation-carrying capsule rather than as a bare function. The capsule contains not only the executable implementation, but also the intent from which it was derived, the capability contract it must satisfy, its dependencies, tests, command-line interface, documentation, sandbox result, lifecycle state, and router metadata. This makes a tool an inspectable operational artifact. The second interpretation concerns tool exposure. Tool Forge Router treats the agent-facing tool set as an intentscoped projection over a governed catalog, not as a complete schema dump. The agent first interacts with a small MCP surface that can search, resolve, describe, and call tools. Full schemas are loaded lazily only for the selected subset. This keeps the catalog large while keeping the model-facing decision surface small. The argument of this paper is twofold. The technical claim is that validation-carrying tool artifacts and intentscoped routing reduce two practical bottlenecks in agentic systems: unverified tool generation and tool-schema token bloat. The conceptual claim is that these mechanisms point to a broader systems abstraction. In the same way that agent runtimes need control layers for memory, identity, and orchestration [11, 13, 21], they also need a control layer for governed executable capability. This paper contributes: (1) a conceptual model of the validation-carrying tool capsule; (2) a system architecture for compiling intent into governed Python tool bundles; (3) a token-efficient MCP-compatible router for catalog-scale tool selection; (4) a reproducible benchmark protocol with explicit routing baselines and live-sandbox generation measurements from the current open-source implementation; and (5) an analysis of limitations and open problems.
2
validated, how they are approved, or how a large tool catalog is exposed without excessive context cost. The issue is analogous to earlier transitions in software systems. Service oriented architectures and microservices separated capabilities into callable interfaces, but required service discovery, observability, deployment policy, and operational ownership [24, 25, 26]. Container orchestration made units of execution easier to deploy, but only because surrounding control planes tracked state, health, scheduling, and policy [27, 28]. Modern data and ML systems similarly learned that model artifacts require metadata, lineage, validation, and reproducible pipelines [29, 30, 31]. Tool-using agents are now encountering the same pattern. The tool schema is not the control plane. A generated integration must be reviewed, tested, sandboxed, versioned, and governed. A third-party MCP server must be imported and approved, not blindly trusted. A large catalog must be retrieved against intent, not loaded wholesale. Without these surrounding mechanisms, tool use remains closer to a demo capability than to an enterprise execution substrate. Tool Forge is motivated by this operational gap. It is not primarily a better prompt for generating Python code. It is a system that separates model-authored synthesis from platform-enforced operationalization. This distinction is central because the model may be cloud-hosted or local, strong or weak, deterministic or variable. The trust boundary should not be the model response. The trust boundary should be the validation and governance process wrapped around the response.
3
Tool Forge Conceptual Framework
Tool Forge rests on a distinction between a tool as a materialized function and a tool as an effective governed capability. In the first interpretation, a tool exists once executable code has been produced. In the second interpretation, a tool exists once the system has established a contract, an implementation, and evidence that the implementation satisfies the contract within a declared runtime boundary. Let I denote a user intent. It may be an informal naturallanguage request or a structured specification containing inputs, outputs, authentication requirements, validation expectations, and live-sandbox values. Tool Forge maps I into a capability contract:
Background and Motivation
The modern tool-using agent is the result of several converging lines of work. Large language models established broad natural-language reasoning and code generation capabilities [1, 2, 3]. Tool-use methods such as ReAct [5], Toolformer [6], Gorilla [7], ToolLLM [8], and HuggingGPT [9] showed that models could plan over, select, and call external capabilities. Agent frameworks and multi-agent systems such as AutoGen, MetaGPT, Reflexion, Voyager, and Generative Agents explored persistent roles, collaborative execution, memory, and feedback loops [11, 12, 13, 14, 15]. At the infrastructure layer, function calling and tool schemas became the practical interface between model reasoning and software action. The Model Context Protocol (MCP) generalizes this interface by defining a common protocol for model applications to connect with tools, resources, and prompts [16]. MCP is important because it gives agent systems a standard interoperability layer. However, protocol-level interoperability does not solve the lifecycle problem of where tools come from, how they are
C = (n, P, K, O, R, H, E) where n is the tool name, P is the parameter set, K is the set of required credentials and environment variables, O is the output contract, R is the runtime class, H contains handling requirements for expected failure modes, and E contains evidence extracted from user intent, documentation, or indexed API material. A Tool Forge tool capsule T is then: T = (C, X, D, U, V, G, A) 2
where X is executable code, D is dependency policy, U is the user surface such as CLI and README, V is validation evidence, G is governance and lifecycle state, and A is audit and provenance metadata. The capsule is not merely a packaging convention. It is the unit that the catalog, router, and downstream agents reason about. This model produces three useful invariants. First, a callable tool should have a machine-readable schema and a dependency declaration. Second, a tool should carry validation evidence rather than requiring the caller to infer trust from the fact that code exists. Third, routing should operate over governed capsules, not over raw functions. The approach also changes the role of the model. The model is not the compiler. It is a synthesis component inside a compiler-like pipeline. The platform owns the intermediate representation, scaffolding, review gates, dependency policy, sandbox execution, and catalog registration. This is especially important for local small language models, which may be useful for privacy and experimentation but should not be asked to author every operational artifact in a bundle. Under this framework, the catalog is a set of tool capsule versions: K = {T1 , T2 , . . . , TN }.
layer is conservative: it provides evidence for reconciliation rather than absolute authority. Generation recipes determine how much surface area the model owns. In full-bundle mode, the model writes the tool implementation, CLI wrapper, tests, README, tool card, requirements, and harness. This is appropriate for strong cloud models when a richer initial bundle is desirable. In tiny mode, the model writes only the core Python implementation. Tool Forge deterministically creates the CLI, tests, metadata, README, requirements, and manifests. Local model mode forces the effective recipe to tiny, reducing prompt size and keeping weaker models away from brittle multi-file generation. The validation stages are intentionally layered. Deterministic review catches structural defects before sandbox execution. Unit tests must mock network calls for external APIs. CLI help checks ensure that the command-line surface matches the capability contract. Live sandbox validation then executes the tool with runtime values. If required credentials or sandbox parameters are absent, the backend pauses and asks for those values instead of pretending it can infer them. The routing loop is implemented by Tool Forge Router. The Router exposes a small MCP-compatible surface: search tools, resolve tools, describe a selected tool, call a tool, and list profiles. Generated Tool Forge tools and imported third-party MCP tools can both be normalized into catalog cards. Imported MCP tools begin in pending review and can be approved, blocked, pinned, or assigned credential mappings. This lets Tool Forge act as a governed tool control plane rather than a collection of one-off MCP servers. The architecture supports a file-backed open-source mode and a hosted system-of-record mode. In the filebacked mode, artifacts, router state, profiles, governance records, sessions, and audit logs are inspectable on disk. In a hosted deployment, Postgres and object storage can become the durable system of record while preserving the same capsule and routing abstractions.
A governance profile Gp defines which capsules may be resolved for a particular agent, tenant, or workflow. A router receives an agent task q and returns an intent-scoped session: Sq = R(q, K, Gp , k) where k is the maximum number of tools to expose. The session is a bounded view over the catalog. Calls outside the session can be rejected. This makes the agent’s available tool set a runtime projection rather than a fixed global list.
4
System Architecture
Tool Forge is organized around two loops: a generation loop and a routing loop. The generation loop receives intent, synthesizes a capability contract, grounds it in documentation where possible, generates code or a full bundle, applies deterministic scaffolding, runs technical review, executes tests, and validates the result in a sandbox. The routing loop indexes the catalog, applies governance policy, resolves tools for a task, lazily exposes full schemas, and audits calls. The generation pipeline begins with intent enhancement only when needed. If the user has already provided structured intent, enhancement is skipped because unnecessary enhancement can introduce unrelated fields or providers. Capability synthesis then produces a structured contract. For API integrations, Tool Forge can extract documentation URLs from the prompt, index relevant pages, and derive endpoint evidence such as HTTP methods, paths, provider vocabulary, and path parameters. This grounding
5
Contract and Evidence Model
The capability contract is the intermediate representation that separates intent understanding from code generation. This intermediate layer is necessary because naturallanguage intent is often underspecified, while executable code is too concrete to be the first reliable representation. A contract can be inspected, reconciled, repaired, and tested before the system commits to an implementation. The contract contains several classes of information. The first is the external shape of the tool: name, description, input parameters, output fields, and CLI flags. The second is the operational shape: required credentials, environment variables, file outputs, runtime side effects, and validation inputs. The third is the failure model: missing credentials, invalid parameters, provider errors, network failures, rate limits, malformed responses, and permission failures. The 3
fourth is evidence: source documentation URLs, endpoint looks plausible while missing edge cases, dependencies, fragments, path parameters, SDK method names, and auth checks, or error handling. If the generated script is user-provided examples. copied directly into an agent runtime, the model response This representation lets Tool Forge avoid two common becomes the trust boundary. Tool Forge replaces that errors. One error is over-generation, where an enhancer or boundary with a sequence of checks. The model may promodel invents inputs that were not part of the requested pose code, but the platform decides whether the resulting tool. Another is under-specification, where a tool is gener- capsule is admissible. ated without an account identifier, parent folder identifier, Dependency policy is one part of this process. Modelauthentication header, or multi-step upload flow that the generated dependency pins can be stale, over-specific, or API actually requires. Contract reconciliation does not incompatible with the execution environment. Tool Forge eliminate those errors, but it creates a place where they normalizes dependency specifications through a policy layer can be detected before code execution. so the behavior is consistent across local and cloud modes. The evidence model is intentionally weaker than a This prevents dependency handling from becoming another formal specification. Most public API documentation is prompt-level convention. prose, examples, and partial reference material, not a Documentation grounding is another part. Many producmachine-checkable theorem about provider behavior. Tool tion API integrations fail not because the model cannot Forge therefore uses documentation grounding as a source write Python, but because it uses the wrong endpoint, of high-impact evidence, not as a source of unquestioned misses an account or folder identifier, confuses API vertruth. If indexed documentation shows an endpoint such as sions, or omits a multi-step upload flow. Tool Forge’s /accounts/{account id}/folders/{folder id}/files/local upload, grounding layer attempts to extract high-impact evidence the system can infer that both account and folder context from source documentation and reconcile it with the capamay be important. If the user prompt contains only a bility contract. It is deliberately generic, not hardcoded to project identifier, the contract can be flagged or amended one provider. Provider-specific playbooks can still exist, before generation. but the core system should not require a hand-authored This evidence layer is also distinct from provider play- playbook for every new API on the Internet. books. A playbook is useful when a flow is known and Validation evidence also improves downstream routing. stable. Documentation grounding is useful when Tool A Router profile can prefer sandbox-validated or approved Forge encounters a new API and must extract enough tools. It can exclude failed, deprecated, or blocked tools. structure to reduce hallucination. The two approaches It can expose only tools whose credential mappings are are complementary. The key architectural requirement is available for a given tenant. In this sense, validation is not that provider-specific knowledge must not become hidden merely a generation-time event. It becomes part of the hardcoding in the core generator. The core should remain runtime selection interface. capable of discovering evidence from arbitrary documentaThe capsule model does not prove semantic correctness. tion, while playbooks can provide stronger contracts for A tool can pass tests and still be incomplete under procommon integrations. duction data, unusual permissions, rate limits, or provider The contract-and-evidence model also makes regression API drift. The claim is narrower and more practical: analysis easier. When a generated tool fails, the failure can validation-carrying capsules make the evidence explicit, be traced to a specific layer: intent enhancement, capabil- machine-readable, and available to the control plane. ity synthesis, evidence extraction, contract reconciliation, code generation, dependency policy, review, or sandbox execution. Without an intermediate representation, those 7 Token-Efficient Tool Routing failures collapse into an opaque model output. Tool routing addresses the consumption side of the lifecycle. A standard MCP server can expose all tools through a 6 Validation-Carrying Tool Cap- tool-list response. This direct exposure is simple and compatible, but it scales poorly for large catalogs. If there sules are N tools and each full schema costs s̄ tokens, naive The defining property of a Tool Forge artifact is that exposure costs O(N s̄) tokens before the agent has begun validation evidence is attached to the tool itself. A gen- the task. Tool Forge Router changes the complexity profile. The erated bundle contains the executable implementation, a command-line wrapper, tests, a test harness, a README, agent first sees a small stable MCP surface. The Router a tool card, runtime requirements, review findings, and then filters the catalog by governance policy, searches comsandbox results. These artifacts create a record that can pact cards, creates an intent-scoped session, and reveals be consumed by humans, CI jobs, catalog registries, and full schemas only for selected tools. If compact cards cost c̄ tokens and the session contains k tools, the conceptual agents. This design addresses a common failure mode in cost becomes: generated-code systems. A model can produce code that O(N c̄) + O(ks̄) 4
with c̄ ≪ s̄ and k ≪ N . If compact-card retrieval is performed locally and only selected schemas are sent to the model, the model-visible schema cost approaches O(ks̄). This design is not merely an optimization. It changes the responsibility boundary. A language model can choose from a list, but if the first operation is to send a large list to the model, tool selection is already entangled with token traffic, latency, and context noise. The Router performs retrieval and policy filtering in ordinary software before the model sees the final decision surface. The Router can combine several implementation strategies behind this interface: lexical scoring, semantic retrieval, curated profiles, lifecycle filters, read/write classification, and future reranking. These are implementation details beneath the control-plane abstraction. The paper’s claim does not depend on any single prefiltering technique; it depends on moving first-stage selection and policy enforcement out of the model context and into a governed software layer. Anthropic’s recent MCP optimization work identifies the same scaling pressure from a complementary direction. Its advanced tool-use guidance argues that agents should avoid stuffing all tool definitions into context and instead use Tool Search for on-demand discovery, Programmatic Tool Calling for executing multi-step tool workflows in code, and Tool Use Examples for improving invocation accuracy beyond JSON schemas [38]. Its code-execution-with-MCP pattern further observes that direct MCP calls make both tool definitions and intermediate results flow through the model context; it proposes presenting MCP servers as code APIs, using progressive disclosure or search tools, filtering and aggregating large results inside the execution environment, and returning only the final useful output to the model [39]. Tool Forge adopts these methods as a model-agnostic control-plane strategy. The Router’s search, resolve, and describe operations correspond to deferred tool loading and progressive disclosure. Tool cards, generated examples, contract ledgers, and validation traces provide the usage evidence that schemas alone cannot express. Router sessions provide the bounded set of tools that may be used for a task. For data-heavy or multi-step workflows, the same catalog can support a code-execution mode in which selected Tool Forge tools and imported MCP tools are exposed as callable APIs inside a sandbox; the model sees the orchestration code and final result rather than every intermediate tool response. The distinction is that Tool Forge ties these token-saving techniques to governance state, sandbox validation, credential mappings, and lifecycle policy, rather than treating them only as inference-time compression. The Router is MCP-compatible because its meta-tools are ordinary MCP tools. It does not require a new wire protocol. It does require a retrieval-oriented agent pattern: resolve tools for the task, inspect selected tools if needed, and then call through the Router. For clients that require direct exposure, selected profiles can mount tools directly,
trading token efficiency for compatibility.
8
MCP and Multi-Agent Orchestration
MCP compatibility gives Tool Forge an agent-facing protocol surface, but the Router changes the operational role of an MCP server. Instead of requiring each tool or small group of tools to be wrapped as a separate MCP server, Tool Forge can serve as a governed aggregation layer over generated tools, human-authored tools, and imported thirdparty MCP tools. This is important for multi-agent systems. In a simple agent, a static tool list may be acceptable. In a multi-agent environment, different agents need different capability projections. A developer agent may require GitHub, CI, and file-system tools. A revenue-operations agent may require CRM, spreadsheet, and email tools. A document agent may require PDF, markdown, and storage tools. A single global schema list forces all of these agents to share context and policy surfaces even when their responsibilities differ. Tool Forge profiles make capability surfaces explicit. A profile can define a curated subset of tools, lifecycle requirements, credential mappings, and imported MCP approvals. The Router then resolves task-specific sessions inside that profile. This creates a two-level selection model: a human or platform owner defines the permitted capability universe, and the Router resolves the task-specific subset. The model is closer to capability-based security than to unrestricted function calling [32, 33]. Imported MCP servers fit naturally into this model. A third-party MCP server may expose dozens of tools, some read-only and some destructive. Tool Forge can introspect or import the tool list, normalize each tool into a card, assign a pending lifecycle state, and require explicit governance before exposure. This lets organizations adopt MCP ecosystems without treating every imported server as fully trusted by default. The same mechanism also supports tool sessions. A session is a short-lived binding between an agent intent and a resolved tool set. If an agent asks for tools to upload a report and notify Slack, the session might include a PDF converter, storage uploader, and Slack message sender. A later call to an unrelated payment refund tool can be denied because it is outside the session. This is a practical control against accidental tool drift during long reasoning chains. The orchestration implication is that agents do not need to carry the entire tool universe in memory. They can ask the Router for the next relevant capability surface. This is compatible with planning agents, workflow agents, and human-in-the-loop systems. It also creates a clean integration point for future HTTP MCP serving, MCP-serverfrom-intent generation, and enterprise policy engines. The current implementation is open source at https: //github.com/nextmoca/tool-forge. The repository in5
cludes the generator, Router, benchmark suites, APIdocument grounding modules, dependency policy, localmodel support, and self-hosting documentation. This matters because the proposed control layer is intended to be inspectable infrastructure rather than a closed demonstration.
after intent-scoped resolution. For RQ2 and RQ3, the end-to-end generation benchmark uses fixed L1, L2, and L3 local-tool suites. Each case provides an intent, expected artifact patterns, and sandbox inputs where applicable. The benchmark invokes Tool Forge, writes a tool bundle, runs the generated harness, runs live sandbox validation, and then scores the generated artifact against required and forbidden patterns. We report 9 Experimental Protocol and Base- generated-bundle count, precision, recall, F1, and sandbox pass count. The E2E benchmark intentionally avoids live lines third-party APIs so that failures measure generation and validation quality rather than provider availability. This The evaluation is designed around three research questions. design makes the benchmark deterministic and inexpensive RQ1 asks whether intent-scoped routing can reduce modelto reproduce, but it also means the E2E result should not visible tool context while preserving tool-selection quality. be generalized to arbitrary SaaS APIs, multi-step OAuth RQ2 asks whether the generation pipeline can produce flows, provider-specific upload protocols, or long-running complete tool bundles, not merely function bodies, that workflow automation. satisfy deterministic artifact checks. RQ3 asks whether We do not claim a matched superiority result against sandbox execution exposes failures that are not captured other tool-generation systems in this version of the paby static artifact scoring. per. Such a claim would require running the same intents, The benchmark suite should be interpreted as an initial constraints, sandbox inputs, and scoring harness against release benchmark. Its purpose is to make the system’s each baseline. Instead, the paper reports two conservative claims inspectable, reproducible, and challengeable at opencomparisons that are directly measured by the repository source launch time. It is not a comprehensive production artifacts: Router exposure versus naive full-catalog expocertification suite and it is not presented as evidence of sure, and deterministic artifact quality versus live sandbox state-of-the-art performance versus other frameworks. The execution. This makes the current evaluation reproducible results are most useful for evaluating whether the proposed and falsifiable while leaving cross-system comparisons to architecture has measurable signal and for identifying the future benchmark work. failure modes that should guide the next benchmark expansion. For RQ1, the Router benchmark constructs synthetic 10 Benchmark Evaluation but structured tool catalogs and task intents. Each case defines a set of expected tools. The Router returns se- We evaluated the current open-source implementation uslected tools for the intent, and the scorer computes true ing the initial offline benchmark suites shipped in the repospositives, false positives, and false negatives. We report itory. The reported numbers are intended as transparent micro-precision, micro-recall, micro-F1, macro-F1, average release measurements: they show the present behavior of task-flow token estimates, and reduction versus a naive Tool Forge on fixed suites, and they define a baseline that full-catalog exposure baseline. Token estimates use the future versions and external contributors can improve or repository’s explicit approximation of characters divided challenge. They are not a claim that Tool Forge dominates by four. This approximation is not tokenizer-identical, other systems under a shared public benchmark. but it is deterministic and sufficient for comparing relative The Lite suite contains 8 fast regression cases over a 250exposure across fixed artifacts. tool synthetic catalog. The L2 suite contains 50 realistic The Router benchmark’s synthetic catalogs are inten- single-tool and multi-tool intents over a 600-tool catalog. tionally controlled. This enables repeatable measurement The L3 suite contains 25 adversarial cases over a 500-tool of catalog size, schema exposure, and confusable tool fami- catalog, including negation, read/write ambiguity, and lies, but it does not fully capture the distribution of real semantically confusable tool families. enterprise tool catalogs, naming conventions, permission The aggregate across all 83 Router cases is micro-F1 structures, or provider-specific edge cases. 0.908 with 99.49estimated task-flow token reduction relaThe primary routing baseline is naive full-catalog schema tive to naive full-catalog exposure. The strongest result is exposure: every tool schema in the catalog is made visible the L2 realistic suite, where Router selection reaches microto the agent. This is a strong practical baseline because F1 0.958 and reduces task-flow tool context by 99.55failure it maximizes recall but scales linearly with catalog size signal is the L3 suite, where micro-F1 drops to 0.786. and pushes unrelated tools into the model context. We Inspection of the L3 failures shows weaknesses around also record compact-catalog and Router-meta-tool sizes. negation and closely related tools: for example, selecting These are not competing agent policies by themselves; file upload when the request asks for text-only Slack notithey are exposure levels that clarify where token reduction fication, opening a pull request when the request asks for enters the system. The relevant comparison is between a GitHub issue, or confusing upload, download, and delete full schema exposure and the Router’s task-flow exposure operations in object storage. 6
These failures are important. They show that the Router’s current retrieval layer is already strong enough for large token reduction and many realistic intents, but not yet sufficient as a sole safety mechanism for adversarial or high-stakes tool selection. Governance profiles, mounted subsets, stricter read/write classifiers, and second-stage semantic reranking are natural next steps. We also ran an end-to-end generated-tool benchmark over 25 fixed local-tool cases: 10 L1 smoke tasks, 10 L2 realistic file and data-processing tasks, and 5 L3 adversarial safety tasks. Unlike the earlier catalog snapshot benchmark, this run invokes the generator, writes full bundles, executes test harnesses, runs live sandbox validation with benchmark-provided inputs, and then scores the generated artifacts against deterministic acceptance patterns. The run generated 25 of 25 bundles with aggregate micro-F1 0.940, precision 1.000, recall 0.887, and macro-F1 0.946. Live sandbox validation passed for 23 of 25 generated bundles. Because these are local-tool cases rather than live third-party integrations, the result measures bundle construction, local validation, dependency handling, and sandbox behavior. It does not establish performance on providers such as Slack, Stripe, Frame.io, Notion, GitHub, or Google APIs. The per-tier generation scores are shown in the generated-tool results table. L1 smoke tasks reached microF1 0.985. L2 realistic tasks reached micro-F1 0.911 and were the hardest portion of the run, primarily due to missing secondary return fields or filesystem-error tests rather than wrong tool identity. L3 adversarial tasks reached micro-F1 0.948 while preserving zero false-positive pattern matches. Two live sandbox validations failed despite generated artifacts being written: one CSV pivot case and one safe YAML case. We report these separately because pattern coverage and live sandbox execution measure related but distinct properties of generation quality.
This governance model matters because enterprise agent systems are judged not only by whether they can act, but by whether they can explain and constrain that action. When an agent calls a tool, the organization needs to know which intent led to selection, which tool version was used, what validation evidence existed, what credential alias was required, what profile permitted the call, and what result was returned. Tool Forge makes these fields part of the artifact and router state. The model is not a replacement for OS-level sandboxing, network egress policy, container isolation, RBAC, or human approval workflows. It is an application control plane that can integrate with those lower-level controls. This distinction is important for production deployments. A generated tool that can move money, delete data, or post to customers should still require stronger approval and monitoring than a local CSV transformation.
12
Limitations and Open Questions
The current implementation has several limitations. First, documentation grounding is constrained by the quality and crawlability of public API documentation. Some providers distribute crucial flow information across multiple pages, examples, and authentication guides. Generic endpoint extraction can help, but it cannot fully replace providerspecific reasoning or human review. Second, the paper does not yet include a matched crosssystem baseline against other agent frameworks or codegeneration products. This is deliberate rather than an omission of a favorable result. A fair baseline would need identical intents, runtime inputs, dependency policies, mock requirements, sandbox execution, and artifact scoring. The current evaluation therefore emphasizes reproducible within-system baselines: full-catalog exposure versus routed exposure, and deterministic artifact checks 11 Governance, Isolation, and En- versus live sandbox execution. Third, local model mode improves privacy and experiterprise Readiness mentation but does not guarantee high-quality synthesis. Tool Forge assumes that generated code, model outputs, Small language models can struggle with long contexts, and imported tool metadata are not intrinsically trusted. JSON output validity, and complex API flows. Tool Forge The primary threats include generated code that performs mitigates this through tiny generation and deterministic unintended actions, generated tests that make live network scaffolding, but the quality of the generated core implecalls, hardcoded secrets, stale dependencies, incorrect API mentation still depends on model capability. flows, imported MCP tools with broad write capability, Fourth, Router retrieval remains an empirical system. and agents that call tools outside their intended session. The L3 benchmark shows that negation and confusable Several controls follow from this assumption. Creden- tool families are still difficult. Better retrieval features, tials are referenced by environment variable or credential explicit operation taxonomies, read/write classification, mapping, not embedded in generated code. Unit tests for policy constraints, and reranking are necessary for highexternal API tools are expected to mock network calls. stakes settings. Sandbox validation requests real runtime values only when Fifth, the generated-tool benchmark in this paper is a necessary. Router sessions bind tool availability to a task fixed 25-case probe, not a full production certification. It and profile. Imported MCP tools begin in pending review. covers L1 smoke, L2 realistic, and L3 adversarial local-tool Audit logs record metadata and argument names rather tasks with a fixed model configuration, but it does not cover than raw secret values. hundreds of third-party APIs, long-running workflows, very 7
Suite
Cases
Tools
Micro-F1
Macro-F1
Avg flow tok.
Naive tok.
Reduction
8 50 25
250 600 500
0.900 0.958 0.786
0.938 0.973 0.780
948.0 972.5 811.1
89665 214464 178165
98.94 99.55 99.54
Router Lite Router L2 realistic Router L3 adversarial
Table 1: Tool Forge Router benchmark results. Token estimates use the repository benchmark convention of characters divided by four. Reduction compares the task-flow context against naive full-catalog schema exposure. Suite Router Lite Router L2 realistic Router L3 adversarial
Tools
Full schema tok.
Compact tok.
Router tok.
Avg flow tok.
Flow reduction
250 600 500
89665 214464 178165
30130 72222 60058
284 284 284
948.0 972.5 811.1
98.94 99.55 99.54
Table 2: Context exposure baselines for routing. Full schema tokens estimate naive exposure of every tool schema; compact tokens estimate catalog-card exposure; Router tokens estimate the stable Router MCP surface; average flow tokens estimate the task-scoped selected-tool context.
Suite E2E L1 smoke E2E L2 realistic E2E L3 adversarial
Cases
Gen.
Sandbox
Micro-F1
10 10 5
10 10 5
10/10 9/10 4/5
0.985 0.911 0.948
log, route, and audit tools. Prec. Recall Tool Forge introduces a validation-carrying toolchain for
1.000 0.971 this layer. It treats a generated tool as a capsule containing 1.000 0.836 intent, contract, implementation, dependency policy, tests, 1.000 0.902
documentation, validation evidence, lifecycle state, and routing metadata. It treats the agent-facing tool context as an intent-scoped projection over a governed catalog rather than a static full-schema list. The current implementation demonstrates promising results for token-efficient tool routing and provides a practical open-source foundation for governed intent-to-tool generation. The benchmark results also expose the next problems clearly: adversarial routing, deeper API grounding, broader end-to-end generation evaluation, stronger sandbox isolation, and additional language targets. These are engineering and research directions, but they do not change the core thesis. If agents are going to run real systems, tools must become validation-carrying, governable, and context-efficient execution artifacts.
Table 3: End-to-end generated-tool benchmark over fixed L1/L2/L3 local-tool suites. “Sandbox” reports live sandbox validations passed out of generated bundles; pattern scores are deterministic artifact checks. large files, or provider-specific authentication failures. The two sandbox failures in the run are useful evidence that deterministic scoring and live execution should be reported together rather than collapsed into a single quality number. Sixth, release-time benchmarks can create a false sense of completeness if their scope is not stated clearly. The present results are best understood as a starting measurement for an open benchmark program. Stronger evidence would include larger E2E suites, repeated runs across model providers and generation recipes, live third-party API tasks with mocked and real validation modes, human-authored baselines, and matched comparisons against other agenttooling systems under the same harness. Finally, validation evidence is not proof. A sandbox can validate one path and still miss production edge cases such as rate limits, permission boundaries, large files, malformed provider responses, or provider API drift. Tool Forge therefore should be viewed as a control-plane layer that makes evidence explicit and enforceable, not as a theorem prover for arbitrary generated code.
13
References [1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. [2] OpenAI. GPT-4 technical report, 2023. [3] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models, 2023.
Conclusion
Agentic systems will not become reliable production software merely by adding larger models or longer context windows. They require operational control layers around the model. One of the most important layers is governed executable capability: the ability to create, validate, cata-
[4] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, 8
Micro-F1 1.0
0.900
0.958
1.0
98.94%
Token reduction 99.55%
99.54%
0.786 0.75
0.75
0.5
0.5
0.25
0.25
0
0 Lite
L2
L3
Lite
L2
L3
Figure 1: Router benchmark trends. L2 realistic tasks preserve high selection quality while L3 adversarial tasks expose semantic-confusion failure modes. All suites preserve large estimated reductions in task-flow tool context versus naive full-catalog exposure. Live sandbox pass rate
Generated-tool Micro-F1 1.0
0.985 0.911
0.948
1.0
100.0% 90.0% 80.0%
0.75
0.75
0.5
0.5
0.25
0.25
0
0 L1
L2
L3
L1
L2
L3
Figure 2: End-to-end generation benchmark trends across L1 smoke, L2 realistic, and L3 adversarial suites. L2 has the largest gap between deterministic artifact quality and live sandbox success, which is consistent with more realistic file-processing edge cases. et al. Retrieval-augmented generation for knowledgeintensive nlp tasks. In Advances in Neural Information Processing Systems, 2020.
Solving ai tasks with chatgpt and its friends in hugging face, 2023. [10] Gregoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: a survey, 2023.
[5] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023.
[6] Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, [11] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Shaokun Zhang, Jiale Liu, et al. AutoGen: Enabling Toolformer: Language models can teach themselves next-gen llm applications via multi-agent conversation, to use tools. In Advances in Neural Information Pro2023. cessing Systems, 2023. [12] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Zhang, Ceyao Wang, Steven Ka Shing Yau, Zi Hen Lin, Liyang Zhou, Chenyu Ran, et al. MetaGPT: Meta programming for a multi-agent collaborative framework, 2023. [8] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. ToolLLM: Facilitating large language [13] Joon Sung Park, Joseph C. O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. models to master 16000+ real-world apis, 2023. Bernstein. Generative agents: Interactive simulacra [9] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng of human behavior. In ACM Symposium on User Li, Weiming Lu, and Yueting Zhuang. HuggingGPT: Interface Software and Technology, 2023.
[7] Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis, 2023.
9
[14] Noah Shinn, Beck Labash, and Ashwin Gopinath. [29] Jerome H. Saltzer and Michael D. Schroeder. The Reflexion: Language agents with verbal reinforcement protection of information in computer systems. Prolearning, 2023. ceedings of the IEEE, 1975. [15] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- [30] Fred B. Schneider. Enforceable security policies. ACM dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and AnTransactions on Information and System Security, ima Anandkumar. Voyager: An open-ended embodied 2000. agent with large language models, 2023. [31] Leslie Lamport. Time, clocks, and the ordering of [16] Anthropic. Model context protocol. https:// events in a distributed system. Communications of modelcontextprotocol.io/, 2024. the ACM, 1978. [17] OpenAI. Function calling and other api [32] Leslie Lamport. Paxos made simple. ACM SIGACT updates. https://openai.com/index/ News, 2001. function-calling-and-other-api-updates/, 2023. [33] Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The google file system. In ACM Symposium [18] LangChain. Langchain. https://github.com/ on Operating Systems Principles, 2003. langchain-ai/langchain, 2023. [34] Jeffrey Dean and Sanjay Ghemawat. Mapreduce: Sim[19] LangChain. Langgraph. https://github.com/ plified data processing on large clusters. In USENIX langchain-ai/langgraph, 2024. Symposium on Operating Systems Design and Imple[20] Swanand Rao, Kiran Kashalkar, Parvathi Somashekar, mentation, 2004. and Priya Krishnan. Aethon: A reference-based replication primitive for constant-time instantiation of [35] Frederick P. Brooks. The Mythical Man-Month. Addison-Wesley, 1995. stateful ai agents, 2026. [21] Martin Fowler. Patterns of Enterprise Application [36] David L. Parnas. On the criteria to be used in decomposing systems into modules. Communications of the Architecture. Addison-Wesley, 2002. ACM, 1972. [22] Sam Newman. Building Microservices. O’Reilly Me[37] Peter Naur. Programming as theory building. Microdia, 2021. processing and Microprogramming, 1985. [23] Thomas Erl. Service-oriented architecture: Concepts, technology, and design. 2005. [38] Anthropic. Introducing advanced tool use on the anthropic api. https://www.anthropic.com/ [24] Brendan Burns, Brian Grant, David Oppenheimer, engineering/advanced-tool-use, 2025. Eric Brewer, and John Wilkes. Borg, omega, and kubernetes. In ACM Queue, 2016. [39] Anthropic. Code execution with mcp: Building more efficient agents. https://www.anthropic.com/ [25] Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, engineering/code-execution-with-mcp, 2025. David Oppenheimer, Eric Tune, and John Wilkes. Large-scale cluster management at google with borg. In European Conference on Computer Systems, 2015. [26] D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems, 2015. [27] Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al. Accelerating the machine learning lifecycle with mlflow. In IEEE Data Engineering Bulletin, 2018. [28] Sebastian Schelter, Dustin Lange, Philipp Schmidt, Meltem Celikel, Felix Biessmann, and Andreas Grafberger. Automating large-scale data quality verification. In Proceedings of the VLDB Endowment, 2018. 10