Route Before Response: Fail-Closed Routing for LLM-Enabled Customer-Service Software

Route Before Response: Fail-Closed Routing for LLM-Enabled Customer-Service Software Yuchen Cao1 , Zhenyu Mao1 , Jacky Wai Keung1 , Yifei Wang1 , Guancheng Lin1 , Jingyu Zhang2,* 1

City University of Hong Kong, Hong Kong {cynthcao2-c, zhenyumao2-c, ywang4748-c, guanchlin4-c}@my.cityu.edu.hk, Jacky.Keung@cityu.edu.hk 2 Hong Kong Metropolitan University, Hong Kong * Corresponding author: fzhang@hkmu.edu.hk

A routing error at this boundary changes executable software behavior. For this study, a route has three fields: an action, an object, and an outcome. The action is one of five service operations: advisory, information lookup, account analysis, support, or a protected operation. When required, the object is a registered identifier used by the service; in our benchmark, it is usually a canonical ETF identifier. The outcome is either DISPATCH or one of four non-dispatch results: CLARIFY , DEFER, HANDOFF, or REJECT. Together, these fields define the final route. A false dispatch sends a request that should stop; a false block stops one that should be dispatched. Even valid JSON may name an unknown object or request an operation forbidden by the current state. These failures define boundary-aware pre-response routing. Its input combines a request and history with a service catalog, registered objects, runtime state, and versioned rules. Its software-visible output is either an allowed action–object dispatch or a specified non-dispatch outcome. Unlike destination selection alone, this task requires the semantic action, referenced object, current state, and final outcome to agree before execution. The objective is to preserve flexible language interpretation while ensuring that the selected service is permitted, every required object is registered, and the decision can be replayed from recorded software inputs. The acceptance I. I NTRODUCTION object is therefore a complete executable route rather than an LLM-enabled customer-service software in financial institu- isolated label. tions increasingly connects one conversational entry point to Prior work covers individual stages. Intent and out-of-scope information lookup, account analysis, advisory services, support methods return labels or abstain, LLM routers choose destinaworkflows, and protected operations. Tool-augmented language tions, and tool-use methods produce calls and arguments [9]– models can interpret open-ended requests and construct calls [12]. Entity linking maps mentions to registered identifiers, and to external services [1]–[4]. These destinations expose distinct runtime controls constrain later transitions [5]–[8]. Without interfaces, identifiers, and runtime preconditions. Together, they a separate pre-dispatch verification layer, these outputs do make the entry point a software integration boundary that not establish a complete route in our setting. Three conditions passes requests, identifiers, and responsibility to downstream remain: the service matches the request, state and policy permit components. it, and any required object resolves to a registered identifier. Yet a service that matches the request is not necessarily A local result can appear correct while the composed system allowed by the current software state and policy, and a refer- dispatches a forbidden service or invalid object [13]–[16]; enced entity may not map to a valid system identifier [5]–[8]. evidence from source-code tasks likewise cautions against Figure 1 fixes the request, history, and referenced object while relying on an LLM’s own verification of flawed outputs [17]. changing only one permission value. One record dispatches We present RouteBR, which stands for Route Before the protected operation; the otherwise identical record ends in Response, as a pre-response controller for this boundary. One rejection. The correct downstream route therefore depends on model call proposes a primary service action and registered runtime information that is absent from the request text. object. The controller keeps that primary choice, so a lower-

arXiv:2611.07534v1 [cs.SE] 5 Nov 2026

Abstract—LLM-enabled customer-service software in financial institutions must choose a downstream service before responding. Classifiers and LLM routers can identify a relevant label or destination, but a composed application must still verify dispatch permission and entity registration. We study this integration problem as boundary-aware pre-response routing. We present RouteBR, a controller in which one model call proposes a primary service action and registered object, then deterministic code checks identifiers and versioned rules before dispatch or a specified nondispatch outcome. Lower-ranked candidates cannot replace a blocked primary route. We construct 600 privacy-preserving scenarios from recurring confidential-request patterns at a real financial institution; three engineers jointly reviewed the scenarios and labels. On the 200-case CONFIRM MAIN split, RouteBR reaches 97.5% final-path exact match. A matched one-call baseline given the same rules reaches 95.5%, whereas a diagnostic baseline without the rule program obtains 94.0% action accuracy but only 41.0% final-path exact match. Across 166 non-dispatch MAIN and FLIP cases, RouteBR produces no false dispatch. The artifact pairs all 1,960 prediction rows with raw receipts while excluding gold and reviewer fields from model requests. These results support separating model proposals from codeowned, replayable execution. Our implementation is available at https://github.com/YgcCoder/RouteBR. Index Terms—large language models, LLM-enabled software, customer service, financial software, service routing

eserved

from natural-language criteria [25], whereas LLM routers select models or services and guarded variants add rejection or "Explain 300ETF (510300) and submit the protected request." prompt-optimized routing behavior [11], [26], [27]. Tool-use service available verified session entity = 510300 benchmarks assess interface choice, arguments, execution, or API collections [2], [3], [12], [28], while τ -bench combines State A State B interaction, tools, policies, and final database state [29]. These permission = false permission = true studies establish the relevance of routing and policy following. REJECT ALLOW RouteBR narrows the evaluated output to one pre-dispatch DISPATCH protected_action(510300) BOUNDED TERMINAL action–object pair or a specified non-dispatch outcome. A wrong outcome is a route error, dispatch when the gold route One runtime predicate changes the admissible route should stop is a safety error, and refusal under ALLOW is an availability error. Table I compares these evaluated outputs, not Fig. 1. Motivating state-conditioned route. The request and object are fixed; changing only permission from true to false changes the result from whether the other systems could be extended. protected-operation dispatch to rejection. Matching the request to a service The distinction also changes the oracle. Router evaluation does not by itself determine an executable route. commonly asks whether the selected destination is appropriate, ranked alternative cannot replace it after a rule blocks dispatch. while tool-use evaluation may continue through arguments, Deterministic code applies the recorded state and policy, and execution, and task outcome. Our controller ends earlier but the object check verifies the registered identifier before the requires a stronger precondition at that point: the selected controller emits either a typed dispatch or a specified non- action, canonical object, versioned state, and terminal must dispatch outcome. Alternatives are recorded for inspection only agree before any response service is invoked. A correct bounded and cannot be dispatched. In this way, the model interprets the terminal cannot erase a wrong semantic action from diagnosis, request while ordinary code owns the final software transition. and a correct action cannot compensate for dispatch under a prohibiting state. This contract-level oracle motivates reporting Our contributions are: path, false-dispatch, and false-block measures together rather • We formulate boundary-aware pre-response routing as a software decision over a service action, registered object, than treating routing as destination accuracy alone. runtime state, and final routing outcome. C. Entity Linking and Runtime Control • We design RouteBR, which separates model proposals Named-entity recognition identifies spans and types; entity from code-owned state checks, closed-list object validation, linking maps mentions to canonical identifiers [5], [30]–[33]. controlled repair, and replayable decision traces. RouteBR uses a smaller closed-list problem: the supplied • We construct a privacy-preserving benchmark of 600 synregistry defines the only dispatchable objects, and ambiguity thetic reconstructions from recurring confidential-request becomes UNRESOLVED . Runtime rails, adaptive guards, and patterns, jointly reviewed by three engineers from the executable specifications separately constrain model-driven institution and organized into MAIN, FLIP, and ENTITY transitions [6]–[8]. RouteBR joins these restrictions after cases. semantic selection: code checks the committed action against • We evaluate complete-path correctness, state-conditioned execution, the closed-list object contract, and backend state and its object against the catalog before invocation. A portability through matched one-call controls, two controlled trace can therefore attribute a bounded path to a named rule, unresolved object, or persistent interface failure rather than to a mechanism checks, and six hosted model backends. generic refusal. This scoped contract complements general NER II. R ELATED W ORK and agent-safety work with behavior that is directly testable at a software dispatch boundary. A. Customer-Service Task Formulations Closed-list linking serves here as an executable interface Financial question-answering benchmarks evaluate answers over documents and tables [18], [19]. Task-oriented dialogue contract. The registry is supplied for the current request, represents services through intents, slots, state, and API argu- and the relevant question is whether a proposed identifier is ments [20]–[22]; intent, out-of-scope, and selective prediction both canonical and permitted by the selected action contract. add labels or abstention [9], [10], [23], [24]. These tasks An out-of-registry string is therefore more than a linking motivate service descriptions, history, and bounded refusal, error: if allowed to cross the boundary, it becomes an invalid but their target is principally an answer, label, or call inferred downstream argument. Conversely, resolving every ambiguity from dialogue. Our boundary instead receives runtime state by refusal can be safe but unavailable. This connection between as a separately versioned software record. This permits paired canonical identity and dispatchability is why RQ3 evaluates tests in which request-side input is fixed while one predicate object accuracy together with final path and false block. Table I compares the reported evaluation objects of each and the admissible output change. formulation. A check means that the cited formulation explicitly B. Semantic Routing and Tool Calling evaluates the property; a cross means that the property lies Routing research evaluates different output objects. Autore- outside its reported evaluation target. Boundary-aware pregressive semantic route planning predicts navigation paths response routing combines the closed object, versioned state, Same Request Different Runtime State

rules with a default ALLOW model the boundary without reproducing institutional policy. Evaluation wraps the controller: DEV, SELECT, and CONFIRM supply cases, backends replace only the proposer, B1 and B2 run in parallel, and path, safety, and cost metrics consume final results and traces. Formulation Output Obj. State Typed Code Flip Table II summarizes the interface surface. The names describe response-service families, not customer-visible business Intent/OOS [9], [23] Label/reject × × × × × units. Four actions are object-sensitive, while support can be Entity linking [5], [31] Canonical ID ✓ × × × × object-free. This difference is encoded in the catalog rather than LLM routing [11] Destination × × × × × inferred after routing. A protected action is not special-cased Guarded routing [26] Service/reject × × × × × in the model prompt; its additional identity and permission Tool use [2], [3], [12] Tool call × × ✓ × × requirements are ordinary policy rules evaluated only after it Runtime control [8] Transition × ✓ ✓ ✓ × RouteBR(ours) Final path ✓ ✓ ✓ ✓ ✓ becomes the committed primary action. The model produces q = ⟨a, o, K⟩, where a ∈ S is typed action contract, deterministic transition owner, and paired the primary action, o ∈ E ∪ {NONE, UNRESOLVED}, and K state test in one pre-dispatch decision. contains at most three unique catalog actions. The contract The comparison identifies the checks required at component requires a = K1 . This equality turns ranking into an executable handoffs. Dialogue models supply request-side structure, routers commitment: K2 and K3 are retained for audit, but never supply candidate destinations, entity linkers supply registered become fallback routes after policy evaluation. identifiers, and runtime-control work supplies executable guards. We define a final path as the pair of an executable action Boundary-aware pre-response routing evaluates how these and object when the terminal is DISPATCH, or as one bounded outputs compose: the semantic stage commits to one current terminal otherwise. Exact-match evaluation therefore compares action, the object remains inside a supplied registry, and the software-visible result, not an internal ranking. Two code reproduces the state transition from recorded inputs. Its proposals with the same action can differ in final path because regression oracle must therefore preserve the relation among their objects differ; two identical proposals can also produce action, object, state, and outcome. Testing only the action can different final paths because z selects different policy outcomes. accept an unsafe dispatch, while testing only the outcome can This definition is what links semantic routing to the integration conceal a wrong but conservatively blocked route. contract exercised by the experiments. Figure 1 illustrates the resulting composition. Both records III. ROUTE BR provide the same request, history, catalog entry, and canonical A. Overview and Formulation ETF. The proposal contract should therefore produce the same Figure 2 places RouteBR between a customer entry protected-action pair. The state gate then reads the only changed point and downstream response services. Its input is I = field, permission, and selects either the named ALLOW rule ⟨x, h, S, E, z, R⟩: request x, bounded history h, versioned or the bounded REJECT rule. Only the former reaches the object service catalog S, closed entity registry E, runtime state z, and contract and dispatch return. This controlled pair holds semantic prioritized rules R. The output is either a dispatch record (a, o) evidence fixed while one versioned runtime fact changes the or one terminal in {CLARIFY, DEFER, HANDOFF, REJECT}. expected path. This decomposition yields three independently testable Customer-facing response generation is outside this boundary. The trust boundary separates three owners. Product engineers interfaces. The proposal interface accepts only catalog and own the catalog and the versioned rule set; the session and registry identifiers. The enforcement interface accepts an service infrastructure own z; the model contributes only a already validated primary action plus a state and rule version, typed semantic proposal. A downstream response service is and returns a named boundary decision. The dispatch interface not addressable from the model adapter. It can be reached only accepts only an ALLOW result and a validated action–object pair. through the controller’s dispatch return, after all contracts have A release can therefore replay policy and object checks from a succeeded. Consequently, a parseable model output is evidence stored proposal without calling the hosted model. Conversely, for semantic selection, not authorization. This division also prompt or backend changes can be tested against frozen state makes configuration changes local: a new rule version can alter TABLE II admissibility without changing the prompt, whereas a catalog C LOSED ACTION CONTRACTS USED BY ROUTE BR. A LL IDENTIFIERS ARE change can be reviewed as an interface migration. VERSIONED CATALOG VALUES . The lower panels represent offline dependencies and evaluaID Response-service family Permitted object type tion. Frozen assets map to distinct control points: the catalog constrains proposal and action validation; rules drive the state A1 Advisory ETF A2 Information lookup ETF or topic gate; the registry supplies candidates and the closed-list check; A3 Account analysis Account or ETF and the typed contract validates proposal and result records. A4 Support None or service The catalog has five action families with stable identifiers and A5 Protected action ETF or account permitted object types; eight synthetic predicates and prioritized TABLE I R EPRESENTATIVE FORMULATIONS AND EVALUATED PROPERTIES . O BJ .: CLOSED OBJECT; S TATE : RUNTIME PREDICATES ; T YPED : ACTION CONTRACT; C ODE : DETERMINISTIC TRANSITION OWNER ; F LIP : PAIRED STATE TEST.

Online Pre-Response Control Customer Request

Semantic Proposal

Deterministic State Gate versioned predicates

primary action

Dispatch

Closed-List Object Check canonical ID + type

response service

Primary Route Commitment

closed-list object

Bounded Terminal

diagnostic top-k

1st

2nd

k-th

Proposal Audit

ON

OFF

NA

ALLOW

BLOCK

N/A

clarify / defer / handoff / reject Blocked primary terminates

Frozen Engineering Assets Service Catalog

Versioned Rules

Object Registry

Reproducible Evaluation Typed Contract

Six Hosted Backends

Dev/ Select/ Confirm

B1 / B2 Controls

M1

M2

M3

M4

M5

M6

Path / Safety / Cost

B1 B2

proposal + commitment

state gate

proposal + object check

proposal + result boundary

frozen cases

parallel systems

results + traces

Fig. 2. RouteBR execution and evaluation framework. The online path commits to one semantic proposal, then code applies state and object checks before dispatch or a specified non-dispatch outcome; lower-ranked candidates are recorded only for inspection and cannot be dispatched. Frozen assets feed their corresponding control points, not the customer-request card. Evaluation wraps and measures the controller rather than entering its online path.

and policy assets. The implementation and evaluation use these seams to separate semantic, state, object-validation, and adapter errors rather than assigning every wrong path to the model. B. Proposal and Primary Commitment The prompt receives request text, dialogue history, catalog descriptions, and candidate entities, but not runtime predicates or rules. It selects the current requested function rather than a hypothetical later action, and resolves an object only through exact code, name, alias, or unique antecedent evidence. The prompt returns one JSON object with action_id, object_id, and topk. This separation prevents the semantic call from adapting its preferred action to whatever runtime route happens to be allowed. The prompt uses the catalog and entity arrays as data, not instructions. It gives priority to a complete current request with an explicit object or imperative expression; emotional language, conditional future operations, and supporting explanations remain secondary cues. For object resolution, exact codes and aliases precede display names and a unique antecedent in the bounded history. Similarity matching and generated identifiers are forbidden. These constraints make every accepted value checkable against the supplied interface, while action and finalpath metrics evaluate semantic correctness separately. The validator rejects unknown or duplicated actions, disagreement between the primary action and K1 , out-of-registry objects, and object types incompatible with the primary action. It permits one format-only repair that receives the invalid output and validation errors but may not replace semantic evidence or closed identifiers. A second failure reaches a configured nondispatch terminal. The accepted proposal and every validation error remain in the trace.

Validation proceeds from syntax to enumerated identifiers and then to cross-field relations. This order separates a malformed provider response from an unknown catalog value, and separates both from an object whose type is incompatible with an otherwise valid action. The trace retains the first failed contract and the supplied value rather than only the eventual terminal. As a result, fail-closed behavior does not erase the maintenance signal: adapter owners can investigate schema failures, catalog owners can inspect action errors, and registry owners can review unresolved or mistyped objects without inferring the cause from a generic refusal. Repair is deliberately narrower than a second routing attempt. The original payload, invalid output, and error list are preserved, and the repair instruction asks only for a contract-conforming representation of the same semantic choice. This prevents repeated calls from searching for an action that happens to pass the policy. If the corrected object still lies outside E, if the primary action changes, or if the output again violates the schema, the controller terminates. The experiments count repair as an additional API call and report first-pass JSON validity separately. Primary commitment is the key compositional invariant. Suppose the request supports a protected action and an advisory alternative. If the protected action is primary but current permission is false, the controller returns the rule’s terminal. It does not reinterpret the lower-ranked advisory candidate as an executable substitute. This prevents semantic ranking from becoming an implicit policy-bypass mechanism. This invariant also clarifies responsibility for mixed-service language. The model must choose which function is current and primary; the policy engine decides only whether that action is admissible. The engine does not rank semantics, and the model does not search the policy space. A wrong primary

action remains a measurable semantic error rather than being concealed by a later fallback. Diagnostic alternatives can still help engineers inspect ambiguity or revise catalog descriptions, but they have no execution authority. C. Deterministic Enforcement and Object Validation

into the rule engine. RQ2 requires both pair members to reach their expected final paths. D. Executable Contract and Implementation

Algorithm 1 summarizes the complete control path. The reference implementation defines typed records for requests, For the committed action a, the policy engine selects the services, entities, rules, traces, and results; a replay backend first matching rule: for tests; an OpenAI-compatible adapter; and a deterministic b(a, z, R) = rj .decision, j = arg min ri .priority. policy engine. Ten unit tests cover normal dispatch, invalid i:ri (a,z)=true (1) JSON, out-of-list objects, primary-route commitment, state The model does not predict b and cannot override it. If removal, object-validation removal, and adapter behavior. These b ̸= ALLOW, RouteBR records the winning rule and returns tests establish control-flow properties, not empirical routing the corresponding terminal. The state-removal intervention in accuracy. Configuration loading follows the same fail-closed rule as RQ2 bypasses this transition while keeping the same semantic proposal, exposing the errors prevented by code-owned policy. request handling. Controller construction rejects an empty Rules are ordered by numeric priority and then stable catalog and rejects DISPATCH as a fallback terminal. The rule identifier. When several rules match, the rule with the policy engine requires at least one rule, rejects duplicate rule smallest priority number determines the routing outcome. In the identifiers, and raises an error if evaluation reaches the end evaluated cases, the active non-default predicates cover safety without an explicit default. Rules are sorted by numeric priority escalation, unsupported requests, service unavailability, unclear and stable identifier before any request is processed. These scope, unresolved entity state, unverified identity, and missing checks move configuration defects to startup or replay instead protected-operation permission. A final empty-condition rule of converting them into an implicit runtime choice. They returns ALLOW, making the policy function total for every also make a policy snapshot self-contained: given an accepted catalog action. This explicit default is important for replay: proposal, state record, and rule version, the winning boundary absence of a matching prohibition is represented by a named is deterministic. rule rather than by missing control flow. The unit tests exercise each executable branch with a When b = ALLOW, the object contract checks whether replay backend. They verify an ordinary dispatch, a blocked a requires an object, whether o belongs to the supplied primary action whose allowed secondary remains diagnostic, registry, and whether its type is permitted by the catalog. An an unresolved object, persistent out-of-list output after repair, unresolved required object returns CLARIFY; object-free support the two safeguard removals, and an object-free service. Adapter paths require NONE. The closed-resolver intervention in RQ3 tests separately check response identity and receipt handling. makes object-sensitive proposals unresolved. It measures the Each assertion targets the public result and trace rather than a availability cost of losing closed-list object validation rather private model explanation. The benchmark then tests the part than silently dispatching an unchecked object. these unit tests cannot establish: whether the hosted model Let b = b(a, z, R) and let ⊥o denote an unresolved object. chooses the correct semantic action and object on varied The final transition is customer-service language. This division keeps construction  guarantees, integration tests, and empirical accuracy as distinct  b ̸= ALLOW, b, evidence. T (q, z) = CLARIFY, b = ALLOW, o = ⊥o , (2)  The implementation is dependency-free Python at the con DISPATCH (a, o), b = ALLOW , Ca (o), troller layer. Immutable catalog, entity, request, and rule records where Ca is the action-specific closed-object contract. Persis- make version changes explicit, while the result object contains tent validation failure is handled before T and reaches the terminal, action, object, reason codes, and a trace. Policy configured fail-closed terminal. The reference implementation evaluation is a stable linear scan over the prioritized rules; enforces three resulting invariants: every dispatch action object membership is a hash lookup over the request’s candidate belongs to S; every required dispatch object belongs to E registry. With the small catalog used here, deterministic work and satisfies Ca ; and no dispatch is returned when the winning is negligible relative to the model call, but the separation also rule is non-ALLOW. These are construction properties of the avoids sending policy reasoning through multiple model turns. The implementation records requested and response-reported control path, while semantic correctness of a and o remains model identities, prompt and input hashes, token usage, latency, empirical. repair status, and the accepted transition. Credentials remain State pairing gives a direct test of this separation. For two outside the run ledger. The controller makes one semantic call inputs that differ only in z, the semantic proposal should per case unless bounded repair is needed, so its call count is remain unchanged. The boundary may change because b is directly comparable with the two one-call baselines. a pure function of the committed action, state record, and rule version. Conversely, if a model changes a for identical Across the frozen matrix, the artifact pairs all 1,960 norlanguage across a pair, the error is visible rather than absorbed malized prediction rows with 1,960 raw provider receipts

Algorithm 1 Route before response Require: request x, history h, catalog S, entities E, state z, rules R Ensure: typed dispatch (a, o) or bounded terminal 1: q ← P ROPOSE(x, h, S, E) 2: q ← VALIDATE O R R EPAIR(q) 3: if q = ⊥ then 4: return FAIL C LOSED 5: end if 6: (a, o, K) ← q; assert a = K1 7: b ← F IRST M ATCHING RULE(a, z, R) 8: if b ̸= ALLOW then 9: return b 10: end if 11: if a requires object and o = UNRESOLVED then 12: return CLARIFY 13: end if 14: assert a ∈ S and o satisfies the closed object contract 15: return D ISPATCH(a, o)

and records 27 run manifests. The adapter validates responsereported model identity, while request bodies exclude gold, annotator, and adjudication fields. This supports deterministic replay from an accepted proposal, provider inspection from a receipt, and aggregate reconstruction from the ledger and frozen hashes. Before release, engineers can compare this ledger with the preceding frozen contract and inspect changed dispatch decisions. This comparison turns release review into a contract-level change audit across model and code boundaries. Fail-closed behavior is configurable only to one of the four non-dispatch terminals; DISPATCH is rejected as a fallback during controller construction. Persistent JSON or contract failure therefore ends at a recorded bounded outcome. Semantically wrong but contract-valid proposals remain visible in the action and final-path metrics and error analysis. This measurement boundary positions RouteBR as the dispatch controller within a larger customer-service system. The versioned assets also define the change surface. A rule edit changes admissibility and triggers FLIP regression; a registry edit changes valid object identifiers and triggers ENTITY regression; a catalog or prompt edit changes semantic selection and triggers MAIN plus the known account-analysis/information-lookup confusion cases. Adapter changes require schema, response-identity, timeout, repair, and fail-closed tests. These mappings keep a policy correction from being evaluated only as a prompt change and preserve the primary-commitment invariant across maintenance releases. Deployment observability follows the same contract. A decision record carries a stable request identifier, catalog and rule hashes, primary proposal, validation result, winning rule, object-resolution result, final path, repair or fallback status, and backend-reported identity. Request text and provider receipts remain subject to institutional access controls; a public artifact can retain schemas, hashes, synthetic cases, and normalized predictions. This separation allows an incident to be replayed from an accepted proposal without invoking the hosted backend again, while preserving the raw receipt for provider-level inspection.

IV. E XPERIMENTS AND E VALUATION A. Research Questions and Evaluation Overview We evaluate RouteBR as a boundary-aware pre-response routing controller in the customer-service setting of a real financial institution. The evaluation separates semantic selection, deterministic policy enforcement, closed-list object validation, and backend choice. A result is useful only when the final path, safety error, and operational trace agree; aggregate accuracy alone cannot distinguish unsafe dispatch from excessive blocking. The evaluation is organized around four integration decisions. A complete path must first be compared with matched one-call designs under the same information. A state-aware controller must then react to a changed runtime predicate while requestside evidence remains fixed. Closed-list object validation must be isolated from state enforcement because it controls a different dispatch precondition. Finally, the model backend is replaceable, so the same controller and test cases must be exercised across providers. These decisions yield four research questions. RQ1—Executable routing correctness. How accurately does RouteBR convert a semantic proposal into a complete executable route compared with matched one-call routing designs? RQ2—State-conditioned execution. Under identical requestside input, does deterministic state enforcement produce the correct state-dependent terminal, and what fails when that enforcement is removed? RQ3—Closed-list object contract. How does closed-list object validation affect object correctness and the availability of otherwise admissible object-sensitive routes? RQ4—Backend portability and operational variation. With the controller and evaluation cases fixed, how stable are routing correctness, safety, interface validity, and operational cost across hosted model backends? Assessment proceeds through four layers. Final-path exact match evaluates the dispatched action or bounded terminal; path macro-F1 and action accuracy expose class and semantic errors. False dispatch counts a dispatch when the gold path is bounded, while false block counts a bounded output when dispatch is admissible. Paired FLIP accuracy requires both state variants to be correct. Object accuracy, valid JSON, calls, tokens, and latency complete the evidence. Each question has a distinct decision criterion. RQ1 accepts no label-only shortcut: the action, object when required, and terminal must form the gold final path. RQ2 treats the two members of each FLIP pair as one test and uses Astate to expose the consequence of removing deterministic authorization. RQ3 evaluates object resolution only on ENTITY and uses A-ground to measure both object correctness and lost admissible dispatch. RQ4 freezes the controller and cases while replacing only the backend profile. This mapping keeps state and object mechanisms separate, prevents the backend screen from being interpreted as an architectural ablation, and keeps SELECT evidence separate from the CONFIRM evidence used for the main claims.

P For a set of n cases, final-path exact match is n−1 i 1[ŷi = be admissible but the object contract still determines whether yi ]. Macro-F1 averages F1 over the five dispatch actions and dispatch can complete. All three views use the same final-path four bounded terminals observed in the benchmark. False type. This common representation permits one controller and dispatch uses only gold non-dispatch cases as its denominator; one scoring implementation to expose different failure sources false block uses only gold ALLOW cases. We report both counts without changing the claimed software output between RQs. and rates so a low rate cannot conceal a small denominator. Gold fields remain separate from every model request FLIP pair accuracy counts a pair only when both members and response receipt. The evaluator reads the frozen action, are exact. Exact McNemar tests compare paired correctness boundary, canonical object, and final path only after the vectors. The artifact records Wilson intervals for final-path controller has emitted its result. It then recomputes denominator exact match and FLIP pair accuracy. Statistical tests do not membership for false dispatch and false block from the treat the two members of a FLIP pair as independent pair-level gold boundary rather than from the prediction. This ordering observations. matters for fail-closed systems: a malformed response still Together, these metrics separate semantic selection, autho- contributes its configured bounded outcome, while a refusal rization, grounding, and availability, while JSON, identity, call, on an admissible case remains a false block. Thus, interface token, and latency fields characterize the adapter and backend failure, conservative fallback, and semantic error are counted as software-visible outcomes rather than filtered before scoring. boundary. Before execution, a read-only audit checks case identifiers, B. Benchmark, Systems, and Protocol label domains, JSON fields, catalog and rule references, and The benchmark contains 600 privacy-preserving customer- the 40 canonical entity identifiers. It rejects normalized request service scenarios reconstructed from recurring patterns in overlap among the protected non-FLIP partitions and requires confidential requests; raw requests and customer identifiers every FLIP identifier to contain exactly two records with remain confidential. Three engineers from the institution jointly identical request-side input, different runtime records, and reviewed the scenarios and current labels for practice relevance. exactly one ALLOW boundary. The audit prints split statistics The cases cover five service families and 40 public ETF and SHA-256 hashes without editing any row. Practitioner entities whose exchange codes, names, and source records are review establishes practice relevance; the automated audit documented in the released entity ledger. Runtime states and protects the structural assumptions used by the paired metrics benchmark policies are synthetic. Labels were reviewed jointly and run scripts. without independent duplication, so inter-annotator agreement The 600 scenarios contain 123 cases for each of advisory, was not computed. information lookup, account analysis, and protected action, Construction begins from recurring task and ambiguity plus 108 support cases. Their gold boundaries comprise 262 patterns instead of copied utterances. Templates vary the ALLOW, 128 CLARIFY, 75 DEFER, 75 HANDOFF, and 60 requested service, cross-service cues, object-reference form, and REJECT cases. Object-reference difficulty includes exact names, runtime boundary, then express each combination as a privacy- codes, anaphora, conflicting mentions, underspecification, and preserving request. Public ETF codes, display names, and object-free requests. The controlled distribution targets conexchange sources provide real closed identifiers. The released tract boundaries and mixed-service language; estimating issue language is reconstructed; customer, account, transaction, and frequency in production traffic requires a separately sampled internal-rule records remain confidential. This design exercises operational dataset. the software interfaces while preserving the institution’s data The 120-case DEV split is used only for prompt and boundary. interface checks. SELECT contains 160 cases (100 MAIN, The construction ledger retains the scenario cell, bounded 40 FLIP, and 20 ENTITY) and compares hosted backends. history, runtime record, candidate registry, cue evidence, action, The disjoint CONFIRM set contains the remaining 320 cases object difficulty, canonical object, boundary, final path, and (200 MAIN, 80 FLIP, and 40 ENTITY) and evaluates systems applicable rule identifiers. These fields separate the natural- and safeguards. Every FLIP subset contains complete samelanguage request from the executable oracle. In particular, a request pairs. In 23 CONFIRM cases, more than one nongold action does not determine a gold terminal without the default rule matches; the frozen numeric priority determines versioned state and rules, and a dispatchable action does not the gold routing outcome. All reported calls use the same determine a complete path without a valid canonical object. frozen prompt, catalog, rules, dataset rows, and deterministic Public entity records also retain their exchange source and validators. verification date, allowing the closed registry to be checked Table III makes the data flow explicit. SELECT and CONindependently of the synthetic request text. FIRM partition the 480 non-DEV cases without overlap. Their MAIN, FLIP, and ENTITY are evaluation views over the action and boundary distributions retain coverage of all five same versioned schema rather than unrelated tasks. MAIN actions and all five boundary values; paired records are assigned samples the ordinary composition of semantic, state, and object as complete units. MAIN exercises the ordinary mixture of decisions. FLIP duplicates request-side evidence but changes services, objects, and boundaries. FLIP consists of two records one state record, making the pair itself the regression unit. with identical request-side input and one state change. ENTITY ENTITY concentrates reference forms for which the action may concentrates exact-code, exact-name, anaphoric, conflicting,

and underspecified references. TABLE III We compare RouteBR with B1 Direct, which predicts F ROZEN BENCHMARK PARTITIONS . M/F/E: MAIN, FLIP, AND ENTITY; A/B: DISPATCHABLE AND BOUNDED CASES . the final route without an explicit rule program, and B2 Direct+Guard, the principal baseline that receives the complete Partition Cases M/F/E A/B Role rules and predicts the route in one call. Both use the same DEV 120 – 50/70 Prompt/interface checks catalog, entities, decoding controls, and output validator. RQ2 SELECT 160 100/40/20 76/84 Backend screening uses the offline A-state construction check, which bypasses CONFIRM 320 200/80/40 136/184 RQ1–RQ3 deterministic enforcement; RQ3 uses A-ground, which disables the closed object resolver. A-state and A-ground are mechanism 320 for RQ2 (three systems and A-state on 80 FLIP cases), checks, while B2 is the end-to-end architecture comparator. All 80 for RQ3 (RouteBR and A-ground on 40 ENTITY cases), evaluated systems terminate at dispatch-boundary records; live and 960 for RQ4 (six backends on SELECT-160). RQ1–RQ3 institutional services remain outside experimental execution. use the same primary backend; RQ4 changes only the backend B1 tests whether an LLM can infer a complete contract profile. Hashes freeze the controller, prompt, SELECT file, and directly from request-side information and state without the CONFIRM file. The receipt and request-field audit described explicit rule program. B2 is a stronger prompt-embedded guard: in Section III verifies the evidence chain for every matrix row. it receives the same catalog, object candidates, versioned The evaluation gates correspond to the deployed contracts state, complete rule list, and output contract as RouteBR, rather than to one aggregate score. MAIN tests the complete cites rule identifiers, and predicts action, object, boundary, action–object–terminal path, FLIP tests whether a changed and terminal itself. The information is therefore matched; the predicate changes the path while request evidence is fixed, and decisive difference is that B2 predicts the transition, whereas ENTITY tests whether closed-list object validation preserves ordinary code computes it in RouteBR. All three systems use admissible dispatch. False dispatch and false block are sliced one model call on a valid first pass and the same format-only by their applicable denominators, while JSON validity, repair, repair rule. identity, calls, and latency characterize the adapter boundary. The comparisons have distinct roles. B1 omits the rule Evidence of improvement therefore requires path correctness program and diagnoses the gap between plausible semantic together with both error types; indiscriminate blocking and routing and the complete contract. B2 controls information label-only success fail this criterion. availability and provides the principal architecture comparison. The protocol also separates semantic and executable accepA-state and A-ground reuse accepted proposals and modify tance. First, the adapter must return contract-valid JSON with one executable mechanism, isolating state and object effects identifiers drawn from the supplied catalog and registry. Second, under the same semantic prompt. The six-backend panel the deterministic controller must reproduce the expected fixes controller and cases. Thus, B2 tests architecture, B1 boundary and object contract from frozen inputs. Third, the diagnoses the semantic-contract gap, the interventions attribute final path is compared with gold, including the exact bounded mechanism effects, and the panel tests portability. terminal. Finally, the ledger checks model identity, repair, The backend panel includes DeepSeek V4 Pro and Flash, fallback, call count, token use, and latency. A case that fails an GPT-4o and GPT-5.2, and two Qwen configurations. Exact earlier interface remains in the denominator with its configured requested and response-reported identifiers, endpoints, and fail-closed outcome. This ordering matches the deployed control request options are retained in the artifact rather than expanded path and prevents interface failures from disappearing during in the paper. Temperature is zero, reasoning modes are disabled, metric aggregation. concurrency is one per task, and one format-only repair is allowed. We present backend screening before RQ1–RQ3 C. Confirmatory Contract Evidence results because the final matrix fixes DeepSeek V4 Pro as RQ1—Executable routing correctness. Table V reports the the architecture backend. Paired system differences use exact matched CONFIRM comparison. On MAIN, RouteBR reaches McNemar tests. 97.5% final-path exact match and 96.3% macro-F1, compared The final experiment matrix predesignates DeepSeek V4 with 41.0% and 36.2% for B1. B1 nevertheless obtains 94.0% Pro for RQ1–RQ3 and uses the disjoint SELECT partition to action accuracy. Its 118 path errors include 96 false dispatches document the backend choice. The screening table is presented among the 126 bounded MAIN cases, although all 200 outputs before the RQ result tables for that reason; it is not pooled are accepted JSON without fallback. Semantic action accuracy with CONFIRM. Requests are executed through provider- therefore does not establish that the action–object–terminal path compatible chat-completion endpoints. Each profile freezes is executable. RouteBR significantly exceeds B1 (p < 0.001). the model name and accepted response identities, and every B2 is the more demanding comparison because it receives case stores the reported identity. Because hosted checkpoints the same state, rules, catalog, object candidates, and output can change behind stable aliases, the requested and response- contract. It reaches 95.5% MAIN exact match, and its difference reported identities define the evaluated service profile and its from RouteBR is not significant (p = 0.344). A well-specified replication scope. one-call guard is therefore competitive. The architectural The matrix contains 1,960 system–case executions before distinction is transition ownership: B2 predicts the boundary repair calls: 600 for RQ1 (three systems on 200 MAIN cases), inside the model output, whereas RouteBR computes it from

TABLE IV BACKEND SCREENING ON SELECT-160. VALUES ARE PERCENTAGES EXCEPT CALLS , TOKENS , AND MEDIAN LATENCY. FD: FALSE DISPATCH ; FB: FALSE BLOCK .

TABLE V E VIDENCE FOR RQ1 AND RQ2 ON CONFIRM. MAIN AND PAIR VALUES ARE PERCENTAGES ; SAFETY COUNTS POOL 200 MAIN AND 80 FLIP CASES . System

Backend

Path FD FB JSON Calls

DeepSeek V4 Pro 97.5 0.0 1.3 DeepSeek V4 Flash 97.5 0.0 2.6 Qwen3.8 Max 97.5 0.0 0.0 GPT-4o 96.9 0.0 2.6 Qwen3.7 Plus 96.2 0.0 1.3 GPT-5.2 95.6 0.0 6.6

100.0 99.4 100.0 100.0 100.0 100.0

MAIN Macro-F1 FLIP pair

FD

FB

Tok./ms

1.00 762/1367 1.01 767/996 1.00 737/1930 1.00 730/1853 1.00 739/2293 1.00 742/2329

RouteBR B1 Direct B2 Direct+Guard

97.5 41.0 95.5

96.3 36.2 94.5

92.5 0/166 1/114 10.0 127/166 2/114 100.0 1/166 2/114

effect on object correctness and admissible dispatch. Unlike state enforcement, it determines whether an admissible objectsensitive request can complete. The two object mismatches made by full RouteBR occur on already bounded CLARIFY cases, so both retain the correct terminal. Four additional account-analysis requests are proposed as information lookup but also remain bounded. The trace therefore preserves diagnostic evidence even when the softwarevisible path is conservative. By contrast, A-ground converts every admissible object-sensitive route to CLARIFY; its zero false-dispatch count is achieved through complete loss of availability on that subset. RQ3 consequently shows that object correctness and false block must be assessed together. Across the 1,000 CONFIRM executions assigned to RQ1– RQ3, no run requires repair, fallback, or a transport-error outcome. Interface reliability therefore does not explain the measured differences among the systems and interventions.

the committed action and versioned state, making the decision directly replayable from the winning rule. RouteBR produces five wrong final paths among the 200 MAIN cases. Four confuse account analysis with information lookup; one support request is proposed as advisory and then clarified because the selected action requires an object. Five additional action errors are masked by a correct non-dispatch terminal. These cases explain why we report both action and final-path measures: a safe terminal can still conceal a semantic defect, while a plausible action can conceal an unsafe path. RQ1 therefore finds that the complete contract, rather than action accuracy alone, is the appropriate acceptance object for pre-response routing. RQ2—State-conditioned execution. The 40 FLIP pairs hold request text, history, catalog, and object evidence fixed while changing one runtime predicate. RouteBR obtains 37/40 correct D. Backend Portability pairs (92.5%); B1 obtains 10.0%, whereas B2 obtains 100.0%. RQ4—Backend portability and operational variation. TaB2 therefore leads this paired measure, 40/40 to 37/40. In ble IV shows that all six backends remain within 1.9 percentage each of RouteBR’s three missed pairs, the proposal confuses points on final-path exact match. DeepSeek V4 Pro, V4 Flash, account analysis with information lookup. The code-computed and Qwen3.8 Max each reach 97.5%; GPT-4o reaches 96.9%, boundary remains correct on all 80 members: the bounded Qwen3.7 Plus reaches 96.2%, and GPT-5.2 reaches 95.6%. member reaches the required terminal, while the admissible None produces a false dispatch on SELECT. Qwen3.8 Max member dispatches the wrong service. The residual error is produces no false block, while V4 Pro and Qwen3.7 Plus semantic, not a rule-engine failure. each produce one. V4 Flash is fastest, but one of the 960 The A-state construction check isolates authorization. By- backend–case executions remains invalid after bounded repair passing deterministic enforcement lowers FLIP exact match and reaches the fail-closed fallback, yielding 99.4% accepted from 96.2% to 46.2%, obtains 0/40 correct pairs, and dispatches JSON and 1.01 calls per case. The narrow path-score spread all 40 members that require a bounded terminal. This mech- indicates contract portability; false blocks, JSON acceptance, anism intervention identifies the consequence of removing and latency distinguish the profiles. code-owned b(a, z, R), while B2 supplies the architecture The latency and token columns describe hosted-service comparison. RQ2 therefore finds that paired same-request tests operation at the time of access and include network and provider expose state-conditioned execution within the frozen equality- variation. Tokens remain close because all profiles receive predicate policies. the same payload and output contract; latency varies more Across MAIN and FLIP, RouteBR produces 0 false dis- substantially across endpoints. The persistent failure remains in patches among 166 bounded cases and one false block among the denominator as the software-visible fallback result. Across 114 dispatchable cases. B1 dispatches 127/166 bounded cases; the six profiles, accepted JSON is 99.4–100%, calls remain B2 dispatches 1/166. These counts complement pair accuracy: 1.00–1.01 per case, and median latency ranges from 996 to B2 is perfect on the FLIP oracle, while RouteBR supplies a 2,329 ms. transition that can be reproduced from versioned code and the The final matrix fixes V4 Pro as a balanced backend. It ties recorded winning rule. the highest path score, produces zero false dispatch and 100% RQ3—Closed-list object contract. On ENTITY, full RouteBR accepted JSON, and is faster than the tied Qwen3.8 Max; Flash reaches 100.0% final-path exact match and 95.0% object accu- is faster but has one persistent contract failure and more false racy. Disabling the resolver in the A-ground construction check blocks. Qwen3.8 Max has the lowest false-block count. These yields 45.0% and 42.5%, and blocks all 22 dispatchable object tradeoffs motivate V4 Pro as the fixed RQ1–RQ3 representative cases. This mechanism check isolates the object contract’s and separate architecture effects from backend changes.

TABLE VI C ONTROLLED CONSTRUCTION CHECKS FOR RQ2 AND RQ3. O BJ . IS OBJECT ACCURACY; FD AND FB USE THEIR APPLICABLE DENOMINATORS .

B. Threats to Validity

Validity depends on information parity, metric fit, and backend stability. B1, B2, and RouteBR share the request, catalog, object candidates, and output schema; B2 also receives System Split Path Obj. FD FB the same state and rules as the primary matched control. ARouteBR FLIP 96.2 100.0 0/40 0/40 state and A-ground each remove one enforcement function A-state FLIP 46.2 100.0 40/40 0/40 as controlled mechanism tests. Final-path exact match, false RouteBR ENTITY 100.0 95.0 0/18 0/22 A-ground ENTITY 45.0 42.5 0/18 22/22 dispatch, false block, and FLIP pair accuracy measure the dispatch boundary; response quality and customer outcomes Backend errors concentrate in semantic selection and exces- form a separate evaluation layer. Exact McNemar tests and sive blocking rather than unsafe progress because deterministic Wilson intervals are retained in the artifact. All 1,960 rows enforcement is shared. This is the intended replacement preserve predictions, backend-reported identities, raw receipts, boundary: changing the model can affect which catalog action and catalog, rule, and split hashes for replay. Hosted-service and object are proposed, but it cannot change rule priority changes trigger the frozen regression suite. or turn a non-ALLOW decision into dispatch. RQ4 therefore C. Scope and Generalizability establishes a replacement protocol for the evaluated profiles: The evaluation is a privacy-preserving, controlled study of rerun the frozen regression suite after changing the proposer. routing contracts. Its 600 reconstructed scenarios come from Evaluation across other languages, service catalogs, and later one financial institution and were jointly reviewed by three hosted checkpoints remains future work. engineers; customer logs remain confidential. Runtime states and policies are synthetic, while 40 ETF identifiers are public. V. D ISCUSSION DEV, SELECT, and CONFIRM are disjoint, and every output is regenerated under frozen hashes; CONFIRM therefore provides A. Engineering Implications and Deployment internal confirmation within this collection. Separately sampled external data are needed to estimate traffic prevalence and RQ1 changes the acceptance object for customer-service cross-institutional performance. routing. B1’s 94.0% action accuracy coexists with only 41.0% Transfer is bounded by the closed ontology, equality predifinal-path exact match, so a service label cannot serve as a cates, single committed object, and hosted backends used in the release gate when software must also select an object and study. The reported metrics focus on routing correctness and terminal. B2 reaches 95.5% versus RouteBR’s 97.5%, and admissibility at the dispatch boundary. End-to-end response the difference is not statistically significant; B2 also wins quality and live operational outcomes form the next evaluation the FLIP-pair comparison 40/40 to 37/40. The engineering layer. Future evaluation should extend the contract to range, distinction is authority and evidence: the proposal remains temporal, and cross-field predicates and composite requests, untrusted, code owns the executable transition, and the winning test an independently reviewed institutional batch, and assess rule and provider receipt remain available for replay. the controller through privacy-preserving shadow deployment. RQ2 and RQ3 separate failure classes hidden by aggregate VI. C ONCLUSION safety. The A-state and A-ground checks confirm designed consequences: removing state enforcement dispatches all 40 We formalize boundary-aware pre-response routing and prohibited FLIP members, whereas removing object validation present RouteBR, where a model proposes a route while blocks all 22 admissible object-sensitive cases. B2 remains the code owns state authorization and closed-list validation. Our architecture comparison, while these contract checks isolate benchmark contains 600 privacy-preserving customer-service state and object effects. State enforcement protects authoriza- scenarios from a real financial institution, jointly reviewed tion; object validation protects grounding and availability. The by three engineers. On 200 MAIN cases, RouteBR reaches three FLIP pairs with a wrong service remain semantic errors. 97.5% final-path exact match; matched B2 remains competitive The boundary localizes maintenance: action confusion be- at 95.5%, while B1’s 94.0% action accuracy yields 41.0% longs to the catalog and proposer, a state error to policy, an complete-path accuracy. RouteBR produces no false dispatch unknown identifier to the registry, malformed output to the across 166 bounded MAIN and FLIP cases, and receipts support adapter, and failed-contract dispatch to the controller. RQ4 replay of all 1,960 matrix rows. These results support completedemonstrates proposer replacement under a frozen controller route acceptance and code-owned execution within the studied contract: path accuracy spans 1.9 points while latency, false closed routing setting. blocks, and semantic choices differ. Each component change Acknowledgement. The work described in this paper was activates its corresponding frozen regression slice. Test selection substantially supported by Hong Kong Metropolitan University should also be examined across objectives and distribution shifts Research Grant (Project Reference No. RD/2026/1.15). This instead of one aggregate criterion [34]. The next validation step work is also supported in part by the Research Grants Council is an external institutional batch followed by privacy-preserving of Hong Kong and research funds from the City University of Hong Kong (9229029, 9229192, 6000871). shadow operation.

R EFERENCES [1] T. Schick, J. Dwivedi-Yu, R. Dessı̀, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” Advances in neural information processing systems, vol. 36, pp. 68 539–68 551, 2023. [2] M. Li, Y. Zhao, B. Yu, F. Song, H. Li, H. Yu, Z. Li, F. Huang, and Y. Li, “Api-bank: A comprehensive benchmark for tool-augmented llms,” in Proceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 3102–3116. [3] Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian et al., “Toolllm: Facilitating large language models to master 16000+ real-world apis,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 9695–9717. [4] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022. [5] L. Wu, F. Petroni, M. Josifoski, S. Riedel, and L. Zettlemoyer, “Scalable zero-shot entity linking with dense entity retrieval,” in Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), 2020, pp. 6397–6407. [6] T. Rebedea, R. Dinu, M. N. Sreedhar, C. Parisien, and J. Cohen, “Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails,” in Proceedings of the 2023 conference on empirical methods in natural language processing: system demonstrations, 2023, pp. 431–445. [7] R. Yang, M. Fu, C. Tantithamthavorn, C. Arora, G. Gulmammadova, and J. Chua, “Adaptiveguard: Towards adaptive runtime safety for llmpowered software,” in 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2025, pp. 3381–3391. [8] H. Wang, C. M. Poskitt, and J. Sun, “Agentspec: Customizable runtime enforcement for safe and reliable llm agents,” arXiv preprint arXiv:2503.18666, 2025. [9] S. Larson, A. Mahendran, J. J. Peper, C. Clarke, A. Lee, P. Hill, J. K. Kummerfeld, K. Leach, M. A. Laurenzano, L. Tang et al., “An evaluation dataset for intent classification and out-of-scope prediction,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 1311–1316. [10] K. Hendrickx, L. Perini, D. Van der Plas, W. Meert, and J. Davis, “Machine learning with a reject option: A survey,” Machine Learning, vol. 113, no. 5, pp. 3073–3110, 2024. [11] I. Ong, A. Almahairi, V. Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. Kadous, and I. Stoica, “Routellm: Learning to route llms from preference data,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 34 433–34 448. [12] S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large language model connected with massive apis,” Advances in Neural Information Processing Systems, vol. 37, pp. 126 544–126 565, 2024. [13] Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. Maddison, and T. Hashimoto, “Identifying the risks of lm agents with an lm-emulated sandbox,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 27 031–27 098. [14] E. Debenedetti, J. Zhang, M. Balunovic, L. Beurer-Kellner, M. Fischer, and F. Tramèr, “Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents,” Advances in neural information processing systems, vol. 37, pp. 82 895–82 920, 2024. [15] Q. Zhan, Z. Liang, Z. Ying, and D. Kang, “Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents,” in Findings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 10 471–10 506. [16] Z. Xue, Y. Zhao, S. Wang, K. Chen, and H. Wang, “A characterization study of bugs in llm agent workflow orchestration frameworks,” in 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2025, pp. 3369–3380. [17] X. Yu, L. Liu, X. Hu, J. W. Keung, J. Liu, and X. Xia, “Fight fire with fire: How much can we trust chatgpt on source code-related tasks?” IEEE Transactions on Software Engineering, vol. 50, no. 12, pp. 3435–3453, 2024. [18] Z. Chen, W. Chen, C. Smiley, S. Shah, I. Borova, D. Langdon, R. Moussa, M. Beane, T.-H. Huang, B. R. Routledge et al., “Finqa: A dataset of numerical reasoning over financial data,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 3697–3711.

[19] P. Islam, A. Kannappan, D. Kiela, R. Qian, N. Scherrer, and B. Vidgen, “Financebench: A new benchmark for financial question answering,” arXiv preprint arXiv:2311.11944, 2023. [20] M. Eric, R. Goel, S. Paul, A. Sethi, S. Agarwal, S. Gao, A. Kumar, A. Goyal, P. Ku, and D. Hakkani-Tur, “Multiwoz 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines,” in Proceedings of the twelfth language resources and evaluation conference, 2020, pp. 422–428. [21] A. Rastogi, X. Zang, S. Sunkara, R. Gupta, and P. Khaitan, “Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 05, 2020, pp. 8689–8696. [22] B. Byrne, K. Krishnamoorthi, C. Sankar, A. Neelakantan, B. Goodrich, D. Duckworth, S. Yavuz, A. Dubey, K. Kim, and A. Cedilnik, “Taskmaster-1: Toward a realistic and diverse dialog dataset,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 4516–4525. [23] I. Casanueva, T. Temčinas, D. Gerz, M. Henderson, and I. Vulić, “Efficient intent detection with dual sentence encoders,” in Proceedings of the 2nd workshop on natural language processing for conversational AI, 2020, pp. 38–45. [24] Y. Geifman and R. El-Yaniv, “Selective classification for deep neural networks,” Advances in neural information processing systems, vol. 30, 2017. [25] E. Zhao, P. Awasthi, Z. Chen, S. Gollapudi, and D. Delling, “Semantic routing via autoregressive modeling,” Advances in Neural Information Processing Systems, vol. 37, pp. 10 060–10 087, 2024. [26] R. Šléher, W. Brach, T. Sloboda, K. Košt’ál, and L. Galke, “Guarded query routing for large language models,” arXiv preprint arXiv:2505.14524, 2025. [27] R. Šléher, W. Brach, K. Košt’ál, and L. G. Poech, “Influence of prompt engineering on small language models for guarded query routing,” arXiv preprint arXiv:2607.24801, 2026. [28] Y. Zhuang, Y. Yu, K. Wang, H. Sun, and C. Zhang, “Toolqa: A dataset for llm question answering with external tools,” Advances in Neural Information Processing Systems, vol. 36, pp. 50 117–50 143, 2023. [29] S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ -bench: A benchmark for tool-agent-user interaction in real-world domains,” arXiv preprint arXiv:2406.12045, 2024. [30] S. Tedeschi and R. Navigli, “Multinerd: A multilingual, multi-genre and fine-grained dataset for named entity recognition (and disambiguation),” in Findings of the association for computational linguistics: NAACL 2022, 2022, pp. 801–812. [31] N. De Cao, G. Izacard, S. Riedel, and F. Petroni, “Autoregressive entity retrieval,” arXiv preprint arXiv:2010.00904, 2020. [32] N. De Cao, L. Wu, K. Popat, M. Artetxe, N. Goyal, M. Plekhanov, L. Zettlemoyer, N. Cancedda, S. Riedel, and F. Petroni, “Multilingual autoregressive entity linking,” Transactions of the Association for Computational Linguistics, vol. 10, pp. 274–290, 2022. [33] J. A. Botha, Z. Shan, and D. Gillick, “Entity linking in 100 languages,” in Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), 2020, pp. 7833–7845. [34] J. Zhang, F. Wang, J. Keung, Y. Liao, Y. Xiao, and L. Ma, “Empirical insights of test selection metrics under multiple testing objectives and distribution shifts,” Proceedings of the ACM on Software Engineering, vol. 3, no. FSE, pp. 1288–1311, 2026.

Related documents

• Grounding LLM Agents in Business Processes: Process-Aware Knowledge Graphs for Reliable Multi-Step Service Invocation#1473669
• Grounding LLM Agents in Business Processes: Process-Aware Knowledge Graphs for Reliable Multi-Step Service Invocation#1468846
• Proof of concept integration of a single-level service-oriented architecture into a multi-domain secure environment#1278949
• Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills#1324947
• Application Bridging for Federated Access Beyond Web (ABFAB) Use Cases#502082