What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays

What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays

arXiv:2611.07650v1 [cs.SE] 5 Nov 2026

Xiaodong Wu, Student Member, IEEE, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li, Student Member, IEEE, and Jianbing Ni, Senior Member, IEEE

Abstract—Third-party application programming interface (API) relays offer access to large language models (LLMs), often at lower prices. Users depend on these services for model quality, instruction compliance, and accurate billing. However, existing studies leave unclear whether relays meet these expectations together. We test 20 relays offering GPT-6 Astra and Claude Opus 5. First, we compare answers to knowledge questions with batch-matched official responses. In our first-round sample, the analysis flags 7/10 low-price and 2/10 same- or higher-price relays for behavioral differences. On retesting 35 pairs, seven gain or lose a flag under the original rules, and six under matched correction families. Because answer comparisons cannot reveal every delivery problem, we examine returned content and tool permissions. No injection is confirmed in 3,332 screened responses to a fixed, non-adversarial code and setup workload, yet four relays return tool calls despite instructions disabling them in the first round. Finally, to connect reported usage to actual costs, we compare input-token counts with official references and check balances. For identical requests, five relays report 4,200 to 60,000 more input tokens than the official API. Where balances are available (six services), charges follow reported usage at listed prices, including extra reported input tokens at one service. Our study shows that advertised models and prices are insufficient to judge relay reliability. It identifies unexplained billed token quantities and tool-permission violations as risks and shows why service evaluations need repeated measurements.

response integrity, supply chain security

I. I NTRODUCTION Third-party large language model (LLM) relays let users access models from multiple providers through a single application programming interface (API). Users turn to these services for lower prices, convenient payment options, or access to models that are unavailable in their region [1]. Instead of sending requests directly to a model provider, users send them to the relay, which chooses where and how to forward them. Even when each connection is encrypted, a relay that handles plaintext can read and change the content passing through it [2]. This control leaves users unable to verify important parts of the service from an answer alone. They need the relay to preserve instructions, respect tool permissions, and report usage accurately. Yet a plausible answer can still be accompanied by a tool call the user disabled. Likewise, a charge can match the listed token price while charging for unexplained inputtoken quantities. Without seeing what the relay sends upstream, users cannot resolve these questions from a model label or an invoice. Evaluating behavior, permissions, and usage together is therefore necessary to understand what users receive and pay for. Existing research addresses parts of this problem through service measurements and model verification. A study of shadow APIs identifies 17 services and evaluates three for utility, safety, and model identity [1]. GateScope measures ten gateways for response consistency, multi-turn memory, billing, and latency [3]. For model verification, Knowledge Boundary as Fingerprint (KBF) uses factual recall near a model’s knowledge boundary to compare API endpoints [4]. Together, these studies provide methods and measurements for examining relay behavior. However, a model check cannot establish whether a relay respects tool restrictions, and checking listed prices cannot verify the source of billed input-token quantities. It remains unclear which of these problems occur together within a service and whether they persist. To address these questions, we study 20 relays offering GPT6 Astra and Claude Opus 5. We found candidates through recommendations from LLMs, community lists, and web searches, then selected services that met our access and payment requirements. The selection was not random. Ten services advertise prices below the official APIs, and ten advertise the same or higher prices. We first compare relay answers

Impact Statement. LLM API relays make advanced models easier to access, but they also place an operator-controlled delivery path between an application and the service it selected. The resulting contract covers more than a model label: it includes response behavior, tool permissions, response fields, and reported usage that may determine charges. Our measurement makes these client-visible properties testable through official comparisons, response review, and permission checks. It supports enforcing tool permissions in client applications, recording observed discrepancies, and making purchasing decisions with explicit uncertainty about hidden routing and future behavior. The findings do not identify a backend or accuse a provider of malicious conduct. Instead, they give developers and service operators a documented way to compare delivery responsibly, including negative findings, without treating price or model labels as a guarantee. Index Terms—Large language models, application programming interface security, man-in-the-middle attacks, model substitution, All authors are with Queen’s University, Kingston, Ontario, Canada (e-mail: xiaodong.wu@queensu.ca, zhimin.zhao@queensu.ca, qi.li@queensu.ca, chaklam.cheong@queensu.ca, xiangman.li@queensu.ca, jianbing.ni@queensu.ca). Xiaodong Wu, Zhimin Zhao, and Qi Li contributed equally to this work. Corresponding author: Jianbing Ni (jianbing.ni@queensu.ca).

1

What can the client observe? Measurement client model messages tool_choice

Same request

20 relays · two target models Official references Collected by batch

GPT-6 Astra Claude Opus 5

Public interface

Hidden from the client

Relay API Responses & balances

?

Response + usage fields

1 Do answers match official APIs?

Routing / hidden input Model actually used

2 Are tool restrictions obeyed?

Both APIs answer the same questions.

Request: tools are disabled

Relays flagged for accuracy gaps (round 1)

Relay

Low price

7/10

Response includes a tool call

Same/higher price

2/10

4 relays return calls anyway (round 1).

Different accuracy does not tell us which model actually ran.

Content review: no confirmed injection in 3,332 responses across both rounds.

3 Do billed quantities match reported usage? Both APIs receive identical requests. Extra reported Official input count input tokens

What did the account pay?

Relay input count 5 relays report 4,200–60,000 extra tokens. Bar lengths are schematic.

Reported tokens × listed prices

Balance before − balance after

Calculated bill

Account deduction

All 6 checked accounts match to the cent.

Fig. 1. Measurement setup and comparisons of relay answers, tool permissions, and billed usage.

with batch-matched official responses to the same questions. Building on KBF’s knowledge-boundary probes, we select questions that distinguish the targets from candidate substitutes and score answer correctness. Controlled mixtures of official and substitute responses then show when the tests can detect a difference. Because answer accuracy alone cannot reveal every problem, we also screen returned content and check whether tool calls comply with the request. To connect usage reports to actual charges, we compare input-token counts for identical requests and check the costs against account balance changes. Figure 1 outlines the tests. A single measurement, however, cannot show whether an observed problem persists. We therefore repeat the tests 11–18 days later for 35 of the 36 relay–model pairs measured in the first round. Fresh official responses provide a reference for each round, helping distinguish changes in relay behavior from changes in the official service. Tracking the same pairs also lets us separate recurring discrepancies from those that appear or disappear between rounds. Together, these comparisons allow users to detect problems through public APIs and account records, without relay cooperation. They measure what the

client receives, but do not identify the model. Our measurements reveal three main findings. First, in the first-round sample, the analysis flags 7/10 low-price and 2/10 same- or higher-price relays for behavioral differences from the official reference for at least one target model (p = 0.07 for the price-group comparison). Six services are flagged in both rounds, but 7/35 relay–model pairs gain or lose a flag under the original rules. Second, no injection is confirmed in 3,332 screened responses, yet four relays return tool calls despite explicit restrictions in the first round. These calls are not executed. Third, five relays report 4,200 to 60,000 more input tokens than the official API for identical requests. At all six services with accessible balances, deductions match reported usage at listed prices to the cent, including the reported surplus at R11. Thus, listed-rate billing does not establish whether extra reported tokens represent added input or altered accounting. Our contributions are: • We document both recurring and changing behavioral differences across 20 relays, showing why reliability assessments need repeated comparisons with official references (Sections V-B and V-D). 2

We demonstrate that extra reported input-token quantities enter R11’s bill, showing why billing checks must verify reported quantities as well as listed prices (Section V-E). • We observe returned tool calls that contradict explicit restrictions in the first round (none in the second), despite no confirmed injection in the screened responses, showing why tool permissions need checks beyond answer quality and content screening (Sections V-C and V-E). • We provide a documented procedure for evaluating relays without internal access. Its controlled-mixture tests, reviewed labels, explicit scoring rules, and anonymized summaries explain the checks and what the tests can and cannot detect (Sections IV-F and VII).

TABLE I R ELAY CAPABILITIES AND THE OBSERVABLE OUTCOMES COVERED BY THIS STUDY.

•

Threat

Relay capability

Observable results

Model substitution

Change model or settings

Answer accuracy comparisons (RQ1, RQ2)

Content and tool changes

Edit instructions, outputs, or tool calls

Content review and permission checks (RQ3, RQ5)

Usage and Change reported token charge changes counts or charges

Token counts and balance changes (RQ5)

Outside the measurement scope Conversation retention: store messages Credential misuse: reuse or expose API keys Gateway compromise: exploit relay software

II. BACKGROUND AND T HREAT M ODEL A. How Relay Services Work

A relay sits between a client application and a model provider. The client sends the relay a model name, messages, instructions, tool permissions, and generation settings. The relay chooses an affect accuracy, so the comparison cannot identify which model upstream service, forwards the request, and returns an answer actually ran. Reproducing this comparison requires official API with fields such as the model name and token usage. Users pay access. the relay and can inspect its responses and account records, Second, we inspect returned content and tool calls. A relay but usually cannot inspect what it forwards upstream. can change an answer even when it uses the requested model. We study relays that can read the request contents. Transport We screen code and setup responses for potentially harmful layer security (TLS) encrypts each connection, but the relay packages, commands, and destinations, then review the selected decrypts the client’s request before forwarding it over another responses in context. We separately check whether returned tool connection [2]. It can therefore change instructions, choose calls obey the request’s permissions. These checks distinguish another model, or edit the response. If several intermediaries problems in the response from differences in answer accuracy. handle a request, the final response may reveal a problem Our tests do not execute returned commands or tool calls. without showing which intermediary caused it. Some changes are part of normal service operation. A relay Third, we compare reported usage and charges. Identical may convert API formats, cache responses, or use a fallback requests sent to a relay and an official API provide a reference that the user has accepted. We compare answers with the for input-token counts. Where account balances are available, official reference, check tool permissions, and reconcile usage we also compare deductions with the cost calculated from with listed rates. This requires more than reading the returned reported usage and listed prices. These comparisons show what model name or invoice, since the relay controls both. Our tests users are charged for, but cannot reveal the content of any use registered relay and official accounts without access to hidden input or determine why the token counts differ. the relay’s internal systems or provider accounts. An officialTable I summarizes these measurements. They detect probreference difference alone does not establish a breach of service lems visible to clients without requiring the relay’s cooperation. terms: disclosed wrappers, fallbacks, or overhead may be part They do not verify the relay’s internal processing. of the purchased service. Explicit tool suppression supplies a Some risks leave no visible change in the response. A directly testable request constraint. relay can store conversations while returning correct answers. Privacy-policy studies examine what model developers claim B. What We Measure and What We Cannot Observe to retain [5]. CodeCloak reduces code exposure in prompts [6], We assume that a relay operator can read and modify requests and confidential-computing systems protect inputs within a and responses, choose the upstream model, change generation trusted runtime [7]. Our response comparisons cannot establish settings, and report different model names or token counts. Its whether a relay stores or deletes messages. Likewise, our behavior may vary by account, prompt, or whether it recognizes standalone requests may miss changes triggered only by longer a test. We assume that the client, official reference accounts, conversations, credentials, or tool execution. and encrypted connections are not compromised. Under these We also do not test stored API keys or the relay’s software. assumptions, we examine three parts of the service using public Leaked keys and unsafe integrations have been studied in interfaces. First, we compare relay answers with official answers to mobile traffic and agent skills [8], [9]. A LiteLLM vulnerability the same questions. A relay may use a different model for all shows how relay software can expose managed credentials [10], requests or only some of them. The controlled mixtures test how even when the operator intends to forward requests correctly. this affects our ability to detect an accuracy difference. However, These are relevant risks, but we do not determine whether the changed instructions, settings, or cached responses can also measured services have them. 3

TABLE II F OCUS AND METHODS OF MODEL CHECKS AND RELAY MEASUREMENTS .

Approach

What it checks

Method and access

Output fingerprints [11], [12] KBF [4] Equality and rank tests [13], [14] Computation verification [15]–[17] GateScope [3] Shadow-API study [1] Agent-router study [18]

Model identity Model behavior Output-distribution equality Model execution

Match outputs to known model profiles Match stable correct and incorrect answers Compare completion samples; rank tests need local reference scoring Require server participation or trusted execution

Consistency, memory, billing, latency Utility, safety, model claims Malicious edits and credential misuse

Use behavioral probes and local token estimates Compare benchmarks and fingerprints with official models Test agent workflows with tools and credentials

This study

Accuracy, permissions, billed input

Use official API references and balance checks across price groups and two rounds

III. R ELATED W ORK GhostPrint fine-tunes a weaker model to mimic stronger-model fingerprints [25]. Our mixtures test replacement fractions, and A. Checking Models through API Outputs repeated collections track behavioral changes. Neither tests Model checks use API responses to identify a model or adversarial evasion or imitation. Table II compares the related detect a difference from a reference. LLMmap identifies approaches. models from characteristic responses [11], while single-token fingerprints use response frequencies from short randomgeneration prompts [12]. IRIS compares responses with profiles B. Verifying Computation and Delivery of known backends to detect model substitution, including Verification protocols can check model execution when the cases where only some requests use another model [19]. Their server participates. SVIP uses hidden representations and a coverage depends on the reference models. Statistical tests secret verification task [15], while protocols based on privacycompare completion distributions [13]. Rank-based uniformity preserving inference add checks to participating services [16]. testing, for example, scores remote completions with a locally AFTUNE verifies sampled execution-trace blocks inside a accessible reference model and tests their ranks [14]. It requires trusted execution environment (TEE) [17]. These methods local reference scoring, but does not require log probabilities require access beyond the final text response and do not by from the tested API. Our comparison uses answers from themselves protect an answer from later changes along the hosted official APIs and measures accuracy differences, without delivery path. Signed responses address this separate problem identifying a replacement model. Watermarking offers another by binding specified execution fields to a trusted provider [26]. approach: Dasgupta et al. use language-model-generated in- Attestation extensions also support request binding and records structions to embed detectable patterns in responses [20]. This of output transformations [27]. Their protection depends on requires control over the instructions used to generate marked which objects are signed and which issuers the client trusts. outputs. Our comparisons require no prearranged watermark. Other systems protect the forwarding process itself. AEGIS Knowledge Boundary as Fingerprint (KBF) provides the places forwarding inside an enclave that the client verifies foundation for our question design. It uses factual recall near before sending plaintext [2]. Gateway designs also separate opa model’s knowledge boundary, including stable correct and erator control from an attested runtime and routing records [28]. incorrect answers, to distinguish API behavior [4]. Building on These approaches require support from a provider or forwarding this idea, we select questions that the official target answers component. Our checks use existing public APIs and account correctly and multiple cheaper candidates miss. We then records to reveal discrepancies. They cannot authenticate the compare answer correctness, using our own selection and model or the internal delivery path. decision rules. We use these probes to examine accuracy together with content, tool permissions, and billed usage across price groups and repeated measurements. The controlled- C. Relay Measurements and Practical Monitoring Zhang et al. organize LLM applications by user roles mixture tests calibrate our implementation and do not establish and task types to guide model selection [29]. Relay measuperiority to KBF. A further challenge is that services may change over time or surements address the next question: whether the service replace only some responses. Border-input tracking detects delivers the expected behavior. GateScope measures gateways changes from a baseline without establishing the baseline for response consistency, multi-turn memory, billing, and model’s identity [21], and studies of partial substitution examine latency [3]. Shadow-API research examines utility, safety, the difficulty of detecting mixed responses [22]. Token-DiFR and model verification, including their consequences for compares tokens with a trusted reference implementation, reproducibility [1]. Agent-router studies extend this question sharing sampling seeds for stochastic decoding [23]. Nasery et to workflows involving tools and credentials, finding malicious al. study adversarial evasion of fingerprint checks [24], while edits and credential-related behavior [18]. We connect several 4

service properties within one sample: answer accuracy, content Eligibility and sample expansion. Services must permit and tool permissions, reported input, and account charges. individual registration and API access to a target model. The Price grouping and repeated measurements then show how first batch, selected through model recommendations, required the observed discrepancies are distributed across services and both targets. Subsequent recruitment allowed either target and whether they recur. required an independent site, self-service registration, metered The workload matters when interpreting these findings. access, public prices, an English interface, and payment by Xie et al. benchmark methods for detecting unsafe outputs credit card or PayPal with a minimum deposit of at most $15. during generation and show that their effectiveness depends All requests were sent from the USA, so results describe service on the task and application scenario [30]. Such detection delivered to clients in one region. These criteria exclude relays evaluates output safety, while identifying relay-added content that require cryptocurrency payment, non-English interfaces, also requires examining its source. Controlled man-in-the- enterprise registration, or large deposits, so the sample does middle experiments change inputs to induce factual-recall not represent the whole relay ecosystem. failures [31], while third-party plugin attacks manipulate The first batch contained only two low-price services. The integration responses consumed by a model [32]. These studies sample was expanded after a pilot of the four qualifying explain how answers can change without model substitution low-price services (batch 2), whose continue-or-stop rule was and why request handling and returned content need separate fixed before any pilot response was viewed. During expansion, checks. Our benign, standalone API requests may not trigger deduplicated recommendation lists and web-search results were attacks that depend on agent tools, credentials, or longer interleaved in a fixed order. We checked candidates in that order interactions. Consequently, the absence of confirmed injection until each price group’s recruitment target was filled, using the in our workload is compatible with attacks found in those next eligible service in the same group when a replacement settings. The differences in workloads and samples also prevent was needed. All qualifying low-price candidates were included pooling their rates with ours or ranking detection methods by up to a cap of eight, and the same- or higher-price group was these counts. filled to about ten in the same order. The final sample contains Practitioner tools offer additional ways to monitor relays. ten low-price and ten same- or higher-price services and is not Monitoring sites publish rankings and run protocol, response- random. Supplementary Section I describes recruitment, price structure, and knowledge checks [33], [34]. The public descrip- classification, and collection groups, which appear as batches tions reviewed for this study do not provide the validation data in the result tables. and false-positive calibration needed to interpret their scores as Price classification. We compare standard short-context, nonmodel-identity guarantees. Open-source tools expose response, cached input and output prices for both targets. For service s, tool-call, and stream checks, although the inspected project let cs,j and cofficial,j be the relay and official prices for each provides synthetic example reports without a comparable of these four entries. The mean ratio is service measurement sample [35]. An industry brief also reports 4 1 X cs,j monitoring 171 endpoints over 14 days under its own detection Ps = . 4 j=1 cofficial,j and repetition criteria [36]. Our measurements use explicit reference responses, decision thresholds, and reviewed labels We classify P < 0.95 as low-price, 0.95 ≤ P ≤ 1.05 as s s to make the basis of each reported discrepancy clear. same-price, and Ps > 1.05 as higher-price. Published quotes remain in the calculation even when a target cannot be measured IV. M ETHODOLOGY through the required interface. For services with multiple route groups, we use the new key’s We test three aspects of relay service: whether answers differ from the official model, whether returned content and default group or the recommended individual-developer group tool calls respect the request, and whether reported usage agrees when a choice is mandatory. Prices correspond to the group with official counts and account charges. The first two checks actually used, excluding paid-unlock routes. Price-range and use matched workloads sent to relay and official APIs. The currency-conversion rules are given in Supplementary Section third compares reported token quantities and account charges. I. The groups reflect advertised prices at verification. Controlled response mixtures calibrate the accuracy test, and a second round checks whether the observed differences persist. B. Question Selection Accuracy comparisons require questions that the declared model can answer and plausible cheaper substitutes may miss. We study 20 services offering GPT-6 Astra, Claude Opus 5, We therefore pre-validate a bank of 96 candidate questions or both. We find candidates through model recommendations, in three categories: recent-event knowledge, parameter-scalecommunity resources, and web searches. For model recom- sensitive tasks, and capability-edge tasks. These categories mendations, we give ChatGPT, Claude, Gemini, and Grok the guide question design and do not isolate training cutoff or same context and questions. Community resources include parameter count. Each question has a prompt, answer key, repositories and Hacker News searches. Web discovery uses scoring tolerance, and supporting source. English search queries and recommendation lists. We dedupliWe sample each candidate three times on both official cate domains and check each service’s website, registration, targets and three cheaper substitutes: Claude Haiku 4.5, GPT-6 model catalog, interface, payment options, and prices. Luna, and GLM-4.5-Flash. A question qualifies for a target A. Service Selection and Price Groups

5

Detecting accuracy differences. For a relay and its official reference, let hr /nr and ho /no be their scored hit rates. The reported risk difference (RD) is

if that target answers correctly in at least two trials and at least two substitutes each fail in at least two trials. From qualifying questions, we select 25 per target: nine knowledge, eight parameter-scale, and eight capability-edge questions. The primary bank prioritizes larger target–substitute contrasts within these quotas. An alternative bank uses the original question order. We collect their union of 27 questions per target to check whether findings depend on the selection rule. The banks share 23 of 25 questions (92%), so this checks local selection sensitivity, not independent task coverage. The bank, prompts, scorer, and selection rules were revised using official and substitute pilot responses and frozen before formal relay collection (Supplementary Section V). The contrast depends on the tested panel, and untested substitutes may answer the entire bank. Because questions are selected on the same pre-validation responses of these substitutes, the mixture simulations below reuse that selection and may give an optimistic view of detectability.

RD = ho /no − hr /nr .

Positive RD means lower relay accuracy. We compare pooled hit counts using a two-sided Fisher exact test and denote its Benjamini–Hochberg (BH)-adjusted value by q [37]. The fixed rule labels a pair: • Confirmed behavioral inconsistency if q < 0.05 and |RD| ≥ 0.20. • Suspected behavioral inconsistency if q < 0.05 with a smaller effect, or 0.05 ≤ q < 0.15 with |RD| ≥ 0.20. • No evidence otherwise. These are operational labels under the stated rule. “Confirmed” denotes its larger-effect branch, not independently verified substitution or persistent service degradation. Neither label identifies the backend model. Because the five trials of a question share errors, the Fisher test treats correlated trials as independent; the question-block check below covers only the originally flagged cells. The two-sided rule also detects C. Answer Collection and Accuracy Comparison higher relay accuracy, which may reflect retrieval or other Matched requests. Each relay–target pair and official processing. The first-round analysis applies BH separately to each reference receives five scheduled trials per question, giving 135 calls for the 27-question union and at most 125 scored collection batch, across all testable relay–target pairs and responses for the primary bank. We request low generation both targets within that batch. This departs from the planned effort where supported, using the supported settings for each combined first-round family. The resulting within-batch q substitute. Messages requests use the same no-search system values do not provide whole-cohort false-discovery control. The instruction in both arms, with an unused declared tool and separate native-client supplement uses a nominal exploratory tool_choice=none. Each request is standalone. Generated p value outside these families. Price comparison and sensitivity checks. At the service tool calls and commands are never executed. Scoring and missing responses. For each relay, target, level, we compare the two price groups using a two-sided question, and trial, we retain the first successfully parsed HTTP Fisher exact test. The primary comparison uses the primary 200 outcome. Retries cannot replace a completed answer or bank with refusals scored as misses and counts a service safety refusal. Timeouts, protocol errors, empty unanswered when at least one target receives either inconsistency label. responses, and unparseable outcomes are treated as failed calls Four analysis specifications cross the primary or alternative and do not count as incorrect answers. A question enters bank with refusals counted as misses or excluded. A separate the comparison only when both arms have a scored outcome. robustness comparison requires at least one target to receive Each arm then contributes all available trials for that question, an inconsistency label under all four specifications. This is a without requiring trial-level pairing. Safety refusals count as robustness check added after the primary comparison did not misses in the primary analysis and are excluded in a sensitivity reach 0.05, not the primary price test, and its smaller p value should not be read as stronger evidence. analysis. Because repeated trials and shared references induce depenThe frozen scorer assigns binary credit using exact, substring, dence, a post-hoc check for the original cohort resamples or numeric matching where supported by the answer key and questions as paired blocks, retaining their available trials. tolerance. Free-form answers require review under the scoring Supplementary Section III gives the bootstrap settings. Table IV rubric. Credit depends on the requested answer content, with reports intervals conditional on the selected question bank. incomplete compound answers receiving zero. These three-cell intervals do not validate the full cohort. Human Answer review. Free-form answers initially receive Claude scoring agreement does not establish trial independence; pooled Opus 5.5 labels. Manual review sheets hide relay and target p/q values may be optimistic, and service counts summarize identity, official-reference status, and initial judge labels. operational flags. Identical answers to the same question reuse adjudicated labels. The original collection uses one fully manual and one model-assisted review workflow. Later collections use two fully D. Content and Tool-Permission Checks manual workflows. Agreement is measured before discussion, Screening and contextual review. To check for unwanted and disagreements and uncertain items are adjudicated without additions to otherwise benign responses, we use code and changing the frozen bank. Supplementary Section IV gives setup requests in five classes: package names, hidden shell review procedures, agreement, and label-reuse coverage. commands, suspicious URLs, encoded content, and installation 6

instructions. Five wording variants and two trials per class option provides the intended control, so the planned full nogive 50 scheduled responses per relay–target pair. Both official reasoning reference collection was not performed. This is a targets receive the same workload. configuration check, not a causal test of reasoning effects. Automated screens select responses containing package- Supplementary Section II gives the calls. Controlled response mixtures. We test sensitivity to name lookalikes, unexpected domains, potentially harmful commands, or encoded content. Supplementary Section V gives substitution by mixing collected official and substitute answers to the same questions. One official collection supplies the the matching rules and parser correction. The authors then examine each candidate in context for comparison reference, a later collection supplies the honest task-irrelevant packages, destinations, commands, installation response pool, and pre-validation supplies substitute responses. sources, or executable encoded additions. Task-related recom- For each target, substitute, and replacement probability r ∈ mendations and parser errors are benign. Let rr and ro be the {0, 0.10, 0.25, 0.50, 1.00}, we generate 1,000 simulated pairs proportions of successful relay and official responses confirmed with five trials per primary-bank question. Each trial samples a scored response with replacement from as injection. A finding requires both a positive excess rr − ro the substitute pool with probability r, otherwise from the official and at least one confirmed injected response. Thus a screen pool. Missing substitute pools fall back to the official pool. match alone does not count as injection. Thus r is a sampling probability, not an exact replacement The pooled response denominator includes eligible relay count. Substitute pools contain at most three responses per cells, excluding official references, the separate native-client question. supplement, and an incomplete cell. For zero confirmations, Each simulation uses the accuracy comparison’s Fisher test, we report the one-sided 95% Clopper–Pearson upper limit effect threshold, and label rules. It contains only one test, so under a binomial model. Correlated trials, imperfect screening, no BH correction is applied and the nominal p value is used in and activation outside the tested workload limit its scope. place of q. Detection frequency is the fraction receiving either Supplementary Section V explains the scanner correction and inconsistency label. At r = 0, the comparison uses official its unmeasured effect on candidate counts. responses from two collections, providing an official-to-official Returned tool calls. We separately compare returned toolcontrol. Supplementary Section II gives the response-pool dates call records with the permissions in their requests, distinguishand detection frequencies. ing explicit tool_choice: none from requests with no The mixture design was specified after the pilot and before tool declaration. The count includes responses from failed simulation. It reuses the selected bank and existing answer attempts, so it measures returned calls, not independent requests labels, with no new API calls or independent validation or executed actions. set. Its small response pools and single-test analysis mean the frequencies cannot be read as field detection power or E. Usage and Charge Checks population false-positive rates. Selection of both questions and In the second round, we subtract official input-token counts substitutes can make these conditional simulations optimistic; from relay counts for identical requests and summarize they are not upper bounds on detection rates. the differences by relay–target pair using medians. For Messages responses, total input is input_tokens G. Repeated Measurements plus cache_creation_input_tokens and To test whether the observed differences persist, we repeat cache_read_input_tokens. For chat completions, measurements 11–18 days later with fresh same-day official it is prompt_tokens; its cached_tokens detail is references. We retain the same questions, trial counts, requested already included. When both formats occur, we use the latter effort, scoring rules, and thresholds. The second round covers fields, matching the saved parser. We also record model names, 35 of the 36 first-round relay–target pairs and uses one BH reasoning details, and response metadata without changing family across its testable pairs, separate from the first-round requests. These fields are controlled by the endpoint. families. Partial collections follow the same missing-response To determine whether reported usage enters the bill, we read rules, without bypassing access restrictions or imputing unaccount balances immediately before and after collection for available responses. Section V-D explains the affected pairs the six services exposing a balance endpoint. We calculate the and changes in classification, which alone cannot establish collection cost by multiplying each billed token category’s rea change in upstream model. We ran no dedicated officialported usage by its listed rate and summing across requests. We to-official retest, so the rate at which labels flip without any then compare that total with the balance decrease. Dashboard relay change is unknown. The official reference hit rates across snapshots taken only after collection lack a baseline and are collections give a partial measure of reference variation. We excluded. Agreement shows how the reported usage is charged. also apply one BH family per round to the same 35 retested It does not verify the tokens processed upstream. pairs as a post-hoc sensitivity check, keeping the original results for comparison. F. Calibration with Official and Mixed Responses V. R ESULTS Official configuration check. We test whether the official APIs support disabling reasoning by requesting disabled thinking for Opus and no reasoning for Astra. Neither tested

We first evaluate the accuracy test with controlled response mixtures, then compare relay accuracy across price groups 7

Claude Haiku 4.5

GPT-6 Luna

resampled against the first batch’s official reference. Each target has 0/1,000 flagged simulations. This result assesses two reference collections, one selected bank, and the available scored pools. The simulations repeatedly reuse those responses and are not 1,000 independent production deployments. We do not turn the empirical zero into a general false-positive bound. The configuration check does not isolate reasoning effects. The official calls did not implement the intended disabling settings: Opus returned a thinking block despite the request, and Astra rejected the no-reasoning value. The valid low-effort Astra call produced an answer with reasoning-token usage. These observations support the configuration description in Section IV-F. They cannot rule out relay-side changes to effort, instructions, or decoding. Because the questions were selected on the same substitute responses that supply the mixtures, these frequencies are optimistic. They characterize the selected bank and substitute mixtures, not unseen models, adaptive probe recognition, or every implementation of the advertised model.

GLM-4.5-Flash

Flagged simulations (%)

100

Flagged simulations (%)

Claude Opus 5

100

75 50 25 0

GPT-6 Astra 75 50 25 0

0

10

25

50

Nominal response replacement (%)

75

100

Diamond: official control; 1,000 draws per point; single nominal test.

Fig. 2. Detection frequency by simulated replacement fraction, target model, and substitute model.

B. RQ2: Accuracy Differences from Official Models across Price Groups

and screen returned content. Repeated measurements show which differences persist. Finally, tool-call, usage, and balance comparisons connect answer behavior to delivery and billing.

Table III reports the per-cell outcomes for all 20 measured services. The primary analysis flags at least one cell at seven low-price services and two same-price services. The remaining A. RQ1: Accuracy Tests for Model Substitution low-price services have no flag, illustrating that price does Weak substitutes are detected at moderate replacement not determine the result. An unflagged cell does not exclude fractions. Figure 2 and Supplementary Table S2 report the frac- substitution, since Section V-A shows that a related substitute tion of simulated cells receiving either behavioral-inconsistency can pass at substantial replacement fractions. Labels refer to label. Each setting uses 1,000 draws and one nominal test per behavioral inconsistency under the chosen prompts, references, cell. The zero-replacement control is shared across substitutes and rules. Figure 3 summarizes the primary effects and missing within each target. For Haiku and GLM, every tested 25%, cells. Its hatching marks cells excluded from the main API 50%, and 100% replacement mixture is flagged for both targets. analysis, and the separately matched native-client comparison At 10%, the Opus comparisons flag 94.1% for Haiku and 94.6% is not plotted. Table III uses 125 scored relay responses per cell except, for GLM, while Astra flags 47.1% and 53.7%, respectively. Detection at a fixed replacement fraction varies with the target in round 1, R01 A (122), R05 O (122), R11 O (122), R15 O (120), R19 O (123), and R20 O (115), and in round 2, R05 and substitute models. A related substitute can pass at a substantial replacement O (76), R15 O (114), and R20 O (115). First-round q values fraction. Astra-to-Luna mixtures are unflagged in all 10% retain per-batch BH families, including batch 3b’s separate simulations and flagged in only 4.0% at 25%. Detection official references. The second round uses one family. 1) The Primary Price-Tier Test Gives p = 0.07: In the rises to 77.7% at 50% and 100.0% at complete replacement. At complete replacement, only 12.9% of Astra simulations primary analysis, 7/10 low-price services have a flagged cell, reach the larger-effect Confirmed branch, with 87.1% receiving compared with 2/10 same- or higher-price services. The relaySuspected. By comparison, Opus-to-Luna mixtures reach 54.2% level two-sided Fisher test gives p = 0.0698, reported as 0.07. detection at 25% and 99.5% at 50%. The panel includes a This is the primary price-tier test, and it does not meet a 0.05 model whose recent knowledge can exceed Astra’s, so the threshold. The proportions suggest an association among the difficult contrast is consistent with the bank’s limited coverage selected services, not a statistically established population price effect or a causal pricing mechanism. of newer knowledge. The robustness analysis counts a service only when at least The difference between a flag and the larger-effect label matters. For Haiku replacing Opus at 25%, detection is 100.0%, one of its cells is flagged in all four specifications. It gives but only 77.7% of simulations meet Confirmed. For Haiku 7/10 versus 1/10 and p = 0.0198, reported as 0.02. That lower replacing Astra at 25%, 52.6% meet Confirmed and 47.4% p value comes from a specification added after the primary meet Suspected. These branches reflect effect thresholds and test missed 0.05, so it is not the primary test and not stronger significance, not a posterior probability that a specific backend evidence. Both comparisons remain conditional on the selected sample and the decision to expand it after preliminary pilot was served. The official-to-official control produces no flags. At zero results. They also weight flags equally: R19 Astra’s RD of replacement, official responses from the second batch are 0.056 (q = 0.041) contributes one service, just as an RD above 8

TABLE III A NSWER ACCURACY AND BEHAVIORAL CLASSIFICATIONS IN BOTH ROUNDS , GROUPED BY ADVERTISED PRICE . A

O

Round 1

Round 2

rr

ro

RD

q

Low price R01 1 R04 1 R08 2 R11 2 R12 3 R13 3 R15 3 R17 3 R19 2 R20 2

0.803 – 0.808 0.744 0.936 – 0.968 0.712 0.936 0.704

0.968 – 0.992 0.992 0.952 – 0.952 0.952 0.992 0.992

+0.165 – +0.184 +0.248 +0.016 – −0.016 +0.240 +0.056 +0.288

<0.001 – <0.001 <0.001 1.000 – 1.000 <0.001 0.041 <0.001

S – S C N – N CCSS S C

Same price R02 1 R03 1 R05 1 R06 1 R09 3 R10 3 R14 3 R16 3b R18 3

0.992 0.960 0.976 0.944 – – 0.944 0.936 0.960

0.968 0.968 0.968 0.968 – – 0.952 0.960 0.952

−0.024 +0.008 −0.008 +0.024 – – +0.008 +0.024 −0.008

0.802 1.000 1.000 0.809 – – 1.000 0.571 1.000

Higher price R07 1

0.960

0.968

+0.008

1.000

Relay

Batch

Round 1

Round 2

L

rr

ro

RD

q

L

RD

+0.272 – +0.008 +0.224 +0.056 – +0.032 +0.032 +0.000 +0.288

C – N CCSS N – N N N C

0.704 0.912 0.840 0.418 0.944 1.000 0.958 0.712 0.976 0.409

0.992 0.992 0.992 0.992 0.968 0.968 0.975 0.968 0.992 1.000

+0.288 +0.080 +0.152 +0.574 +0.024 −0.032 +0.017 +0.256 +0.016 +0.591

<0.001 0.023 <0.001 <0.001 1.000 0.317 1.000 <0.001 0.368 <0.001

C S S C N N N C N C

+0.264 +0.056 +0.184 +0.280 +0.208 – +0.026 +0.248 −0.024 +0.583

C NNSS SSCC C CNCN – N C N C

N N N N – – N N N

+0.248 +0.016 +0.024 +0.040 – – +0.032 +0.056 +0.040

CCSS N N N – – N N N

0.976 0.976 0.984 0.952 0.952 0.776 0.880 0.968 0.976

0.992 0.992 0.992 0.992 0.968 0.968 0.968 0.992 0.968

+0.016 +0.016 +0.008 +0.040 +0.016 +0.192 +0.088 +0.024 −0.008

0.809 0.809 0.809 0.389 1.000 <0.001 0.049 0.571 1.000

N N N N N S SNNN N N

+0.024 −0.016 +0.045 −0.008 +0.016 +0.200 +0.000 +0.024 −0.024

N N N N N C N N N

N

+0.016

N

0.960

0.992

+0.032

0.554

N

+0.000

N

L

RD

L

A: GPT-6 Astra; O: Claude Opus 5. rr , ro : relay and official hit rates; RD: ro − rr . L lists four classifications in order: (1) primary bank, refusals counted as misses; (2) primary bank, refusals excluded; (3) alternative bank, refusals counted as misses; (4) alternative bank, refusals excluded. One letter means all four agree; CCSS means C in (1)–(2) and S in (3)–(4). C/S/N: Confirmed/Suspected/No evidence of behavioral inconsistency. Hit rates, RD, and q use the primary bank with refusals counted as misses; only L compares all four specifications. Bold: q < 0.05; dash: outside the main cohort or unmeasured.

0.5 does. Table III retains effect sizes so that these unequal fect from model substitution. We therefore report behavioral deficits remain visible. Figure 4 keeps the two specifications inconsistency and leave the causal backend claim unresolved. separate. Not every classification level is invariant. R17 Astra is 2) Large Deficits, Refusals, and Wrapper Effects: Broad Confirmed in both primary-bank analyses and Suspected in deficits persist in several cells. R01 Opus has a hit rate both alternative-bank analyses. It remains flagged in all four. of 0.704 against 0.992 officially, RD 0.288, and q < 0.001. This is the CCSS entry in Table III: changing the bank changes Its failures span knowledge and capability tasks. The pilot the label from C to S, but all four analyses still detect a adds larger Opus deficits at R20 (0.409 versus 1.000, RD difference. The first letter gives the primary result used in 0.591) and R11 (0.418 versus 0.992, RD 0.574). Their Astra the price comparison. The remaining letters show sensitivity cells are also flagged. Third-batch R17 is flagged for both to refusal handling and question selection. A C-to-S change targets, with primary RD 0.240 for Astra and 0.256 for Opus. therefore preserves a flag, while a change to N removes it. These deficits motivate investigation, without identifying the The robustness tally counts a cell only when no specification backend weights or excluding changed instructions, retrieval, returns N. The original three flagged API cells retain their labels across all four specifications. or execution settings. The same-price R14 flag depends on refusal handling. 3) Missing Cells and the Separate Client-Matched SuppleIts Opus primary hit rate is 0.880 against 0.968, RD 0.088, ment: The main table leaves four Astra cells missing. R04 and q = 0.0487. Only the primary bank with refusals scored requires an unmodified first-party client and lies outside the as misses is flagged. Excluding protocol-level refusals removes original compatible API protocol. R13 and R10 expose Astra the flag, as does the alternative bank. This cell accounts for only through the Responses interface, which is excluded to keep the difference between the primary and all-four service-level the interface consistent with the official chat-completions arm. tallies. This pattern is refusal-sensitive behavioral inconsistency. R09 changed key/group access during collection. Its partial It does not identify a replacement model. responses are excluded because no whole-cell collection under The same-price R10 flag survives all four specifica- one consistent configuration is available. These cases leave the tions. Its Opus rate is 0.776 against 0.968, RD 0.192, and accuracy comparison unresolved. q < 0.001. The service exposes a coding-client-style wrapper, R04’s separate client-matched comparison for the first batch with responses referring to disabled tools. Those integration gives 117/125 against 123/125, RD 0.048, and nominal twodifferences are relevant alternative explanations. Persistence sided p = 0.1023. Its alternative bank gives 119/125 against across scoring specifications cannot separate a wrapper ef- 123/125 and p = 0.2811. Refusal exclusion does not change 9

R01

+0.165 S

+0.288 C

R04

n/a

+0.080 S

R08

+0.184 S

+0.152 S

R11

+0.248 C

+0.574 C

R12

+0.016

+0.024

R13

n/a

-0.032

R15

-0.016

+0.017

R17

+0.240 C

+0.256 C

R19

+0.056 S

+0.016

R20

+0.288 C

+0.591 C

R02

-0.024

+0.016

R03

+0.008

+0.016

R05

-0.008

+0.008

R06

+0.024

+0.040

R07

+0.008

+0.032

R09

n/a

+0.016

R10

n/a

+0.192 S

R14

+0.008

+0.088 S

R16

+0.024

+0.024

R18

-0.008

-0.008

−0.1

0.0

0.2

0.4

Risk difference (official minus relay hit rate)

TABLE IV Q UESTION - LEVEL OFFICIAL - MINUS - RELAY ACCURACY GAPS AND BOOTSTRAP INTERVALS .

Relay Target R01 R01 R04

Flagged in all four analyses

7/10

Low price

Same or higher 0%

7/10 2/10 1/10

25%

50%

75%

Relays with at least one flagged cell

95% interval

0.288 [0.128, 0.464] 0.184 [0.080, 0.304] 0.080 [0.016, 0.160]

C. RQ3: Injected Content in Code and Setup Responses

0.6

Fig. 3. First-round official-minus-relay accuracy differences for the 20 services, grouped by advertised price. C/S: Confirmed/Suspected (primary analysis). n/a: cells outside the main analysis.

Primary analysis

Claude Opus 5 GPT-6 Astra Claude Opus 5

Mean gap

each question’s official and relay trials together. Table IV reports gaps that weight questions equally, with 95% percentile intervals. Missing calls explain why R01 Astra’s question-mean gap is 0.184 while its pooled-trial RD is 0.165. Supplementary Section III gives the resampling settings. Grouping also preserves the pairing between the official and relay answers to each question: a resampled question contributes both sides of the comparison. The resulting intervals describe uncertainty across questions within this bank, without treating repeated answers as new independent questions. All three intervals remain above zero, supporting the direction of these accuracy deficits after grouping repeated answers. The intervals cover only the original flagged cells, are conditional on the selected questions, and are unadjusted for multiple comparisons. They do not resolve shared-reference dependence or the separate batch-level correction families, and cannot identify the cause of the accuracy gaps.

Same or higher price

Claude Opus 5

Low price

GPT-6 Astra

100%

Fig. 4. Proportion of relays with behavioral differences by price group under the primary and robustness analyses.

these comparisons. The client-matched supplement does not replace a missing main-cohort cell or enter the price-tier test as a new relay. 4) Sensitivity to Repeated Answers: Repeated answers to one question can share errors. The original-cohort check therefore resamples the 25 shared questions as blocks, keeping 10

Table V summarizes the 1,686 relay-arm responses from the first-round code and setup workload and the candidates selected for review. There are zero confirmed injections. The one-sided 95% Clopper–Pearson upper limit is 0.00178, or 0.18%. This bound applies under the binomial model to confirmations in the measured workload. It does not cover all malicious edits, account histories, or production workloads. Two authors classify all 64 first-round candidates (48 relay-arm, 16 official-arm) across the four collections as benign. Legitimate task-related packages, known example destinations, and parser errors account for reviewed detections. The initial batch includes package-extraction false positives caused by shell continuations and requirements-file arguments. No candidate is promoted to an injection merely because the detector flagged it. Every measured cell’s confirmed rate and excess over its official reference are zero. The original compatible-API workload contributes 546 relay responses. Its separate client-matched supplement has 50 further relay responses and 50 official responses, all without candidates. These supplement responses are not included in the 1,686 cohort denominator. Neither are official arms nor the incomplete R09 Astra cell. The second round adds 1,646 relay-arm responses with 43 automatic candidates. Two authors classify 40 identically as benign. Three split labels on optional scikit-learn suggestions for loading Iris are adjudicated benign, matching seven earlier candidates of the same form, one from an official arm. No round-2 injection is confirmed. Pooling both rounds gives 0/3,332 confirmed events. Assuming independent binomial

Astra effects vanish: R08 from 0.184 to 0.008 and R17 from 0.240 to 0.032, while both services’ Opus cells stay flagged. Upstream routing, model updates, or account-level allocation could produce such changes, which these measurements cannot Low price Same or higher price distinguish. Relay A O Relay A O A same-price service changes. R02 Astra moves from RD R01 0/47 4/50 R02 0/50 2/50 −0.024 to +0.248. Its round-2 errors include statements that R04 – 4/49 R03 0/50 5/50 the 2025 Nobel Prize in Literature and Super Bowl LX had R08 0/50 3/50 R05 – – not yet occurred, plus wrong counting and arithmetic answers, R11 0/50 0/42 R06 0/50 5/50 while other trials of the same items are correct. The pattern R12 0/50 2/50 R09 – 2/50 is consistent with part of the traffic reaching a model with an R13 – 1/50 R10 – 1/50 R15 0/50 6/50 R14 0/50 1/50 earlier knowledge boundary, without identifying that model. R17 0/50 1/48 R16 0/50 3/50 R12 Opus is flagged only when refusals count as misses: 29 R19 0/50 3/50 R18 0/50 1/50 of its 135 responses refuse benign items, against one in the † R20 0/50 3/50 R07 0/50 1/50 official arm, and no behavioral difference is detected when Total 0/397 27/489 Total 0/350 21/450 refusals are excluded. Entries: relay-arm candidates/analyzed responses (the 16 official-arm candidates Two rounds bound, but do not characterize, change. The are excluded). A: GPT-6 Astra; O: Claude Opus 5. † : higher-price service; short-interval repeat of R01 Astra within the first batch (95/121, dash: unmeasured or excluded. RD 0.183) matched its primary result (98/122, RD 0.165) at short range, with shared questions, a shared reference, and outcomes, the one-sided 95% upper limit for confirmations is three backfilled keys. Official reference hit rates range from 0.09%; it is not a bound on actual malicious content. 0.952 to 0.992 for Astra and from 0.968 to 1.000 for Opus Repeated prompts and services need not generate indepen- across collections. This observed variation is smaller than dent Bernoulli outcomes. The exact binomial interval is reported an RD of 0.20 but comparable to effects near the Suspected with that assumption, while unknown screening sensitivity boundary. Without a dedicated official-to-official retest, the further limits its reach. The workload excludes executed tools, rate of flag changes without service changes is unknown. Two secrets, and long histories that could activate conditional rounds cannot distinguish service changes from reference and injection. A zero confirmation is a bounded negative result response variation or estimate how often routes change, and under these conditions, not certification that any service is safe. a stable label does not establish that every request used the Without rerunning both parsers on the initial workload, we same route. cannot quantify how the correction changed candidate counts. TABLE V F IRST- ROUND CONTENT- SCREENING CANDIDATES AND ANALYZED RELAY RESPONSES , GROUPED BY ADVERTISED PRICE .

E. RQ5: Tool Permissions, Usage Reports, and Charges D. RQ4: Accuracy Differences over Time

Table VI separates systematically tallied tool-call records The second round ran 11–18 days after the first-round from qualitative protocol, text, and account observations. Four collections (Section IV-G). Of the 36 first-round main cells, services return tool calls despite tool_choice: none. The 35 were retested. R13 Opus requires a named coding client. counts are 1 at R04, 24 at R11, 1 at R10, and 4 at R17. Table III compares results across rounds. The returned tool name is the unused declared noop. A fifth Six services are flagged in both rounds. Of the 35 service, R08, returns two web_search call records when recollected cells, 9 are flagged in both rounds, 5 lose their no tool declaration was supplied. These counts include calls primary flag, 2 gain one, and 19 have no flag in either. At the returned during failed attempts, not tool executions or rates. No service level, R01, R08, R10, R11, R17, and R20 are flagged official-arm response in either round returned a tool call, and in both rounds. R04, R14, and R19 are no longer flagged, and 24 of the 30 relay-arm calls come from one service. Qualitative R02 and R12 are newly flagged. The round-2 primary tier observations have no systematic count, and unknown counts comparison gives 6/9 low-price versus 2/10 same- or higher- are not treated as zero. Token surpluses in Table VI are secondprice services, again p = 0.0698, with R13 unmeasured. The round median differences from official counts for identical requests in the accuracy workload, with the content-screening all-four count gives 5/9 versus 2/10 (p = 0.17). The original seven flag transitions use different BH families workload identified separately. A returned tool call is not a completed action. The client across rounds. With one 35-pair family in each round, six pairs gain or lose a flag: R19 Astra is no longer flagged in round 1 executed none of these calls. The records cannot distinguish (q = 0.089), while all other primary labels are unchanged. relay modification, upstream parameter noncompliance, or This post-hoc check isolates correction-family effects, while output translation. These counts include failed attempts and response dependence and reference variability remain. Supple- cannot be read as rates among independent scored responses. mentary Section III gives the calculation; anonymous counts Protocol handling changes the measurement denominator. accompany the paper. R19 returned server-sent events to a non-streaming request. Lost flags are mostly small effects. Three of the five lost The parser handles this format without changing request flags had first-round RDs below 0.09 (R04 O, R14 O, R19 A), bytes, while unparseable outcomes are excluded from accuracy consistent with movement around the threshold. Two larger scoring. Treating them as incorrect answers would conflate 11

TABLE VI O BSERVED TOOL CALLS , RESPONSE FIELDS , REPORTED USAGE , AND ACCOUNT CHARGES .

Observation

Services (count)

Supported conclusion and limit

Tool calls despite suppression

Round 1, O: R04 (1), R11 (24), R10 (1), R17 (4); round 2: none Round 1, A: R08 (2); round 2: none

Returned noop calls contradict tool_choice: none; execution and modification point are unknown. Returned web_search calls; actual retrieval is unverified.

Search calls without supplied tools

Streaming for a non-streaming request R19

Stream semantics differ; says nothing of backend identity. Format difference, not a substitute fingerprint. content_filter treated as a safety refusal, not a failed call.

<think> tags in output Cross-format refusal representation

R01 (A) R15 (O)

Key disabled after safety refusal Client-restricted access

R05 (three keys, both rounds) R13 (O, round 2)

Sequence observed; operator rationale unknown. Requires a named coding client; not bypassed.

Input-token surplus (round 2)

R01, R11, R20 (A, 4.38k, 4.22k cached); R08 (A, 4.22k; O, 734); R10 (O, 59.5k); R16 (O, 274); R11 (O, content screening, 10.6k); R20 (O, content screening, 5.4k) R15 (O, −355), R20 (O, −275), R04 (O, −97) R05 (all), R12 (101/185)

Relay-reported usage: hidden input or altered accounting, not which.

Input-token deficit Dated model name for A Balance vs. reported usage (round 2)

Equal at R03, R04, R11, R12, R17, R19

Relay-side token counting. Two returned names for one request at R12; route unverified. Charges follow reported usage at listed prices; others not reconcilable.

A: GPT-6 Astra; O: Claude Opus 5.

compatibility with capability. Conversely, R15’s translated content_filter response is a safety refusal and counts as a miss in the primary analysis. Some apparent instruction clues have known confounders. R04 Opus mentions a request not to search. Our client supplied the same no-search instruction to both relay and official Messages arms. The phrase alone therefore does not show that the relay added instructions. Coding-client-style text at R10 likewise warrants checking for added instructions. It does not identify a replacement model. The table also states what account enforcement and reasoning-format observations can and cannot establish. Reported usage reveals unexplained token quantities. On the same Astra item, R01 and R20 report 4,416 input tokens and R11 4,418, with 4,224 cached in each, where the official API reports 36. R08 reports a 4,216-token cached surplus. A fixed, cached surplus of this size is consistent with an added system prompt, but altered accounting can produce the same reports. R10 Opus reports a median surplus of about 59,500 tokens, mostly cache reads, in line with its coding-client wrapper. Smaller surpluses (R08 and R16 Opus) and deficits (R15, R20, and R04 Opus) indicate relay-side additions or token counting. All five services with surpluses above 4,000 tokens are flagged in both rounds, but R08’s Astra cell has a surplus without a round-2 flag. A reported surplus thus need not coincide with lower measured accuracy. Billing follows the reported usage. For the six services exposing a balance endpoint, the balance change equals the cost implied by their reported usage at listed prices to the cent. R11 is the only service shared by this six-service group and the five services reporting more than 4,000 extra input tokens. Its extra 12

reported quantities therefore enter the bill, without establishing that corresponding hidden text existed. Whether the other four services bill their surplus cannot be determined. Both usage and balances are endpoint-controlled. The balance comparisons use collection totals across billed token types, as illustrated in Figure 1; Supplementary Table S3 gives R11’s token categories, rates, and reconciliation. Returned model names mostly match the request, with aggregator prefixes. R12 returns a dated Astra snapshot name for 101 of 185 responses and the undated name for the rest. No round-2 response contains a tool call, including R11, which returned 24 in round 1. These findings coexist at the service level across workloads and rounds; they do not establish a shared cause or co-occurrence within one request. RQ5 detects permission and delivery differences, but does not assess retention, credential misuse, software patches, or authentication of upstream requests. VI. D ISCUSSION A. Price and Relay Reliability Low-price services account for most persistent accuracy differences in our sample, which was expanded after a low-price pilot, but the results do not establish price as a reliable predictor of service quality. The primary price comparison does not reach the 0.05 significance threshold. Selection was nonrandom, and retesting the same services adds no independent sample. Lower prices can reflect efficiency, promotions, or different account arrangements, as well as changes that reduce answer quality. Conversely, matching the official price does not ensure that a relay preserves the requested service. Price is therefore useful for organizing comparisons, but choosing a higher-priced relay

cannot replace checking its answers, tool permissions, and billed usage.

D. Requirements for Verifiable Relay Services Our measurements identify what users can check today and what remains hidden. A returned model name or usage count is a statement from the endpoint, not independent verification of processing. To close this gap, a service would need to connect the client’s request to the processing and charges it reports. Useful records should cover the requested model and permissions, disclosed additions to the input, returned tool calls, and billed token categories. Users could then distinguish an agreed processing cost from an unexplained charge and determine whether a tool restriction survived forwarding. The record should also distinguish upstream output from changes made during forwarding. Otherwise, a valid provider signature could leave later changes and the final charge unexplained. Provider signatures can authenticate covered fields, and attested forwarding can constrain how participating infrastructure handles a request [2], [26], [28]. Their value depends on what they cover and which component the user trusts. Signing an answer alone does not verify its billed usage, and signing a token count does not by itself establish that the count was measured correctly. Such mechanisms require support unavailable through the ordinary relay APIs we tested. They can complement the behavioral, permission, and billing checks studied here.

B. Accuracy Differences and Model Identity An accuracy difference establishes a mismatch in observed answer quality, without identifying the model that produced it. Model substitution is one possible cause, but changed instructions, refusal policies, reasoning settings, retrieval, and caching can also change answers. Removing an explicit refusal instruction in our early pilot coincided with Opus scores rising from 6/40 to 20/40 (Supplementary Section V). This suggests prompt sensitivity, without isolating a causal effect, reinforcing the wrapper explanation. The sensitivity checks help separate some of these explanations: a difference that disappears when refusals are excluded has a different interpretation from one that persists across question banks and refusal rules. Even a persistent difference cannot isolate model substitution from an undisclosed wrapper. Likewise, passing the test only shows that the selected questions did not reveal a difference, and the mixture results show that a related cheaper model can pass at moderate replacement fractions. Services that evade checks or imitate fingerprints pose further challenges [24], [25]. These results support using accuracy tests to identify services needing further checks, while keeping claims about backend identity separate from measured behavior.

VII. E THICS C. Recommendations for Relay Users Measurements use registered accounts and benign knowledge, Users should check billed quantities alongside advertised reasoning, code, and setup requests. Returned commands and rates. For identical requests, the official API supplies a tool suggestions are not executed. Endpoint access restrictions reference for reported input tokens. A surplus warrants checking are not bypassed. No confirmed injection was found in the for added instructions or different token counting, without tested workload, so the study’s injection-disclosure criterion assuming that every reported token was processed upstream. was not met. We anonymize all measured services and retain Where balances are available, comparing their decrease with the the identity mapping privately for confidential verification. calculated charge shows whether reported usage enters the bill. The manuscript does not name the measured services, their Calculate charges separately for cached input and other token domains, or their route groups. The accompanying data contain categories because their rates can differ. Agreement with the anonymized results and plot inputs for both rounds, including balance confirms the charge calculation, but does not validate accuracy, content screening, usage, and billing summaries. Full the reported quantity. Accuracy comparisons should also be repeated with fresh requests, responses, failure logs, and annotations remain private official responses. Keeping the questions, request settings, and because they can expose service identities and account inforscoring rules fixed helps make changes interpretable. When mation. The supplied anonymous hit counts and analysis script a difference appears, users should first repeat the comparison allow recomputation of the common-family comparison. Other under the same conditions before changing the configuration. collection and scoring scripts and the complete frozen bank are Retaining the settings and responses allows later checks of not supplied, so the summaries alone do not allow independent whether a change concerns answer correctness, refusals, or rescoring, recalculation of every result, or reproduction of another returned field. Public monitoring services can help historical endpoint behavior. locate candidates, but their ratings cannot replace a comparison VIII. C ONCLUSION under the application’s own requirements [33], [34]. Tool permissions require enforcement in the client. An We studied 20 LLM API relays across two rounds using application should reject calls that violate its requested per- official APIs as references. The measurements found recurring missions and inspect tool names, arguments, commands, and accuracy differences, changes between rounds, and returned destinations before execution. Our finding of returned calls tool calls that violated explicit restrictions despite no confirmed despite no confirmed injection shows why content screening injection in the screened responses. Usage and balance checks alone is insufficient. Plugin and tool-metadata attacks also also showed that extra reported input-token quantities enter involve integration paths beyond standalone responses [32], R11’s bill, while their source remains unverified. These findings [38]. These checks should remain separate when reporting show why relay evaluation must connect answer quality, request results: correct answers do not establish correct billing, and a compliance, and billing: success on one check does not establish plausible answer does not authorize a tool call. that the full service meets the user’s expectations. For users, 13

the practical implication is to verify token quantities as well as prices, check returned tool calls before execution, and repeat comparisons as services change. For researchers, the study provides a documented way to relate these problems within the same services through public APIs and account records. These checks make delivery problems observable without internal access, while leaving their causes unresolved. More transparent services would link requested instructions, reported usage, and charges to verifiable records of processing, allowing users to assess whether they received the service they paid for. ACKNOWLEDGMENT OpenAI Codex [39] was used to polish the language in the abstract, Impact Statement, main text, and supplementary material. The authors take responsibility for the final manuscript. R EFERENCES [1] Y. Zhang, Y. Jiang, Z. Chen, M. Backes, X. Shen, and Y. Zhang, “Real money, fake models: Deceptive model claims in shadow APIs,” arXiv preprint arXiv:2603.01919, 2026. [2] S. Xie, Q. Wu, H. Lu, Z. Sun, Q. Wu, B. Qin, and Q. Wang, “The proxy knows too much: Sealing LLM API routers with attested TEEs,” arXiv preprint arXiv:2606.16358, 2026. [3] G. Lin, Y. Wan, S. Pei, T. Xu, K. Xu, and G. Xue, “Behavioral consistency and transparency analysis on large language model API gateways,” in Proceedings of the 2026 ACM Internet Measurement Conference (IMC ’26), 2026, pp. 563–573. [4] Y. Fang, Y. Feng, B. Li, and M. Zhou, “KBF: Knowledge boundary as fingerprint for language model and black-box API auditing,” arXiv preprint arXiv:2605.29524v3, 2026. [5] J. King, K. Klyman, E. Capstick, T. Saade, and V. Hsieh, “User privacy and large language models: An analysis of frontier developers’ privacy policies,” in Proceedings of the Eighth AAAI/ACM Conference on AI, Ethics, and Society (AIES). AAAI Press, 2025, pp. 1465–1477. [6] A. Finkman Noah, A. Shapira, E. Bar Kochva, I. Maimon, D. Mimran, Y. Elovici, and A. Shabtai, “CodeCloak: A method for mitigating code leakage by LLM code assistants,” in Proc. 28th European Conference on Artificial Intelligence (ECAI). IOS Press, 2025, pp. 4418–4427. [7] C. Li, I. Gim, and L. Zhong, “Confidential prompting: Privacy-preserving LLM inference on cloud,” arXiv preprint arXiv:2409.19134, 2024. [8] P. Gao, L. Wang, Y. Liu, K. Liu, F. Yang, and Y. Zhang, “Mind your key: An empirical study of LLM API credential leakage in iOS apps,” arXiv preprint arXiv:2606.12212, 2026. [9] Z. Chen, Y. Zhang, Y. Liu, G. Deng, Y. Li, Y. Zhang, J. Ning, L. Y. Zhang, L. Ma, and Z. Li, “How your credentials are leaked by LLM agent skills: An empirical study,” in Proc. 41st IEEE/ACM International Conference on Automated Software Engineering (ASE), 2026, pp. 536–548. [10] BerriAI, “SQL injection in proxy API key verification (CVE-202642208),” GitHub Security Advisory GHSA-r75f-5x8p-qvmc, Apr. 2026. [11] D. Pasquini, E. M. Kornaropoulos, and G. Ateniese, “LLMmap: Fingerprinting for large language models,” in Proceedings of the 34th USENIX Security Symposium (USENIX Security 25). USENIX Association, 2025, pp. 299–318. [12] T. Bruckner, “One token is enough: Fingerprinting and verifying large language models from single-token output distributions,” arXiv preprint arXiv:2607.10252, 2026. [13] I. Gao, P. Liang, and C. Guestrin, “Model equality testing: Which model is this API serving?” in Proceedings of the 13th International Conference on Learning Representations (ICLR), 2025. [14] X. Zhu, Y. Ye, T. Qiu, H. Zhu, S. Tan, A. Mannan, J. Michala, R. Popa, and W. Neiswanger, “Auditing black-box LLM APIs with a rank-based uniformity test,” in Proceedings of the 14th International Conference on Learning Representations (ICLR), 2026. [15] Y. Sun, Y. Li, Y. Zhang, Y. Jin, and H. Zhang, “SVIP: Towards verifiable inference of open-source large language models,” arXiv preprint arXiv:2410.22307, 2024. [16] A. Pal, L. Zahran, W. Gvozdjak, A. Potti, and M. Goldblum, “Privacypreserving mechanisms enable cheap verifiable inference of LLMs,” arXiv preprint arXiv:2602.17223, 2026.

14

[17] H. Jin, C. Zhang, H. Yu, S. Shi, N. Zhang, Y. T. Hou, and W. Lou, “Trusting what you cannot see: Auditable fine-tuning and inference for proprietary AI,” in Proceedings of the 35th USENIX Security Symposium (USENIX Security 26). Baltimore, MD: USENIX Association, Aug. 2026, pp. 6067–6086. [18] H. Liu, C. Shou, H. Wen, Y. Chen, R. J. Fang, and Y. Feng, “Your agent is mine: Measuring malicious intermediary attacks on the LLM supply chain,” arXiv preprint arXiv:2604.08407, 2026. [19] Y. Zhang, Z.-H. Zhang, and H. Qin, “Which model is actually serving you? IRIS: Budgeted black-box auditing of model substitution and routing dilution in LLM gateways,” arXiv preprint arXiv:2607.20860, 2026. [20] A. Dasgupta, A. A. Tanvir, and X. Zhong, “Watermarking language models through language models,” IEEE Transactions on Artificial Intelligence, vol. 7, no. 3, pp. 1642–1651, Mar. 2026. [21] T. Chauvin, C. Lalanne, E. Le Merrer, J.-M. Loubes, F. Taiani, and G. Tredan, “Token-efficient change detection in LLM APIs,” in Proceedings of the 43rd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 306. PMLR, 2026, pp. 13 247–13 277. [22] W. Cai, T. Shi, X. Zhao, and D. Song, “Are you getting what you pay for? Auditing model substitution in LLM APIs,” arXiv preprint arXiv:2504.04715, 2025. [23] A. Karvonen, D. Reuter, R. Rinberg, L. Marks, A. Garriga-Alonso, and K. Warr, “DiFR: Inference verification despite nondeterminism,” arXiv preprint arXiv:2511.20621, 2025. [24] A. Nasery, E. Contente, A. Kaz, P. Viswanath, and S. Oh, “Are robust LLM fingerprints adversarially robust?” in IEEE Conference on Secure and Trustworthy Machine Learning, SaTML. IEEE, 2026, pp. 326–352. [25] J. Zhang, X. Li, and S. Wang, “Your “Pro” LLM subscription may actually be “Free”: Exposing fingerprint spoofing risks in LLM inference services,” arXiv preprint arXiv:2606.16100, 2026. [26] M. Luo, Z. Zhang, Z. Liu, Y. Xie, Z. Zhang, D. H. H. Yeung, W. I. Lai, P. Chen, M. Wen, and D. She, “Rewriting the response path: Silent tampering and provider-signed defense in BYOK LLM agents,” arXiv preprint arXiv:2605.02187v2, 2026. [27] Y. Guan, “AEX: Non-intrusive multi-hop attestation and provenance for LLM APIs,” arXiv preprint arXiv:2603.14283, 2026. [28] F. Wang and Z. Tian, “Evidence-bound gateway-path provenance for third-party LLM inference,” arXiv preprint arXiv:2606.22560, 2026. [29] Y. Zhang, M. Zhao, Y. Zhang, and Y.-m. Cheung, “Trending applications of large language models: A user perspective survey,” IEEE Transactions on Artificial Intelligence, vol. 7, no. 4, pp. 1835–1852, Apr. 2026. [30] X. Xie, J. Song, Z. Zhou, Y. Huang, D. Song, and L. Ma, “Online safety analysis for LLMs: A benchmark, an assessment, and a path forward,” IEEE Transactions on Artificial Intelligence, vol. 7, no. 3, pp. 1626–1641, Mar. 2026. [31] A. Fastowski, B. Prenkaj, Y. Li, and G. Kasneci, “Injecting falsehoods: Adversarial man-in-the-middle attacks undermining factual recall in LLMs,” arXiv preprint arXiv:2511.05919, 2025. [32] W. Zhao, V. Khazanchi, H. Xing, X. He, Q. Xu, and N. D. Lane, “Attacks on third-party APIs of large language models,” in ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024, arXiv:2404.16891. [33] Hvoy AI, “AI API relay monitoring and rankings,” https://www.hvoyai. com/, 2026, accessed: Oct. 23, 2026. [34] LLMTest, “LLM API relay verification and rankings,” https://llmtest.cn/, 2026, accessed: Oct. 23, 2026. [35] API Relay Audit contributors, “API Relay Audit: Local security audit for AI API relays and LLM proxies,” https://github.com/toby-bridges/apirelay-audit, 2026, open-source software. Accessed: Oct. 23, 2026. [36] BazaarLink Research, “Model substitution in the black-box LLM API resale market: A 14-day, 171-endpoint, 625-probe empirical measurement brief,” https://bazaarlink.ai/research/llm-resale-substitution-2026, 2026, industry research brief, updated Jul. 29, 2026. Accessed: Oct. 23, 2026. [37] Y. Benjamini and Y. Hochberg, “Controlling the false discovery rate: A practical and powerful approach to multiple testing,” Journal of the Royal Statistical Society: Series B (Methodological), vol. 57, no. 1, pp. 289–300, 1995. [38] X. Li and X. Gao, “A first look at the security issues in the model context protocol ecosystem,” in Proceedings of the 56th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), 2026. [39] OpenAI, “Codex,” https://openai.com/codex/, 2026, accessed: Oct. 9, 2026.

Supplementary Material What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays TABLE S2 D ETECTION PERCENTAGES FROM 1,000 SIMULATED CELLS PER REPLACEMENT SETTING .

TABLE S1 C OLLECTION DATES AND OFFICIAL REFERENCES IN O CTOBER 2026.

Group

Relay dates

Reference date

Services

Batch 1 Batch 2 Batch 3 Batch 3b Round 2

Oct. 2–3 Oct. 5 Oct. 7 Oct. 9 Oct. 20

Oct. 2 Oct. 5 Oct. 7 Oct. 9 Oct. 20

7 4 8 1 19

Target

Substitute

10%

25%

50%

100%

O O O

Haiku 4.5 Luna GLM-4.5-Flash

94.1 2.1 94.6

100.0 54.2 100.0

100.0 99.5 100.0

100.0 100.0 100.0

A A A

Haiku 4.5 Luna GLM-4.5-Flash

47.1 0.0 53.7

100.0 4.0 100.0

100.0 77.7 100.0

100.0 100.0 100.0

A: GPT-6 Astra; O: Claude Opus 5.

I. S ERVICE S ELECTION AND C OLLECTION

Candidates were identified through model recommendations, expansion responses were collected, without changing the initial community resources, and English web searches. The initial and pilot assignments. model-recommendation search gave ChatGPT, Claude, Gemini, Foreign-currency prices use the last available official midand Grok the same context and two questions. After domain point exchange rate. Route-group and currency checks were deduplication, seven of 15 candidates qualified: two low-price, completed after first-round scoring but before the cross-group four same-price, and one higher-price service. The exclusions price comparison. The affected service remained in the samewere four enterprise-only services, one unavailable site, two price group. For the second round, the billed route group was without target support, and one without API access. checked before scoring. R17’s group charged half the official The low-price pilot used community repositories, English rate for both targets, retaining its low-price classification. web searches, and Hacker News. Four services qualified, reducing the planned five-service pilot and proportionally scaling II. C ALIBRATION D ETAILS its continuation threshold. Preliminary outcomes triggered Official configuration check. The second-batch calls reexpansion through deduplicated, interleaved recommendation lists and web-search results. The expansion added nine services: quested disabled thinking for Opus and no reasoning for Astra four low-price and five same-price. Paid placements were on a multiplication prompt. Opus accepted the request but excluded. Candidates were checked in order against the stated returned a thinking block and reported 182 thinking tokens. access, payment, and deposit requirements, with replacements Astra rejected the unsupported no-reasoning value with HTTP 400. The study’s low setting returned an answer with reported drawn from the same price group. Collection groups and timing. Table S1 identifies the groups reasoning tokens. Neither option disabled reasoning, so the full used in the main result table. The question bank and selection no-reasoning reference collection was not performed. Controlled response mixtures. Table S2 reports detection rules were fixed before the initial relay collection. Batch 1 includes next-day responses compared with the October 2 frequencies for the tested replacement fractions. Each percentreference; it is not entirely a same-day comparison. Subsequent age counts Confirmed or Suspected behavioral inconsistency groups used same-day official references. The second round among 1,000 resampled cells under a single nominal Fisher retained the questions and analysis rules with fresh official test, without multi-cell adjustment. The reference uses the first batch’s official responses, and the honest response pool responses. The initial API analysis uses a Benjamini–Hochberg (BH) uses the second batch’s official responses. Substitute responses family of 13 relay–target pairs. Later first-round groups each reuse pre-validation. A missing substitute pool falls back to the use a separate family, departing from the planned combined official pool. The nominal fraction is a sampling probability, not first-round correction. Round 2 uses one family across its 35 a fixed number of replacements in each cell. These frequencies testable pairs. The separate native-client comparison uses its apply only to the selected response pools. own official reference and nominal exploratory p-values. III. Q UESTION -L EVEL AND C OMMON -FAMILY C HECKS Price and route selection. Prices correspond to the key’s default route group, or the recommended individual-developer The original-cohort check resamples the 25 shared questions group when a choice is required. Paid-unlock routes are as blocks, keeping available official and relay trials within excluded. For quoted price intervals, the lower bound for the each question. It uses 20,000 bootstrap draws with seed comparable short-context setting is used. The four-quote mean 20261005 and reports unadjusted 95% percentile intervals. ratio in the main methods allows a 5% margin around the Questions receive equal weight, including when their trial official price for the same-price group. This rule was set before counts differ because of missing calls. RQ2 reports the three 15

originally flagged cells’ intervals, which remain conditional on supplement answers, 59 match reviewed answers, two apply the selected question bank. the clarified Go initializer rule, and nine retain initial labels. Common-family sensitivity check. For the 35 pairs mea- Matching reviewed answers and applying a rubric do not sured in both rounds, we recompute two-sided Fisher p-values constitute direct human review of every response. These later from the primary-bank hit counts and apply BH separately decisions do not reselect the frozen bank. Within each target’s to each round’s same 35 pairs. The RD and label thresholds primary bank, 84 non-automatic pre-validation labels inform are unchanged. Six pairs gain or lose a flag, compared with target or substitute scores. Reviewed-answer matches cover seven under the original families. Only R19 Astra’s first-round 42 for Astra and 49 for Opus. Of the 63 non-automatic label changes, from S to N (q = 0.0893). The accompanying substitute-pool labels per target used in the mixtures, 23 and anonymous counts and script reproduce this post-hoc check; 30, respectively, match reviewed answers. The remaining labels it retains pooled-trial dependence and is not a new reliability retain initial model scores; automatic scores are excluded from these denominators. Label transfer changed none of the valid validation. pre-validation scores, so it did not change frozen eligibility. This does not independently validate the unmatched labels. IV. A NNOTATION AGREEMENT AND L ABEL R EUSE Per-batch agreement. Agreement is computed before discussion on blind sheets that hide relay, target, official-reference status, and initial labels. Reasoning blocks are removed, and identical answers are grouped by question after case and whitespace normalization. Final adjudicated labels are stored separately from the submitted labels. Batch 1. One author labels fully manually and the other manually checks model prelabels item by item. The sheets contain 153 distinct field answers covering 524 API labels and 33 short-repeat labels, plus 100 sampled pre-validation answers. Of 253 items, 252 have two binary labels and 245 agree (97.22%, Cohen’s κ = 0.943). Seven disagreements and one uncertainty receive author adjudication. The field/repeat subset has κ = 0.897 and the pre-validation sample κ = 0.940. Batch 2. Both authors annotate fully manually. Among 105 distinct answers, 103 have two definite labels and 102 agree, with κ = 0.981. One disagreement and two uncertain items are adjudicated using the same compound-answer and initializer rules as batch 1. Batch 3. Both authors annotate fully manually. The main sheet contains 108 items, including three uncertain items. All 105 items with two definite labels agree (κ = 1.0). The uncertain items are adjudicated. A later six-item supplement also has complete agreement. Its single-label distribution makes κ undefined, not 1.0. The batch-3b sheet contains 16 distinct answers. Another 120 labels reuse identical, already adjudicated answers. All 16 items agree (κ = 1.0). Round 2. Both authors annotate fully manually. The sheet contains 234 distinct answers. Another 1,011 labels reuse identical adjudicated answers. Of 233 items with two labels, 228 agree (κ = 0.956). Five disagreements and one missing label are adjudicated under the batch-1 compound-answer, initializer, and hedging rules. Both authors classify 40 of 43 injection candidates as benign. The three disagreements are also resolved as benign after review. Across first-round batches, both authors classify all 64 response-screening candidates as benign. This count includes official-arm candidates. There is complete observed agreement but no category variation, so injection-review κ is undefined. Shared rubric errors remain possible. Label reuse. Reviewed-answer matches cover 174 of 480 valid non-automatic pre-validation labels. The remaining 306 retain initial model labels. Of 70 free-form native-client 16

V. Q UESTION S ELECTION AND R ESPONSE P ROCESSING Question-bank revisions. Before formal relay collection, official and substitute pilot responses led to three changes: replacing DeepSeek Flash, whose initial rough matching scored 30/40 recent-event answers, with GLM-4.5-Flash (0/40 on those items), removing an instruction to answer unknown questions with an explicit refusal, and adjusting automated scoring to ignore Markdown emphasis and accept a final answer line. The refusal instruction produced 6/40 correct Opus answers in an early pilot, compared with 20/40 after removal. The bank and selection rules were fixed before formal relay collection. Luna’s documented knowledge cutoff is later than Astra’s, so recent-event questions alone cannot distinguish them. Capability questions must do so. Request handling and scoring. An early endpoint observation of unexpected search-tool availability led to the unused declared tool and explicit tool suppression in Messages requests. Both relay and official arms receive the same no-search instruction, so an answer mentioning that instruction alone does not establish relay modification. The response parser supports server-sent events even when the request disables streaming. Parser corrections did not change request bytes. Outcomes that could not be parsed are excluded from answer scoring. Complete correct answers receive credit despite reservations about verification, while incomplete compound answers receive zero. The clarified Go initializer rubric accepts paraphrases that convey the intended initial value without explicitly claiming an incorrect two-argument syntax. Second-round metadata collection records identifier prefixes, fingerprints, and selected headers without changing request bytes. Content screening and parser correction. The package screen compares imports and installation names with expected libraries and marks lookalikes at similarity at least 0.8. Other screens inspect execution and dynamic-evaluation constructs, domains outside a whitelist with subdomain-boundary checks, decodable encodings, custom package indexes, and downloadto-shell patterns. Encoding checks include prose and comments. The other screens inspect code blocks. The initial package scanner falsely treated shell continuation characters and requirements-file arguments as package names. The corrected parser was used for subsequent collections without changing requests. All candidates from both scanner versions received contextual review. Because the initial

workload was not rescanned with both versions, the effect of the correction on candidate counts cannot be quantified. VI. R11 U SAGE AND C HARGE R ECONCILIATION Table S3 includes both workloads and recorded attempts with metadata, matching the original collection-total calculation. Ordinary input excludes cache reads and writes. For each model, the calculated cost is (Ti pi + Tr pr + Tw pw + To po )/106 ; token categories and rates are listed separately to avoid counting cached input twice. The total is $1.20107395, against a recorded deduction of $1.2011, a difference of $0.00002605 at the stored precision. This reconciles endpoint-reported quantities and charges, not upstream processing or undisclosed text. TABLE S3 R11 ROUND -2 REPORTED TOKENS AND RATES IN USD PER MILLION TOKENS .

Astra (188 calls) Opus (185 calls) Category

Tokens

Ordinary input 63,575 Cache read 756,388 Cache write 0 Output 25,011 Cost (USD)

Rate

Tokens

Rate

1.50 180,067 0.15 722,046 1.875 0 7.50 78,293

1.25 0.125 1.5625 6.25

0.39640320

0.80467075

17

Related documents

• Using LLMs in Software Design: An Empirical Study of GitHub and A Practitioner Survey#1036689
• Exploring Individual Factors in the Adoption of LLMs for Specific Software Engineering Purposes#141723
• The New Editorial Gatekeepers: Understanding LLM-Based Interfaces, Their Benefits, Risks and Design#94317
• Clone What You Can't Steal: Black-Box LLM Replication via Logit Leakage and Distillation#996874
• When the Model Retires: An Empirical Study of LLM Migration in Open-Source Applications#1105846