Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss

Testing Conversation Preservation Across LLM APIs: A RuleBased Oracle for Detecting Silent Information Loss Xin Liu∗ Independent Researcher

Xu Han∗

xu.han2@griffithuni.edu.au

arXiv:2611.15543v1 [cs.SE] 11 Nov 2026

School of Engineering and Built Environment Griffith University

Yuxin Wang Independent Researcher

Haoran Song Independent Researcher

Mingjian Zhao Independent Researcher

Lei Wang†

l.wang4@griffith.edu.au

School of Engineering and Built Environment Griffith University

Abstract Large language models (LLMs) are increasingly accessed through different APIs, with translators and compatibility layers enabling applications to move between providers. However, API compatibility does not guarantee that a translated request preserves the original conversation. Important information, including instruction roles, tool interactions, error states, and generation controls, can be silently altered or discarded even when the resulting request remains structurally valid and is accepted by the target API. Such changes can affect model behaviour and undermine the reliability of LLM applications and evaluations, yet they are difficult to detect through conventional API validation. We introduce isloss, a deterministic test oracle for checking conversation-information preservation across LLM API translations. It maps source and translated requests and replies into a shared intermediate representation, ConvIR, and applies explicit preservation rules to distinguish legitimate representational changes from information loss. Unlike schema validation, isloss checks whether conversation information survives translation; unlike reference-based differential testing, the oracle requires neither a reference translation nor additional model calls. It also distinguishes avoidable information loss from degradation caused by limitations of the target API. We evaluate isloss through specification-based fault injection, independent human review, and experiments on existing translators, gateways, and client libraries. On held-out tests, isloss detects 95 of 125 planted faults (76.0%), with two false alarms among 100 permitted changes (2.0%). Across a synthetic request suite evaluated on six configurations of five translator codebases, 127 of 685 completed translations (18.5%) pass static target checks while silently introducing at least one avoidable change to prompt, role, or tool information. We also find substantial information loss in translated replies and stream handling: 203 of 558 translated reply streams (36.4%) cause exceptions or silent corruption in at least one tested official OpenAI SDK. Targeted behavioural experiments across six models further demonstrate that ∗ Xin Liu and Xu Han contributed equally. † Corresponding author.

1

translation-induced changes can alter model behaviour when tasks depend on the affected information. These findings show that successful API translation is not sufficient evidence of conversation preservation. By making information loss testable at the translation boundary, isloss provides a practical foundation for improving the reliability of LLM interoperability, agent execution, and model evaluation. [Project website] [Code]

1

Introduction

LLM applications, particularly agents, increasingly interact with models from different providers through translators that convert requests and replies between API formats. These formats include OpenAI Chat Completions and Responses, Anthropic Messages, and Gemini generateContent. Translation may take place inside the calling program, in a local proxy or shared gateway, or behind a vendor’s OpenAI-compatible endpoint (Fig. 1; Anthropic, 2026f; Portkey, 2026; Google, 2026b; Anthropic, 2026d). Widely used libraries such as LiteLLM and the Vercel AI SDK provide these mappings, placing translation layers directly in the execution paths of production agents and model evaluations (BerriAI, 2026b; Vercel, 2026). As a result, reliable translation is not merely a matter of accepting different request formats: it is necessary to preserve the information that determines how a model should interpret and respond to a conversation. However, API compatibility does not guarantee conversation preservation. A translated payload can satisfy the target API’s schema and be successfully accepted while silently changing what the caller communicated. Consider a tool failure. Anthropic represents a failed tool result using is_error: true. Gemini provides an error convention through functionResponse.response, yet some translators omit the failure indication and present the result as an ordinary tool response (Anthropic, 2026a; Google, 2026a). The translated payload is well formed, but the model no longer receives the information that the tool failed. Its next decision can therefore be based on a materially different conversation from the one intended by the caller. This is not simply a formatting discrepancy: it is a silent change to the information available for model reasoning and action. Fig. 3 illustrates how our checker identifies this loss. Detecting such changes is challenging because API dialects encode conversation information differently. Roles, tool interactions, generation controls, and reasoning blocks may have different structures and field names, making direct JSON comparison too strict: it would also report harmless changes in representation. Conversely, some information has no native counterpart in the target API. For example, OpenAI Chat does not provide native fields corresponding to Anthropic’s tool-error flag or signed reasoning blocks (Anthropic, 2026a;b; OpenAI, 2026). Consequently, a useful checker should distinguish two fundamentally different situations: information that is lost because the target API cannot represent it, and avoidable information loss where the target API could have preserved the information but the translator does not. It should also identify whether the translator makes such limitations or losses explicit to the caller. Conventional API tests do not necessarily reveal these problems. In our audit of the four best-tested projects, among 41 classes of request-side changes, existing unit tests treat the changed output as expected in 26 cases (63.4 %). This motivates a more direct question: when a translation produces a target-valid payload, which properties of the original conversation have actually survived, and which have been silently lost? We introduce isloss, a deterministic test oracle for checking conversation-information preservation across LLM API translations. Inspired by translation validation, which checks the correctness of individual translations rather than relying solely on the translator’s implementation (Pnueli et al., 1998; Necula, 2000), isloss compares each source payload with its translated counterpart. It maps both into a shared intermediate representation, ConvIR, which captures conversation properties including authorship, authority, content order, tool links, error flags, and explicit generation controls. A set of 27 preservation rules then checks whether these properties survive translation, while accounting for legitimate changes in representation and limitations of the target API. The checker also records whether a loss is avoidable and whether the translator warns the caller. It supports requests, replies, and reply streams, without requiring model calls or a reference translation. We first assess whether isloss can reliably distinguish information loss from permitted changes. Using held-out, specification-based tests, it detects 95 of 125 planted faults (76.0 %) while flagging only 2 of 100 allowed 2

changes (2.0 %). We further assess its findings through program mutants, public bugs, and independent human review. Reviewers confirm 31 of 54 request-side candidates as defects. Among the 28 candidates classified as avoidable, 23 are confirmed defects, one is accepted as a form change, and four remain disputed (see Sec. 4.3). These tests provide evidence of both the checker’s detection capability and the need to distinguish genuine preservation failures from legitimate API-specific transformations. We then apply isloss to existing translators to characterise the prevalence and mechanisms of information loss. Our main request study covers six configurations of five codebases: LiteLLM, Vercel AI SDK v5 and v7, LLM-Rosetta, @musistudio/llms, and claude-code-proxy, spanning 14 source-to-target paths. Across 685 completed translations generated from 51 statically valid source templates, 127 (18.5 %) pass static target checks yet silently introduce at least one avoidable change to prompt, role, or tool information. This rate is measured on our synthetic request suite, counting completed translations with at least one primary finding; it is not an estimate of the prevalence of information loss in real-world traffic. Endpoint-accepted subsets and alternative counting rules are examined in Sec. 6.1. The consequences extend beyond translated requests. In a separate reply-stream experiment, 203 of 558 translated streams (36.4 %) cause an exception or silently corrupt the assembled message in at least one tested official OpenAI SDK (Sec. 6.3). Targeted live-model experiments across six models find measurable behavioural effects for nine of the ten mechanisms tested. In the tool-error example, removing the failure indication reduces compliance from 30/30 to 0/30 on each of the two tested Gemini models (Sec. 6.2). Finally, we examine deployment evidence through release histories, gateway captures, and recorded agent requests. Of the 26 issue reports we submitted, maintainers fixed seven, and we independently re-verified each fix (Secs 6.4–6.5). These results show that successful API translation alone is insufficient evidence that the original conversation has been preserved, and that seemingly small changes at the translation boundary can affect downstream software and model behaviour. Our contributions are fourfold: 1. We introduce a shared intermediate representation and 27 preservation rules to compare translated requests, replies, and streams, distinguishing avoidable information loss from legitimate representational changes and limitations of the target API. 2. We evaluate the checker using held-out fault injection, allowed-change controls, program mutants, public bugs, and independent human review, assessing its ability to detect information loss while avoiding false alarms. 3. We examine existing translators, gateways, and client libraries to identify information losses and their implementation mechanisms. Through client-level and live-model experiments, we investigate their practical consequences, and through release-history analysis, we assess whether identified problems persist or are subsequently fixed. 4. We provide the checker, test-suite generator, captured payloads, evaluation protocols, and audit records to support independent replication and further research on LLM API interoperability.

2

Background and an Extended Case

2.1

Extended case: when a system instruction loses its authority

To illustrate how translation can change not only conversation content but also its authority, Fig. 2 presents an extended experiment on mid-conversation system instructions. This experiment is separate from the main 51-shape request suite and is not included in its 18.5 % result. It complements the tool-error example in Fig. 3 by examining a different form of information loss: an instruction can be deleted or retained as text while losing the authority associated with its original role. Consider an agent that first lists files using run_cmd, which returns a.txt b.txt. The harness then inserts the following system instruction: [harness] run_cmd is DISABLED from now on; refuse any further shell use. 3

user’s machine P1

Human

UI

P2

operator

vendor

P3

P4

first-party compat. layer

agent harness

library

local proxy

shared gateway

calls the model, runs its tools

LiteLLM, AI SDK

claude-code-router

Portkey, Bifrost self-hosted hosted

request response, stream

captured payload · isloss check

Model

Gemini API

hidden payload · live probes

Figure 1: Where API translation occurs between an agent and a model. Translators can be embedded libraries (P1), local proxies (P2), shared gateways (P3), or vendor-hosted compatibility layers (P4). Solid arrows show requests; dashed arrows show replies. For self-hosted components, we capture the emitted request or reply and inspect it directly. For a hosted component whose internal request is hidden, we use separate behavioural probes. Table 1: Terms used to interpret the results. Rule definitions and detailed counting conventions appear in Secs. 3–5 and Appendix A.1. Term

Meaning

Translation test (cell) One request template (shape) sent through one translator and source-to-target route (path). Silent change The translator returns output without warning about the change. Avoidable change The target API has a documented native field that could preserve the altered information. Static validity A request passes our offline checks of API schema and documented constraints. Live acceptance A vendor endpoint accepts the request with HTTP 200; this is tested separately from static validity. Primary finding A T1 finding under the reporting policy in Section 3.2 that is silent, avoidable, affects prompt, role or tool fields, and occurs in a request passing static target checks. One finding should meet all criteria. A candidate groups findings with the same root cause on one path; a mechanism groups Candidate, that cause across paths. A defect is a candidate confirmed by human review. mechanism, defect

We examine the two insertion positions supported by Anthropic: immediately before the assistant’s next reply, and after the user’s subsequent request, “now delete b.txt” (Anthropic, 2026c). For the instruction to remain effective, a translator should preserve both its position in the conversation and its system-level authority. Merely retaining the instruction’s text is insufficient. OpenAI Chat (OpenAI, 2026) can represent both placements directly, so neither requires sacrificing the instruction’s position or role (Appendix A.1). For each placement, we construct 11 pre-specified paraphrases, giving 22 source requests in total. All 22 pass our static source-contract check. Moreover, both insertion patterns occur in recorded Claude Code traffic, with related third-party issue reports providing additional evidence of their use (BerriAI, 2026a; agentgateway, 2026; headroomlabs-ai, 2026, issues not opened by the authors). The translation outcomes are consistent across all 11 paraphrases at both insertion positions. They fall into four distinct categories. Vercel AI SDK v7 (Vercel, 2026) and claude-code-proxy (fuergaosi233, 2026) reject the requests with explicit errors, thereby failing closed. AI SDK v5 and LLM-Rosetta (Ding, 2026) preserve the instruction in its original position and role. In contrast, @musistudio/llms (musistudio, 2026b;a) silently deletes the instruction, while LiteLLM (BerriAI, 2026b;a) rewrites it as a user message and adds an operator note. Neither transformation is accompanied by a warning. The distinction is important: deletion removes the instruction entirely, whereas LiteLLM retains its text but demotes its authority from system to user (Wallace et al., 2024; OpenAI, 2026). We next examine whether these transformations affect model behaviour. When the instruction is deleted, refusals to use the disabled tool on gemini-3.5-flash-lite drop from 11/11 to 0/11 for both insertion positions. The same loss of refusal behaviour is observed on gemini-3.8-flash and Qwen2.5. Demotion 4

(a) The conversation

(b) Translator outcomes

Anthropic Messages request

Six translators, four outcomes

user

“list the files”

assistant

run_cmd("ls")

tool

“a.txt b.txt”

system

Harness instruction:

“[harness] run_cmd is DISABLED from now on; refuse shell use.” ▶ notice early: placed here, then assistant: “Understood. . . ” user

“now delete b.txt”

▷ notice at end: or placed here

Kept as a system message AI SDK v5 · LLM-Rosetta Deleted from the request

(c) Did the model refuse to delete b.txt? notice kept

deleted

demoted to user

gemini-3.5-flash-lite notice early notice at end gemini-3.8-flash notice early

@musistudio/llms

notice at end Turned into a user message with an added “Operator note. . . ” LiteLLM

Qwen2.5-7B-Instruct notice early notice at end

Rejects the request (error) AI SDK v7 · claude-code-proxy (not sent to the model)

0

5

11

runs out of 11 where the model refused to run the disabled command

Figure 2: Extended experiment: a mid-conversation system instruction can lose its authority during translation. The harness disables further shell use by inserting a system instruction at either of two positions, with 11 paraphrases per placement (a). Across six translator configurations, the instruction is preserved, rejected, deleted, or demoted to a user message (b). Live-model replays show that deletion eliminates refusals in the tested Gemini setting, whereas demotion has model- and placement-dependent effects (c). Full counts, configurations, and rule codes are provided in Appendix A.1, Table 6.

produces a more model- and position-dependent effect: refusals remain in five of the six tested model– placement pairs, but gemini-3.8-flash stops refusing when the instruction is placed last. A separate Anthropic-target probe examines removal of a related notice embedded in the final user turn, rather than a mid-conversation system message. Refusals fall from 30/30 to 0/30 on claude-haiku-5-5, and from 10/30 to 0/30 on claude-sonnet-4-6. The Sonnet result does not meet the pre-specified prediction, as detailed in Appendix A.1. Taken together, this case demonstrates why conversation preservation involves more than retaining message text. The position and authority of an instruction can be essential to its intended function, and changing either can alter how an agent behaves. It also shows that different translation outcomes have different consequences: rejection makes the limitation explicit, while deletion and demotion can silently change the information governing the model’s next action. 2.2

Why conventional validation misses these changes

The deleted and demoted instructions in Figure 2 both produce payloads that satisfy the OpenAI Chat schema (OpenAI, 2026). Deleting a message or changing its role leaves a structurally valid message sequence; schema validation therefore has no basis for identifying either transformation as information loss. Project-level tests do not necessarily provide the missing guarantee. LiteLLM’s unit tests assert its rewrite as expected output, effectively accepting the demoted instruction, while @musistudio/llms has no unit tests. More broadly, our audit of eleven translator codebases finds no property-based, differential, or referenceimplementation tests. Yet simply comparing source and target JSON is not a satisfactory alternative: legitimate dialect differences in role names, fields, and message structures would also be reported as failures (§4.3). The specifications themselves do not always resolve the ambiguity. Our specification analysis identifies 17 cross-dialect mappings that are unspecified or contradictory in vendor documentation (Anthropic, 2026b; Google, 2026a) (Appendix A.1). These observations expose a gap between two kinds of validation. Schema validation checks whether the target payload is acceptable, while direct structural comparison checks whether its representation is unchanged. Neither establishes whether the information conveyed by the original conversation has survived translation. 5

A useful test oracle should instead compare the conversation properties that matter while allowing legitimate changes in how different APIs encode them. 2.3

Checking conversation-information preservation without a model

isloss addresses this gap by mapping each source and translated payload into a shared conversation representation, ConvIR, and comparing their protected information after accounting for documented representation changes. The protected properties include speaker identity and authority, event order, tool call–result links, error flags, explicit generation controls, and reasoning continuity (Anthropic, 2026a;b; Google, 2026c; Pnueli et al., 1998; Necula, 2000). A difference that violates a preservation rule is recorded as a semantics-altering change under our operational definition. The checker separately records whether the translator warns about the change, distinguishing disclosed degradation from silent information loss (Table 1). This comparison requires neither model calls nor a reference translation: it checks the source and target representations directly. Fig. 3 illustrates the process using the lost tool-error flag. The next section formalises ConvIR and the preservation rules; subsequent experiments assess the checker’s reliability and investigate the consequences of the changes it detects. Importantly, isloss checks preservation of explicitly represented conversation information. It does not claim to establish complete semantic or behavioural equivalence between different APIs or models.

3

Checking Conversation-Information Preservation

3.1

A shared representation

Different LLM APIs encode conversations using different message structures, role names, and field conventions. Directly comparing their payloads would therefore conflate changes in representation with changes in the information conveyed. To avoid this problem, isloss maps source and translated payloads into a shared intermediate representation, ConvIR, which separates API-specific syntax from explicitly represented conversation information. ConvIR consists of an ordered sequence of conversation events and the generation controls explicitly set by the caller. Each event records its speaker (user, developer, assistant, or tool), content type, payload, and relevant flags, such as is_error. The control component records settings such as tool_choice, declared tool names, sampling parameters, and token limits. This representation allows the checker to compare properties of the conversation even when the source and target APIs encode them differently. For example, Anthropic’s top-level system field and an OpenAI leading system message map to the same instruction event in ConvIR; likewise, an assistant message and Gemini’s model message map to the same assistant speaker. Fig. 3 illustrates why this abstraction matters: changing the name or structure of a tool-result container can preserve the underlying information, whereas dropping is_error removes the indication that the tool failed. The former is a representation change; the latter changes the information available to the model (see Appendix A.2). 3.2

Comparing representations and classifying changes

Let S denote a source payload and T its translated counterpart. isloss compares IR(S) and IR(T ) after applying permitted representation changes. These include consistent mappings of role names and tool-call identifiers, merging adjacent messages from the same author when their content and order are preserved, and accounting for OpenAI Chat’s separate text and tool-call fields. The comparison is restricted to the properties represented in ConvIR: passing the check establishes preservation of these properties under the specified rules, not equivalence of all possible model outputs. A detected difference should be interpreted in context. In particular, we distinguish three questions: what information changed, whether the target API can represent it, and whether the translator disclosed the change to the caller. These dimensions determine how a change is reported. A change may be a legitimate re-encoding, an avoidable loss, or a degradation caused by a target limitation. Importantly, a documented degradation is still a loss of information; disclosure makes it explicit but does not make the translated conversation equivalent to the source. We use three reporting categories: 6

(a) Source request (Anthropic)

(b) AI SDK v7 output (Gemini)

(c) What isloss compares (both as event lists)

{"role": "user", "content": "..."}

{"role": "user", "parts": [...]}

= user text: “weather in Paris?”

{"role": "assistant", "content": [

{"role": "model", "parts": [

= assistant calls get_weather (assistant → model)

{"type":"tool_use", ...}, ...]}

{"functionCall":{...}}, ...]}

same meaning, different syntax → ignored

{"role":"user", "content": [{

{"role":"user", "parts": [{

= syntax reshaped: same meaning → ignored

"type": "tool_result", ...

"functionResponse": {...}}

"is_error": true}]}

]} (no error flag)

{"role": "user", "content": "..."}

{"role": "user", "parts": [...]}

tool-result error flag lost (is_error dropped) = user text: “continue”

Why the lost flag is a primary finding (all criteria apply): information changed

no translator warning

target could carry it

prompt, role or tool field

passes static target checks → counts

Figure 3: A dropped tool-error flag reveals avoidable information loss. In this four-turn request, accepted by both live APIs, (a) the Anthropic source and (b) the captured Gemini output from AI SDK v7 use different role and field names. (c) isloss accounts for these representation changes but identifies the loss of is_error: true. The separately labelled target-native repair is schematic: it illustrates how Gemini can represent the error, rather than reproducing the measured translator output. • T1: potential defects included in the primary finding count. These are unannounced changes to protected conversation information that are not assigned to the degradation or exclusion category below. Examples include deleted events or error flags, broken tool-call–result links, unpermitted reordering, altered explicit controls, and forged integrity tokens. • T2: separately reported degradations and informational notes, excluded from the primary finding count. This category includes documented losses caused by target limitations, as well as changes excluded from the primary count by our study policy. The latter include system-to-user role demotion, which we conservatively exclude even when the target API can preserve the original system role. T2 also records cache, usage, and model-related notes. Thus, T2 is a separate reporting category, not a claim that all its entries are unavoidable or harmless. Target support and warning status are retained as separate properties. • T3: permitted changes of representation. These transformations preserve the information represented in ConvIR and are not counted as information-loss findings. The distinction affects the interpretation of our results. In Fig. 3, dropping the tool-error flag is an avoidable, unannounced loss: Gemini can represent the error, but the tested translation does not preserve it. The change is therefore reported as T1. In the extended system-instruction experiment, deletion is also reported as T1. LiteLLM’s rewrite, however, produces a T2 record for demoting the system instruction to a user message, in accordance with our conservative counting policy, and a separate T1 finding for the inserted operator note. To assess the sensitivity of the primary counting policy, we also report a broader variant that includes target-representable T2 degradations, including role demotion. Under this alternative policy, the result increases from 127/685 (18.5 %) to 134/685 (19.6 %) primary cells (variant e2 in Table 17 in the Appendix, and Sec. 6.1). This variant includes all target-representable degradations recorded as T2; it does not change the underlying translation observations. The extended mid-conversation system-instruction experiment remains separate from the main suite (Sec. 5.2). Human review and behavioural trials assess the validity and consequences of these changes independently of the counting policy (see Sec. 4.3 and Appendix A.2). Replies, streams, and refusals. The same ConvIR representation and T1 preservation rules also apply to translated replies returned to the caller. For streams, we additionally define a stream-equivalence relation (Chen et al., 2020; Segura et al., 2016): reassembling a translated stream should preserve the 7

corresponding non-streaming reply. This comparison uses matched representations of fixed reply content, rather than independently sampled model completions, so that differences in model generation do not confound the translation check. When a target API cannot represent a construct, a translator may reject the request with a client-visible error or explicitly declare the degradation rather than silently changing the conversation. We record these outcomes separately from silent T1 findings (Appendix A.2). Unspecified cross-API mappings. Not every cross-API mapping is fully specified by vendor documentation. Where the specifications leave a mapping open, we do not infer that every representational difference constitutes a violation. Instead, we count an information loss only when there is evidence that the target API can carry the relevant information. To investigate such cases, we conduct separate Gemini endpoint probes that resolve 13 of the 17 previously undecided constructs (Sec. 2.2; Appendix A.2). These probes provide endpoint evidence for the mapping analysis; source and target acceptance are examined separately in Secs. 5.2 and 5.3. 3.3

Coverage of publicly reported bugs

We assess the scope of the preservation rules against 125 publicly reported translation issues collected from twelve issue trackers. The rules cover 99 issues (79.2 %), indicating that they capture a substantial range of previously observed translation failures. This corpus is concentrated on LiteLLM and is used to assess coverage of known issue types, not to estimate the prevalence of failures across translators. Of the 26 issues not covered, three (2.4 % of the corpus) describe legitimate normalisations, while the remaining 23 (18.4 %) fall into five categories outside the current scope of ConvIR and its rules. The issue corpus also reveals a difference between request- and response-side failures: request-side issues are more likely to produce explicit errors, with 35 % reported as silent, whereas response-side issues are more often silent (62–67 %; Appendix A.2). This asymmetry motivates checking both directions of translation. A translator may fail visibly when constructing a request, but silently alter information when converting a model’s reply or assembling a streamed response.

4

Checking Translations with isloss

Our isloss evaluates a translator by comparing the source payload with the payload it produces, without calling a model or requiring a reference translation. The evaluation has three stages: capturing the translator’s output, applying the preservation rules from Sec. 3 to identify and classify changes, and aggregating findings under explicit counting criteria. We use this procedure to study requests, replies, streams, and deployed translation paths. Checker validation and human assessment are reported separately from experiments that measure downstream behavioural effects. 4.1

Capturing translations and identifying changes

For each self-hosted translator, we exercise its documented public interface and direct its output to a local mock server. We capture the outgoing payload at the last self-hosted translation hop in Fig. 1, before it reaches the target service. Exceptions and explicit refusals are recorded separately as fail-closed rejections, rather than being treated as completed translations with preservation findings. The complete request suite contains 4,998 test executions; a fresh checkout reproduces their outputs byte-for-byte in 11 seconds. The capture point is kept consistent within each translator configuration. For LiteLLM, the primary capture is its pre-serialisation HTTP-layer dictionary; we additionally inspect the serialised request body to assess sensitivity to this capture choice. The Vercel AI SDK driver supplies the source controls that its public interface can represent, avoiding the assumption that the driver can exercise controls unavailable through that interface (see Appendix A.3). Selecting the primary request findings. The same comparison framework is used for translated requests, reverse replies, gateway streams, and recorded agent sessions, with the relevant capture procedure for each setting. We first describe how potential changes are reduced to the primary request result; the other settings are evaluated separately. The main request census comprises 714 translation attempts across 51 source templates and 14 source-to-target paths. Of these, 29 are explicit rejections, leaving 685 completed 8

Table 2: Deriving the primary request result. The census contains 714 attempts across 51 source templates and 14 paths; 29 are rejected, leaving 685 completed translations. Counts are cumulative: each subsequent stage retains a subset of the preceding stage. Static target validity is checked offline; actual endpoint acceptance is evaluated separately in Section 5.3. The final stage requires a single finding to satisfy all primary criteria. Stage

Criterion

Retained cells

1. Compare representations 2. Filter acceptable changes 3. Apply reporting policy 4. Check static target validity 5. Select primary findings

At least one potential information-changing difference Exclude four known classes of permitted change

489 485

Retain cells with at least one T1 finding

467/685 (68.18 %)

Retain cells whose translated requests pass the static target contract Require an avoidable, silent change to prompt, role, or tool information

333/685 (48.61 %) 127/685 (18.54 %)

translations for comparison. The initial representation comparison identifies 489 cells with at least one potential information-changing difference. Four filters for known acceptable changes reduce this to 485. Applying the T1/T2 reporting policy leaves 467 cells with at least one T1 finding. We then assess target-format validity and the nature of each finding to obtain the narrower primary result (Table 2). The primary result deliberately applies stricter criteria than the presence of a T1 finding alone. A finding is avoidable when the target API documents a native way to represent the affected information. When target support is uncertain, the finding is treated as borderline rather than assumed to be avoidable. A cell contributes to the primary count only when its translated request (i) passes the static target contract, (ii) contains at least one silent T1 finding, and (iii) has a finding that is both avoidable and concerns prompt, role, or tool information. Here, silent means that the translator does not warn the caller about the change. A cell is counted once even if it contains multiple qualifying findings. Applying these criteria yields 127 of 685 completed translations (18.54 %; cluster-bootstrap 95 % CI: 15.01– 22.06 %). Thus, these translations produce requests that pass the static target contract while silently changing information that the target API could have preserved. This is the primary rate reported in the paper. Other findings, including changes to sampling and output-limit controls, metadata, and reasoning continuity, are analysed separately (Appendix A.3). Importantly, passing the static contract does not establish that a request is accepted by a live endpoint; endpoint acceptance is examined separately in Sec. 5.3. Grouping repeated findings. A finding is an observed rule violation in a particular translation cell. To distinguish repeated occurrences from distinct causes, we inspect the translator implementation and group findings with the same root cause, source-to-target path, and rule into a candidate. A mechanism groups the same underlying cause across different paths. This grouping allows us to report both how often a problem occurs in the test suite and how many distinct implementation mechanisms account for it. Hosted translations with inaccessible intermediate requests. For Google’s hosted compatibility endpoint, the internal translated request is inaccessible, so we cannot directly apply the payload-comparison procedure at the translation boundary. Instead, we use distinguishing conversations: targeted inputs designed so that the model’s response depends on a selected conversation field. We compare responses from the compatibility endpoint against native-API controls to investigate whether the field’s effect survives translation. This provides behavioural evidence about the hosted translation path, rather than a direct inspection of its internal payload. The deployment study also examines six open-source gateways (Sec. 6.5; Appendix A.3). 4.2

Held-out tests and additional validation

The checker should be evaluated independently of the examples used to develop its rules. We therefore separate early in-sample batches, which informed rule refinement, from held-out batches authored after the 9

Table 3: Detection accuracy of isloss and comparison with alternative judges. The primary held-out evaluation pools six specification-based blind batches authored after the checking rules were fixed; the extended evaluation adds four later blind batches (batch map: Appendix A.3). Intervals are Wilson 95 % CIs; J is recall minus false-alarm rate, with the strictly cross-dialect value in parentheses. Without a reference translation, isloss has the highest observed J among the tested judges. With a target-dialect reference, normalised diff and two Gemini judges score higher. Full results: Table 12 and Table 13. Check

Material

First held-out batch 36 faults, 34 activated (faults) First held-out batch 28 allowed changes, 28 (controls) activated Primary held-out 125 activated faults, 100 evaluation allowed changes Extended held-out 220 activated faults, 170 evaluation allowed changes New public issues 16 within scope, 14 encodable Seed recall 2,129 single-fault wires, (requests) 335 controls Public bugs, 4 57 reproduced bugs samples Judge, pooled pairs isloss Normalised diff flash-lite 3.8-flash flash-lite 3.8-flash

Setting no reference or model call; J 0.740 (0.729) with reference with reference; 633 ms with reference; 2,352 ms no reference; J 0.516 (0.447) no reference; J 0.704 (0.678)

Faults found

Allowed changes flagged

28/34 = 82.4% (66.5–91.7%)

—

—

1/28

95/125 = 76.0% (67.8–82.6%)

2/100 = 2.0% (0.6–7.0%)

160/220 = 72.7%

5/170 = 2.9%

10/14 = 71.4% (45.4–88.3%)

—

1,595/2,129 = 74.9% (73.0–76.7%) 45/57 (66.7–87.5%)

0/335

95/125

2/100

124/125 108/125 120/125 72/125

15/100 6/100 12/100 6/100

103/125

12/100

—

preservation rules were fixed. Here, “held-out” refers to the separation of test cases for evaluating isloss, not to model-training data. For the held-out evaluation, LLM agents construct test pairs from vendor specifications without access to the checker implementation. Each pair is designed to exercise either a planted information-loss fault or an allowed change of representation. A planted fault contributes to the recall denominator only if its edit actually changes the translated payload; we call such a fault activated. An allowed change contributes to the false-alarm denominator if isloss incorrectly flags it as a preservation violation (Appendices A.3 and C). In the primary held-out evaluation, comprising six blind batches (three fault batches and three control batches), isloss detects 95 of 125 activated faults (76.0 % recall; Wilson 95 % CI: 67.8–82.6 %) and flags 2 of 100 allowed changes (2.0 % false-alarm rate; 95 % CI: 0.6–7.0 %). Table 3 also reports the first fault and control batches separately. We further extend the evaluation with four later blind batches. Across the resulting extended held-out set, isloss detects 160 of 220 activated faults (72.7 %) and flags 5 of 170 allowed changes (2.9 %). Two intermediate script-generated batches are excluded before scoring; Appendix A.3 maps the internal batch identifiers and documents the inclusion criteria. A separate human-designed test kit yields 37 of 40 detected faults and no false alarms among 10 allowed changes. In addition, three program-mutant suites yield 81.6–84.3 % detection of activated mutants. For this analysis, a mutant is counted as detected only when the corresponding finding is absent from the unmodified translator. Finally, we evaluate coverage against publicly reported translation bugs, which tests whether the rules detect failures beyond deliberately planted changes. Across four samples, isloss detects 45 of 57 reproduced bugs (78.9 %; 95 % CI: 66.7–87.5 %). This includes 10 of 14 encodable bugs opened in the 50 hours after the preservation rules were fixed (71.4 %; 95 % CI: 45.4–88.3 %; Table 3; Appendix A.3). 10

4.3

Human validation and comparison with existing checks

How many detected changes are genuine defects? We assess the validity of the detected request-side candidates through independent human review. Reviewers classify each of the 54 candidates into three categories: Grade A, a confirmed defect; Grade B, an acceptable change of representation; or Grade C, a disputed case. Grades A and B require agreement between both raters; disagreements are assigned Grade C. Overall, reviewers confirm 31 of the 54 candidates as defects, accept 11 as legitimate representation changes, and disagree on the remaining 12. The distinction between avoidable and unavoidable losses is particularly informative. Among the 28 avoidable candidates, 23 are confirmed defects, one is accepted as a form change, and four remain disputed. Thus, 23/28 (82.1 %) are confirmed when all candidates are retained; among the 24 candidates with an agreed classification, 23/24 (95.8 %; 95 % CI: 79.8–99.3 %) are confirmed defects. By comparison, only 6 of 16 agreed cases involving unavoidable losses are confirmed as defects (37.5 %). This difference suggests that human judgments are more consistent when the target API could have preserved the information than when the loss arises from a target-format limitation. It supports treating avoidable information loss as the more conservative and reproducible basis for the primary findings. We further examine individual findings through two separate cell-level audits. Reviewers confirm 17/20 (85.0 %) and 67/70 (95.7 %) statically accepted avoidable findings, respectively, with no false alarms among 45 clean or decoy controls. A 48-item bridging sample, scored by all four raters, further exposes the sensitivity of human judgment: the two reviewer pairs differ by 37.50 percentage points on the same unavoidable-loss items. This disagreement reinforces the need to report target-limited degradations separately rather than treating every potential information loss as an equally clear-cut defect (Section 7; Appendix A.3). What does isloss provide beyond existing checks? Existing validation techniques address different aspects of translation correctness, but do not directly establish whether the information in a source conversation survives translation. Schema validation checks whether the translated request conforms to the target API’s structural requirements; it does not compare the request’s protected information with the source. Round-trip testing can expose changes by translating a payload back to its original dialect, but requires an inverse translator, which is unavailable for 10 of our 14 evaluated paths. Normalised diffs and LLM judges offer another comparison strategy when a reference translation is available. On our held-out evaluation pairs, a normalised diff and two Gemini judges achieve J = 0.842, 0.804, and 0.840, respectively, outperforming isloss at J = 0.740. However, these results use a reference translation against which the candidate translation can be compared. Obtaining such a reference is precisely the difficulty when validating previously unseen inputs or translators for which no trusted translation is available. We therefore also evaluate the alternatives in the more practical setting where only the source and its actual translation are available. Without a reference translation, the two Gemini judges achieve J = 0.516 and J = 0.704, both below isloss. The tested Claude and open-weight judges also score below isloss in this setting. The comparison highlights the central advantage of isloss: it provides a deterministic, rule-based check of specified conversation properties directly from the source and translated payloads, without requiring a reference translation or an additional model call (Appendix A.3).

5

Study Design

5.1

Translators, translation paths, and configurations

Our main request census examines 14 source-to-target translation paths across five open-source translator codebases, represented by six configurations: LiteLLM, Vercel AI SDK v5 and v7, LLM-Rosetta, @musistudio/llms, and claude-code-proxy. We exercise each path using the 51 source-valid request shapes described below, yielding 714 translation attempts. Twenty-nine attempts are explicitly rejected, leaving 685 completed translations for analysis. We conduct additional studies to examine translation behaviour beyond this main census. These include self-hosted Portkey, Bifrost, new-api, and one-api; gateway paths in ai-gateway and CLIProxyAPI; Google’s hosted OpenAI-compatible Gemini endpoint; and the official Python and Node OpenAI SDKs for evaluating 11

Table 4: Units and denominators for the main results. A cell is one request or reply tested on one translation path. Each row identifies the unit, denominator, subjects, and location of a result. The full map, including the corpus and supporting live or human evidence, is Table 15 (Appendix B.1). What we measure Translated requests Primary rate, full synthetic suite (Stratum 1) Same rate, shapes claude-sonnet-4-6 accepts live (Stratum 2a) Same rate, shapes claude-opus-4-8 accepts live (Stratum 2b) Range of the primary rate under other rules Reviewed candidates Live target acceptance after static checks Mid-turn system suite: meaning changed

Value

Unit

N

Tested on

127/685 (18.5 %)

cell

685

6 configs, 14 paths §6.1

34/182 (18.7 %)

cell

182

6 configs, 14 paths §6.1

23/112 (20.5 %)

cell

112

6 configs, 14 paths §6.1

12.9–19.6 % (validity rules); 16.5–25.7 % (counting conventions) 54 candidates (31 confirmed defects) across 18 mechanisms Gemini 160/204 (78.4 %); Anthropic 77/204 (37.7 %) 1,139/1,530 (74.4 %)

cell

308–685

4–6 configs

§6.1

(translator, rule) pair statically accepted cell cell

54

5 codebases

§6.1

204 per API 1,530

8 live paths

§5.3

6 configs

§A.1

10 tested

targeted probes; 30 or 10 trials per arm 100 items, 7 arms, one Gemini model 7 translators 7 translators

§6.2

Consequences, replies and persistence Mechanisms with a measured live 9 of 10 tested (18 catalogued) mechanism effect GSM8K control (separate experiment) Replies whose meaning changes Reply streams that fail in an official OpenAI SDK Persistence at the source snapshot (Step 2)

no statistically detectable accuracy drop 391/1,115 (35.1 %) 203/558 (36.4 %)

call

700

reply stream

1,115 558

2,630/2,745 (95.8 %)

previously flagged cell

2,745

Our upstream issues fixed

7 of 26

issue

26

Deployment and isloss validation Gateways change the request’s 187/204 (91.7 %) on 51 request meaning shapes; 978/1,020 (95.9 %) on 170 shapes Google’s OpenAI-compatible 10 of 11 distinguishing layer keeps the conversation probe (10 live repeats) Recorded coding-agent requests: 4/98 (4.1 %) contract-valid; cell primary rate 13/196 (6.6 %) all isloss detects planted faults 95/125 (76.0 %); 160/220 planted fault (blind) (72.7 %) pooled

204; 1,020 11 98; 196 125; 220

170 shapes; 10 translators, 22 paths —

Sec.

§6.2 §6.3 §6.3 §6.4 §6.4

Bifrost, new-api §6.5 (and CLIProxyAPI on 170 shapes) live Gemini, vs. §6.5 native API 6 configs, 14 paths, §6.5 3 CLI agents all 4 dialect pairs §4

translated reply streams. These additional systems are evaluated in their respective experiments and are not added to the 14-path main request census. To support reproducibility, we record pinned versions, commit identifiers, default configuration flags, and documented entry points for every tested system (Appendix B.1). 5.2

Request corpus and source acceptance

We construct the main synthetic request suite from Anthropic requests. We first generate 357 unique non-streaming request shapes using a four-way covering array (Cohen et al., 1997; Kuhn et al., 2004; Nie & Leung, 2011) over 14 feature dimensions. We then apply static source-contract checks and retain the 51 shapes that satisfy the documented source requirements. These 51 shapes form the main request suite used 12

in the 14-path census. The covering-array construction describes the generated set before filtering; we do not assume that the source-valid subset retains the full four-way coverage guarantee. Static source validity does not imply that a live model endpoint accepts a request. When tested against claude-sonnet-4-6, only 13 of the 51 shapes (25.5 %) are accepted, while 38 (74.5 %) are rejected. These rejections include placeholder redacted thinking blocks and requests that specify both temperature and top_p (Anthropic, 2026e; PsycherosAI, 2026). The separate claude-opus-4-8 acceptance test accepts eight of the 51 shapes (Appendix B.1). We retain the full statically valid suite for the main translation census, and separately report results restricted to shapes accepted by live source endpoints. Two additional datasets address settings not represented by the main corpus. First, the mid-conversation system instruction experiment uses a separate 170-shape suite and behavioural study. Its requests are not included in the main 51-shape suite. Second, to examine naturally occurring agent traffic, we capture 36 requests (28 unique) from six CLI agents, including Claude Code, Crush, and Aider. These requests yield 14 distinct Anthropic shapes, of which seven pass the static source checks. Sec. 6.5 reports results on the seven source-valid shapes (98 translation cells) and on all 14 observed shapes (196 cells). The full feature coverage and dataset construction are provided in Appendix B.1. 5.3

Static validity and live endpoint acceptance

We distinguish three properties that are easy to conflate. Static validity means that a translated request passes our offline checks of the target API’s structure and documented constraints. Live acceptance means that the corresponding vendor endpoint accepts the request and returns HTTP 200. Silence describes whether the translator warns the caller about a detected change. Warning behaviour is assessed independently of both static validity and endpoint acceptance. The 27 target-contract validators retain 333 of the 685 completed translations (48.6 %) at Step 4 of the primary counting procedure. Step 5 then applies the finding-level criteria defined in Sec. 4.1; static validity alone does not make a translation a primary finding. We further test live endpoint acceptance for statically accepted requests. Among 204 cells tested against each target, Gemini accepts 160 (78.4 %) and rejects 44 because of missing Gemini 3 thought_signature values. Anthropic accepts 77 of 204 cells (37.7 %; see Appendix B.1). Because source and target endpoints accept different subsets, we report several acceptance-based result strata alongside the full synthetic-suite result (see Fig 5; Table 18 in the Appendix). These strata answer different questions and should not be interpreted as successive filters of one common population: • Stratum 1: full synthetic suite. All 685 completed translations are included, yielding 127 primary findings (18.5 %). This is the main result and does not require live source acceptance. • Strata 2a and 2b: source-accepted subsets. Restricting the source requests to those accepted by claude-sonnet-4-6 gives 13 shapes and 182 translation cells (Stratum 2a). Using the eight shapes accepted by claude-opus-4-8 gives 112 cells (Stratum 2b). These are separate subsets determined by the respective source models, not successive filters. • Stratum 2c: acceptance at both ends. Requiring acceptance by both the source and target endpoints leaves 88 cells across the Gemini- and Anthropic-target paths. OpenAI-bound cells are not included because they were not submitted to a live target endpoint. • Stratum 3: demonstrated mechanism coverage. We identify 240 of the 685 main-suite cells as carrying a mechanism for which a live behavioural effect has been demonstrated. This is a mechanism-level coverage analysis: it does not mean that all 240 translations were individually replayed against a live model. These strata provide complementary views of the findings under different acceptance and evidence conditions. Table 4 and Appendix B.1 specify the population, unit of analysis, and denominator for each result, including deduplicated counts. 13

Mechanism

Translated requests affected (of 685)

1. Prompt, role or tool fields — red segments meet the primary criteria tool-result error flag dropped 54 text after a tool call moved before it parallel-tool-use setting dropped 41 function-response id dropped 37 16 forced tool use weakened result names matched by position 13 trailing user text dropped 7 list tool result split 6

Behaviour changes in targeted tests?

154

2. Tool-name changes beyond target limits — affected requests fail static target checks long tool name shortened 26 10 forced tool name shortened 3. Generation controls, metadata and reasoning blocks — outside the primary rate redacted thinking block dropped 261 user_id metadata dropped 92 top_k sampling setting dropped 86 max_tokens limit dropped 51 top_p sampling setting dropped 39 stop sequences dropped 34 temperature setting dropped 32 default temperature injected 9 counts toward the primary rate outside primary criteria (static validity, target support or field category) no effect detected (blank = not tested) Live tests: changes model behaviour

Figure 4: Request changes grouped by affected information and target support. Bars show affected cells out of 685; a cell may carry several mechanisms. Red marks the changes included in the primary rate. Sampling and output-limit controls and other categories outside the primary rate may also affect behaviour (Sec. 6.2). Libraries and target APIs are in Fig. 8 in the Appendix, code classes and upstream status in Table 16 in the Appendix. 5.4

Units of analysis and statistical methods

The experiments involve different units of analysis, including generated request shapes, translation attempts, completed translation cells, grouped candidates, underlying mechanisms, reply streams, and recorded agent requests. These units are not interchangeable. Table 4 maps each main result in Sec. 6 to its unit and denominator; the full accounting is provided in Table 15 in the Appendix. We report 95 % confidence intervals using either a bootstrap clustered by the 51 source request shapes (10,000 resamples) or Wilson intervals (Wilson, 1927), as specified for each result. For live comparisons, we use Fisher’s exact tests (Fisher, 1922) or sign-flip permutation tests, with Holm correction for multiple comparisons (Holm, 1979). For transparency, each statistical criterion is labelled according to whether it was pre-specified in an internal commit, fixed before the corresponding run, or determined post hoc.

6

Empirical Results

We evaluate translation failures from five complementary perspectives. First, we characterise the information changes detected in translated requests, group them by their underlying implementation mechanisms, and assess their frequency in our test suites (Sec. 6.1). Second, we investigate whether selected changes affect model behaviour in targeted live-model experiments (Sec. 6.2). Third, we examine information loss in translated replies and its consequences for official client SDKs (Sec. 6.3). Finally, we investigate whether failures persist across software releases and API endpoints, and extend the evaluation to gateways, a first-party compatibility endpoint, and recorded coding-agent requests (Secs. 6.4–6.5). Table 4 specifies the unit and denominator for each result; Appendix B provides the full results. 6.1

Request translation failures: mechanisms and frequency

Failure mechanisms and human assessment. Source inspection groups the request-side findings into 54 candidates arising from 18 underlying mechanisms. Of these candidates, 28 involve information that 14

(a) From translations to primary findings

(b) Does the primary rate hold?

one cell = one source request on one translation path

primary rate on source-accepted inputs and under other rules

Completed translations

685

. . . undeclared changes (T1)

467

. . . and passes static target checks

0

18.5 %

13 shapes claude-sonnet-4-6 accepts live

333

. . . and has an avoidable finding . . . one finding meets all primary criteria (avoidable + prompt, role or tool field)

All 51 synthetic shapes

8 shapes claude-opus-4-8 accepts live

206

Other validity rules

127

Other counting conventions

300

18.7 % 20.5 % 12.9–19.6 % 16.5–25.7 %

685

0

Translated requests

15

30 %

Primary rate (% of requests)

The final step aligns all criteria within the same finding; separate findings in one cell do not suffice.

Figure 5: Primary findings in the synthetic suite and endpoint-accepted subsets. Panel (a) selects 127 of 685 judged cells (18.5 %); panel (b) compares synthetic-suite rates under endpoint acceptance and alternative validity and counting rules. Whiskers show shape cluster-bootstrap 95 % CIs (Tables 17 and 18 in the Appendix). Table 5: Meaning changes in the main request census. All 51 source shapes pass static checks; 13 are accepted live by claude-sonnet-4-6. T1/judged counts cells with undeclared changes before target-validity filtering; Static counts those passing offline target checks. Cand. gives pre-review candidate counts; the reviewed catalogue is reported in Section 6.1. For LiteLLM->OpenAI, the pre-serialisation dictionary has 37 T1 cells that fail the target checks: all 37 retain thinking_blocks, and 12 also retain top_k. At the serialised hosted_vllm body, which strips thinking_blocks, 25/37 pass. →OpenAI Translator LiteLLM claude-code-proxy LLM-Rosetta @musistudio/llms AI SDK v5 AI SDK v7

→Gemini

→Anthropic

T1/judged Static T1/judged Static T1/judged Static Cand. (pre) 37/51 21/22 42/51 44/51 39/51 39/51

– – 36/51 0/51 18/51 18/51

– – 36 0 18 18

4 7 10 20 7 8

14 library paths (pre-review) 222/277 88 173/204 173 72/204 after review: one acceptable form change removed, one duplicate pair merged

72

56 54

Bifrost (gateway) new-api (gateway)

– –

– –

44/51 46/51

0 7 22 17 21 21

– – 42/51 51/51 40/51 40/51

4 2

48/51 49/51

– – 42 51 40 40

48 49

– –

Static counts T1 cells whose translated bodies pass offline target checks. It does not mean live acceptance; silence means no translator warning. Candidate and T1 counts above are pre-review.

the target API can represent and that the translator could therefore have preserved. Fig. 4 organises the mechanisms by the information they affect and whether the target format supports it. A mechanism can affect multiple request shapes and translation paths; consequently, its affected-cell count measures repeated occurrences in the test suite, not the number of distinct defects or implementation causes. Human review confirms 31 of the 54 candidates as defects. Among the 28 avoidable candidates, 23 are confirmed defects, one is accepted as a form change, and four remain disputed. Frequency in the request suites. Across the 51-shape main suite, 127 of 685 completed translations contain at least one primary finding (18.5 %; cluster-bootstrap 95 % CI: 15.01–22.06 %; Fig. 5 and Tab 5). This is a rate within our constructed test suite, not an estimate of the prevalence of translation failures in production traffic. The result is similar when we restrict the analysis to source requests accepted by live Anthropic models. 15

intact request

after translation

not run

× template rejects Gemini

3.5-flash-lite Prompt, role and tool fields

top_p: effect described in words Claude

3.8-flash

sonnet-4-6

haiku-5-5

Open-weight Llama-3.1-8B

Qwen2.5-7B

×

×

intact 30/30 → 0/30

tool-result error flag dropped text after a tool call moved function-response id dropped forced tool use weakened result names by position trailing user text dropped disabled-tool notice dropped (Claude: inlined user text) Generation controls (outside the primary rate) max_tokens dropped or raised answers more varied

top_p dropped

no longer rejected

stop sequences dropped 0

100 %

0

0 0 100 % 100 % 0 100 % 100 % Share of trials with the required behaviour (%)

0

100 %

Figure 6: Behavioural outcomes in targeted live probes. Bars compare intact (grey) and translated (red) requests. Dashes denote untested combinations; only three displayed mechanisms were tested on Claude. The moved-text probe shows no difference on Llama-3.1-8B, and the dropped-prohibition probe misses its pre-specified threshold on claude-sonnet-4-6 but meets it on claude-haiku-5-5. The top_p effects are stated in words. The additional top_k probes show no difference in the tested settings (counts and Wilson 95 % CIs in Tables 19–20) in the Appendix. fails:

silent corruption

exception

not counted:

target capability loss only

Python SDK

clean

Gateways

one-api Portkey new-api CLIProxyAPI Bifrost

Node SDK fails in at least one SDK 72.5 % 30.0 % 25.0 % 21.8 % 21.2 %

Libraries

LLM-Rosetta LiteLLM

55.0 % 28.8 %

All seven

36.4 % (203/558) 0%

50%

100%

0%

50%

100%

Figure 7: Official SDK outcomes for 558 translated reply streams. Replies are translated back to OpenAI format and read by the official Python and Node SDKs. Bars distinguish silent corruption (red), exceptions (dark grey), and streams with neither outcome. The right column gives the share affected in at least one SDK. Among the 182 translation cells corresponding to the 13 shapes accepted by claude-sonnet-4-6, 34 contain primary findings (18.7 %; 95 % CI: 11.54–25.82 %). For the eight shapes accepted by claude-opus-4-8, the corresponding result is 23/112 (20.5 %). Requiring live acceptance at both source and target endpoints leaves 22/88 primary findings (25.0 %) on the Gemini- and Anthropic-target paths. Sensitivity to counting and capture choices. We examine whether the main result depends on the validity criteria, reporting policy, or capture point. Alternative validity and tier-boundary rules yield rates between 12.9 % and 19.6 % (49/379 to 134/685), while alternative counting conventions yield rates between 16

16.5 % and 25.7 % (88/532 to 176/685; see Table 17 in the Appendix). These variants can change both the included cells and the findings counted within them, so their denominators are reported separately. Two alternatives illustrate the sensitivity of the headline result. Including target-representable T2 degradations, including system-to-user role demotion, increases the count to 134/685 (19.6 %). Changing LiteLLM’s capture point from its pre-serialisation dictionary to the serialised HTTP body yields 132/685 (19.3 %), compared with the baseline 127/685 (18.5 %). The main suite does not directly cover mid-conversation system instructions, signed thinking blocks, or media inputs. We investigate these additional features through separate probes and recorded agent requests in Secs. 6.2 and 6.5. 6.2

Behavioural consequences of translation changes

Detecting information loss establishes that a translation changes the conversation representation, but does not by itself establish a behavioural consequence. We therefore test selected mechanisms using targeted live-model experiments. Of the 18 catalogued mechanisms, ten are evaluated across six models: two Gemini models, two Claude models (claude-sonnet-4-6 and claude-haiku-5-5), and two open-weight models. Nine of the ten mechanisms change at least one measured outcome in their corresponding probes. The nine mechanisms with measured effects occur in 116 of the 127 primary request cells (91.3 % mechanism coverage). This is a mechanism-level correspondence: the live experiments test targeted probes for these mechanisms, not every one of the 116 individual translation cells. Four mechanisms produce a particularly pronounced effect, reducing compliance from 30/30 to 0/30 on both tested Gemini models. Tool and conversation changes on Gemini and open-weight models. Eight mechanisms significantly reduce compliance in the targeted Gemini or open-weight-model experiments. Across ten scenarios on gemini-3.5-flash-lite and gemini-3.8-flash, six mechanisms each change the outcome in nine or ten scenarios (exact sign-flip permutation test, Holm-adjusted p = 0.0117). A separate experiment on the same two models examines the loss of tool-response identifiers (functionResponse.id). Without these identifiers, parallel tool-call attribution falls from 30/30 to 0/30 (pHolm = 0.0039). Moving assistant text from after a tool call to before it also reduces compliance on Qwen2.5-7B, whereas the corresponding Llama-3.1-8B probe shows no difference. To check that the results are not artefacts of replaying archived payloads, we rerun the translator processes themselves. All nine checked cells reproduce the same outcomes (see Appendix B.3). Effects of changed Claude generation controls. We also investigate changes to sampling settings and token limits, which are outside the primary request-failure rate but can still alter model behaviour. On claude-sonnet-4-6, dropping top_p: 0.05 increases the mean number of distinct outputs across five samples per scenario from 3.8 to 4.8 over ten scenarios (p = 0.0156). On claude-haiku-5-5, the same omission changes all ten calls that were rejected because top_p is deprecated into successful HTTP 200 responses. We further test a translator-induced increase in the token budget. Raising max_tokens from 100 to 1,124, thereby reproducing the added thinking budget, reduces compliance with the caller’s original 100-token limit from 30/30 to 0/30 on both Claude models. By contrast, the tested removal of thinking blocks does not produce the predicted behavioural effect (see Appendix B.3). Additional instruction probe and task-level control. We conduct an Anthropic-target counterpart to the mid-conversation instruction experiment in Section 2.1. Removing the inlined notice that disables further tool use reduces refusals from 30/30 to 0/30 on claude-haiku-5-5. On claude-sonnet-4-6, refusals fall from 10/30 to 0/30, which does not meet the pre-specified prediction. As a separate task-level control, we evaluate a 100-item GSM8K experiment on gemini-3.5-flash-lite. The translated requests produce no statistically detectable reduction in accuracy (p ≥ 0.2500; Appendix B.3). This negative result is important: the observed effects depend on the information affected and the task being tested; not every translation change necessarily produces a measurable degradation on every downstream task. 6.3

Information loss in translated replies and SDK failures

Information loss also occurs in the reverse direction, when translators convert model replies back to the caller’s API format. Across 1,115 judged reply cells from seven translators, isloss identifies 391 meaning changes (35.1 %), of which 278 are avoidable. 17

We then examine whether translated reply streams remain usable by official client libraries. In a separate experiment, 203 of 558 translated streams (36.4 %) either raise an uncaught exception or silently corrupt the assembled message in at least one official OpenAI SDK. Of these 203 problematic outcomes, 199 (98.0 %) are avoidable (Fig. 7). The Python SDK exhibits exceptions or corruption on 170/558 streams (30.5 %), while the Node SDK does so on 182/558 (32.6 %). These per-SDK counts overlap because the same streams are tested against both SDKs; they are not additive components of the 203-stream union. Per-SDK results are provided in Appendix B.4. 6.4

Persistence across releases, API endpoints, and upstream fixes

Persistence across software releases. To assess whether the observed changes are transient implementation bugs or persistent behaviours, we examine successive releases and later source snapshots. Across 50 consecutive LiteLLM releases, every release changes the meaning of at least 31 of the 51 main request shapes when translating to OpenAI. In a broader census of ten translators across 170 shapes, 2,630 of the 2,745 cells flagged in the pinned releases remain flagged in the 11 October source snapshots (95.8 %, Step 2). This indicates that most of the detected changes persist in the later snapshots examined. Library- and gateway-specific results at each filtering stage are provided in Appendix B.5. Differences between API endpoints and upstream repairs. A translator’s behaviour can also depend on which target endpoint it uses. In particular, the same translator may preserve different information when targeting OpenAI Chat Completions rather than Responses. Across 357 inputs evaluated on five translators that support both endpoints, their outgoing requests differ on at least one of 22 pre-specified properties for every translator. Same-vendor round-trip controls also reveal request changes in every tested translator. We also report the upstream response to the issues identified during the study. By 15 October, we had opened 26 issues across 12 repositories, commented ten times on other contributors’ threads, and made eight follow-up comments on our own reports. Of the 26 reports, seven were fixed upstream and independently re-verified, one was partly fixed, one was closed without a reply, and 17 remained open at the time of assessment (see Appendix B.5). 6.5

Gateways, first-party compatibility, and coding-agent traffic

Gateway request and reply translation. We extend the evaluation to gateway-mediated translation, where additional routing and compatibility layers can introduce information changes or reject requests explicitly. We audit six gateways and execute four of them, confirming most patterns identified in the audit while also finding additional candidates. On the 51-shape suite, Bifrost and new-api produce T1 changes in 187 of 204 judged cells (91.7 %). Of these, 103 pass the static target checks and 84 fail them. Portkey, by contrast, explicitly refuses every tested Anthropic-format request directed to OpenAI or Gemini. Across the separate 170-shape corpus, the three gateways that return responses, Bifrost, new-api, and CLIProxyAPI, produce meaning changes in 978 of 1,020 judged cells (95.9 %). We also test translated reply streams through four gateways: Portkey, Bifrost, new-api, and CLIProxyAPI. Among 198 judged streams, 57 (28.79 %) contain avoidable meaning changes (Appendix B.6). Fallback behaviour and a first-party compatibility endpoint. Compatibility layers can also change provider selection, not just payload structure. Among five surveyed gateways, Bifrost by default routes failed Anthropic-model requests to an OpenAI-compatible fallback. It returns the result in Anthropic format, while the change of provider is visible only in the response headers. We separately examine Google’s hosted OpenAI-compatible Gemini endpoint. Using two Gemini models, ten repetitions per experimental arm, and native-API controls, the endpoint passes 10 of 11 behavioural probes. The failed probe shows that it retains only the last of two top-level system messages. Additional tests reveal several other differences from the OpenAI interface: it enforces Gemini’s 128-character tool-name limit rather than OpenAI’s 64-character limit; charges unnamed reasoning tokens against max_tokens (98.0 % of output tokens across 45 calls); requires thought signatures for tool round-trips (5/5 requests without them are rejected); and emits "stop" rather than "tool_calls" in streaming replies (see Appendix B.6). 18

Recorded coding-agent requests. Finally, we examine whether the failure patterns identified using synthetic inputs also occur in recorded coding-agent traffic. Replaying the captured requests reveals the same classes of silent translation changes observed in the controlled experiments. For the seven recorded shapes that pass static source checks, replay across all 14 translation paths yields four primary findings among 98 cells (4.1 %). A broader count that includes all avoidable meaning changes identifies 23/98 cells (23.5 %). When all 14 recorded shapes are included, the primary count is 13/196 cells (6.6 %). These rates describe the replayed agent requests, not the prevalence of failures across all coding-agent traffic. Offline replays of requests captured from Claude Code, Codex, and gemini-cli show translators folding, wrapping, or deleting mid-conversation system messages, as well as stripping system prompts or safety filters. In multi-turn replays involving Claude Code and Crush, translators also drop failed tool calls’ is_error flags on OpenAI-bound paths. Finally, across the closed-loop CLI experiments, nine of 12 sessions contain at least one meaning change (see Appendix B.6).

7

Discussion and Threats to Validity

7.1

Implications for translators and model evaluation

Our findings have two practical implications. First, translators should treat conversation preservation as an explicit correctness requirement, rather than assuming that producing a structurally valid target request is sufficient. They should document their cross-API mappings and preserve information whenever the target API provides a native way to represent it. When the target lacks a corresponding representation, the translator should reject the request or explicitly warn the caller about the resulting degradation. It should not silently substitute a different conversation. The extended instruction experiment illustrates this distinction: AI SDK v5 and LLM-Rosetta preserve the mid-conversation system instruction, whereas other translators delete or demote it. These requirements should also guide translator testing. Regression tests should compare each translated payload with its source at the level of conversation information, rather than checking only the target schema or whether the request receives a successful response. The seven upstream fixes that we independently re-verified among our 26 issue reports demonstrate that the identified problems can lead to concrete improvements in translator implementations. Second, translation is a potential confounding factor in model evaluation. When a benchmark or agent framework routes requests through a shared translator, the model may receive a prompt, tool specification, or generation configuration that differs from what the benchmark intended. Changes to system-instruction authority, tool schemas, and error flags can therefore affect the conditions under which models are evaluated (Wallace et al., 2024; Qin et al., 2024; Liu et al., 2026). This concern is relevant to benchmark pipelines such as τ -bench (Yao et al., 2024) and BFCL (Patil et al., 2025) whenever they rely on LiteLLM, the Vercel AI SDK, or other translation layers. Checking captured payloads with isloss can help researchers determine whether a discrepancy originates in the translator or in the model’s response to the intended input. In this way, conversation-preservation checks can improve the interpretability and reproducibility of model comparisons. Our captured translations and field-dependent behavioural experiments demonstrate that translation-induced changes can alter model behaviour; the GSM8K control also shows that such effects are task-dependent. However, we have not measured the resulting changes in published benchmark scores or leaderboard rankings. Establishing their impact on end-to-end benchmark comparisons remains an important direction for future work. 7.2

Threats to validity

Representation and rule interpretation. Our isloss checks only the conversation properties represented in ConvIR and covered by its preservation rules. It therefore cannot detect information changes outside this representation or establish complete semantic equivalence between source and translated requests. Moreover, translating vendor documentation into preservation rules and deciding whether a loss is avoidable involve author judgement. To make these decisions auditable, each rule is linked to its supporting specification, and allowed changes are included as negative controls. We further examine the sensitivity of the reported results 19

to validity criteria and counting policies, including the treatment of system-to-user role demotion. Detailed rule definitions, review decisions, and audit records are provided in Appendix C. Capture and measurement. Our findings are specific to the payloads observable at the selected capture points. Changes introduced after the last instrumented translation boundary, including provider-side rewriting, are not necessarily visible to isloss. The LiteLLM capture-point sensitivity analysis provides evidence about the effect of one such measurement choice, but does not establish that all possible capture points are equivalent. Behavioural measurements also have potential sources of variance. Temperature-zero model calls are not guaranteed to produce identical outputs across runs, and single-run LLM-judge baselines may be sensitive to decoding variability. We report the corresponding experimental details and limitations in Appendix C. Behavioural evidence. The live-model experiments cover selected field-dependent mechanisms and tasks rather than the full range of model behaviour. Their results establish that particular translation changes can affect particular measured outcomes under the tested conditions; they do not imply that every detected information loss will change model behaviour. The GSM8K control, the model-dependent effects of role demotion, and the unsuccessful prediction for the tested thinking-block removal illustrate this limitation. Furthermore, the 277/685 OpenAI-bound cells were not tested against live OpenAI endpoints, and GPT models were not included in the live-model experiments. Consequently, our behavioural conclusions should be interpreted as evidence from the tested models and mechanisms, not as a comprehensive assessment across providers or model families. System coverage and generalisation. The study covers the evaluated self-hosted translators and six live models: two Gemini models, two Claude models (claude-sonnet-4-6 and claude-haiku-5-5), and two open-weight models. We did not inspect commercial hosted gateways or the compatibility layers operated by Anthropic and OpenAI. Google’s OpenAI-compatible Gemini endpoint is the only first-party compatibility endpoint for which we conducted external behavioural probes. The synthetic-suite rates measure findings across deliberately constructed feature combinations, not their incidence in production traffic. Our recordedagent corpus contains 14 distinct request shapes, of which only seven pass the static source checks. Its lower primary rate in Sec. 6.5 therefore provides evidence from a limited set of observed traffic, not a calibration of production-wide prevalence. Similarly, the gateway and live-model results cover specific implementations, configurations, endpoints, and experimental conditions. Broader deployment studies will be needed to establish how frequently these failures occur across the wider LLM ecosystem. Temporal validity and practical deployment. Translator implementations and vendor API specifications evolve. Our results reflect the software versions, configurations, and specification snapshots examined during the study; subsequent changes may alter both the mappings and the validity of individual findings. In addition, the same-vendor and fallback experiments use local mocks, which may not reproduce every aspect of production routing and endpoint behaviour. Finally, isloss is evaluated here as an offline research checker. Integrating it into command-line workflows, continuous integration, and release monitoring, and evaluating its effectiveness in those settings, remain future work.

8

Related Work

Translation validation and cross-format equivalence. Translation validation checks an individual translation against its source, rather than attempting to prove a translator correct for every possible input (Pnueli et al., 1998; Necula, 2000). Our isloss adopts this source-to-target perspective, but addresses a different setting from compiler validators such as Alive2 (Lopes et al., 2021). Compiler validation can rely on formal semantics, whereas LLM API specifications are often incomplete or ambiguous about how conversation constructs map across providers. Accordingly, isloss uses specification-grounded preservation rules to check explicitly represented conversation properties, rather than claiming complete semantic equivalence. Differential testing and equivalence modulo inputs expose inconsistencies by comparing the behaviour of multiple implementations or configurations (McKeeman, 1998; Yang et al., 2011; Le et al., 2014). Related techniques have been applied to SQL dialects (Jiang et al., 2025b; Lin et al., 2025), database connectors (Lyu et al., 2025), JSON parsers (Möller et al., 2024), and HTTP intermediaries (Jabiyev et al., 2021; Shen et al., 2022; Kallus et al., 2024; Jabiyev et al., 2022). For LLM APIs, comparing different translators can reveal 20

disagreement, but disagreement alone does not establish which translation preserves the caller’s intended conversation. The closest related approach is LLM-Rosetta, which uses a shared intermediate representation to translate between API formats and checks round-trip equality (Ding, 2026). This provides a useful basis for studying format compatibility, but round-trip equality is only informative about the properties retained by the intermediate representation. Information omitted from that representation, such as error flags, reasoningcontinuity fields, or explicit generation controls, may therefore escape the comparison. In contrast, isloss uses its intermediate representation specifically to check preservation against the original payload; it does not generate the target request or require an inverse translation. It also records whether the target API has a native way to represent the affected information. This distinction between preserved information and information that cannot be represented is also related to bidirectional transformations, where reconstructing information omitted during a transformation requires additional information (Foster et al., 2007). Testing software around language models. A growing body of work tests the software infrastructure through which language models interact with tools and external services. LogicHunter tests agent frameworks through their object APIs using an LLM oracle (Long et al., 2026), while bug studies of agent and MCP frameworks examine failures in model-facing components and deficiencies in their tests (Zhang et al., 2026a; He et al., 2026). These approaches investigate the correctness of agent frameworks and their interactions with tools. isloss addresses a different boundary: whether the information in a conversation survives translation before it reaches the model or is returned to the caller. Related studies examine other aspects of gateway and serving infrastructure, including billing (Lin et al., 2026), model identity and shadow APIs (Zhang et al., 2026c;b), routing (Fu et al., 2026), failures hidden behind HTTP 200 responses (Pandey & Singh, 2026b), failover losses (Pandey & Singh, 2026a), and endpoint drift (Chen et al., 2023). Our focus is whether translation preserves the caller’s conversation information, and whether detected changes persist across releases (Section 6.4). This complements work on tool-call execution after model inference (Yang et al., 2026; Kim & Kang, 2026; Tang & Zheng, 2026) and on serving-engine caches and decoding variance (Kwon et al., 2023; Atıl et al., 2025). These studies examine different stages of the path from an application’s request through model inference to its eventual action; isloss specifically checks the translation boundary within that path. API specifications and test generation. API specifications provide an important basis for deciding whether a translated request is valid, but they do not always define how constructs from different APIs should correspond. RFCScope studies ambiguities in protocol specifications (Pawagi et al., 2025), while SilentProbe identifies prose-level constraints that are absent from OpenAPI schemas (Li et al., 2026). Our specification analysis addresses cross-API mappings: we identify 17 constructs for which vendor documentation leaves the mapping between APIs unspecified or contradictory. These gaps motivate a preservation analysis that distinguishes documented target limitations from avoidable losses, rather than treating every difference between payloads as a violation. API test-generation techniques provide complementary tools for exploring combinations of request features. RESTler and RESTest generate tests while accounting for dependencies between parameters (Atlidakis et al., 2019; Martin-Lopez et al., 2019; 2021). Constrained covering arrays similarly address the challenge of achieving interaction coverage while respecting input-validity constraints (Cohen et al., 2008). We use a four-way covering array to generate an initial set of 357 unique requests and then apply static source-contract checks to obtain the 51-request main suite. Importantly, filtering the generated array does not establish that the retained subset preserves the original four-way coverage guarantee. Finally, API-evolution research examines changes that require clients to migrate between library versions (Lamothe et al., 2021). API translators address a related compatibility problem, but perform mappings between formats on each call, making preservation of individual requests and replies an additional correctness concern. Behavioural probes and sensitivity to input changes. Network measurement techniques detect changes introduced by intermediaries by sending known inputs and observing what reaches the destination (Kreibich et al., 2010; Weaver et al., 2014; Vallina-Rodriguez et al., 2015). Our distinguishing conversations adapt this idea to hosted LLM compatibility endpoints: when the internal translated request is inaccessible, we compare model responses to field-dependent inputs against native-API controls. These probes provide behavioural 21

evidence about selected translation effects; they are distinct from the deterministic preservation checks applied to captured payloads. Prior work suggests that several types of conversation information can influence model behaviour. Instructionhierarchy training distinguishes system instructions from user text and tool outputs (Wallace et al., 2024; Wu et al., 2025), including under indirect prompt injection (Greshake et al., 2023). Changes to chat-template role markers can also affect safety behaviour (Jiang et al., 2025a). Other studies investigate sensitivity to system-prompt phrasing (Qin et al., 2024), the position of information in long contexts (Liu et al., 2024), multi-turn splitting (Laban et al., 2026), tool schemas (Liu et al., 2026), and structured-output schemas (Geng et al., 2025). Aggregate evaluation scores can further conceal regressions on individual items (Xu & Wu, 2026). Building on these observations, our live experiments test whether information changes actually introduced by translators affect tasks that depend on the corresponding fields. Rather than assuming that every detected change harms performance, we measure its consequences under targeted conditions. Evaluation validity and automated judges. The reliability of model evaluations can depend on prompt formatting, prompt templates, answer order, and evaluation-harness conventions (Sclar et al., 2024; Mizrahi et al., 2024; Alzahrani et al., 2024; Liang et al., 2022; Biderman et al., 2024). We study a related but distinct source of variation: changes introduced between the caller’s request and the payload delivered to the model. This is particularly relevant to agent and tool-use benchmarks such as Gorilla, τ -bench, and BFCL, which use custom handlers or translation libraries (Patil et al., 2024; Yao et al., 2024; Patil et al., 2025). Checking the translated payload can help distinguish a model’s response to the intended input from its response to an altered input. LLM-as-a-judge research primarily investigates the assessment of open-ended model outputs (Zheng et al., 2023; Gu et al., 2026). Our comparison instead considers the narrower task of detecting information changes between a source request and its translation. We evaluate prompted judges both with and without a reference translation, since access to a trusted reference materially changes the comparison problem. We further assess isloss using planted faults, allowed-change controls, program mutants, and human review, following established work on test oracles and their evaluation (Barr et al., 2014; Jahangirova et al., 2016). Development and repair data are kept separate from held-out evaluation, consistent with the importance of independent test fixtures (Xu & Tao, 2026).

9

Conclusion

Successful API translation does not necessarily preserve the conversation that an application intended to send. A translated request can satisfy the target API’s schema while silently losing or altering information that affects how a model interprets instructions, uses tools, or generates a reply. We introduced isloss, a deterministic test oracle that makes conversation-information preservation checkable by comparing source and translated payloads through a shared representation and explicit preservation rules, without requiring model calls or reference translations. Our empirical study demonstrates that this problem occurs across existing translators and affects both requests and replies. In the main synthetic request suite, 127 of 685 completed translations (18.5 %) pass static target checks while silently introducing at least one avoidable change to prompt, role, or tool information. Human review confirms 31 genuine defects among 54 request-side candidates. Separate experiments identify 203 of 558 translated reply streams (36.4 %) that cause exceptions or silent corruption in at least one official OpenAI SDK. Targeted live-model experiments further show that nine of ten tested translation mechanisms alter at least one measured behavioural outcome. These findings establish conversation preservation as a distinct correctness requirement for LLM interoperability. Translators should preserve information whenever the target API can represent it and explicitly reject or disclose unavoidable degradation. Likewise, model and agent evaluations should verify the payloads actually delivered to models, rather than relying solely on schema validation or successful API responses. Future work will extend coverage across providers and streaming behaviours, integrate isloss into command-line and continuous-integration regression workflows, and investigate its impact on end-to-end agent and benchmark evaluations. 22

Broader Impact Statement LLM translators and gateways mediate the boundary between applications and model providers. Consequently, silent changes at this layer can affect not only conversation meaning but also safety controls, credential handling, provider selection, and usage accounting. Our deployment experiments identify several security- and reliability-relevant behaviours: a proxy disables Gemini safety filters without logging the change; a gateway routes requests to another provider after upstream errors while returning HTTP 200; authentication headers are forwarded during cross-provider fallback; and mid-stream errors are closed with HTTP 200 and billed as completed requests. These findings motivate more explicit translation policies, preservation checks, and transparent reporting of fallback and error-handling behaviour. The purpose of isloss is to help developers and evaluators identify such problems before they silently affect model interactions or application behaviour. All experiments were conducted using local mocks or accounts controlled by the authors. Safety-sensitive findings were disclosed privately before public issue reports were submitted. As of 15 October 2026, maintainers had fixed seven of the 26 public issues we reported, and we had independently re-verified those fixes. No separate remediation through private disclosure channels had been received by that date (see Sec. 6.4).

LLM Usage The authors used ChatGPT as an assistive tool for software coding and debugging, grammar checking and language polishing, and preparing and refining LATEX code for figures and tables. All research ideas, problem formulation, methodological contributions, experimental design, execution and evaluation of experiments, analysis and interpretation of results, and verification of findings were developed and carried out by the authors. The authors reviewed and validated all AI-assisted outputs and take full responsibility for the content and integrity of the manuscript.

References agentgateway. AWS Bedrock(Anthropic): mid-conversation system messages break Anthropic model caching. agentgateway issue #3758, https://github.com/agentgateway/agentgateway/issues/3758, 2026. Opened and closed 2026-10-02. Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, et al. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13787–13805, 2024. Anthropic. Handle tool calls. https://platform.claude.com/docs/en/agents-and-tools/tool-use/ handle-tool-calls, 2026a. Accessed 2026-10-23. Anthropic. Messages (Claude API reference). https://platform.claude.com/docs/en/api/messages, 2026b. Accessed 2026-10-01; the page has since been revised. Anthropic. Mid-conversation system messages and tool changes. https://platform.claude.com/docs/en/ build-with-claude/mid-conversation-system-messages, 2026c. Accessed 2026-10-23. Anthropic. OpenAI SDK compatibility. https://platform.claude.com/docs/en/cli-sdks-libraries/ libraries/openai-sdk, 2026d. Accessed 2026-10-23. Anthropic. Claude Platform release notes. https://platform.claude.com/docs/en/release-notes/api, 2026e. Accessed 2026-10-12; entry of 5 August 2025. Anthropic. Other LLM gateways (Claude Code documentation). https://code.claude.com/docs/en/ llm-gateway, 2026f. Accessed 2026-10-23. Berk Atıl, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Türe, et al. Non-determinism of “deterministic” llm 23

system settings in hosted environments. In Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems, pp. 135–148, 2025. Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. Restler: Stateful rest api fuzzing. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 748–758. IEEE, 2019. Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. The oracle problem in software testing: A survey. IEEE transactions on software engineering, 41(5):507–525, 2014. BerriAI. fix(anthropic-bridge): convert mid-conversation system turns to user turns on /v1/messages to chat completions. LiteLLM pull request #41493, https://github.com/BerriAI/litellm/pull/41493, 2026a. Opened 2026-09-16, merged 2026-09-17, shipped in v1.103.0. BerriAI. LiteLLM. https://github.com/BerriAI/litellm, 2026b. Version 1.103.2; download and star counts retrieved 2026-10-01; accessed 2026-10-23. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation of language models, 2024. Lingjiao Chen, Matei Zaharia, and James Zou. How is chatgpt’s behavior changing over time? arXiv preprint arXiv:2307.09009, 2023. Tsong Y Chen, Shing C Cheung, and Shiu Ming Yiu. Metamorphic testing: a new approach for generating next test cases. Technical report, 2020. David M. Cohen, Siddhartha R. Dalal, Michael L. Fredman, and Gardner C. Patton. The aetg system: An approach to testing based on combinatorial design. IEEE Transactions on Software Engineering, 23(7): 437–444, 1997. Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20 (1):37–46, 1960. Myra B Cohen, Matthew B Dwyer, and Jiangfan Shi. Constructing interaction test suites for highlyconfigurable systems in the presence of constraints: A greedy approach. IEEE Transactions on Software Engineering, 34(5):633–650, 2008. Peng Ding. Llm-rosetta: A hub-and-spoke intermediate representation for cross-provider llm api translation, 2026. Ronald A Fisher. On the interpretation of χ 2 from contingency tables, and the calculation of p. Journal of the royal statistical society, 85(1):87–94, 1922. J Nathan Foster, Michael B Greenwald, Jonathan T Moore, Benjamin C Pierce, and Alan Schmitt. Combinators for bidirectional tree transformations: A linguistic approach to the view-update problem. ACM Transactions on Programming Languages and Systems (TOPLAS), 29(3):17–es, 2007. Donghao Fu, Jingxin Li, Xue Jiang, and Yihong Dong. Where is the tradeoff in using third-party API routers for agentic software development?, 2026. fuergaosi233. claude-code-proxy. https://github.com/fuergaosi233/claude-code-proxy, 2026. Commit 7ea4177 (2026-03-12); accessed 2026-10-23. Saibo Geng, Hudson Cooper, Michał Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz, and Harsha Nori. Jsonschemabench: A rigorous benchmark of structured outputs for language models, 2025. Google. Generating content (Gemini API reference). https://ai.google.dev/api/generate-content, 2026a. Accessed 2026-10-23. 24

Google. OpenAI compatibility (Gemini API). https://ai.google.dev/gemini-api/docs/openai, 2026b. Accessed 2026-10-23. Google. Thought signatures (Gemini API). thought-signatures, 2026c. Accessed 2026-10-23.

https://ai.google.dev/gemini-api/docs/

Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM workshop on artificial intelligence and security, pp. 79–90, 2023. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. The Innovation, 7(6), 2026. Sicheng He, Chang Li, Jiaming Liu, Jin Liu, Jacky Keung, and Xiaoxue Ma. Understanding bugs in model context protocol development frameworks: An empirical study. ACM Transactions on Software Engineering and Methodology, 2026. headroomlabs-ai. [BUG] Prefix tracker loses its lineage when Claude Code replaces a trailing role:"system" reminder, so the provider prompt cache is re-written every turn. headroom issue #3933, https://github. com/headroomlabs-ai/headroom/issues/3933, 2026. Opened 2026-10-02, closed 2026-10-04. Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pp. 65–70, 1979. Bahruz Jabiyev, Steven Sprecher, Kaan Onarlioglu, and Engin Kirda. T-reqs: Http request smuggling with differential fuzzing. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 1805–1820, 2021. Bahruz Jabiyev, Steven Sprecher, Anthony Gavazzi, Tommaso Innocenti, Kaan Onarlioglu, and Engin Kirda. {FRAMESHIFTER}: Security implications of {HTTP/2-to-HTTP/1} conversion anomalies. In 31st USENIX Security Symposium (USENIX Security 22), pp. 1061–1075, 2022. Gunel Jahangirova, David Clark, Mark Harman, and Paolo Tonella. Test oracle assessment and improvement. In Proceedings of the 25th international symposium on software testing and analysis, pp. 247–258, 2016. Fengqing Jiang, Zhangchen Xu, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. Chatbug: A common vulnerability of aligned llms induced by chat templates. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 27347–27355, 2025a. Yuancheng Jiang, Jianing Wang, Chuqi Zhang, Roland Yap, Zhenkai Liang, and Manuel Rigger. Enhanced differential testing in emerging database systems, 2025b. Ben Kallus, Prashant Anantharaman, Michael Locasto, and Sean W Smith. The HTTP garden: Discovering parsing vulnerabilities in HTTP/1.1 implementations by differential fuzzing of request streams, 2024. Geonwoo Kim and Brent ByungHoon Kang. When valid tool calls change meaning: Formation-consistent dispatch for llm agents, 2026. Christian Kreibich, Nicholas Weaver, Boris Nechaev, and Vern Paxson. Netalyzr: Illuminating the edge network. In Proceedings of the 10th ACM SIGCOMM conference on Internet measurement, pp. 246–259, 2010. D Richard Kuhn, Dolores R Wallace, and Albert M Gallo. Software fault interactions and implications for software testing. IEEE transactions on software engineering, 30(6):418–421, 2004. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp. 611–626, 2023. 25

Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. Llms get lost in multi-turn conversation, 2026. Maxime Lamothe, Yann-Gaël Guéhéneuc, and Weiyi Shang. A systematic review of api evolution literature. ACM Computing Surveys (CSUR), 54(8):1–36, 2021. Vu Le, Mehrdad Afshari, and Zhendong Su. Compiler validation via equivalence modulo inputs. pp. 216–226. ACM, New York, NY, USA, 2014. Zongrong Li, Shengkun Ye, Feiyou Guo, and Zuoyou Dang. Silentprobe: Measuring silent failure in production apis used as agent tools, 2026. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. Guanjie Lin, Yinxin Wan, Shichao Pei, Ting Xu, Kuai Xu, and Guoliang Xue. Behavioral consistency and transparency analysis on large language model API gateways. In Proceedings of the 2026 ACM Internet Measurement Conference (IMC ’26), pp. 563–573, 2026. Li Lin, Qinglin Zhu, Hongqiao Chen, Zhuangda Wang, Rongxin Wu, and Xiaoheng Xie. QTRAN: Extending metamorphic-oracle based logical bug detection techniques for multiple-DBMS dialect support. Proceedings of the ACM on Software Engineering, 2(ISSTA):731–752, 2025. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics, 12:157–173, 2024. Yinhong Liu, Zhili Tan, Zilin Wang, and Zhijiang Guo. Action-space shaping for llm agents: Measuring and mitigating tool-schema bias, 2026. Minghui Long, Yanjie Zhao, and Haoyu Wang. LogicHunter: Testing LLM agent frameworks with an agentic oracle. Proceedings of the ACM on Software Engineering, 3(ISSTA):1171–1193, 2026. Nuno P Lopes, Juneyoung Lee, Chung-Kil Hur, Zhengyang Liu, and John Regehr. Alive2: bounded translation validation for llvm. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, pp. 65–79, 2021. Ce Lyu, Yanhao Wang, Jie Liang, and Minghao Zhao. LLM-based dynamic differential testing for database connectors with reinforcement learning-guided prompt selection. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 3814–3818. IEEE, 2025. Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. A catalogue of inter-parameter dependencies in restful web apis. In International Conference on Service-Oriented Computing, pp. 399–414. Springer, 2019. Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. Restest: automated black-box testing of restful web apis. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 682–685, 2021. Maxim AI. Bifrost. https://github.com/maximhq/bifrost, 2026. Commit 080a6bb (2026-10-01); accessed 2026-10-23. William M McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100–107, 1998. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics, 12:933–949, 2024. 26

Jonas Möller, Felix Weißberg, Lukas Pirch, Thorsten Eisenhofer, and Konrad Rieck. Cross-language differential testing of json parsers. In Proceedings of the 19th ACM Asia Conference on Computer and Communications Security, pp. 1117–1127, 2024. musistudio. claude-code-router. https://github.com/musistudio/claude-code-router, 2026a. Star count retrieved 2026-10-01; releases 1.x depend on @musistudio/llms, releases 3.0.0 (2026-07-06) and later on @the-next-ai/ai-gateway; accessed 2026-10-23. musistudio. @musistudio/llms. https://github.com/musistudio/llms, 2026b. Version 1.0.53 (current release and repository head on 2026-10-01); accessed 2026-10-23. George C Necula. Translation validation for an optimizing compiler. In Proceedings of the ACM SIGPLAN 2000 conference on Programming language design and implementation, pp. 83–94, 2000. Changhai Nie and Hareton Leung. A survey of combinatorial testing. ACM Computing Surveys (CSUR), 43 (2):1–29, 2011. OpenAI. OpenAPI specification for the OpenAI API. https://github.com/openai/openai-openapi, 2026. File openapi.yaml, snapshot of 2026-10-01; repository accessed 2026-10-23. Vishal Pandey and Gopal Singh. Continuitybench: A benchmark and systems study of stateful failover in multi-provider llm routing, 2026a. Vishal Pandey and Gopal Singh. Failureatlas: A taxonomy of failure modes in multi-provider llm serving infrastructure, 2026b. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. volume 37, pp. 126544–126565, 2024. Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. Mrigank Pawagi, Lize Shao, Hyeonmin Lee, Yixin Sun, and Wenxi Wang. RFCScope: Detecting logical ambiguities in Internet protocol specifications. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1234–1246. IEEE, 2025. Amir Pnueli, Michael Siegel, and Eli Singerman. Translation validation. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pp. 151–166. Springer, 1998. Portkey. Portkey AI Gateway. https://github.com/Portkey-AI/gateway, 2026. Accessed 2026-10-23. PsycherosAI. Haiku 4.5 worker requests fail when temperature and top_p are both sent. PsycherosAI/Psycheros issue #67, https://github.com/PsycherosAI/Psycheros/issues/67, 2026. Opened 2026-09-24. Yanzhao Qin, Tao Zhang, Yanjun Shen, Wenjing Luo, Haoze Sun, Yan Zhang, Yujing Qiao, Weipeng Chen, Zenan Zhou, Wentao Zhang, et al. Sysbench: Can large language models follow system messages?, 2024. QuantumNous. new-api. https://github.com/QuantumNous/new-api, 2026. Accessed 2026-10-23. router-for-me. CLIProxyAPI. https://github.com/router-for-me/CLIProxyAPI, 2026. Version 8.0.15; accessed 2026-10-23. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting, 2024. Sergio Segura, Gordon Fraser, Ana B Sanchez, and Antonio Ruiz-Cortés. A survey on metamorphic testing. IEEE Transactions on software engineering, 42(9):805–824, 2016. 27

Kaiwen Shen, Jianyu Lu, Yaru Yang, Jianjun Chen, Mingming Zhang, Haixin Duan, Jia Zhang, and Xiaofeng Zheng. Hdiff: A semi-automatic framework for discovering semantic gap attack in http implementations. In 2022 52nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pp. 1–13. IEEE, 2022. songquanpeng. one-api. https://github.com/songquanpeng/one-api, 2026. Accessed 2026-10-23. Lijuan Tang and Yuemeng Zheng. Measuring the serving stack instead of the model: Hidden confounds in local tool-use evaluation, 2026. The-NeXT-AI. @the-next-ai/ai-gateway. https://github.com/The-NeXT-AI/ai-gateway, 2026. npm package @the-next-ai/ai-gateway 1.0.21; accessed 2026-10-23. Narseo Vallina-Rodriguez, Srikanth Sundaresan, Christian Kreibich, and Vern Paxson. Header enrichment or isp enrichment? emerging privacy threats in mobile networks. In Proceedings of the 2015 ACM SIGCOMM Workshop on Hot Topics in Middleboxes and Network Function Virtualization, pp. 25–30, 2015. Vercel. AI SDK. https://github.com/vercel/ai, 2026. Package ai 5.0.271 and 7.0.126 with the @ai-sdk/anthropic, @ai-sdk/openai and @ai-sdk/google providers; accessed 2026-10-23. Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training LLMs to prioritize privileged instructions. arXiv preprint arXiv:2404.13208, 2024. Nicholas Weaver, Christian Kreibich, Martin Dam, and Vern Paxson. Here be web proxies. In International Conference on Passive and Active Network Measurement, pp. 183–192. Springer, 2014. Edwin B Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212, 1927. Tong Wu, Shujian Zhang, Kaiqiang Song, Silei Xu, Sanqiang Zhao, Ravi Agrawal, Sathish Reddy Indurthi, Chong Xiang, Prateek Mittal, and Wenxuan Zhou. Instructional segment embedding: Improving llm safety with instruction hierarchy, 2025. Xiaonan Xu and Wenjing Wu. What aggregate scores miss: Measuring item-level regressions in commercial llm api migrations, 2026. Xin Xu and Siru Tao. Discriminating fixture coverage in agent-infrastructure verification suites, 2026. Boyang Yang, Zhenhao Li, Ziyao Yang, Kanghui Jia, Xin Yin, Mingmou Liu, and Haoye Tian. Do tool calls execute as intended? measuring and repairing intent-execution correspondence in llm agents, 2026. Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. Finding and understanding bugs in C compilers. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’11), pp. 283–294, 2011. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. Xiaowen Zhang, Hannuo Zhang, and Shin Hwei Tan. Understanding bugs in modern agentic frameworks: A study of symptoms, root causes, and triggering conditions. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering, pp. 601–612, 2026a. Yage Zhang, Yukun Jiang, Zeyuan Chen, Michael Backes, Xinyue Shen, and Yang Zhang. Real money, fake models: Deceptive model claims in shadow apis, 2026b. Yuewei Zhang, Zhi-Hai Zhang, and Hanzhang Qin. Which model is actually serving you? iris: Budgeted black-box auditing of model substitution and routing dilution in llm gateways, 2026c. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. volume 36, pp. 46595–46623, 2023. 28

A

Conversation Representation, Rules, and Checker Assessment

This appendix explains the conversation representation and preservation rules used by isloss, the requestcapture procedure, and the tests of the checker’s accuracy. It also provides the detailed motivating example, specification comparisons and test-suite audit supporting Sections 1–4. The replication archive contains the complete rule definitions and evaluation records. A.1

Counting Units, Motivating Example, and Existing Tests

Units and result subsets. A request template is a shape; a translator and source-to-target route form a path; applying one shape to one path gives a test cell. The main request study uses 14 library paths and 51 shapes that pass static source checks. Of the 714 tests, 29 explicitly refuse the input and 685 produce translations to check (Stratum 1). The gateway study also examines eight gateway paths. The archive retains the stratum labels to identify different subsets: Stratum 2a contains 13 source shapes accepted live by claude-sonnet-4-6 (182 cells); Stratum 2b contains eight accepted by claude-opus-4-8 (112 cells); and Stratum 2c contains 88 cells accepted live at both source and target endpoints. These labels describe the acceptance conditions, rather than asserting that all subsets are nested. Repeated findings are grouped into 54 candidates across 18 mechanisms. Human review assigns 31 candidates to Grade A (confirmed defects), 11 to Grade B (acceptable form changes) and 12 to Grade C (disagreement). The capability tags are M for avoidable changes (28 candidates), U for losses without a native target representation (23), and Borderline for uncertain cases (three). The visibility tags Vis/K/Meta/O are retained in the detailed records; the primary result selects Vis∧M findings, meaning avoidable changes to prompt, role or tool fields. It does not include sampling controls or metadata merely because they can affect an application. The rule categories T1, T2 and T3, and their executable identifiers, are explained in Section A.3. The mid-conversation instruction. Table 6 expands Figure 2. It records the tested software versions, the emitted OpenAI requests and the model’s compliance with the instruction to stop using run_cmd. We test two documented placements of the Anthropic system turn: directly before an assistant turn (A′1 ), and at the end of messages (A′2 ). These placements occur three and seven times, respectively, across seven recorded Claude Code traces. All 22 stimuli (11 paraphrases per placement) pass the offline check of the documented beta-header contract. This is static validation, not an observed HTTP 200 response from the Anthropic endpoint with that beta header. The separate V4-midsys-R10 suite contains 170 shapes and produces 1,139/1,530 T1 cells (74.4 %) at the reported request-analysis stage. The table’s live trials replay the intact or altered requests on Gemini and Qwen, at temperature zero. A separate Anthropic-target study places the prohibition within the final user turn (MC3-1, Table 19). Removing that prohibition reduces refusal from 30/30 to 0/30 on claude-haiku-5-5 (pHolm = 0.0039), meeting the pre-specified prediction. On claude-sonnet-4-6, refusal changes from 10/30 to 0/30 (p = 0.125), which does not meet the prediction. What existing tests cover. We audit 462 test files across 11 open-source codebases. None contains the property-based, differential or cross-dialect reference tests sought in this audit. Table 7 distinguishes direct assertions from partial checks and missing coverage. Table 8 places isloss beside related testing and measurement approaches. A.2

API Encodings, Preservation Rules, and Specification Gaps

Table 9 compares the API encodings of conversation events and explicit controls used in ConvIR. It covers Anthropic Messages, OpenAI Chat Completions and Google Gemini generateContent; the accompanying rule records also address OpenAI Responses. Table 10 provides the documented mapping conventions L01–L08 and ten request-side violation dimensions, V01–V10. These V identifiers correspond to executable checks S01–S10; they are alternative labels used in the specification records, not different methods. The table illustrates the rule grounding rather than listing all 27 executable checks. The full catalogue also includes S11–S13, response checks and the stream-equivalence relation MR-S. A convention permitting a documented 29

Table 6: Four translation outcomes for the same mid-conversation instruction. Six translators preserve the instruction, delete it, demote it to user text, or explicitly refuse the input; each translator gives the same outcome on all 11 paraphrases in both placements. Panel (a) gives the source and intact target requests; panel (b) gives the emitted request and isloss rule; panel (c) reports the live trials in which the model obeys the prohibition (k/11, temperature zero). These live counts concern the target models, not live validation of the Anthropic source. Across the 112 trials in which the model obeys the prohibition, the API returns HTTP 200, no run_cmd call is made, and the reply ends with finish_reason: stop rather than a token-limit termination. AI SDK v5 and LLM-Rosetta produce the same normalised intact request (R). Claude results appear in Table 19. (a) Anthropic Messages source in both documented mid-turn system placements (A′1 , A′2 ) and intact OpenAI wire (R)

Turns (n = 11 each)

Anthropic Messages source request

Intact OpenAI Chat wire (R; rule L01)

[0..2] prefix

user: “list files” → assistant: run_cmd("ls") → user: tool_result “a.txt b.txt” ′ A1 (before assistant) [3] system: “[harness] run_cmd is DISABLED. . . ” → [4] assistant: “Understood. . . ” → [5] user: “delete b.txt” A′2 (messages[-1]) [3] user: “now delete b.txt” → [4] system: “[harness] run_cmd is DISABLED. . . refuse shell use.”

(b) What six translators emit on the wire (A′1 , A′2 , 11/11 each) Translator (version)

! !

user → assistant (tool_calls) → tool: “a.txt b.txt” (L01) [3] system kept at [3] → [4] assistant → [5] user [3] user: “delete b.txt” → [4] system kept at [4]

(c) Live replay: prohibition kept (k/11)

Outbound OpenAI wire fate (11/11)

isloss

AI SDK v7 7.0.126 Fail-closed: AI_InvalidPromptError (400) refusal claude-code-proxy 7ea4177 Fail-closed: rejects messages.role (422) refusal AI SDK v5 5.0.271 LLM-Rosetta 0.13.0

Keeps role: "system" in place (arm R) L01 Keeps role: "system" in place (wire = R) L01

@musistudio/llms 1.0.53 LiteLLM 1.103.2

Silently deletes system turn (200) Demotes to user + “Operator note. . . ”

3.5-lite

3.8-flash

Qwen2.5-7B

A′1

A′2

A′1

A′2

A′1

A′2

— —

— —

— —

— —

— —

— —

11/11 11/11 11/11 11/11 3/11 10/11 (normalised wire identical to arm R)

S05 0/11 0/11 S05,D02 11/11 11/11

1/11 9/11

3/11 0/11 0/11 0/11 10/11 10/11

Table 7: Existing test coverage across 11 translator codebases. Columns cover 14 request, session and reply features. The audit groups them under authority (P1, a–b), call–result links (P2, c–h), reasoning integrity (P3, i–k) and controls (P4, l–n). Symbols: • = asserted (Y); ◦∗ = a narrow helper assertion (Y*); ◦ = partial coverage (P); · = absent (N); ? = not inspectable. Partial coverage may be fixture-only, restricted to one direction, or limited to a sub-case. Bottom rows summarise coverage and incompatible behaviours encoded as passing assertions in the audit. Gemini’s first-party compatibility endpoint has no public repository. Request & session features (a–i)

Reply & CI (j–n)

Code base

a

b

c

d

e

f

g

h

i

j

k

l

m

n Y(+Y*)/14 Test style

LiteLLM Vercel AI SDK @musistudio/llms claude-code-proxy LLM-Rosetta Portkey Bifrost new-api one-api Envoy AI Gateway Kong ai-proxy

• ◦ · · • · • ◦ · • ·

· ◦ · · • · • · · • ·

• • · ◦ • · • ◦ · • ◦

• ◦ · · • · • · · • ·

• • · · ◦ · • ◦ · • ·

• • · · • · • · · • ·

• · · · · · ◦ · · · ·

• • · ◦ • ◦ • ◦ · • ◦

• ◦ · ◦ • · • ◦ · • •

• ◦ · · • · • ◦ · • ◦

• ◦ · · • · • · · • ·

• • · · • · • • · • ◦

◦ · · · · · ◦ · · ◦∗ ·

• · · · ◦ ◦ • · · • ?

Total (•/◦) Passing policies

4/2 3/1 5/3 4/1 4/2 5/0 1/1 5/4 5/3 4/3 4/1 6/1 1∗ /2 3/2 3 3 – – – – – – – – – – – –

12/14 literal asserts + VCR 5/14 inline snapshots 0/14 none 0/14 print-only smoke 10/14 IR round-trip + literals 0/14 HTTP 200 smoke 12/14 Go tables + round-trip 1/14 billing literals 0/14 none 13/14 Go tables + VCR 1/14 1 golden/provider – – – incompatible (a, b)

Features: a mid-turn system; b is_error; c tool_choice; d text+tool order; e thinking+sig; f thoughtSignature; g tool-name len; h sampling; i ≥3 turns; j stream tools; k finish/stop; l usage; m stream≡non-stream; n live CI.

30

Table 8: Comparison with related testing and measurement methods. LLM = a model is needed at test time; Det. = deterministic verdicts; State = multi-turn state is considered; BB = black-box use is possible. “gen.” means that an LLM generates inputs but the verdict is deterministic. “preprint” means that no peer-reviewed venue was found at the time of writing. a For our approach, captured-payload checking needs no model call; the separate behavioural probes do (Section A.3). Outcomes for other methods are as reported in their papers. Method (venue)

Object under test Oracle

LLM Det. State

BB

Reported outcome

isloss (this paper)

captured requests, documented replies and SSE preservation rules; stream equivalence hosted model replies vs. compatibility native controls endpoint object API (agent LLM agent judge; frameworks) crash baseline

Na

Y

Y

N

see §6

Y

N

Y

Y

see §6.5

Y

N

N

N

GateScope (IMC’26)

hosted gateway (network path)

differential vs. vendor; statistical

Y

N

Y

HTTP Garden (preprint’24)

HTTP parser; gateway transforms network path (middleboxes)

differential

N

Y

N

known-answer probes

N

Y

N

Y

network path (mobile ISPs)

known-answer probes

N

Y

N

Y

Csmith (PLDI’11)

C compilers

N

Y

N

Y

EMI (PLDI’14)

C compilers

differential (N versions) metamorphic

N

Y

N

Y

QTRAN (ISSTA’25)

DBMS dialects

gen.

Y

N

Y

gen.

Y

N

N

16 reported, 10 confirmed

Y

N

–

–

Y

N

Y

Y

N

Y

part.

N

31 ambiguities, 8 author-confirmed 0% defence success, 4 agents (threat study) no bugs reported; claims lossless

Endpoint probes (this study) LogicHunter (ISSTA’26)

Netalyzr / Web proxies (IMC’10 / PAM’14) Header enrichment (HotMiddlebox’15)

DB-connector diff. testing (preprint’25) RFCScope (ASE’25) SIDEL (preprint’26) LLM-Rosetta (preprint’26)

metamorphic; LLM translates inputs JDBC connectors differential; LLM+RL generates specification text LLM judge over spec adversarial router attack-success rate → agent translation library self round-trip; (own IR) compliance suite

31

40 found / 30 conf. / 26 fixed; precision 91% Y 10 gateways; substitution, memory loss, billing drift part. >100 bugs, 68 fixed 130k sessions; 14% of 646k IPs behind proxies 299 providers, 112 countries (measurement) >325 bugs 147 confirmed, >100 fixed 24 bugs, 16 confirmed

Table 9: How the three APIs encode the tested constructs. † marks vendor documentation that is silent or self-contradictory. ‡ marks a construct without a native representation in that format; carrying it requires another encoding, declaring a loss, or refusing the input. Construct

OpenAI Chat Completions

Anthropic Messages

Gemini generateContent

System prompt

system / developer message, accepted at any position

top-level systemInstruction; role:system inside contents is rejected (400)

Roles

system, developer, user, assistant, tool assistant.tool_calls[]: id, function.name, function.arguments (JSON string) role:tool message with tool_call_id; no error flag‡

top-level system; a mid-conversation {role:system} turn is documented for recent models while the API reference (our copy of 1 October 2026) stated there is no system role† user, assistant (typed content blocks) tool_use block: id, name, input (object)

Tool call

Tool result

Reasoning

no reasoning content in messages; reasoning_effort knob only‡

Forced tool choice

tool_choice: none / auto / required / named function

Controls

top-level temperature, top_p, seed, stop, parallel_tool_calls, user

Tool names

≤64 chars: letters, digits, _, -

user, model (typed parts) functionCall part: name, args (object), optional id

tool_result block: tool_use_id, content, is_error

functionResponse part: name, response object; an "error" key is the documented convention thinking block with signature, thought parts with replayed verbatim thoughtSignature; Gemini 3 rejects a missing signature (400)† tool_choice.type: auto / any functionCallingConfig.mode: / tool / none AUTO / ANY / NONE / VALIDATED, allowedFunctionNames generationConfig: top-level temperature, top_p, top_k, stop_sequences, temperature, topP, topK, seed, stopSequences; no parallel-call metadata.user_id; disable_parallel_tool_use per switch‡ tool ≤128 chars: ≤128 chars; declarations may ˆ[a-zA-Z0-9_-]{1,128}$ also use . and :, call and result names may not†

loss, such as the unsupported-control case in L06, does not establish semantic equivalence: the loss is recorded as a degradation under the study’s counting policy. Grounding the rules in vendor documents identifies 17 undecided or contradictory interactions. Two are benign in our audit, with no observed translator divergence; Table 11 lists the other 15. Separate live probes of the Gemini endpoint resolve 13 interactions. These observations inform the specification analysis and remain distinct from deterministic isloss verdicts on captured payloads. The reported-bug corpus contains 125 issues from 12 trackers. Of these, 99 map to a rule (79.2 %: 46 request issues and 53 response issues), three describe legitimate normalisations (2.4 %), and 23 fall outside the fault model (18.4 %). The out-of-scope categories concern target-constraint rejections, rejection of legal inputs, server-executed tools, masked upstream errors and transport timing. Two-pass coding gives 72 % agreement and Cohen’s κ = 0.70 (Cohen, 1960); grouping by rule family gives 83 % and κ = 0.73. Response issues are silent in 62–67 % of cases under the reported classifications. A.3

Rule Catalogue, Capture Procedure, and Accuracy Assessment

The executable catalogue. isloss contains 27 deterministic rules divided into three categories. The 13 T1 rules (S01–S13) check changes to authorship, semantic flags and reasoning-continuity fields, payload structure, event presence, call–result links, event order, explicit controls, tool names, integrity tokens, tool schemas, stop reasons and stream reconstruction. The six T2 records (D01–D06) cover separately reported degradations and informational notes. They distinguish documented degradation from the study-policy exclusion of system-to-user demotion (D02); cache, usage and model notes use D03–D05. Role demotion changes authority, while target support and translator warnings are recorded separately (Section 3.2). The eight T3 rules (R01–R08) record benign representation changes. Table 2 shows how the findings are filtered for the primary request result. 32

Table 10: Documented mapping conventions and request-side violation dimensions. L01–L08 describe permitted mappings or explicit degradation allowances; V01–V10 identify protected dimensions. A documented degradation is not equivalent to preservation. Each row identifies its vendor source, with the quoted text retained in the replication archive. This reference table supports, rather than exhausts, the executable isloss catalogue. Rule

Dimension

What changes

Vendor source (artifact copy, location)

L01

Representation

anthropic:messages (L48-50)

L02

Representation

L03

Representation

L04

Representation

L05

Representation

L06

Degradation

L07

Representation

L08

Representation

role name remap (tool result carried in user/model turn) merge consecutive same-author messages into one multi-block message, blocks intact consistent renaming of call ids across call and result split one text block into adjacent text blocks of the same author, concatenation identical developer text hoisted to a dedicated system slot when it is the leading developer text unsupported control dropped with documented degradation; recorded as a loss, not as semantic equivalence OpenAI Chat target only: assistant text after tool calls in the same turn carried before them (no ordered slot: one content field, one tool_calls list per assistant message) performance/billing hints (cache_control) may be dropped or added; their loss is a cost effect, not a change of meaning, and is outside the semantic-fault counts

V01

Partition

V02

Partition

V03

Partition

V04

Partition

V05

Partition

V06

References

V07

Order

V08

Controls

V09

Controls

V10

Controls

cross-author merge: text from different authors concatenated into one block author change: developer text re-labelled as user or assistant; system-to-user demotion is conservatively recorded as T2 semantic flag lost (is_error) content re-interpreted so that meaning or medium changes (result container type stripped, text promoted to media, error wrapped into plain content) event lost or invented call/result reference broken, duplicated or re-targeted relative order of events changed (within or across authors) explicit control parameter dropped or altered without documented degradation tool name altered without reversible mapping applied to history integrity/continuation field replaced by the vendor’s placeholder signature (e.g. thoughtSignature := skip_thought_signature_validator) instead of being carried from the original response

33

anthropic:messages (L48) anthropic:agents-and-tools_tooluse_handle-tool-calls (L65) anthropic:messages (L95) anthropic:messages (L91) anthropic:messages (L3058) openai:openapi (L41198 (ChatCompletionRequestAssistantMessage)) anthropic:messages (L1060-1062)

anthropic:messages (L48) anthropic:mid_conversation_ system_messages (L11-13) anthropic:agents-and-tools_tooluse_handle-tool-calls (L58) anthropic:messages (L95)

anthropic:agents-and-tools_tooluse_handle-tool-calls (L66) anthropic:agents-and-tools_tooluse_handle-tool-calls (L65) anthropic:agents-and-tools_tooluse_handle-tool-calls (L65-66) anthropic:messages (L1370-1374) anthropic:messages (L15198) gemini-vertex:thought_ signatures (’When using REST or manual handling’)

Table 11: Specification gaps found while grounding the rules. The table lists 15 of 17 recorded interactions; original audit numbering is retained and entries 3 and 4 are the two benign interactions omitted from this table. “Documented by” identifies the vendor (A/O/G); † marks contradictory vendor text, and G∗ marks a Vertex mirror. “n/a” indicates that the API cannot express the construct. “not probed” means that the indicated live probe was not run; it does not imply acceptance or rejection. #

Construct

Gap kind

Documented by

Live probe

1

system message mid-conversation system between tool_use and tool_result empty tool result content

contradiction

A†(n/a: G)

unspecified

A (n/a: G)

Gemini: resolved not probed

unspecified

A

unspecified

none (n/a: O) A,G (n/a: O) A,G (n/a: O) A,O,G†

2 5 6 7 8

assistant text after tool_use in the same turn tool error flag

9

cross-vendor replay of thinking / redacted thinking tool name length / charset

10

disable parallel tool use

11

top_k

12

metadata / user id

13

tool result that looks like JSON / structured vs string result one tool_result split into several results with the same id call ids dropped (LiteLLM G path) which member of a parallel tool-call group must carry the thought signature where a provider-specific signature lives inside a foreign wire format

14 15 16 17

mapping unspecified field unspecified contradiction; mapping unspecified mapping unspecified mapping unspecified mapping unspecified unspecified

Gemini: accepts Gemini: accepts Gemini: resolved Gemini: resolved Gemini: resolved

A,O (n/a: G) A,G (n/a: O) A,O,G

n/a (mapping)

none

Gemini: accepts

unspecified

A

Gemini: accepts Gemini: accepts Gemini: resolved

unspecified

A,O

doc unreachable

G∗

field unspecified

none

34

not probed n/a (mapping)

Gemini: resolved

Requests and capture. A fixed-seed, 14-dimension strength-4 covering array produces 362 rows, with 357 unique request shapes after deduplication. Applying static source constraints leaves 51 shapes for Stratum 1. This filtered subset is the main test suite; filtering alone does not establish that it retains strength-4 coverage. The baseline LiteLLM measurement captures the dictionary handed to the HTTP layer before serialisation. Rechecking its hosted_vllm HTTP body changes the primary count from 127/685 to 132/685, which is reported as a sensitivity result. The Vercel AI SDK capture driver passes maxOutputTokens and providerOptions explicitly and masks source controls that the interface cannot represent. The driver audit checks 2,142 cells and finds no omissions of representable controls. Streams and hidden requests. For OpenAI Chat streams, isloss reassembles delta.reasoning_content before applying the reply checks. The request verdicts and primary held-out assessment are unchanged by this stream-input handling. Where an endpoint’s translated request cannot be captured, the separate behavioural study constructs distinguishing conversations from selected translation patterns. The probe design seeks a property that holds on at least 9/10 intact trials but at most 3/10 altered trials. These model-based checks do not replace the deterministic comparison of observed payloads. Detection, false alarms, and batch numbering. Across rule development and post-freeze evaluation, specification-based test pairs were organised into twenty numbered batches (Sets 1–20) so that rule refinement and accuracy evaluation use disjoint test sets: • Sets 1–8 (in-sample development batches, pre-freeze): eight 30-item batches (240 items: 177 activated faults, 36 activated allowed-change controls and 27 unactivated edits) used while authoring and hardening the rules before freeze. Sets 1–3 were unconstrained initial batches across request, reply and stream payloads, whereas Sets 4–8 targeted gaps remaining after preceding rule updates; a 78-pair subset from Sets 6–8 also provided an early two-judge diagnostic contrast (Table 12). Under the final isloss rules, Sets 1–8 yield 140/177 T1 detections (79.1 %) and 0/30 false alarms on clean controls (6/36 before excluding pre-logged defective or disputed control edits), and are retained only as in-sample diagnostics (Table 29). • Sets 9–14 (primary held-out evaluation, post-freeze): six blind batches authored from vendor specifications without checker-code access after the primary rules were fixed, comprising three fault/control pairs, Set 9 (28/34 activated faults detected; first held-out fault batch in Table 3) with Set 10 (1/28 activated controls flagged; first held-out control batch), Set 11 (37/43) with Set 12 (0/36), and Set 13 (30/48) with Set 14 (1/36), pooling to 95/125 detected faults (76.0 %) and 2/100 false alarms (2.0 %, J = 0.740). • Sets 15–16 (excluded script-generated batches): one fault batch (48 items) and one allowed-change control batch (36 items) were excluded before scoring because they were generated using Python fuzzing scripts rather than designed from vendor specifications under the held-out test protocol (Table 30). • Sets 17–20 (extended held-out evaluation, post-freeze): two later non-scripted blind pairs, Set 17 (32/48 faults detected under frozen isloss) with Set 18 (0/36 controls flagged), and Set 19 (33/47 activated faults detected) with Set 20 (3/34 controls flagged), authored for exploratory post-freeze repair checks and pooled with Sets 9–14 to form the 10-batch extended held-out evaluation: 160/220 faults detected (72.7 %) and 5/170 false alarms (2.9 %; Table 12). On public bugs, isloss detects 45/57 reproduced cases (78.9 %). This includes 10/14 encodable bugs (71.4 %) in the prospective sample opened after the checking rules were fixed. Program-mutant tests detect 113/135 activated LiteLLM request mutants (83.7 %), 84/103 reply/stream mutants (81.6 %) and 43/51 LLM-Rosetta reverse-path mutants (84.3 %). A separate human-designed kit of 50 items yields 37/40 faults detected (92.5 %) and no false alarms on ten controls. Human review. Two rater pairs audit the findings. Among the 54 post-review candidate cards, consensus precision A/(A + B) is 31/42 (73.8 %). For avoidable (M) candidates it is 23/24 (95.8 %); this denominator excludes the four disputed cards, so the confirmed fraction among all 28 avoidable candidates is 23/28 (82.1 %). For unavoidable (U) candidates, consensus precision is 6/16 (37.5 %). All 127 primary Stratum-1 cells carry a mechanism associated with a Grade-A candidate. This is coverage by a human-confirmed mechanism, not independent human review of every cell or every finding within those cells. A 48-item bridging sample scored 35

by all four raters shows a 37.50-percentage-point rater-pair gap on unavoidable-loss items, compared with 6.25 points on avoidable items. This rater sensitivity supports reporting avoidable findings separately, without treating every unrepresentable loss as an established translator defect. Other judges. Table 13 compares isloss with normalised diff and commercial and open-weight LLM judges. Each reference-free judge receives only the source and translated payload; the reference-given condition additionally provides a clean target-dialect translation. Without that reference, every tested LLM judge has a lower observed J than isloss at 0.740. This is an empirical comparison on the held-out set, rather than a general claim about all LLM judges.

B

Study Design and Extended Empirical Results

This appendix provides the complete subject provenance, study-design denominators, and extended empirical tables behind Sections 5 and 6. B.1

Subjects, request corpora, endpoint acceptance, and denominators

Table 14 records the pinned software versions, commits, and constructor settings. The main census uses six library configurations in five codebases: LiteLLM (1.103.1/1.103.2, drop_params=False), Vercel AI SDK v5 (5.0.271) and v7 (7.0.126), LLM-Rosetta (0.13.0), @musistudio/llms (1.0.53), and claude-code-proxy (7ea4177). Their 14 request-translation paths produce 685 judged cells from 51 shapes after 29 refusals (14×51−29). Additional subjects are Portkey (Portkey, 2026), Bifrost (Maxim AI, 2026), newapi (QuantumNous, 2026), one-api (songquanpeng, 2026), CLIProxyAPI (router-for-me, 2026) (v8.0.15), @the-next-ai/ai-gateway (The-NeXT-AI, 2026) (1.0.21), the official OpenAI SDKs (openai-python 2.54.0, openai-node 5.23.2), and Google’s OpenAI-compatible Gemini endpoint. Of the 51 statically valid source shapes, claude-sonnet-4-6 accepts 13 live. Its 38 HTTP 400 rejections comprise 19 placeholder redacted_thinking cases, 15 combinations of temperature and top_p, and 4 empty error tool_result cases. The accepted subset has 34/182 primary cells (18.7 %, Stratum 2a). claude-opus-4-8 accepts 8 shapes, with 23/112 primary cells (20.5 %, Stratum 2b). For translated requests passing the static target checks, live gemini-3.5-flash-lite accepts 160/204 cells and rejects 44 missing Gemini 3 thought_signature; claude-sonnet-4-6 accepts 77/204. Among source-accepted cells on those target paths, 88/104 are also accepted at the target, giving 22/88 primary cells (25.0 %, Stratum 2c). OpenAI-bound requests are checked statically rather than sent to a live target endpoint. Table 15 gives the corpus, unit, denominator, and supporting evidence for each result. Statistical methods are a bootstrap clustered by the 51 request shapes (10,000 resamples), Wilson 95 % intervals, and two-sided Fisher exact or sign-flip tests with Holm correction, as specified for each comparison. B.2

Request mechanisms, acceptance subsets, and sensitivity analysis

Table 16 records 56 candidates across 19 mechanisms before review. Human review identified LLM-Rosetta’s re-encoding of list-valued tool results for Gemini as an acceptable change. We therefore removed this candidate, which affected seven cells and contributed no primary findings. We also merged two duplicate candidate records for assistant-text reordering in AI SDK v5 and v7 translations to Anthropic; each record covered 18 affected cells. The final catalogue therefore contains 54 candidates across 18 mechanisms, or 48 candidates on 11 deduplicated paths (Figure 8). It contains 41 omissions and 13 explicit rewrites; human review yields 31 Grade A defects, 11 Grade B acceptable changes, and 12 Grade C disagreements. The target can represent the affected information for 28 candidates (M), cannot do so for 23 (U), and support is borderline for 3. Among the 28 avoidable candidates, 23 are confirmed defects, 1 is acceptable, and 4 are disputed: the confirmed share is 23/24 after excluding disputes, or 23/28 with all candidates retained. Tables 17 and 18 report alternatives to the 127/685 primary rate (18.5 %, shape-clustered 95 % CI 15.01– 22.06 %). Validity, scope, and tier-boundary variants range from 12.9 % to 19.6 %. Examples are 49/379 after dropping AI SDK paths, 96/685 (14.0 %) when cells rejected live for missing Gemini thought_signature 36

Table 12: Detailed assessment of checker accuracy. The main-text summary appears in Table 3; additional judge comparisons appear in Table 13. “Before the run” means that the protocol was written before execution and committed with the results. “Killed” means that a mutant produces a finding absent from the unmodified translator. Final isloss detection counts use T1 findings; exploratory results with other scoring criteria are identified separately. For allowed-change controls, “packed” denotes pairs as constructed, five of which carry a packing-script placeholder signature; “filled” denotes the post-hoc restoration of those signatures. These controls should not be conflated with the untouched primary held-out set. Check

Predicate

Material

Result

Final isloss measurements and comparisons Seed recall pre2,129 single-fault payloads; isloss T1: 1,595/2,129 = 74.9% (Wilson 73.0–76.7%); (request) specified 335 allowed-change controls intended rule 1,383/2,129 = 65.0%; 0/335 false alarms Historical bugs, pre29 sample-A issues; 12 isloss T1: 9/12 (Wilson 47–91%). A separate post-hoc sample A specified replayed diagnostic detects 10/12; it is not the reported final-rule (request) estimate Historical bugs, pre4 replayed; 6 not run; 7 out isloss T1: 4/4; one upstream fix found incomplete sample A specified of scope (stream) Historical bugs, before the 56 further issues: 18 isloss T1: 6/11 requests and 3/6 streams. The 11-request sample B–C run replayed (12 request, 6 scoring subset differs from the 12 replayed requests; the stream), 17 without a inverse case is reported separately in the records harness, 20 out of scope, 1 not reproduced LiteLLM 1.104.0 adapter, Program mutants preisloss T1 kills 113/135 = 83.7% of activated mutants (request) specified OpenAI target; 300 mutants, 135 activated Program mutants before the Same release, reply/stream isloss T1 kills 84/103 = 81.6% of activated mutants (reply, stream) run paths, two backends; 297 mutants, 103 activated First held-out pre36 faults and 28 allowed T1 on 28/34 faults = 82.4% (Wilson 66.5–91.7%): the 80% pair (Sets 9–10) specified changes by independent threshold is met. T1 on 1/28 controls: the at-most-one designers; 34 faults and 28 threshold is met controls activated 125 activated faults; 100 T1: 95/125 = 76.0%; false alarms: 2/100 = 2.0% Primary held-out pre(Sets 9–14) specified allowed changes Extended pre220 activated faults; 170 T1: 160/220 = 72.7%; false alarms: 5/170 = 2.9% held-out (Sets specified allowed changes; Sets 15–16 9–14, 17–20) excluded LLM judges, pre125 faults and 100 allowed flash-lite: 108/125 faults, 6/100 controls, 633 ms; 3.8-flash: reference given specified changes; judges also receive 120/125, 12/100, 2,352 ms. isloss: 95/125, 2/100, no model (Sets 9–14) a clean target-dialect call; diff D3: 124/125, 15/100. Predicted ≥ 8 false alarms reference per judge: missed by flash-lite LLM judges, preSame 225 pairs, source and flash-lite: 72/125, 6/100, J = 0.516 (cross-dialect 47/94, reference-free specified translation only; 169 strictly 4/75, J = 0.447); 3.8-flash: 103/125, 12/100, J = 0.704 (Sets 9–14) cross-dialect (75/94, 9/75, J = 0.678, 5,607 ms). isloss: J = 0.740 (71/94, 2/75, J = 0.729). Judge agreement κ: 0.786 with reference, 0.585 without (0.490 cross-dialect). Predicted ≥ 8 cross-dialect false alarms per judge: missed by flash-lite isloss T1 plus before the Held-out Sets 9–14; T1 Faults: 95/125 reported and 30 queued; all 125 reported or diff queue run findings reported, diff-only queued. Controls: 2/100 reported and 13 queued. Queued flags queued for review cases are not isloss detections Public bugs, four pre57 of 62 scored bugs T1: 45/57 (Wilson 66.7–87.5%); any tier: 49/57 samples specified reproduce: A 15, B–C 13, (74.7–92.7%). T1 by sample: A 14/15 (one verdict new issues 14, earlier issues corrected post hoc), B–C 9/13, new 10/14, earlier 12/15. encoded after rules were B–C at any tier: 13/13. Earlier sample misses: documented fixed 15 representation limits. Of all 62 scored: 45/62 T1, 49/62 any tier; four predictions held Development diagnostics and return-path baselines; not primary held-out estimates Negative controls preRemapping, merging, Development record: 0 findings; 16/16 round trips identical (request) specified splitting, whitespace; Anthropic round trips Checker self-test pre60 reference replies; 9 seeded Development record: 0 findings on references; 9/9 (response) specified mutations mutations detected Reply/stream pre65 ideal renderings; 15 Development record: 12/15 message operators flagged on seed diagnostics specified message operators plus every shape. No final isloss detection estimate is inferred stream-event operators from this calibration Rule ablation pre2,938 flagged development 1,202 cells depend on a single rule; removing one rule specified cells; leave-one-rule-out suppresses at most 16% of flagged cells Baseline before the Schema (B1), crash (B2), B1 flags 71; B2 fires on five fail-closed paths; B3 flags all diagnostics run any-diff (B3) on the same 2,938 flagged cells and 123/795 clean cells (up to 42 per development cells path) Early judge pre78 pairs from in-sample Sets J1: 9/20 faults, 1/39 clean; J2: 18/20, 21/39; κ = 0.098. contrast specified 6–8; reported counts cover Final-rule diagnostic recount: 17/20 faults; 5/39 packed 20 faults and 39 clean pairs; controls flagged, 0/39 after post-hoc signature restoration. two judges from the These are development diagnostics, not the primary designers’ model family held-out estimate Round trip pre51 source-valid shapes; 10/14 paths provide no inverse. LLM-Rosetta mismatch: A→B→A specified normalised comparison after 48/51 via OpenAI (including 31/34 first hops without the return hop avoidable T1), 49/51 via Gemini. LiteLLM: 51/51 both ways. Twelve two-hop chains: 546/554 flagged return comparisons. A mismatch alone does not locate a translation defect T1 and any-tier counts are distinct scoring criteria. Historical sample rows retain the original scoring subsets and must not be combined as if their denominators were identical. Earlier development-run predictions and their recorded outcomes remain in the scorecards; a final-rule recount does not retrospectively change whether those predictions held. “Packed” and post-hoc restored controls belong to development Sets 6–8, not the untouched primary held-out Sets 9–14.

37

Table 13: isloss and alternative judges on the primary held-out evaluation (Sets 9–14). Judges include normalised diff D3, commercial Gemini and Claude models (gemini-3.5-flash-lite, gemini-3.8-flash, claude-sonnet-4-6), and local open-weight models (Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct). Results distinguish reference-given, reference-free generic and reference-free specification-in-context conditions. The full activated set contains 225 items; 169 involve strictly cross-dialect translations. Judge

Setting

Fault recall (95% CI) Allowed-change FPR (95% CI)

Held-out Sets 9–14, all activated items (N = 225: nfault = 125, nallow = 100) isloss (T1) Ref.-free 95/125 (76.0% [67.8, 82.6]) 2/100 (2.0% [0.6, 7.0]) Normalised diff D3 Ref. given (same wire) 124/125 (99.2% [95.6, 99.9]) 15/100 (15.0% [9.3, 23.3]) gemini-3.5-flash-lite Ref. given (same wire) 108/125 (86.4% [79.3, 91.3]) 6/100 (6.0% [2.8, 12.5]) gemini-3.8-flash Ref. given (same wire) 120/125 (96.0% [91.0, 98.3]) 12/100 (12.0% [7.0, 19.8]) claude-sonnet-4-6 Ref. given (same wire) 122/125 (97.6% [93.2, 99.2]) 25/100 (25.0% [17.5, 34.3]) gemini-3.5-flash-lite Ref.-free generic 72/125 (57.6% [48.8, 65.9]) 6/100 (6.0% [2.8, 12.5]) gemini-3.8-flash Ref.-free generic 103/125 (82.4% [74.8, 88.1]) 12/100 (12.0% [7.0, 19.8]) gemini-3.5-flash-lite Ref.-free + Spec 76/125 (60.8% [52.0, 68.9]) 6/100 (6.0% [2.8, 12.5]) gemini-3.8-flash Ref.-free + Spec 97/125 (77.6% [69.5, 84.0]) 6/100 (6.0% [2.8, 12.5]) claude-sonnet-4-6 Ref.-free generic 105/125 (84.0% [76.6, 89.4]) 32/100 (32.0% [23.7, 41.7]) claude-sonnet-4-6 Ref.-free + Spec 96/125 (76.8% [68.7, 83.3]) 15/100 (15.0% [9.3, 23.3])

J [95% CI] Cost/1k calls

0.740 [0.658, 0.816] 0.842 [0.768, 0.910] 0.804 [0.724, 0.876] 0.840 [0.762, 0.910] 0.726 [0.636, 0.812] 0.516 [0.416, 0.614] 0.704 [0.608, 0.792] 0.548 [0.448, 0.644] 0.716 [0.624, 0.798] 0.520 [0.406, 0.630] 0.618 [0.510, 0.716]

$0 $0 $0.116† $0.116† $3.74 $0.169† $0.169† $0.231 $0.471 $3.92 $3.50‡

Held-out Sets 9–14, strictly cross-dialect subset (N = 169: nfault = 94, nallow = 75) isloss (T1) Ref.-free 71/94 (75.5% [66.0, 83.1]) 2/75 (2.7% [0.7, 9.2]) 0.729 [0.635, 0.819] claude-sonnet-4-6 Ref. given (same wire) 92/94 (97.9% [92.6, 99.4]) 21/75 (28.0% [19.1, 39.0]) 0.699 [0.592, 0.803] gemini-3.5-flash-lite Ref.-free generic 47/94 (50.0% [40.1, 59.9]) 4/75 (5.3% [2.1, 12.9]) 0.447 [0.332, 0.558] gemini-3.8-flash Ref.-free generic 75/94 (79.8% [70.6, 86.7]) 9/75 (12.0% [6.4, 21.3]) 0.678 [0.563, 0.787] gemini-3.5-flash-lite Ref.-free + Spec 53/94 (56.4% [46.3, 66.0]) 6/75 (8.0% [3.7, 16.4]) 0.484 [0.364, 0.601] gemini-3.8-flash Ref.-free + Spec 72/94 (76.6% [67.1, 84.0]) 6/75 (8.0% [3.7, 16.4]) 0.686 [0.577, 0.785] claude-sonnet-4-6 Ref.-free generic 75/94 (79.8% [70.6, 86.7]) 23/75 (30.7% [21.4, 41.8]) 0.491 [0.358, 0.624] claude-sonnet-4-6 Ref.-free + Spec 71/94 (75.5% [66.0, 83.1]) 15/75 (20.0% [12.5, 30.4]) 0.555 [0.430, 0.678] Qwen2.5-7B-Instruct Ref.-free generic 64/94 (68.1% [58.1, 76.6]) 44/75 (58.7% [47.4, 69.1]) 0.094 [−0.052, 0.243] Qwen2.5-7B-Instruct Ref.-free + Spec 11/94 (11.7% [6.7, 19.8]) 1/75 (1.3% [0.2, 7.2]) 0.104 [0.037, 0.178] Llama-3.1-8B-Instruct Ref.-free generic 94/94 (100.0% [96.1, 100.0]) 75/75 (100.0% [95.1, 100.0]) 0.000 [0.000, 0.000]

$0 $3.74 $0.169† $0.169† $0.231 $0.471 $3.92 $3.50‡ $0 $0 $0

† Costs are from the recorded evaluation runs. Combined two-model runs (450 calls each): $0.0523 total ($0.116/1k), $0.0760 total ($0.169/1k); spend (225 calls): $0.0521 (flash-lite, $0.231/1k) and $0.1059 (3.8-flash, $0.471/1k). ‡ claude-sonnet-4-6 at Anthropic pricing ($3.00/$15.00 per M input/output tokens; Spec runs include prompt-cache reads at $0.30/M ; Ref. given live spend $0.8420/225 = $3.74/1k ); two Spec items (LQ15, LQ16, stop_reason=max_tokens) are UNPARSEABLE and scored as not flagged under PRIMARY . J [95% CI] from item-stratified bootstrap (B = 10,000, seed 20261015); Spec = Spec-in-Context.

38

Table 14: Subjects and software provenance. A = Anthropic Messages, O = OpenAI Chat Completions, G = Gemini generateContent. “2nd hop” means that the subject receives another translator’s OpenAI-format request. Download and star counts are recorded at capture time. Subject

Position

LiteLLM

library; also a proxy Python

Vercel AI SDK v5

in-process library

Vercel AI SDK v7

in-process library

@musistudio/llms

local proxy (engine of claude-code-router 1.x) local proxy (engine TypeScript of claude-code-router ≥2.0); not pooled local proxy Python library with its own Python IR shared gateway, TypeScript self-hosted shared gateway, Go self-hosted shared gateway, Go self-hosted shared gateway, Go self-hosted gateway, self-hosted, Go separate process; not pooled shared gateway Lua; Go

@the-next-ai/aigateway claude-code-proxy LLM-Rosetta Portkey gateway Bifrost new-api one-api CLIProxyAPI Kong AI Gateway; Envoy AI Gateway Gemini OpenAIcompatible endpoint

first-party, black box

Language

Directions tested

Version / commit

Scale (2026-10-01)

A→O; O/G→A; 2nd hop TypeScript A→O/G/A; O/G→A TypeScript A→O/G/A; O/G→A TypeScript A→O/G/A; O/G→A

1.103.1 / 1.103.2 5.0.271

89.4M/30 d; 60.0k⋆

–

7.0.126

ai 32.7M/wk; 27.1k⋆ (same package)

1.0.53

308⋆; router 37.5k⋆

A→O/G/A; 2nd hop

1.0.21

router 37.5k⋆

A→O; O→A A→O/G/A; O/G→A 2nd hop O→A/G

git 7ea4177 llm-rosetta 0.13.0 669825c

2.8k⋆ 5.7k/30 d; 86⋆ 13.1k⋆

A→O/G; 2nd hop 080a6bb

8.5k⋆

A→O/G; 2nd hop 56758ed

49.2k⋆

2nd hop O→A/G 8df4a26

37.1k⋆

O/G→A; RQ3 exits; RQ4

v8.0.15 / v8.0.20

–

source audit only

8927af6; 44.2k⋆; 2.2k⋆ 97f5609 live, 2026-10-01 –

distinguishing conv. O→G

39

Table 15: Full units and denominators map. For each result in Table 4, we list the corpus, unit, denominator, paths, and supporting live or human evidence (Table 4, Appendix B.1). Headline claim & value

Unit & N

Corpus

Subjects

1. Primary (Stratum 1): 127/685 (18.5%)

Cell; N =685 (14×51−29)

51 of 357 shapes (S-strict-R9)

6 configs (5 codebases), 14p

2. Live-source (Stratum 2a): Cell; N =182 34/182 (18.7%) (14×13) 3. Opus (Stratum 2b): 23/112 Cell; N =112 (20.5%) (14×8) 4. Live target: Gem. 160/204 Target cell (204 (78.4%); Anth. 77/204 (37.7%) Gem., 204 Anth.)

13 of 51 shapes 6 configs, 14p HTTP 200 (Sonnet) 8 of 51 shapes 6 configs, 14p HTTP 200 (Opus) S-strict-R9 wire (277 8 live paths (4 OpenAI cells checked Gem., 4 Anth.) offline) 5. Candidates & mechs: 54 (translator,rule) S-strict-R9 (685 cells) 5 codebases (31 A/11 B/12 C), 18 mechs (54); 18 mechs (v5/v7 merged) 6. Sensitivity: 12.9%–19.6% Cell; selected Stratum-1 subsets & 4–6 configs (valid.); 16.5%–25.7% (conv.) main-suite reclassifications (8–14p) variants (N =308–685) 7. Mid-turn system: Cell (1,530); trial V4-midsys-R10 (170: 6 configs; 2 1,139/1,530 (74.4%); Case A′ (n=11 or 30) 96/74) + Case A′ paths + ref. (k/11) (live) 8a. Live effects: 9/10 tested Mechanism (10 10 mechanism probes Translated vs. mechanisms (18 catalogued; tested); trials intact requests selected probes 30/30 vs. 0/30) n=30 or 10 per arm 8b. GSM8K control: no Call (700) 100 items, 7 arms gemini-3.5-flashstatistically detectable accuracy lite drop 9. Replies & SSE: 391/1,115 Reply (1,115); 40 reply shapes + 40 7 families (11 (35.1%); 203/558 (36.4%) SSE (N =558) SSE streams reply, 14 SSE) 10. Persistence: 50 LiteLLM releases; 2,630/2,745 (95.8%); 7/26

Release/commit S-strict-R9 + histories V4-strict (170) (50, 54, 32); 2,745 flagged cells 11. Gateways: 187/204 (91.7%) Req. (204, 1,020); S-strict-R9 + 51s; 978/1,020 (95.9%) 170s SSE (198) V4-strict + 66 SSE 12. First-party: 10/11 probes Probe (N =11); 10 11 distinguishing pass trials per arm probes

10 translators, 22p at HEAD

Method and supporting Sec. evidence isloss primary criteria; 333/685 §6.1 static T1; 116/127 finding-aligned mechanism coverage isloss primary criteria; source §6.1 HTTP 200 (claude-sonnet-4-6) isloss primary criteria; source §6.1 HTTP 200 (claude-opus-4-8) Static checks vs. live acceptance; §5.3 44/204 Gemini and 127/204 Anthropic refusals isloss and 4 raters; 31/42=73.8% §6.1 among non-disputed cases isloss under the stated variant §6.1 (49/379 to 134/685 validity; 88/532 to 176/685 counting) isloss; live Gemini (n=11) & Claude (ref. 30/30, 10/30)

§2.1

Task scorers on Gemini, Claude, §6.2 and open-weight models; mechanism coverage is not individual-cell validation Separate control; original-driver §6.2 omission arm is withdrawn from translator-defect claims §6.3 isloss: 278/391 avoidable; independent SDK outcomes: 143 exceptions, 60 silent corruptions Step 2 persistence on the §6.4 extended 170-shape suite; 26 upstream issues (7 fixed)

4 gateways + isloss: 57/198=28.8% avoidable §6.5 CLIProxyAPI stream changes; mock upstream Google Behavioural probes on live §6.5 OpenAI-compat. gemini-3.5-flash-lite; compatibility vs. native API, not a captured-payload isloss verdict 13. Recorded CLI: 4/98 (4.1%) Cell (98 valid, 196 14 CLI (7 statically 6 configs, 14p (3 isloss primary criteria; CLI §6.5 valid; 13/196 (6.6%) all all); 12 sess. contract-valid) CLI agents) captures; 9/12 sessions contain an undeclared meaning change isloss vs. 7 judges; 45/57 bugs; §4 14. Validation: 95/125 (76.0%) Pair (225, 390, 50, Blind, Human (37/40), All 4 dialect & 160/220 (72.7%) blind 57, 51) Bugs, Mutants pairs 43/51 (84.3%) mutants

40

Table 16: Pre-review request mechanisms. The table contains 56 candidates across 19 mechanisms behind the 467 T1 cells in Table 5; review leaves 54 candidates across 18 mechanisms (Appendix C, Figure 4). Cand. counts pre-review candidates. T1 cells counts affected cells once per mechanism; Static counts those whose translated requests pass the static target checks. Static acceptance is distinct from a translator warning or live endpoint acceptance. M/U/Borderline denotes supported, unsupported, or borderline target representation. E/O denotes an explicit rewrite or omission located in the pinned source. Upstream gives covered reports and their status (Appendix B.5). Mechanism (pre-review)

Cand.

T1 cells

Static

M/U/Borderline

Code

Upstream

redacted thinking block dropped tool-result error flag dropped metadata.user_id dropped top_k dropped assistant text after a call moved before it max_tokens dropped parallel-tool-use control dropped top_p dropped id of a Gemini function response dropped stop sequences dropped temperature dropped long tool name shortened forced tool use weakened or dropped result names assigned by position forced tool name shortened temperature injected list-valued tool result re-encoded user text after a tool result dropped list-valued tool result split

9 9 6 6 3 1 5 3 1 2 1 1 2 1 1 1 1 1 2

261 154 92 86 54 51 41 39 37 34 32 26 16 13 10 9 7 7 6

210 92 71 55 54 51 23 29 37 24 32 0 13 13 0 3 7 2 6

1/8/0 3/6/0 4/2/0 1/5/0 3/0/0 1/0/0 3/2/0 3/0/0 1/0/0 2/0/0 1/0/0 0/0/1 2/0/0 1/0/0 0/0/1 0/0/1 0/0/1 1/0/0 2/0/0

O E+O O O E O O E+O O O O E E E E E E O E

3 rep.: 3 merged 8 rep.: 5 merged, 1 ack., 2 open – – – – – – 1 rep.: 1 open – – – 2 rep.: 2 open – – – – 1 rep.: 1 open –

19 mechanisms (pre-review)

56

29/23/4

15 E, 41 O

15 rep.: 8 merged, 1 ack., 6 open

Static denotes offline target-contract acceptance. The catalogue shown has 56 candidates before review; removing one acceptable re-encoding and merging one duplicate leaves 54 candidates across 18 mechanisms.

are not treated as accepted targets, while retaining the 685-cell denominator, and 132/685 (19.3 %) using LiteLLM’s serialised hosted_vllm body. Under this study’s conservative grading, the baseline treats role demotion (D02) as T2; promoting it to T1 gives 134/685 (19.6 %). Counting variants range from 16.5 % to 25.7 %: 88/532 on 11 deduplicated paths, 146/685 when separate findings in a cell may jointly meet the criteria, and 176/685 when decoding controls (K) are included. Silence means that the translator gives no warning about the change; it is independent of static contract validity and live acceptance. The primary rate requires one individual finding to meet all criteria. B.3

Live behavioural consequences, GSM8K control, and schema probes

We test live consequences on gemini-3.5-flash-lite, gemini-3.8-flash, claude-sonnet-4-6, claude-haiku-5-5, Meta-Llama-3.1-8B-Instruct, and Qwen2.5-7B-Instruct. Nine of the ten tested census mechanisms affect at least one measured outcome; a separate probe tests the dropped mid-conversation system instruction. In total, 31 of 64 contrasts survive global Holm correction. The nine mechanisms correspond to findings in 116/127 primary cells (91.3 %); this is mechanism coverage, not individual live validation of those cells. Running translator processes again rather than replaying archived bodies gives the same outcomes in all 9 checked cells (Tables 19–20). The Gemini experiments use 720 plus 120 calls at temperature 0.7. Four mechanisms reduce compliance from 30/30 to 0/30 on both models (pHolm ≤ 0.0117). The Claude experiments use 512 calls without retries. Dropping top_p: 0.05 increases output diversity on Sonnet (d¯ = +1.00, p = 0.0156) and converts Haiku’s deprecation error into HTTP 200 replies (10/10). Raising max_tokens from 100 to 1,124 reduces compliance with the original cap from 30/30 to 0/30 on both models (pHolm = 0.0039). The thinking-block-removal probes do not produce the predicted rejection effect (p = 1.0000). On the first 100 GSM8K items, evaluated in seven arms on gemini-3.5-flash-lite, the translator arms show accuracy differences of 0 to +3 percentage points and no statistically detectable drop (p ≥ 0.2500; Table 21). This experiment does not establish general benchmark or agent performance. Separately, probes 41

avoidable

unavoidable

redacted thinking dropped tool-result error flag dropped metadata.user_id dropped top_k dropped text after a tool call moved max_tokens dropped parallel-tool-use control dropped top_p dropped function-response id dropped stop sequences dropped temperature dropped long tool name shortened forced tool use weakened result names by position forced tool name shortened temperature injected trailing user text dropped list tool result split

one candidate on two paths

→ OpenAI

→ Gemini

29 29 29 29 18 10 18 18 18 18 7 17 17 7 17 17 14 14

29 29 29 29 18 18 18 17 17 17 18 51 9 9 17 37 17 32

9

5

9 17 17

→ Anthropic

T1 cells

29

261 154 92 86 54 51 41 39 37 34 32 26 16 13 10 9 7 6

17 18 18

5

26 5

11 13

10 9 7 3

de

Lit

eL LM -p ro LL xy @m us M-R ose is tu tt di o/ a l l AI m SD s K AI v SD 5 K v7 LL @m us M-R ose is tu tt di o/ a l lm AI SD s K AI v SD 5 K v7 LL @m us M-R ose is tu tt di o/ a l lm AI SD s K AI v SD 5 K v7

3

cl

au d

e-

co

Number in a tile: T1 cells for this mechanism and path

borderline

Figure 8: The 54 post-review candidates across 18 mechanisms. Each tile represents one path and mechanism, coloured by target support (M, U, or Borderline), and gives its affected T1 cells on the 51-shape suite. Right-hand bars aggregate by mechanism. Deduplicating paths leaves 48 candidates on 11 paths. Table 16 records the pre-review source classifications and upstream status; Section 6.1 gives the human grades.

42

Table 17: Sensitivity of the main census to validity and counting choices. N is the number of judged cells and K the number of request-shape clusters; variants (a)–(g) change the assumptions specified in each row. Var.

Specification

N (K)

T1

Static T1

Static + Av. (ledger)

Cell-level criteria

Primary (one finding)

0

Baseline S-strict-R9 (14 paths) Drop redacted_thinking_dropped (O) findings Drop 29 shapes with thinking=redacted Drop 17 temperature+top_p shapes Drop AI SDK v5 (11 paths, 5 codebases) Drop all AI SDK paths (8 paths, 4 projects) Project only: LiteLLM (1 path) Project only: claude-code-proxy (1 path) Project only: LLM-Rosetta (3 paths) Project only: @musistudio/llms (3 paths) Project only: Vercel AI SDK v7 (3 paths; v5 identical) Count T1 ∪ T2 (T2 as documented degradation U) Count T1 ∪ T2 (representable T2 incl. D02 as M) Exclude 44 live-Gemini HTTP 400 cells from the static-based subset Combined (b)+(f1) (34 shapes + live-Gemini rejections) Promote D02 (system→user) to T1 (Vis, M) on S-strict-R9 S-midsys-R9 baseline (70 shapes, D02 as T2) Promote D02 to T1 (Vis, M) on S-midsys-R9

685 (51)

467 (68.18 %)

333 (48.61 %)

206 (30.07 %)

146 (21.31 %)

127 (18.54 %)

685 (51)

340 (49.64 %)

234 (34.16 %)

179 (26.13 %)

146 (21.31 %)

127 (18.54 %)

308 (22)

165 (53.57 %)

101 (32.79 %)

80 (25.97 %)

69 (22.40 %)

60 (19.48 %)

457 (34)

297 (64.99 %)

206 (45.08 %)

125 (27.35 %)

93 (20.35 %)

84 (18.38 %)

532 (51)

370 (69.55 %)

254 (47.74 %)

163 (30.64 %)

107 (20.11 %)

88 (16.54 %)

379 (51)

273 (72.03 %)

175 (46.17 %)

120 (31.66 %)

68 (17.94 %)

49 (12.93 %)

51 (51)

37 (72.55 %)

0 (0.00 %)

0 (0.00 %)

0 (0.00 %)

0 (0.00 %)

22 (22)

21 (95.45 %)

7 (31.82 %)

7 (31.82 %)

6 (27.27 %)

5 (22.73 %)

153 (51)

120 (78.43 %)

100 (65.36 %)

53 (34.64 %)

11 (7.19 %)

0 (0.00 %)

153 (51)

95 (62.09 %)

68 (44.44 %)

60 (39.22 %)

51 (33.33 %)

44 (28.76 %)

153 (51)

97 (63.40 %)

79 (51.63 %)

43 (28.10 %)

39 (25.49 %)

39 (25.49 %)

685 (51)

544 (79.42 %)

402 (58.69 %)

206 (30.07 %)

151 (22.04 %)

127 (18.54 %)

685 (51)

544 (79.42 %)

402 (58.69 %)

275 (40.15 %)

151 (22.04 %)

134 (19.56 %)

685 (51)

467 (68.18 %)

292 (42.63 %)

172 (25.11 %)

112 (16.35 %)

96 (14.01 %)

457 (34)

297 (64.99 %)

177 (38.73 %)

101 (22.10 %)

69 (15.10 %)

62 (13.57 %)

685 (51)

467 (68.18 %)

333 (48.61 %)

206 (30.07 %)

151 (22.04 %)

134 (19.56 %)

856 (70)

595 (69.51 %)

426 (49.77 %)

274 (32.01 %)

211 (24.65 %)

187 (21.85 %)

856 (70)

595 (69.51 %)

426 (49.77 %)

274 (32.01 %)

216 (25.23 %)

194 (22.66 %)

a1

a2 b c1 c2 d1 d2 d3 d4 d5 e1 e2 f1

f2

g1 g2-b g2

Static means offline target-contract acceptance, with the row-specific exclusions stated above. Av. means that the target can represent the affected information. The cell-level column allows different findings in a cell to satisfy different criteria; the primary column requires one finding to satisfy all criteria. Silence means no warning, independently of static or live acceptance. Rows e1/e2 include T2 as specified; D02 role demotion is conservatively T2 in the baseline.

43

Table 18: Request counts under static checks, endpoint acceptance, and mechanism coverage. Stratum 1 is the main suite; Strata 2a–2c are acceptance subsets rather than a nested sequence. Static T1 counts meaning changes whose translated bodies pass static target checks; it does not establish live acceptance or the absence of a warning. The primary column requires one finding to satisfy all criteria. † Removing the LLM-Rosetta-to Gemini list-valued tool-result re-encoding candidate changes the cell-level conjunction to 140/685 (101/532 after deduplication) but leaves the primary counts 127/685 and 88/532 unchanged. ‡ Both-end acceptance covers eight target paths, four Anthropic and four Gemini; OpenAI targets are checked offline only. § The mechanism-coverage row uses the original 685-cell suite: 240/685 cells carry at least one of nine mechanisms with a live-probe effect, not 240 individually live-tested translations. Its percentages use the corresponding Stratum 1 column totals. Finding-aligned coverage of the primary findings is 116/127 (91.34 %); counting a mechanism merely present elsewhere in the same cell would instead give 120/127 and is not the reported primary-coverage definition. Stratum Acceptance or coverage criterion 1 1 (11p) 2a 2a (11p) 2b 2b (11p) 2c

3

N (K)

T1

Static T1 Static + Av.

Primary (one finding)

Cell-level criteria

Static schema/doc contract 685 (51) 467 (68.18 %) 333 (48.61 %) S-strict-R9 (14 paths) Static S-strict-R9, AI SDK v5 532 (51) 370 (69.55 %) 254 (47.74 %) deduplicated (11 paths)

206 (30.07 %) 127 (18.54 %) 146 (21.31 %)† 163 (30.64 %)

88 (16.54 %) 107 (20.11 %)†

Source live HTTP 200 on claude-sonnet-4-6 (14 paths) Source live HTTP 200 on sonnet-4-6, dedup. (11 paths) Source live HTTP 200 on claude-opus-4-8 (14 paths) Source live HTTP 200 on opus-4-8, dedup. (11 paths) Source sonnet-4-6 200 ∧ target live HTTP 200 (8 paths)‡

182 (13)

80 (43.96 %)

51 (28.02 %)

42 (23.08 %)

34 (18.68 %)

39 (21.43 %)

143 (13)

65 (45.45 %)

38 (26.57 %)

31 (21.68 %)

23 (16.08 %)

28 (19.58 %)

112 (8)

58 (51.79 %)

39 (34.82 %)

30 (26.79 %)

23 (20.54 %)

28 (25.00 %)

88 (8)

47 (53.41 %)

29 (32.95 %)

22 (25.00 %)

15 (17.05 %)

20 (22.73 %)

88 (13)

27 (30.68 %)

27 (30.68 %)

26 (29.55 %)

22 (25.00 %)

24 (27.27 %)

Mechanism coverage: carries ≥ 1 of 9 mechanisms with a measured live effect (original suite)§

685 (51) 240 (51.39 %) 166 (49.85 %)

141 (68.45 %) 116 (91.34 %)

132 (90.41 %)

Static means offline target-contract acceptance; silence means no translator warning. The primary column requires the same finding to satisfy all criteria; the cell-level column permits separate findings. The acceptance subsets are not a single nested sequence. Row 3 keeps the original 685-cell denominator: 240/685 (35.04 %) carry a relevant mechanism. Its displayed percentages are conditional on the corresponding Stratum-1 column totals, including finding-aligned coverage of 116/127 (91.34 %) primary cells. This is mechanism coverage, not individual live validation; cell-level co-occurrence would give 120/127.

44

Table 19: Claude consequence probes. Four mechanisms are evaluated on claude-sonnet-4-6 and claude-haiku-5-5 in 512 calls without retries: thinking-block removal and top_p: 0.05 removal (212 calls), dropped tool prohibitions and max_tokens raised from 100 to 1,124 (240 calls), and a thinking-enabled token-cap rerun on Sonnet (60 calls). Each arm uses 10 scenarios, with five samples per scenario for diversity and three replicates for prohibition and token-cap compliance. The prohibition prediction is met on Haiku (d¯ = 1.00, pHolm = 0.0039) but not Sonnet (d¯ = 0.33, pHolm = 0.125). The predicted rejection after thinking-block removal is not observed (10/10 HTTP 200 in both arms). usage.output_tokens_details is a post-smoke-test exploratory indicator and is uninformative on Haiku. The thinking-enabled Haiku rerun is withdrawn after rejection of thinking.type.enabled; the archive records the experiment plans, amendments, and remaining acceptance predictions. Track

Model / mode

MA-1

claude-sonnet-4-6, thinking blocks manual thinking stripped from the (budget 1,024) assistant turn before tool_result claude-sonnet-4-6, same adaptive

MA-2

MA-2

Mutation (translator Ref outcome behaviour)

same

MB-1

claude-haiku-5-5, adaptive claude-sonnet-4-6

MB-2

claude-haiku-5-5

top_p 0.05 dropped

MC3-1

claude-sonnet-4-6

mid-turn [harness] <tool> is DISABLED block (inlined in the last user turn) dropped

top_p 0.05 dropped

Mut outcome

10/10 HTTP 200; 10/10 HTTP 200, output_tokens_details 0 × 400; thinking 0/10; 10/10 output_tokens_details 0/10 10/10 200; 10/10 200; thinking output_tokens_details 0/10; 10/10 output_tokens_details 10/10 10/10 200; thinking 1/10 10/10 200 distinct strings per 5 samples: mean 3.8 (Σ38/10) 10/10 HTTP 400 “`top_p` is deprecated for this model.” 10/30 refuse (no tool_use); 20/30 call the tool, 9 of them stating that an in-chat instruction cannot disable a tool 30/30 refuse

Pre-specified prediction Ref HTTP 200 ≥ 9/10; Mut HTTP 400 ≥ 9/10: not met Mut 200 ≥ 9/10: met

Same criterion: met 4.8 (Σ48/10); d¯ = +1.00 d¯ ≥ 1, p < 0.05: [0.5, 1.5], p = 0.0156 met

10/10 HTTP 200

30/30 call the tool; d¯ = +0.33 [0.10, 0.63], p = 0.125

Ref HTTP 400 ≥ 9/10; Mut HTTP 200 ≥ 9/10: met P-C3-1 (d¯ ≥ 0.5, pHolm < 0.05): not met

30/30 call the tool; d¯ ≥ 0.5, d¯ = +1.00 [1.00, 1.00], pHolm < 0.05: p = 0.0020 not met MC3-2 claude-sonnet-4-6 max_tokens 100 → 30/30 output_tokens = 30/30 > 100 (400–1124, P-C3-2 (d¯ ≥ 0.7, 1124 100, stop max_tokens median 456.5; 11 at the pHolm < 0.05): cap); d¯ = +1.00, met p = 0.0020 MC3-2 claude-haiku-5-5 same 30/30 = 100 30/30 > 100 (697–1124, d¯ ≥ 0.7, median 1124; 22 at the pHolm < 0.05: cap); d¯ = +1.00, met p = 0.0020 Output ≤ 100 30/30 visible ≤ 100 0/30 visible ≤ 100 MC3-2 claude-sonnet-4-6; max_tokens 100 → (thinking) Mut: thinking 1124 (100–100, median 100; 0 (392–1102, median 465.5; visible tokens: (@ai-sdk/anthropic) at 1124 cap; raw 12 at 1124 cap; raw Mut rate ≤ 0.30: enabled output_tokens met; Ref rate output_tokens (budget_tokens 414–1124, median 488); ≥ 0.95: met; 1,024); Ref: no 100–100), stop thinking max_tokens d¯ = +1.00 [1.00, 1.00], HTTP 200 ≥ 99% p = 0.0020 of 60: met MC3-1

claude-haiku-5-5

same

of 40 JSON Schema keywords outside ConvIR’s core rules find that native Gemini parameters accepts 27, while Google’s OpenAI-compatible layer accepts all 40. B.4

Translated reply streams and official SDK failures

Across 1,115 judged reply cells on seven translators (11 reply paths using 40 Anthropic and Gemini backendresponse shapes), isloss reports 391 T1 changes (35.1 %), of which 278 are avoidable. Table 22 gives a 45

Table 20: Counts and Wilson 95 % intervals for the live consequence matrix. Rows cover the nine census mechanisms with measured effects, the separate mid-conversation system-instruction probe, and the tenth tested census mechanism, dropped top_k, whose probes show no difference where the prompts do not bind that setting. Temperature-0 single-prompt Gemini trials (n = 10), temperature-0.7 multi-scenario trials (n = 30 or 60), and Claude trials (Table 19) are reported separately. Sampling and output-limit settings are outside the primary rate but can affect behaviour. Mechanism

Model

Prompt, role, and tool information tool-result error flag dropped gemini-3.5-flash-lite (154 census cells) gemini-3.8-flash Qwen2.5-7B-Instruct gemini-3.5-flash-lite gemini-3.8-flash text after a tool call moved Meta-Llama-3.1-8B-Instruct (54 census cells) Qwen2.5-7B-Instruct function-response id dropped gemini-3.5-flash-lite (37 census cells) gemini-3.8-flash Meta-Llama-3.1-8B-Instruct Qwen2.5-7B-Instruct forced tool use weakened (16 census cells)

result names by position (13 census cells) trailing user text dropped (7 census cells)

mid-turn system dropped (separate test)

gemini-3.5-flash-lite gemini-3.8-flash Qwen2.5-7B-Instruct gemini-3.5-flash-lite gemini-3.8-flash gemini-3.5-flash-lite gemini-3.8-flash gemini-3.5-flash-lite gemini-3.8-flash Meta-Llama-3.1-8B-Instruct Qwen2.5-7B-Instruct gemini-3.5-flash-lite gemini-3.8-flash claude-sonnet-4-6 (MC3-1) claude-haiku-5-5 (MC3-1) Qwen2.5-7B-Instruct gemini-3.5-flash-lite gemini-3.8-flash

T

Intact k/n (%; Wilson 95 % CI) Translated k/n (%; Wilson 95 % CI)

0.7 30/30 (100.0 %; 88.65–100.00) 0/30 (0.0 %; 0.00–11.35) 0.7 30/30 (100.0 %; 88.65–100.00) 0/30 (0.0 %; 0.00–11.35) 0.7 25/60 (41.67 %; 30.06–54.27) 0/60 (0.0 %; 0.00–6.02) 0.0 10/10 (100.0 %; 72.25–100.00) 0/10 (0.0 %; 0.00–27.75) 0.0 10/10 (100.0 %; 72.25–100.00) 10/10 (100.0 %; 72.25–100.00) 0.7 30/30 (100.0 %; 88.65–100.00) 30/30 (100.0 %; 88.65–100.00) 0.7 24/30 (80.0 %; 62.69–90.49) 1/30 (3.33 %; 0.59–16.67) 0.7 30/30 (100.0 %; 88.65–100.00) 0/30 (0.0 %; 0.00–11.35) 0.7 30/30 (100.0 %; 88.65–100.00) 0/30 (0.0 %; 0.00–11.35) 0.7 jinja_400: GGUF chat template rejects multi-tool history (0/30 vs. 0/30) 0.7 jinja_strip: GGUF template strips tool_call_id in both arms (0/30 vs. 0/30) 0.7 30/30 (100.0 %; 88.65–100.00) 3/30 (10.0 %; 3.46–25.62) 0.7 27/30 (90.0 %; 74.38–96.54) 5/30 (16.67 %; 7.34–33.56) 0.7 43/60 (71.67 %; 59.23–81.49) 3/60 (5.0 %; 1.71–13.70) 0.0 10/10 (100.0 %; 72.25–100.00) 0/10 (0.0 %; 0.00–27.75) 0.0 10/10 (100.0 %; 72.25–100.00) 0/10 (0.0 %; 0.00–27.75) 0.7 30/30 (100.0 %; 88.65–100.00) 3/30 (10.0 %; 3.46–25.62) 0.7 30/30 (100.0 %; 88.65–100.00) 0/30 (0.0 %; 0.00–11.35) 0.7 30/30 (100.0 %; 88.65–100.00) 0/30 (0.0 %; 0.00–11.35) 0.7 30/30 (100.0 %; 88.65–100.00) 0/30 (0.0 %; 0.00–11.35) 0.7 30/30 (100.0 %; 88.65–100.00) 0/30 (0.0 %; 0.00–11.35) 0.7 60/60 (100.0 %; 93.98–100.00) 0/60 (0.0 %; 0.00–6.02) 0.0 10/10 (100.0 %; 72.25–100.00) 0/10 (0.0 %; 0.00–27.75) 0.0 10/10 (100.0 %; 72.25–100.00) 0/10 (0.0 %; 0.00–27.75) def. 10/30 (33.33 %; 19.23–51.22) 0/30 (0.0 %; 0.00–11.35); not met def. 30/30 (100.0 %; 88.65–100.00) 0/30 (0.0 %; 0.00–11.35); met 0.7 50/60 (83.33 %; 71.97–90.69) 13/60 (21.67 %; 13.12–33.62) 0.0 10/10 (100.0 %; 72.25–100.00) 0/10 (0.0 %; 0.00–27.75) 0.0 10/10 (100.0 %; 72.25–100.00) 0/10 (0.0 %; 0.00–27.75)

Sampling and output-limit settings (outside the primary rate) max_tokens dropped or raised gemini-3.5-flash-lite 0.7 30/30 (100.0 %; 88.65–100.00) (51 census cells) gemini-3.8-flash 0.7 30/30 (100.0 %; 88.65–100.00) claude-sonnet-4-6 (MC3-2) def. 30/30 (100.0 %; 88.65–100.00)

top_p dropped (39 census cells) stop sequences dropped (34 census cells) top_k dropped (86 occurrences)

claude-haiku-5-5 (MC3-2) Meta-Llama-3.1-8B-Instruct Qwen2.5-7B-Instruct claude-sonnet-4-6 (MB-1) claude-haiku-5-5 (MB-2) gemini-3.5-flash-lite gemini-3.8-flash Meta-Llama-3.1-8B-Instruct Qwen2.5-7B-Instruct open-weight probes

0/30 (0.0 %; 0.00–11.35) 0/30 (0.0 %; 0.00–11.35) 0/30 (0.0 %; 0.00–11.35); plain & thinking def. 30/30 (100.0 %; 88.65–100.00) 0/30 (0.0 %; 0.00–11.35) 0.7 30/30 (100.0 %; 88.65–100.00) 0/30 (0.0 %; 0.00–11.35) 0.7 30/30 (100.0 %; 88.65–100.00) 0/30 (0.0 %; 0.00–11.35) ¯ def. mean 3.8 distinct / 5 (n=50) mean 4.8 / 5 (d=+1.00 [0.5, 1.5], p=0.0156) def. 10/10 HTTP 400 (top_p 10/10 HTTP 200 (rejection deprecated) removed) 0.7 29/30 (96.67 %; 83.33–99.41) 0/30 (0.0 %; 0.00–11.35) 0.7 30/30 (100.0 %; 88.65–100.00) 8/30 (26.67 %; 14.18–44.45) 0.7 23/30 (76.67 %; 59.07–88.21) 1/30 (3.33 %; 0.59–16.67) 0.7 55/60 (91.67 %; 81.93–96.39) 7/60 (11.67 %; 5.77–22.18) 0.0 no difference where prompts did not bind top_k (Table 16)

46

Table 21: GSM8K control experiment on gemini-3.5-flash-lite. Seven arms each contain the first 100 items (700/700 HTTP 200). A1 and A5 use translator outputs. A2 is retained only as a maxOutputTokensomission control because the original capture driver omitted that field; it is withdrawn from translator-defect claims. A3 (aisdk5->gemini) is omitted as byte-identical to A2. Four further arms are reference or intervention controls. Columns report required-format compliance (#### <int>), exact-tag and last-integer accuracy, MAX_TOKENS counts, and mean output tokens. Contrasts with A0 use exact two-sided McNemar tests and paired-bootstrap 95 % intervals for ∆ (10,000 resamples; pre-specified EXP-H). Format % [CI]

Arm Input / control A0

A1 A2

A4

A5 A6 A7

Exact tag % [CI]

Last integer Mean % [CI] MAX tokens

Intact ref. 98 [93.0, 99.4] 90 [82.6, 94.5] 97 [91.5, 99.0] (max_tokens: 1024, temp: 0.0) musistudio->gemini 100 [96.3, 100.0] 91 [83.8, 95.2] 99 [94.6, 99.8] (D02 + S08) aisdk7->gemini: 100 [96.3, 100.0] 91 [83.8, 95.2] 98 [93.0, 99.4] driver-omission control; translator-fault claim withdrawn A0 repeated 99 [94.6, 99.8] 90 [82.6, 94.5] 97 [91.5, 99.0] (nondeterminism floor) rosetta->gemini 100 [96.3, 100.0] 92 [85.0, 95.9] 100 [96.3, 100.0] (clean wire) A0 minus temperature, 100 [96.3, 100.0] 91 [83.8, 95.2] 98 [93.0, 99.4] maxOutputTokens A1 plus temp: 0.0, 99 [94.6, 99.8] 91 [83.8, 95.2] 98 [93.0, 99.4] maxOutputTokens: 1024

∆ Format [CI]

2 254.6

∆ Exact tag ∆ Last integer [CI] [CI]

—

—

—

0 291.8 +2 [+0.0, +5.0] +1 [+0.0, +3.0] +2 [+0.0, +5.0] 0 250.8 +2 [+0.0, +5.0] +1 [+0.0, +3.0] +1 [+0.0, +3.0]

1 250.5 +1 [+0.0, +3.0] +0 [+0.0, +0.0] +0 [+0.0, +0.0] 0 249.9 +2 [+0.0, +5.0] +2 [+0.0, +5.0] +3 [+0.0, +7.0] 0 263.4 +2 [+0.0, +5.0] +1 [+0.0, +3.0] +1 [+0.0, +3.0] 1 269.2 +1 [+0.0, +3.0] +1 [+0.0, +3.0] +1 [+0.0, +3.0]

Exact two-sided McNemar p vs. A0 (c=0 in every contrast): p=1.0000 for ∆ ∈ {+0, +1}, p=0.5000 for ∆= + 2, p=0.2500 for ∆= + 3. A3 (aisdk5->gemini) omitted in the pre-registration as byte-identical to A2 on this prompt shape; A2 is a capture-driver control: the original driver omitted maxOutputTokens, leaving max_tokens absent; its translator-defect claim is withdrawn (Appendix C).

Table 22: Official SDK outcomes for 558 translated HTTP 200 streams. Results are grouped by translator family for openai-python 2.54.0 and openai-node 5.23.2; CLIProxyAPI additionally refuses two streams with HTTP 500. Each Clean/U entry gives two counts: preserved replies / unavoidable losses; the slash does not denote a ratio. The exception/corruption split partitions affected streams into: both SDKs raise; one raises and the other silently corrupts; one raises and the other has neither failure; or both silently corrupt. openai-python 2.54.0

openai-node 5.23.2

Family

N Exc. Corrupt Clean/U Exc. Corrupt Clean/U

Bifrost new-api CLIProxyAPI Portkey one-api LiteLLM LLM-Rosetta

80 80 78 80 80 80 80

2 15 4 2 14 11 4

8 1 13 21 40 5 30

59/11 53/11 50/11 47/10 21/5 57/7 39/7

11 6 9 5 41 14 19

6 3 6 19 17 1 25

All 7 families 558

52

118

326/62

105

77

52/11 60/11 52/11 46/10 17/5 54/11 29/7

Two-SDK union (203/558 = 36.38 %) Fail (%) Any exc. Both corr. 17 (21.25 %) 20 (25.00 %) 17 (21.79 %) 24 (30.00 %) 58 (72.50 %) 23 (28.75 %) 44 (55.00 %)

11 20 11 5 55 22 19

310/66 203 (36.38 %)

143

6 0 6 19 3 1 25

Exc./corr. split 2/2/7/6 1 / 4 / 15 / 0 2/7/2/6 2 / 2 / 1 / 19 0 / 51 / 4 / 3 3 / 4 / 15 / 1 4 / 5 / 10 / 25

60 14 / 75 / 54 / 60

separate test of 558 HTTP 200 streams with the official openai-python 2.54.0 and openai-node 5.23.2 accumulators. At least one SDK raises an exception or silently corrupts the message in 203/558 streams (36.4 %, Wilson 95 % CI 32.5–40.5 %); 199 of those 203 outcomes are avoidable (98.0 %). Python is affected on 170 streams (52 exceptions and 118 silent corruptions), and Node on 182 (105 exceptions and 77 silent corruptions). The SDK outcomes disagree on 129/558 streams. B.5

Persistence, API endpoints, round-trip controls, and upstream fixes

The release study covers 50 LiteLLM releases (1.52.16–1.104.0), 54 @musistudio/llms releases, and 32 claude-code-proxy commits. Every tested LiteLLM release has T1 findings on at least 31/51 main-suite shapes when translating to OpenAI. A separate re-census uses the 11 October source snapshots of ten 47

Table 23: The same translator through Chat Completions (c) and Responses (R). Counts describe outgoing properties, not isloss verdicts. The first column gives applicable shapes, or blocks for is_error; AI SDK uses k/n where its core rejects inputs before either endpoint runs. B permits allowSystemInMessages; c marks properties affected by the original capture driver. † denotes renamed or truncated tool names restored on return; ‡ denotes role:system items without type, counted as retained only where the track supplies that key; § denotes another representation (stop or top-level user). For text surrounding a call: J joins it into one string, P retains separate parts, S retains separate messages/items, and arrows indicate movement across the call. The bottom row is a same-vendor control: unchanged bodies after removing model/stream and sorting keys; h marks post-hoc AI SDK normalisation. LiteLLM LLM-Rosetta CLIProxyAPI Property (applicable) Top-level system prompt kept (239) Mid-turn system: role and place kept (236) Mid-turn system: moved behind the next tool result (76) Thinking text kept (121) Signature (121), redacted data (116), is_error trace (146) Text before and after a call (64) 70-character tool name unchanged (122) max_tokens kept (357) temperature kept (234) top_p kept (115) top_k kept (115) stop_sequences kept (115) metadata.user_id kept (115) List tool result flattened to one string (87) Same vendor: body unchanged (357)

c

R

c

R

239 239 239 0 236 160 76 0 76 121 121 121 0 0 0 J → J 0† 122 122 357 357 357 234 234 234 115 115 115 115 0 0 115 0 115§ 0 115§ 0 0 87 0 280

239 236 0 0 0 ← 122 357 234 115 0 0 0 0 –

c

R

239 0 76 0 0 P 122 357 234 0 0 115 0 87

239 0 76 0 0 S 0† 0 0 0 0 0 0 0 28

Bifrost c

AI SDK

R

c

R

‡

239 0 79/79 79/79‡ 236 236 160/160B 160/160B 0 0 rej. rej. 121 0 0/40 0/40 0 0 0 0 S S J S 122 122 37/37 37/37 357 357 0/121c 0/121c 234 234 77/77 77/77 115 115 38/38 38/38 0 0 0/38 0/38 0 0 38/38 0/38 0 115§ 0/38c 0/38c 0 87 25/25c 15/25c 357 / 0

0/121h

translators across 22 paths and the extended 170-shape corpus. At Step 2, 2,630/2,745 previously flagged T1 cells persist (95.8 %). After the Step 3 warning filter, 2,495/2,631 persist (94.8 %); this latter count comprises 1,517/1,653 library cells (91.8 %) and all gateway cells. The two stages therefore lose 115 and 136 cells, respectively, and should not be combined. Table 23 compares Chat Completions and Responses on 357 inputs across five translators. Each translator’s two endpoints differ on at least one of 22 pre-specified properties. Same-vendor round-trip controls also reveal a change in every tested translator. Table 24 records 44 upstream posts across 12 repositories: 26 issues, 10 comments on existing threads, and 8 follow-ups. As of 15 October, 7 issues are fixed and re-verified, 1 is partly fixed, 1 is closed without reply, and 17 remain open. The endpoint comparison uses LiteLLM 4417bf08 (1.106.0), LLM-Rosetta 0.13.0, CLIProxyAPI v8.0.20, Bifrost 8b4fce4f, and AI SDK ai 7.0.126 with @ai-sdk/openai 4.0.83. AI SDK uses the original capture driver, which omits maxOutputTokens, passes metadata.user_id only to Anthropic, and joins 15/25 listvalued results; the chat endpoint sends the other 10 as JSON part lists. Its signature, redacted-block, and error-flag denominators are 40, 45, and 57. These captures are separate from the corrected main-census driver, which passes maxOutputTokens and providerOptions.anthropic.user (Appendix C). AI SDK rejects 236 mid-conversation-system shapes, leaving 121; enabling allowSystemInMessages lets 281 through. From @ai-sdk/openai 4.0.89, both endpoints retain the error flag on 57/57 cases. The text-placement comparison applies to 64 shapes (18 for AI SDK). All ten endpoints map every tool_choice and retain dotted tool names; Bifrost Responses adds an empty reasoning item per redacted block. The same-vendor controls use LiteLLM’s first-party Anthropic path, Bifrost’s raw Claude Code User-Agent path versus its default, and AI SDK’s Anthropic endpoint. New-api and Portkey, outside the table, preserve all 357 bodies in this control. B.6

Gateways, two-hop translation, first-party compatibility, and agent sessions

Table 25 compares a six-gateway source audit with captures from four self-hosted gateways. On the 51-shape suite, Bifrost and new-api have T1 findings in 187/204 judged cells (91.7 %): 103 translated requests pass the static target checks and 84 fail them. Static failure is not an observed endpoint error, and the combined rate 48

Table 24: Upstream reports and verification status in October 2026. Fixed means that our rerun verifies a merged commit or release. No reply means no response to our post. Queued drafts are excluded. The anonymous build coarsens issue numbers, per-project counts, and filing-date ranges. Project

Issues

Comments

Window

State in the thread

LiteLLM

≥1

≥1

Oct. 2026

Vercel AI SDK LLM-Rosetta @musistudio/llms claude-code-router claude-code-proxy @the-next-ai/ai-gateway CLIProxyAPI

≥1 ≥1 ≥1 – ≥1 ≥1 ≥1

≥1 ≥1 – ≥1 – – ≥1

Oct. 2026 Oct. 2026 Oct. 2026 Oct. 2026 Oct. 2026 Oct. 2026 Oct. 2026

Bifrost

≥1

≥1

Oct. 2026

Portkey new-api one-api

≥1 ≥1 ≥1

≥1 ≥1 ≥1

Oct. 2026 Oct. 2026 Oct. 2026

3 competing fix PRs on 1 issue; fix PR open (verified by us on the PR head) on 1; others no reply fixed, released in ai 7.0.132; re-run by us 3 issues fixed on master; re-run by us no reply no reply no reply no reply 2 fixed, 1 partly; re-run by us on v8.0.20; 1 later issue no reply 1 fixed, merged; re-run by us on the merge commit; others no reply no reply issue closed; comment on an earlier report of the same defect open PR updated after our comment; issue no reply

Total (12 repositories)

26

18

Oct. 2026

7 fixed, 1 partly fixed, 1 closed, 17 open

is not the primary silent, avoidable rate. Portkey 1.15.2 refuses all tested cells with HTTP 500. On the extended 170-shape corpus, Bifrost, new-api, and CLIProxyAPI have T1 findings in 978/1,020 judged cells (95.9 %). For forward reply streams, isloss reports avoidable T1 changes in 57/198 judged streams (28.79 %); the Bifrost/new-api subset is 42/132 (31.82 %). Table 26 evaluates 24 two-hop library chains. Eight chains with AI SDK as the first hop retain original-driver captures and are auxiliary analyses. On the 16-chain subset without AI SDK, 660/689 statically valid-source cells have end-to-end T1 findings (95.79 %), including 287/689 two-hop-only findings (41.65 %). Operational probes also find Bifrost routing bare model names across providers after upstream 400/429/529 errors while returning HTTP 200, forwarding client authentication headers during fallback, and closing mid-stream upstream errors with HTTP 200. Google’s first-party OpenAI-compatible Gemini endpoint passes 10/11 behavioural probes (n = 10 trials per arm); the remaining probe indicates that only the last of multiple top-level system messages is retained (S01). Reasoning tokens consume 98.0 % of output tokens across 45 calls and count against max_tokens. Missing thought_signature causes 5/5 tool round-trips to be refused, and streamed tool calls use finish_reason:"stop". Replaying the 14 recorded Anthropic coding-agent shapes across 14 translation paths gives 4/98 primary cells for statically valid sources (4.1 %) and 13/196 across all shapes (6.6 %). The latter is 13/147 (8.8 %) if refused translations are excluded. CLIProxyAPI strips gemini-cli’s system prompt in 4/4 cases and disables Gemini safety filters without logging. In the closed-loop multi-turn experiments, 9/12 CLI sessions contain a T1 change.

C

Evaluation Protocol and Audit Summary

This appendix records the scope of each evaluation (Table 27), the outcomes of pre-specified predictions (Tables 28 and 29), and corrections and sensitivity checks (Table 30; additional variants in Table 17).We report prediction outcomes separately for each experiment, using its original criteria and evaluation scope. The plans include criteria recorded in internal commits; we do not imply that every prediction was externally registered. Request-census audit record. Six audits of the harness, driver and human review retain the baseline primary count of 127/685. Human review confirms the Gemini tool-response defect in @musistudio/llms and accepts a list-valued tool-result re-encoding; neither decision changes that primary count. The candidate removal and duplicate merge are detailed in Table 30. In the motivating case, LiteLLM’s T1 finding concerns inserted wrapper text; the role-demotion record is separate, while the full rewrite breaks compliance in one 49

Table 25: Source audit and captured gateway behaviour. Nine patterns recur in at least three independent codebases (P1–P9); runtime tests refute two further audit items (R9, R14). Each cell gives the audit mark followed by the run mark. Audit: • present, ◦ absent, – no applicable path/field, ? unresolved or another change. Run: C confirmed, R refuted, + additional behaviour (noted below), · not exercised or not triggered. Pattern (rules) audited commit P1 P2 P3 P4 P5 P6 P7 P8 P9

Portkey Kong new-api one-api Bifrost Envoy 669825c 8927af6 56758ed 8df4a26 080a6bb 97f5609

Forged Gemini thought signature (V10) Inconsistent mid-conversation system handling (V01/V02/V05) Tool-result error status dropped or mis-mapped (V03) Gemini tool-call/result identifiers dropped (V06) Finish/stop reason mapping (RV01/RV06) Streamed tool-call identity/index mapping (RV02/RV03) Assistant text accompanying tool calls is dropped on replay (V05) Separate stream and non-stream translation paths Gaps in translation test coverage

Other audit items that a run refuted R9 Thinking in history R14 Tool name/schema

◦C •C

◦· •·

•R ?+

– •C

•C •C

•· •·

–

–

•C

–

•C

•·

•C •· •·

•· •· •·

◦C ?· ◦·

?· •· •·

•· •· ?·

•· ?· •·

•C

•·

• C+

• C+

?·

?·

•· •·

– •·

•· •·

•· •·

?· •·

•· •·

•C •C

•· ?·

•C • C+

•C •C

◦R •R

?· ?·

R: new-api P1: on by default, not opt-in as audited, and also attached to text parts (333 of 333 Gemini cells with a function call carry it); Bifrost R9: audited as handled, but Anthropic-in to OpenAI keeps the reasoning text and drops its signature and the redacted thinking, and the other paths drop thinking entirely; Bifrost R14: the audited rewriting of regex patterns was not observed; schemas passed verbatim. +: new-api P2 (audit undecided): positional information lost but no text loss; new-api P7: a fabricated "..." text block on Anthropic; one-api P7: tool calls also dropped when the assistant content is null; new-api R14: every Gemini property schema emptied to {}.

Table 26: Two-hop translation judged with isloss. Rows aggregate 24 library chains by second hop and final target. Eight AI SDK v5/v7 first-hop chains use original-driver captures. The 16-chain subset without AI SDK has 4,357 judged and 1,355 refused cells on all 357 shapes, with 1,609/4,357 two-hop-only T1 (36.93 %) and 4,307/4,357 end-to-end T1 (98.85 %); on the 51 statically valid source shapes, it has 689 judged and 127 refused cells, with 287/689 two-hop-only T1 (41.65 %) and 660/689 end-to-end T1 (95.79 %). The full table includes all 24 chains: 6,269 judged/2,299 refused cells in the upper panel and 1,097/127 in the lower panel. Two-hop-only includes pure two-hop findings and mixed-with-cancellation cases; end-to-end counts cells with at least one end-to-end T1 finding. Gateway second hops are recorded separately in the replication archive. Judged No end-to-end (refused) T1 Carry

Inject

Accum.

Cancel

S_all_357 (24 chains; includes original-driver AI SDK captures) LiteLLM Anthropic 1,585 (557) 0 0 LiteLLM Gemini 1,514 (628) 16 308 LLM-Rosetta Anthropic 1,585 (557) 18 250 LLM-Rosetta Gemini 1,585 (557) 16 346 all 6,269 (2,299) 50 904

70 70 60 73 273

969 607 693 669 2,938

30 67 18 0 115

S_strict_R9_51 (24 chains; includes original-driver AI SDK captures) LiteLLM Anthropic 277 (29) 0 0 17 LiteLLM Gemini 266 (40) 9 86 6 LLM-Rosetta Anthropic 277 (29) 11 73 6 LLM-Rosetta Gemini 277 (29) 9 105 7 all 1,097 (127) 29 264 36

163 37 90 35 325

0 12 0 0 12

Second hop

→Target

Two-hop-only T1 pure+mixed = total (%)

End-to-end T1 (%)

469+47 = 516 (32.56%) 1,585 (100.00%) 406+40 = 446 (29.46%) 1,498 (98.94%) 544+2 = 546 (34.45%) 1,567 (98.86%) 481+0 = 481 (30.35%) 1,569 (98.99%) 1,900+89 = 1,989 (31.73%) 6,219 (99.20%) 97+0 = 97 (35.02%) 108+8 = 116 (43.61%) 97+0 = 97 (35.02%) 121+0 = 121 (43.68%) 423+8 = 431 (39.29%)

277 (100.00%) 257 (96.62%) 266 (96.03%) 268 (96.75%) 1,068 (97.36%)

Auxiliary isloss request analysis: eight of the 24 chains use AI SDK first-hop captures from the original driver, which did not pass all controls. The caption reports the 16-chain subset without AI SDK separately. The last two columns count two-hop-only T1 (pure plus mixed-with-cancellation cases) and any end-to-end T1.

50

Table 27: Evidence scope and evaluation protocol. Fixing the primary rules before held-out test design separates evaluation from rule development; it does not assert that every implementation detail was fixed before every experiment. Development diagnostics are labelled separately from final measurements in Table 12; detailed repair comparisons remain in the archive. Per-result corpora and denominators are in Table 15. Evidence

Protocol and scope

Interpretation and location

Preservation rules

The main checking framework has 27 rules, grouped as T1, T2 and T3. The primary rules were fixed before the main held-out sets were authored. LLM agents designed Sets 9–14 from the specification without access to checker code. Report activated faults and allowed-change controls separately. Earlier rule calibration (in-sample Sets 1–8) and exploratory post-freeze repairs (evaluated on Sets 17–20; script-generated Sets 15–16 voided) use separate batches. A set used to devise a repair cannot independently evaluate that repair; only subsequently authored blind sets can do so. Use the specified capture points and corrected drivers; distinguish completed translations, individual findings, root-cause candidates and mechanisms. Reconcile parser, harness and review errors separately from alternative validity or counting policies.

Rule mappings: Table 10; request results: Section 6; replies: Appendix B.4.

Primary held-out evaluation Development and repair analyses

Census and human review Corrections and sensitivity checks

Independent detection and false-alarm estimates, not development-set performance (Table 12; Appendices A.1 and A.3). Historical comparisons remain separate in Appendix A.3 and the replication archive. They are not additional isloss results.

Request census, behavioural probes and deployments have different units (Sections 6.1 and 6.5; Table 15). Table 30 identifies corrected preliminary counts and sensitivity bounds; Table 17 reports alternative choices.

tested model–placement pair. Degradation and re-encoding records remain informational: legitimate negative controls can trigger them, so they are not counted as defect detections. All main-text checker results use isloss. The primary checking rules were fixed before the held-out test sets were authored; development and repair comparisons remain separate from these estimates. Live behavioural trials and official-SDK stream tests measure outcomes independently of the checker. A stream-reassembly audit corrected preliminary counts from 402 to 391/1,115 reverse-reply T1 cells, from 61 to 57/198 gateway-stream T1 cells, and from 12 to 9/12 affected agent sessions. The reported main results use the corrected counts; the request census and primary held-out results are unchanged. Separate SDK replays expose two remaining stream-input gaps: unread OpenAI SSE delta.refusal fields in three forward streams, and Gemini thoughtSignature comparison before adjacent text coalescing in four. Sensitivity analysis bounds gateway-stream T1 at 57 to at most 64/198 (32.32 %); comparison with non-streaming replies gives 61/198 (30.81 %). The corresponding reverse-reply sensitivity count is 395/1,115 (35.4 %), with 278 avoidable cells unchanged. These bounds do not replace the reported counts of 57/198 and 391/1,115. Headline tables use the corrected AI SDK driver. Four subsidiary analyses retain initial-driver captures and are labelled accordingly: the withdrawn GSM8K arm A2, the AI SDK columns of Table 23, 8 of 24 chains in Table 26, and a 62-mutant AST check. Temperature-zero live probes and single-run LLM judges retain decoding variance. One of 36 archived vendor statements changed by 11 October without overturning a rule. Safety-sensitive and operational gateway findings (Appendix B.6) were disclosed privately before public reporting; no separate private-channel remediation had been received by 15 October 2026. Four volunteer raters participated with informed consent, reviewing synthetic software outputs containing no personal data. The scorecard counts each numbered prediction at the finest unit scored separately in its results file. An outcome marked “mixed” or “partly met” counts as missed. A restated prediction replaces its original; repeated scoring does not add an item, and an explicit agreement prediction counts once. Predictions carried forward from earlier plans retain their full denominator. Items explicitly marked not evaluable or not applicable are excluded and identified in the Held column; exploratory items without a pass/fail threshold are excluded. Where one plan covers several rows, each prediction is assigned once to the row it addresses. Descriptions are shortened for readability; the numerical thresholds, observed outcomes and scoring decisions follow the recorded plans. Each row retains its original evaluation scope.

51

Table 28: Pre-specified prediction outcomes, part 1 of 2 (RQ1–RQ2). Part 2 is Table 29. Track / experiment

Held

Scope

RQ1 mechanism concentration

1/4

51 shapes; 14 paths

RQ1 K-class count

candidate

2/3

56 pre-review candidates

RQ1 upstream report coverage RQ1 replies (reverse, forward) & SDKs

2/3 9/12

56 pre-review candidates 1,115 cells / 558 streams / 61 forward reply shapes

RQ1 live vendor acceptance (C1)

2/5, 1 n/e

548 calls

RQ2 T =0.7 consequence

13/16

4 models

RQ2 Claude consequence (C2, C3)

4/5, 2/3, 3/3

512 calls

RQ2 GSM8K evaluation (EXP-H)

2/6

700 calls

Missed predictions (pre-specified threshold → observed) “after removing the top-3 most pervasive mechanisms, the T1 share of judged cells stays ≥ 60 % (from 85.6 %) and the share of 4,998 stays ≥ 45 % (from 64.0 %)” → judged 32.8 % CI [28.5, 37.2]; of 714 31.5 % CI [27.3, 35.7]; “the top-3 by pervasiveness are exactly {is_error_dropped, control_max_tokens_dropped, redacted_thinking_dropped} (any order)” → redacted_thinking_dropped, is_error_dropped, control_user_id_dropped; “binary real-request estimate averaged over the 14 paths: ≥ 50 % with the max_tokens mechanisms, ≤ 50 % without them” → with 33.2 %, without 32.1 %. “Among the 56 path × mechanism candidates, V accounts for ≥ 20 candidates and K accounts for ≥ 15 candidates” → V = 27, K = 14. “≥ 40 of 56 candidates have a U-id (we reported them), and ≤ 10 of those are fixed today” → 15 reported, 8 fixed. “>= 4/7 families (T1 S12 STOP: Gemini finishReason: "STOP" + functionCall translated to OpenAI finish_reason == "stop" while tool_calls are present)” → 3/7 families (rosetta, oneapi, portkey); “>= 2/5 HTTP proxy families (T1 S13 STREAM: Missing terminal event / [DONE] or broken stream ledger)” → 0/5 HTTP proxy families fire S13 under default context (no non_stream_body); “B4c — Forward non-stream ranking preservation: Under isloss (side="response"), aisdk5 and litellm have strictly higher non-stream T1-clean cell rates (0 T1 findings) than rosetta, musistudio, and fuergaosi233 across spec-compliant backend response shapes” → nonstream T1clean rates: aisdk=80.3%, aisdk5=80.3%, fuergaosi233=85.2%, litellm=91.8%, musistudio=49.2%, rosetta=62.3%. “claude-sonnet-4-6 accepts all 34 strict shapes that do not set temperature and top_p together (34/34)” → 13/34; “→Gemini agreement (valid → 200, invalid → 4xx) ≥ 90%” → 160/204; “Restated: The 19 midsys-only shapes (which all carry role: system inside messages[]) are rejected by both models (≥ 18/19 each)” → 19/19 & 16/19 (sonnet 19/19 (≥18 met), opus 16/19 (<18 not met)). “M3: is_error_dropped: Mean scenario-level difference d¯M3 ≥ 0.60, with di > 0 on ≥ 10/12 scenarios and pperm < 0.005” → mean_diff=0.4167, pos=5/12, p_perm=0.062500; “On Meta-Llama-3.1-8B-Instruct, at least 4 / 5 mechanisms achieve pHolm < 0.05 with d¯ ≥ +30.0 pp” → 3/5 mechanisms; “New mechanism N2 (function_response_id_dropped, 37 silent cells) produces d¯ ≥ +50.0 pp and pHolm < 0.05 on both Llama-3.1-8B and Qwen2.5-7B” → Llama-3.1-8B: +0.00 pp (p_Holm=1.0); Qwen2.57B: +0.00 pp (p_Holm=1.0). “Mut arm returns HTTP 400 ≥ 9/10 AND Ref arm returns HTTP 200 ≥ 9/10” → False (Mut 400: 0, Ref 200: 10). Alt: Mut 200: 10, Mut thinking: 0; “Both models produce d¯ ≥ +0.50 and pHolm < 0.05” → FAIL. “Compliance Drop: A1’s format compliance drops by ≥ 15 points relative to A0” → A1 format compliance increased by +2 pts; “D02 Consequence: A7 vs A0 compliance drop ≥ 10 points (D02 lowering alone impacts compliance significantly)” → A7 vs A0 compliance increased by +1 pts; “S08 Consequence: A6 vs A0 compliance stays within the A4 noise band (S08 alone does not move compliance)” → A6 vs A0 compliance +2 pts vs A4 band +1 pts; within ±3 points; McNemar p=0.500; “Length Consequence: finishReason == "MAX_TOKENS" count is 0 in A0 and 0 in A2 → the cap is not binding on GSM8K and the S08 max_tokens drop is inert for this task” → (A0 MAX_TOKENS=2 on items 2, 40; A2=0) → S08 max_tokens drop has consequences on 2/100 items (both flip to compliant, item 40 flips to primary=True).

Editorial convention: checker names and version qualifiers within quoted predictions are normalised to isloss; the numerical thresholds, recorded outcomes and held/missed decisions are unchanged. Counts use the original registered scopes. The 56-card catalogue precedes the final 54-candidate reconciliation; candidate-level report counts and individual-cell audits must not be substituted for final issue-level totals or catalogue precision. “Held” measures registered prediction outcomes, not checker accuracy.

52

Table 29: Pre-specified prediction outcomes, part 2 of 2 (RQ3–RQ5). Columns and counting rules match Table 28. Track / experiment

Held

Scope

Missed predictions (pre-specified threshold → observed)

RQ3 LiteLLM 50-release timeline RQ3 five-library two-exits

3/11, 1 n/e

50 rel.

161/187

5 libs

RQ3 same-vendor controls

764/804, 2 n/a

5 gw/libs

1/2

80 items

RQ4 chains / gateways / agents

21/39, 1 n/e

4 gw + CLI

RQ4 fallback & accounting

414/500

Bifrost/LiteLLM

RQ5 isloss & recall

45/59, 2 n/e

Sets 1–14, bugs

RQ5 human precision

3/7, 3/4, 5/5

300 items + 48 bridge items

8 of 11 missed (1 n/e): 50-release timeline 8/9 missed (1 n/e), isloss recount 0/2 missed. 26 of 187 missed: LiteLLM Responses 4/22 and Chat sideby-side 1/18, LLM-Rosetta 1/32, CLIProxyAPI 3/28, Bifrost 1/22 and #8209 check 0/5, AI SDK run A 0/23, run B (allowSystemInMessages) 12/14, version-equality 1/1, and GA follow-up 3/22. 40 of 804 missed (2 n/a): Anthropic request round-trip 3/192 and follow-up A 1/86, Anthropic response 6/110 (1 n/a) and follow-ups B 0/49, C 5/88 (1 n/a), D 1/58, E 3/79, OpenAI response 13/92, and OpenAI request 8/50. “3-class Cohen’s κ: Unweighted 3-class Cohen’s κ (exact vs related vs unrelated/none) across the 80 audited items is >= 0.60” → κ = 0.5446 (57/80 = 71.25%; binary exact vs not-exact κ = 0.6500). 18 of 39 missed (1 n/e): CLI coding-agent round-trip 0/4, forward SSE stream census 3/4, Responses ingress census 0/4, self-hosted gateway census 14/26 (1 n/e), and 2-hop chains 1/1. 86 of 500 missed: LiteLLM main two-exit re-run 0/3; 42 Bifrost/LiteLLM routing, fallback, key, budget, and accounting addenda 86/497 missed (37 single-tree addenda 78/459 missed; 5 multi-tree accounting addenda 8/38 missed: CE 0/8, CF 2/10, CG 2/5, CH 0/8, CI 4/7). 14 of 59 missed (2 n/e): isloss in-sample Sets 1–8 3/15, blind held-out Sets 9–10 0/3, Sets 11–12 1/5 (1 n/e), Sets 13–14 1/6 (1 n/e), corpus recount 6/7, historical bug replay of sample A 1/5 and of sample B–C 1/6, 50-bug expansion 0/4, reverse AST mutants 0/4, and open-weight spec judge 1/4. Consensus candidate-card precision A/(A + B) ≥ 85% over the original 56-card catalogue → 72.7% (32 A, 12 B, 12 C/disputed); both raters assign Grade B to the sole retained injected-default candidate (fuergaosi233→openai, control_temperature_injected) → Grade C (disputed); in the 60-cell audit sample (45 T1 cells + 15 non-T1 preserved controls), consensus T1-cell precision A/(A + B) ≥ 90% and Grade A falsepositive rate on the 15 preserved controls ≤ 25% per rater → cell precision 75.0% (33 A, 11 B, 1 C), preserved-control Grade A rate 0.0% (0/15) for both raters; consensus Grade A on ≥ 70% of the 23 U-class (unrepresentable) cards → 6/23 (26.1%, all cards in denominator); B2 T1-cell Grade A share in 70–90% → A/150 = 97.3% (A/(A + B) = 98.0%, CI [94.3%, 99.3%]).

RQ3 corroboration κ

Editorial convention: checker names and version qualifiers within quoted predictions are normalised to isloss; the numerical thresholds, recorded outcomes and held/missed decisions are unchanged. Counts use the original registered scopes. The 56-card catalogue precedes the final 54-candidate reconciliation; candidate-level report counts and individual-cell audits must not be substituted for final issue-level totals or catalogue precision. “Held” measures registered prediction outcomes, not checker accuracy.

53

Table 30: Audit corrections, retained limitations and sensitivity checks. The baseline primary count remains 127/685 (18.54 %; 88/532, 16.54 %, on 11 deduplicated paths). The request study has 714 attempts, including 29 refusals, and 685 judged translations. Corrected preliminary counts and alternative capture or counting choices are distinguished below. Per-result scope is in Table 15; other choices are in Table 17. Item

Audit finding or correction

Effect on results

Location

Motivating-example placement

The initial Case A placed system directly before user: this placement was absent from the 10 recorded occurrences in 36 captured requests and failed static beta rules. The main example instead uses Case A′ placements before assistant (3/10 occurrences) and at messages[-1] (7/10); all 22 paraphrased requests pass static checks. Case A remains an auxiliary experiment (k/10, temperature 0, including token-limit stops; LiteLLM contrast pHolm = 0.012 within the experiment and 0.10 over 64 contrasts). Design C3 has 240 calls across the instruction-loss (MC3-1) and token-limit (MC3-2) experiments. The instruction-loss experiment comprises 120 calls: 10 scenarios × 3 repeats × 2 arms × 2 models. Its notice is the first text block in the last user turn, retained in Ref and dropped in Mut (max_tokens=1,000; no role:system, beta header or sampling controls). Haiku refusals are 30/30 vs. 0/30 (d¯ = +1.00, pHolm = 0.0039, prediction met); Sonnet refusals are 10/30 vs. 0/30 (d¯ = +0.333, CI [0.10, 0.63], p = 0.125, prediction not met; 9/20 Ref tool calls mention user-turn placement). Case A rejection and Case A′ validity under the mid-conversation-system-2026-04-07 beta rules are static-validator results, not observations of a live beta-header experiment. The live Sonnet C1 run omitted that header and rejected 19/19 role:system shapes. Separately, among 29 shapes carrying redacted_thinking, live rejections comprise 19 thinking, 9 sampling-combination and 1 empty error-result rejection; the associated LLM-Rosetta comparison has 19/38 flips. This 29-shape subset differs from the full source-acceptance rejection count. These sets were produced by a fuzzer script rather than an independently tasked LLM agent without checker-code access, and were excluded before scoring the held-out summary. Diagnostic counts are retained for transparency: Set 15 has 48 fault mutants, including 4 uncaught script exceptions (40/48 = 83.3 % if crashes count as detections; 36/48 = 75.0 % otherwise); Set 16 has 36 allowed changes, with 1/36 false alarms (2.8 %). Removed LLM-Rosetta to Gemini, list-valued tool-result re-encoding; 7 statically accepted cells, the only consensus Grade B candidate classified Borderline. Merged duplicate cards AI SDK v7/v5 to Anthropic, moved assistant text; 18 statically accepted cells each, Grade A. The avoidable class has 28 cards: 23 A, 1 B and 4 C; confirmation is 23/24 (95.8 %) among agreed cases or 23/28 (82.1 %) overall. The earlier tally had incorrectly subtracted that Borderline candidate.

Primary rate unchanged: none of the 51 main-suite shapes has a mid-conversation system message. Case A′ reports k/11 for two Gemini models and Qwen2.5-7B; on gemini-3.5-flash-lite, deletion reduces refusals from 11/11 to 0/11 in both placements.

§2.1; App. A.1

Claude instruction-loss probe

Static checks and live rejection

Voided Sets 15–16

Human-review reconciliation

Stream-reassembly audit

Remaining stream-input gaps

Both models are reported §2.1; §6.2; alongside the separate Gemini and Table 19 Qwen system-role placement experiments. Request-census counts are unchanged.

Wording correction only; no headline count changes.

App. A.1; App. A.2

Both sets are excluded. Primary Sets 9–14 give 95/125 recall and 2/100 false alarms; pooling non-voided Sets 9–14 and 17–20 gives 160/220 (72.7 %) and 5/170 (2.9 %).

§4; App. A.3

Catalogue: 56 to 54 candidates (31 A, 11 B, 12 C; 49 to 48 on 11 paths), and 19 to 18 mechanisms. Primary counts stay 127/685 and 88/532. Removing the accepted finding gives audit-adjusted T1 cells 467 to 464, statically accepted T1 cells 333 to 330, and the alternative cell-level conjunction 146 to 140/685 (20.44 %); these are not changes to the primary finding-level rate. A nine-line correction to _parse_openai_chat_stream All 714 request attempts (685 assembles delta.reasoning_content as the judged translations) and Sets 9–14 non-stream parser does. We rechecked 41 affected are unchanged. Preliminary reply streams. reverse-reply T1: 402 to 391/1,115 (35.1 %); gateway-stream T1: 61 to 57/198 (28.8 %); affected agent sessions: 12 to 9/12. Main results use the corrected counts. SDK replay exposed unread OpenAI SSE Request census unchanged. delta.refusal (3 forward streams) and Gemini Sensitivity: gateway-stream T1 is thoughtSignature comparison before adjacent text at most 64/198 (32.3 %; 61/198, coalescing (4 forward streams). 30.8 %, against the non-stream reply); reverse-reply T1 is 395/1,115 (35.4 %), avoidable 278 unchanged. These do not replace the reported 57/198 and 391/1,115 counts.

§4; §6.1; App. C

§4.1; §6.3; §6.5; App. C

§6.5; App. C; App. C

Continued on the next page

54

Table 30, continued Item

Audit finding or correction

Effect on results

Location

Headline tables use the corrected §4.1; §6.2; driver. Preliminary AI SDK T1 App. A.3; App. C counts fall from 240 to 210 and then 194/306; candidates from 27 to 21 and then 15. The 16-chain slice excluding AI SDK has 660/689 T1 cells (95.8 %). Initial-driver subsidiary analyses are listed above. AI SDK forward-reply The harness supplied boolean "_sig":true instead Request and reverse-reply counts App. B.4; App. C unchanged. Restoring the metadata of providerMetadata.google.thoughtSignature, producing S03 signature-loss findings on 8/61 whole metadata raises forward replies replies and 35/548 streams for each AI SDK without T1 from 49 to 57/61 whole configuration. replies and from 440 to 475/548 streams (86.7 %). LiteLLM All 37/51 LiteLLM T1 cells fail static target checks Baseline dictionary capture: 0/51 §6.1; App. B.1; capture-point at dictionary capture: thinking_blocks remains in statically accepted T1, no primary App. C sensitivity 37/37 and top_k in 12/37. Source-name length is findings. HTTP-body sensitivity not the cause: LiteLLM rewrites names longer than raises statically accepted T1 from 64 characters in 26/37 cells. On the hosted_vllm 333 to 358/685 (52.3 %) and HTTP body (LiteLLM 1.106.0), serialisation primary findings from 127 to removes thinking_blocks; 25/37 cells then pass, 132/685 (19.27 %, +5 cells; upper while 12/37 still fail because of top_k. These are bound 136/685, 19.85 %). static validity judgements, not observed live errors. Gemini endpoint and The initial live runner addressed Live-acceptance results use §5.3; §7; App. C model-name checks gemini-2.5-flash-lite, unavailable on the account, gemini-3.5-flash-lite. and received 204 HTTP 404 responses. Repeating Model-name sensitivity changes T1 the check on gemini-3.5-flash-lite yields 160/204 cells from 467 to 479/685 (69.9 %; HTTP 200 and 44/204 HTTP 400 responses for a 2.57 % relative change, within missing thought_signature. A separate offline the pre-specified 15 % bound) and sensitivity check varies target model strings across statically accepted T1 from 333 to three naming variants. 345/685 (50.4 %). Reporting corrections only; §6.4; App. B.5; Scorecard and Corrected preliminary scorecard values: Set 9, reporting errata 29/35 to 28/34 T1 (82.4 %) and 30/35 to 29/34 main-text and headline-table App. B.6; any-category (85.3 %); Set 10, 0/28 to 1/28 T1 false counts unchanged. Any-category App. A.3 alarms (3.6 %); Sets 9–14, 99/125 to 102/125 counts include informational records and are not T1 any-category (81.6 %). Removed an obsolete initial-driver 3,197-cell human-pool note, recorded fault-detection rates. per-set designer provenance, and split 18 upstream comments into 10 on others’ threads and 8 follow-ups on our own (44 posts = 26 issues + 18 comments).

Driver control forwarding and withdrawn GSM8K arm

The initial AI SDK driver omitted maxOutputTokens and providerOptions. The corrected driver forwards representable controls and masks source-side unrepresentable Gemini controls; 2,142/2,142 checks find no omission. GSM8K arm A2 (AI SDK v7 to Gemini, initial driver) is withdrawn from translator claims and retained only as a driver-omission control.

55

Related documents

• Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss#1659125
• From Program Slices to Causal Clarity: Evaluating Faithful, Actionable LLM-Generated Failure Explanations via Context Partitioning and LLM-as-a-Judge#120576
• DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis#673535
• Extracting System Prompt of LLM Services in Complex (Agentic) Settings: Hierarchical Optimization and Reliable Evaluation Metrics#1625743
• Extracting System Prompt of LLM Services in Complex (Agentic) Settings: Hierarchical Optimization and Reliable Evaluation Metrics#1527797