arXiv:2605.29524v1 [cs.CR] 28 May 2026
KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing Yijia Fang
Yiqing Feng
Bingyu Li
Mingxun Zhou†
Beihang University China
Xidian University China
Beihang University China
HKUST Hong Kong SAR, China
Abstract—Relay and reseller APIs increasingly intermediate access to large language models (LLMs), but users have no direct way to verify that a claimed endpoint is actually serving the advertised model. We introduce KBF, a lowcost black-box auditing protocol that fingerprints model APIs using stable numerical recall near the knowledge boundary. Across 16 production LLM endpoints, KBF flags all 155 economically relevant substitutions without rejecting any samemodel controls, remains stable under deployment variation, detects high-separation mixed-routing attacks when only 5– 10% of traffic is substituted, and finds that 7 of 27 platform– model cells in a six-platform shadow API audit are statistically inconsistent with their reference endpoints, with inconsistencies concentrated on premium Claude endpoints.
1. Introduction Relay and reseller APIs are now a common access path for large language models (LLMs). Users buy credits from a third-party service and send requests through a blackbox endpoint, while the relay chooses the upstream model provider. Established aggregators offer legitimate benefits, such as unified billing and simple endpoint switching across models. At the same time, the growing token demand of agentic workloads has pushed users toward newer and lessaudited resellers that advertise lower prices or broader model availability. This convenience creates a basic trust problem: the endpoint may not be serving the model it claims to serve. A relay can silently replace an expensive advertised model with a cheaper backend, or mix reference-consistent and substituted traffic to reduce cost while making the behavior harder to notice. The incentive is direct: flagship models can be orders of magnitude more expensive than budget models, and recent measurement work shows that deceptive model claims in shadow APIs already occur at meaningful scale, affecting both research reproducibility and service reliability [1]. The practical need for auditing has already produced an ecosystem of informal checks. Users ask the model to identify itself, pose ad hoc “tricky” questions, or rely on †. Corresponding author.
community auditing services with opaque methodologies 1 . These checks are not a sound basis for security decisions. Self-identification is unreliable because LLMs often misstate their own identity even in benign settings [2]. Hand-crafted challenge prompts can be brittle, model-specific, and easy for a provider to evade by special-casing; public auditing services often expose neither their decision rule nor their conflict-of-interest posture. This motivates a concrete technical question: given black-box access to an official reference API and a suspect relay API, can an auditor test whether the relay is serving the claimed model? We deliberately study the access level available to an ordinary relay user. The auditor can send prompts to the official endpoint and the suspect endpoint, but cannot inspect the served model, observe token probabilities, read hidden system prompts, or see server-side routing logs. This is a setting strictly more challenging and yet more practical than the “white-box” or the “grey-box” setting addressed by many model-provenance and fingerprinting proposals, where the verifier can inspect model internals, observe richer outputs such as logits, or embed secret triggers or watermarks during model preparation [3], [4], [5], [6], [7], [8], [9]. These mechanisms are valuable when the model owner or serving platform cooperates, but they do not apply to relay users facing a potentially uncooperative intermediary. We envision that a useful audit protocol in this setting should satisfy four requirements: • High true-positive rate: the protocol should detect economically meaningful model substitution while maintaining a very low false-positive rate; in this setting, a false accusation against an honest provider is costly. • Robustness to deployment variation: the suspect endpoint may use unknown system prompts, decoding parameters, output-length limits, RAG wrappers, or agent-style interfaces. The audit should not require the auditor to recover the exact deployment configuration. • Benign and low-friction queries: the protocol should avoid prompt injection, harmful-content requests, or other security-sensitive probes that may be blocked, rate-limited, or answered adaptively. Audit traffic should resemble ordinary user traffic. 1. For example, hvoy.ai and llmtest.cn. This is not an endorsement: the authors have no connection with any existing auditors and/or relay services.
Economic practicality: the protocol should be cheap for auditors to run repeatedly, while making selective evasion expensive for dishonest relays even when the protocol is public. Existing black-box techniques do not satisfy these requirements simultaneously. Output-distribution tests such as MET [10] and ZeroPrint [11] compare response distributions over many queries, but their fingerprints are sensitive to deployment context and show high variance under wrappers and system-prompt changes. Behavioral fingerprinting methods such as LLMmap and LLMPrint show that fewquery model identification is possible [12], [13], but they rely on fixed behavioral probes, including prompt-injectionstyle or safety-sensitive interactions, that are poorly suited to routine third-party auditing. Standard benchmark evaluation is conceptually straightforward but too expensive and too slow to serve as a practical relay audit. Given the status quo, we ask: Can we build a black-box relay audit that is discriminative, robust to deployment variation, benign to run, and economically practical?
toward increasingly obscure and specialist-only facts. Each candidate is then re-queried in a short audit format under multiple deployment configurations; only candidates with stable recall are retained. To remove probes that are stable but weakly discriminative, KBF optionally performs contrastive screening against a smaller or cheaper model (e.g., a 9billion parameter model) and discards candidates on which the contrast model agrees with the reference. The result is a compact set of configuration-invariant, model-specific boundary probes, and can be easily applied to audit any chat-completion or agentic API in a black-box setting. Empirical results. Our empirical evaluation shows that KBF meets the four requirements for a black-box API auditing protocol. It is cheap to run repeatedly, highly discriminative across economically relevant substitutions, stable under deployment variation, sensitive to mixed routing, and actionable on real shadow APIs. More specifically, we evaluate KBF on 16 production LLM endpoints accessed through OpenRouter, covering eight mainstream model families and three price tiers. Reference probes are generated from provider-pinned endpoints under multiple system prompts and decoding temperatures; audits are stress-tested under role prompts, temperature changes, and RAG-style wrappers. We compare against three representative black-box baselines: MET, LLMmap, and ZeroPrint. The main findings are as follows:
•
1.1. Our Contribution Main observation. Our key observation is that useful audit signal appears near the knowledge boundary. When queried about numerical facts near the edge of factual recall, models often do not produce arbitrary noise. Instead, they produce stable, model-distinct values, including repeatable wrong answers. Operationally, these responses behave like persistent model-specific parametric associations. This makes boundary numerical recall a stronger audit signal than selfidentification, style, or high-variance continuation behavior. It is also operationally attractive: the probes can be ordinarylooking factual queries, and the comparison does not require embeddings or semantic similarity models, which can introduce their own instability [14], [15]. Technical Contribution: KBF auditing protocol. Based on this observation, we design Knowledge-Boundary Fingerprinting (KBF), a black-box relay-auditing protocol built around stable numerical recall near the knowledge boundary. KBF has three stages. First, in an offline probe-generation stage, the auditor uses the official reference API to construct a model-specific candidate set. Second, KBF screens candidates for stability across reference configurations and, when useful, for contrast against cheaper substitute models. Third, in the online audit, the auditor queries the suspect endpoint, compares the returned numerical values against the reference consensus under domain-specific tolerances, and applies a statistical test to decide whether the suspect endpoint is consistent with the claimed model. The main technical challenge is to generate probes that are simultaneously stable for the reference model, discriminative against likely substitutes, and cheap to obtain. KBF addresses this challenge with an adaptive frontier search. For each knowledge domain, the reference model is asked to propose candidate numerical facts from domainspecific themes. Across rounds, the generation prompts move
High true-positive rate at low cost. KBF flags all 155 economically relevant substitutions at p < 0.05, including all 12 within-family downgrades, while producing no falsepositive rejections on same-model controls. A full audit of all 16 models costs $0.39 after a one-time probe-generation cost of about $22. • Robustness to deployment variation. Under six shared configurations covering role prompts, temperature changes, and RAG-style wrappers, KBF produces 0/30 false positives and detects 60/60 substitutions. The baselines either false-positive heavily under benign configuration changes or miss substituted models. • Mixed-routing detection. KBF remains effective when a relay only substitutes a fraction of requests. In our two-round partial-routing experiment, most economically relevant model substitutions reach at least 80% TPR when 20–40% of traffic is rerouted. Even in the hardest setting, where the auditor does not know the substitute and the reference and substitute have highly similar capabilities, KBF exceeds 95% TPR once the rerouting fraction reaches 50%. • Real-world shadow API findings. We deploy KBF on six shadow API platforms and audit 27 platform–model endpoints for about $10. KBF finds 7 endpoints whose outputs are statistically inconsistent with the corresponding reference endpoint, with the flags concentrated on premium proprietary models. It also reveals tier-dependent serving behavior: a default tier is statistically inconsistent with the official endpoint, while a higher-priced tier for the same advertised model is consistent. •
In summary, we make the following contributions:
2
1) We identify knowledge-boundary numerical recall, including repeatable wrong values, as a stable and modeldistinct signal for distinguishing LLM APIs without relying on self-identification, style, logits, or privileged provider metadata. 2) We design KBF, an end-to-end black-box auditing protocol that turns this signal into compact probe sets through adaptive frontier search, configuration-stability filtering, and optional contrastive screening against likely substitutes. 3) We evaluate KBF on 16 production models and show that it detects economically meaningful substitutions, including within-family downgrades and mixed routing, while maintaining low false-positive risk under deployment variation. 4) We apply KBF to real-world relay APIs and flag several endpoints whose behavior is statistically inconsistent with the claimed reference endpoint, illustrating how KBF supports practical third-party investigation without provider cooperation. 5) We release code, data, and ready-to-use benchmark probe sets at https://github.com/Ooo0ption/KBF.git. The public probes support reproducibility and method comparison. For operational audits, fresh private probes should be generated for the claimed reference endpoint and model version.
In this paper, we focus on two types of deviation behaviors: Full substitution: Osus = Osub , ( Oref Mixed routing: Osus = Osub
Here Osub is a substitute backend and α ∈ [0, 1] is the substitution rate. In the general case, the adversary’s routing strategy may depend on the query, requested configuration, load, user account, or request timing. The goal is a precision-first statistical test. A useful relay audit should reject H0 for an honest endpoint with very low probability, while retaining high power against economically meaningful substitutions. It does not need to prove exact output equality or identify the backend implementation. It must determine whether a particular relay endpoint remains behaviorally consistent with the claimed reference endpoint under deployment variation that an outside auditor cannot fully control. Non-goals. We list several non-goals for clarification here. • We do not target generic model identification. Model fingerprinting techniques [10], [11], [12], [13] often ask which model in a database produced an output. Relay auditing starts from a specific advertised endpoint and a queryable official reference for that endpoint. The probes can therefore be reference-specific rather than universal. • We do not treat upgrade directions as part of the relayfraud threat model. If a service claims a budget endpoint while serving a more expensive or more capable endpoint, the service is outside the economic substitution scenario studied here. Such cells are not used to estimate the falsepositive rate; false positives are measured on same-reference consistency controls. • We do not aim for unbiased substitution-rate recovery in all mixed-routing settings. Our mixed-routing analysis instead provides a conservative estimate under the routing assumptions stated in Section 3.6. • We do not provide cryptographic attestation that a provider executed a particular model binary or inference stack; such guarantees require provider cooperation through verifiable inference, zero-knowledge proofs, or trusted hardware [6], [7], [8], [16], [17].
2. Problem Formulation and Preliminaries 2.1. Black-Box Relay Auditing We study whether a third-party relay API serves the model it advertises. The auditor has black-box access to two randomized oracles: Oref (x, c)
with probability 1 − α, with probability α.
and Osus (x, c).
Here Oref is the official reference endpoint, Osus is the suspect relay endpoint, x is a user query, and c ∈ C is a requested deployment configuration, such as a system prompt, temperature, decoding parameter, maximum output length, or provider-routing option. Both endpoints may be randomized: repeated calls with the same (x, c) can differ because of sampling, batch serving, nondeterministic inference, quantization, or provider-side wrappers. The reference endpoint is the claim anchor. The suspect relay may ignore or modify the requested configuration by adding hidden system prompts, changing decoding parameters, wrapping the request, or routing to a different upstream provider. Auditing goal. A protocol Audit adaptively queries both endpoints and outputs S AME or D IFFERENT. The hypotheses are H0 : Osus ≈ Oref vs. H1 : Osus ̸≈ Oref .
2.2. Access and Threat Model We consider a two-endpoint audit system, as shown in Figure 1. The auditor interacts with both the official reference endpoint and the suspect relay endpoint through their public APIs, while the suspect relay controls only its own backend serving path. The auditor may choose queries and configurations for both endpoints. For the reference endpoint, we assume these requests are applied up to ordinary provider-side nondeterminism. For the suspect endpoint, the auditor can submit the same requests but cannot assume they are honored. The auditor has no access to weights, logits, training data,
Here ≈ is operational: the suspect endpoint is considered consistent with the reference endpoint only if its audited behavior remains within the reference endpoint’s measured self-variation. KBF tests endpoint consistency for an advertised reference endpoint.
3
FPR is the primary safety metric because a false accusation against an honest provider is costly. For mixed routing, TPR depends on the substitution rate α and the substitute model. Robustness. A robust protocol remains stable under benign deployment variation, including system-prompt changes, temperature changes, interface wrappers, nondeterministic inference, and routine provider implementation details. Robustness requires low reference self-noise across repeated queries and configurations. Cost and evasion. Audit cost counts API calls and tokens. We distinguish offline enrollment cost, paid once to construct a reference fingerprint for a model version, from online audit cost, paid per suspect endpoint. Evasion cost captures how difficult it is for a dishonest provider to pass the audit without serving the claimed endpoint. Small fixed prompt sets are easier to recognize and special-case; many renewable, natural-looking probes impose higher operational cost. The probe sets released with this paper are therefore intended for benchmarking and reproducibility. In operational audits, the auditor should generate fresh private probes for the target reference version and rotate them across repeated audits.
Official Provider
Reference model
?
Probes
Results Claims Same Compare
Auditor ?
Flagged
Suspect model
Probes
Results Suspect Relay Operator
Unflagged
Figure 1: System model for black-box relay auditing. The auditor compares a claimed reference endpoint with a suspect relay endpoint using only public API access. inference code, provider metadata, or server-side routing logs. The adversary is the suspect relay operator. It controls the backend serving logic of Osus and may serve a cheaper model, mix several backends, ignore decoding parameters, add hidden prompts or wrappers, change quantization, or route traffic through different providers. The adversary may know the auditing algorithm and any public probe sets, and may query the official reference endpoint. It cannot control the official reference endpoint, the auditor’s local computation, or the communication between the auditor and the APIs. Adaptive behavior. A dishonest relay may try to recognize audit probes and route only those probes to the claimed model, or answer them from an external database. KBF does not provide cryptographic security against perfect probe recognition. Its defense is operational: probes are numerous, renewable, and ordinary-looking factual queries, making selective special-casing harder than evading a small fixed benchmark or visibly adversarial prompts. Auxiliary retrieval (RAG) and tools. KBF fingerprints the served API endpoint, not isolated parametric weights. If the reference endpoint uses retrieval or tools, that behavior is part of the reference. Conversely, if a suspect relay secretly augments a claimed bare-model API with retrieval, tools, or an external fact database, it is no longer serving the same endpoint under the same deployment conditions. KBF does not attempt to separate parametric memory from retrieval in a purely black-box setting.
2.4. Existing Black-Box Fingerprinting Solutions Existing black-box methods differ in the evidence they extract from API outputs. We summarize their operational intuition below. We further compare against these three operational styles in Section 4. Implementation details are in Appendix B. Behavioral-query fingerprints. LLMmap is closest in spirit to network service fingerprinting [12]. It sends eight handdesigned probes that cover model self-description, training metadata, weak alignment, harmful requests, malformed multilingual text, and prompt-injection-style banner grabbing. The responses are embedded together with the queries and passed through a learned trace encoder. In the open-set mode, the encoder outputs a vector signature and identifies the endpoint by nearest-neighbor search over an enrolled fingerprint database. This design is strong for quick model discovery, but relay auditing requires a stricter decision against one advertised reference. The probe set is visible and easy to special-case, and nearest-neighbor margins can collapse among close model variants or under benign wrapper changes. Distributional equality tests. Model Equality Testing (MET) asks whether two APIs induce the same completion distribution on a fixed task [10]. Its main experiment samples short prefixes from multilingual Wikipedia pages, asks the model to continue each paragraph, and collects repeated completions at a high sampling temperature to expose distributional differences. MET encodes the completions as strings and applies a two-sample MMD test with a Hammingstyle kernel; a large inter-sample distance leads to rejection. This makes MET the closest baseline in statistical form because it gives a calibrated hypothesis test. Its weakness is that the output distribution of an endpoint is shaped by more than the underlying model. Role prompts, wrappers, output-length controllers, and decoding settings can all move
2.3. Audit Metrics We use the following criteria to evaluate relay-auditing protocols. Error rates. The two primary statistical metrics are the false-positive rate (FPR) and true-positive rate (TPR): FPR = Pr[Audit(Oref , Osus ) = D IFFERENT | H0 ], TPR = Pr[Audit(Oref , Osus ) = D IFFERENT | H1 ].
4
the sampled distribution. When the suspect endpoint and the reference endpoint use different deployment settings, MET can reject an honest same-model comparison and produce high false-positive rates (Section 4.4). Perturbation-based fingerprints. ZeroPrint fingerprints local response sensitivity [11]. This solution builds base queries from HumanEval code-completion prompts, then creates perturbed queries by replacing selected words with nearby GloVe neighbors. For each base query, ZeroPrint compares the embedding difference in the input to the embedding difference in the model’s response and fits a ridge-regression estimate of a local Jacobian. The aggregated Jacobian is the fingerprint, and model pairs are compared by Pearson similarity. This signal is more structural than direct answer matching, but it depends on the perturbation distribution, the sentence-embedding model, and a similarity threshold that remains calibrated across endpoint configurations. Secret-prompt and injection-based fingerprints. Other methods use secret prompts, injected triggers, adversarial suffixes, or owner-controlled challenge sets [13], [18], [19]. They are useful when the owner can plant or protect the challenge material. Ordinary relay auditors do not control training or deployment, and visible trigger sets can be blocked or special-cased once discovered. Detector-based fingerprints. Detector-based fingerprints train classifiers over model outputs [20]. They are strongest when the candidate set and labeled data are stable. Relay APIs change through new endpoints, wrappers, silent updates, and provider-specific serving stacks; detector scores therefore require continual retraining and extra calibration before they support a reference-endpoint consistency claim.
Figure 2b shows both kinds of retained probes when GPT5.4 is used as the reference endpoint. The lower-panel probes would be discarded by a factual-accuracy benchmark, but they remain valid for endpoint auditing: a suspect endpoint that gives the ground-truth value still mismatches the reference if the reference consistently returns a different boundary value.
3.2. Algorithm Overview KBF turns boundary recall into an actionable endpointconsistency audit protocol. The auditor enrolls a claimed reference endpoint, measures the reference endpoint’s ordinary self-noise, and then tests whether a suspect endpoint reproduces the enrolled boundary behavior. The protocol relies only on black-box access to the reference API and the suspect API. The protocol has three stages. 1) Probe generation. KBF searches numerical domains for facts near the reference endpoint’s knowledge boundary. A probe is enrolled when the reference endpoint gives a valid, stable answer through reference-consistency checks. For a reference endpoint Oref , the fingerprint stores Pref = {(qi , di , ai , matchdi )}ni=1 .
Here qi is a short numerical audit prompt, di is its domain, ai is the reference endpoint’s consensus completion for qi , and matchdi is the domain-specific comparison rule. The reference consensus supplies the value compared later during auditing. 2) Self-calibration. KBF re-queries the reference endpoint with the enrolled prompts under the audit configuration. The resulting self-discrepancy count estimates ordinary reference-side variation. For confidence level γ , KBF converts this count into the CPγ upper bound p0 , which sets the null tolerance for a later S AME decision. 3) Suspect endpoint audit. KBF sends the enrolled prompts to the suspect endpoint, parses its answers under the same domain rules, and counts invalid or nonmatching responses. It reports D IFFERENT when the suspect discrepancy count exceeds what calibrated reference selfnoise can explain.
3. Knowledge Boundary Fingerprinting (KBF) 3.1. Intuition: Boundary Recall as Signal KBF starts from a simple intuition: useful audit signal appears near a model’s knowledge boundary. Common facts have little separating value because capable models usually agree on them. Facts that are too obscure are also poor probes because the reference endpoint itself becomes unstable. The useful regime lies between these extremes: the reference endpoint repeatedly commits to a concrete numerical value, while other endpoints return different values, invalid outputs, or no stable value at all. Figure 2a illustrates this intuition. This boundary behavior is useful for relay auditing in two ways. For facts just inside the boundary, the reference endpoint usually recalls the correct value stably, and weaker substitutes often fail to reproduce it. For facts just outside the boundary, the reference endpoint may instead commit to a stable but factually wrong value. These wrong-but-stable completions are still useful: KBF is not grading factual accuracy, but testing whether the suspect endpoint reproduces the claimed endpoint’s boundary-recall behavior. This matters in real relay markets because price, access, and availability do not always track raw capability: a cheaper backend may sometimes know facts that the claimed reference endpoint gets wrong.
Algorithms 1 and 2 make this pipeline precise. Algorithm 1 enrolls the reference endpoint through probe discovery and self-calibration. Algorithm 2 applies the online audit to a suspect endpoint and returns S AME or D IFFERENT. The main engineering challenge is to make this statistical testing method more practical. Probe search should concentrate on the useful boundary: common facts carry little signal, and far-obscure facts are unstable. We also want to reduce expensive reference calls in the candidate screening process. Batching and parsing must keep short numerical answers aligned with their prompts. Calibration must absorb ordinary deployment variation without hiding endpoint substitution. Section 3.7 describes these implementation choices after the core protocol.
5
Stable and factually correct probes • The boiling point of phosphorus oxychloride at 1 atm is 106 ◦ C. • The diploid chromosome number of Myrmecia croslandi can be 2. • The number of rounds in Threefish-1024 is 80.
Common facts Stable recall inside boundary Stable wrong recall just outside boundary
Stable but factually incorrect probes • The semi-major axis of (455502) 2003 UZ413 is 43.12 AU.
Unstable / no recall
Knowledge boundary
Ground truth: JPL SBDB lists about 39.4 AU, which lies outside the astronomy domain’s ±5% match interval around 43.12.
Useful auditing bands
(a) Near knowledge boundary recalls as fingerprinting signal.
•
The Stirling number of the first kind s(11, 5) unsigned is 269325. Ground truth: the unsigned Stirling number s(11, 5) equals 3416930.
(b) Screened KBF probes for GPT-5.4.
Figure 2: Knowledge-boundary intuition and example probes. KBF retains regimes where the reference endpoint commits to stable numerical values, including stable but factually wrong values just outside the boundary. In the probe examples, the underlined red value is the stable completion compared during auditing; lower-panel ground-truth annotations are for interpretation only. Astronomical ground truth is from JPL SBDB [21].
3.3. Phase 1: Probe Generation
Adaptive candidate proposal. For each domain d, KBF runs a sequence of search rounds indexed by a difficulty tier t. Each round asks the reference endpoint to propose structured candidate records for that domain, typically as name | value pairs. The process is adaptive: the prompt includes the difficulty tier and a short exclusion list of names already proposed in earlier rounds. This feedback tells the reference endpoint what has already been covered, reduces repeated proposals, and pushes later rounds toward more obscure candidates near the boundary of stable recall. Reference-consistency checks. KBF parses the proposed records, removes duplicate names, rejects values outside the domain range, and renders the remaining records into short audit prompts qi . It tests each candidate with referenceconsistency checks Ccons . These checks re-query the reference endpoint under several benign configuration changes, such as prompt variants or decoding settings. For a candidate prompt qi , these responses form a set Ai . The candidate survives when the responses parse as valid values for domain d and agree under matchd . KBF stores the consensus value ai with the prompt, domain, and match rule:
Stage I of Algorithm 1 constructs the reference probe set. Domain Specification. KBF searches independently within numerical domains. A domain d defines one kind of numerical recall task. Each domain specifies a prompt template, a valid numerical range, a difficulty schedule, and a comparison rule matchd . The comparison rule maps two parsed numerical values to agreement or disagreement. Values that fail to parse or fall outside the valid range are invalid and fail to match. The same matchd is used during reference-consistency checks, self-calibration, and online auditing. Table 1 gives representative definitions. The particular domain library is an engineering choice. Our implementation uses 15 domains, covering typical knowledge areas including chemistry, biology, astronomy, math (e.g., OEIS sequences [22]), programming and cryptography. Auditors can add or replace domains as long as the facts are numerical, parseable, and capable of producing stable boundary recall. matchd (a, â)
Domain
Valid range
Material boiling point Astronomy facts Chromosome count
[−300, 600] |a − â| ≤ 3 ◦ C [0, 1015 ] relative error ≤ 5% [1, 2000] exact integer agreement [1950, 2030] exact year agreement [0.01, 5000] relative error ≤ 10%
Programming release year Drug half-life
(qi , d, ai , matchd ).
The enrolled value is the reference endpoint’s stable completion under the audited interface. Progress tracking and stopping. KBF tracks search progress with the stable-count history Hd . After each round, it appends the number of surviving probes |Sd,t | to Hd . The predicate StopDomainCriteria(Hd ) is a domain-level stopping rule over this history. In our implementation, the predicate fires after two consecutive zero-yield rounds once the domain has enrolled at least five stable probes. This rule
TABLE 1: Example domains and matching rules used by KBF.
6
Algorithm 1: KBF Reference Fingerprint Construction Oracles: Reference endpoint Oref Inputs: Search space D with domains, audit prompt templates, and match rules; difficulty schedule T ; reference-consistency checks Ccons ; audit configuration caudit ; self-calibration confidence γ Output: Reference fingerprint Fref = (Pref , kself , p0 )
Algorithm 2: KBF Endpoint Audit Oracles: Suspect endpoint Osus Inputs: Reference fingerprint Fref = (Pref , kself , p0 ); audit configuration caudit ; significance level α Output: S AME or D IFFERENT 1 N ← |Pref |; k ← 0 2 foreach (qi , di , ai , matchdi ) ∈ Pref do 3 âi ← Osus (qi , caudit ) 4 if âi is not a domain-valid match to ai under matchdi then 5 k ←k+1
S TAGE I: P ROBE D ISCOVERY Pref ← ∅ 3 foreach domain d ∈ D do 4 matchd ← domain matching rule 5 Hd ← empty stable-count history for domain d 6 foreach search round t ∈ T do 7 Pd,t ← candidate probes proposed by Oref for domain d at difficulty t 8 Sd,t ← ∅ 9 foreach candidate probe qi ∈ Pd,t do 10 Ai ← responses from Oref to qi under check configuration c ∈ Ccons 11 if Ai is valid and stable under matchd then 12 ai ← Consensus(Ai , matchd ) 13 Sd,t ← Sd,t ∪ {(qi , d, ai , matchd )} 1 2
14 15 16 17
p ← Pr[X ≥ k | X ∼ Binomial(N, p0 )] if p < α then 8 return D IFFERENT 9 else 10 return S AME 6 7
from Phase 1 and uses the same audit configuration caudit that will later be used against suspect endpoints. This gives an endpoint-specific estimate of ordinary reference-side noise under the deployed interface, including nondeterministic generation, quantization, cache effects, provider-side wrappers, and other serving details outside the auditor’s control. Self-test. For each enrolled tuple (qi , di , ai , matchdi ) ∈ Pref , KBF queries Oref on qi under caudit and parses the returned value ãi . The self-discrepancy count kself increments when ãi is invalid for domain di or when it is valid but fails to match ai under matchdi . Every enrolled probe already has a valid reference consensus, so the self-test denominator is |Pref |. Clopper–Pearson null bound. KBF turns the self-test result into a conservative null bound using the one-sided Clopper– Pearson (CP) upper confidence bound [23]. Let γ denote the self-calibration confidence level; we use γ = 0.99 in all experiments unless stated otherwise. For k discrepancies in n enrolled probes, define
Hd ← Append(Hd , |Sd,t |) Pref ← Pref ∪ Sd,t if StopDomainCriteria(Hd ) then break
S TAGE II: S ELF -C ALIBRATION kself ← 0 20 foreach (qi , di , ai , matchdi ) ∈ Pref do 21 ãi ← Oref (qi , caudit ) 22 if ãi is not a domain-valid match to ai under matchdi then 23 kself ← kself + 1
18 19
24 25
CPγ (k, n) = Beta−1 (γ; k + 1, n − k).
Under the binomial model, each enrolled probe is treated as an independent discrepancy trial with the same unknown reference error rate p. The CPγ value is a γ -level one-sided upper confidence bound on this rate: it is the largest p still compatible with observing only k discrepancies in n self-test queries. Equivalently, if the true reference error rate were larger than this bound, observing k or fewer discrepancies would fall in the lower 1 − γ tail. The enrollment algorithm sets p0 = CPγ (kself , |Pref |).
p0 ← CPγ (kself , |Pref |)
return (Pref , kself , p0 )
keeps search focused on productive boundary regions and limits reference-query cost. The output of Phase 1 is Pref : a set of reference-stable numerical probes concentrated near the endpoint’s knowledge boundary. Phase 2 calibrates the ordinary self-noise of this exact set.
The bound is model-specific and probe-count-aware: a stable reference endpoint with many enrolled probes receives a tight null bound, while a noisier endpoint receives a wider bound. The fingerprint passed to online auditing is Fref = (Pref , kself , p0 ). The implementation also stores raw
3.4. Phase 2: Self-Calibration Phase 2 measures how often the enrolled probe set disagrees with the reference endpoint itself. KBF fixes Pref
7
3.6. Adaptive Routing Extension
consensus responses and per-probe self-test outcomes for diagnostics. Dependence among probes. The binomial rule is a simple primary calibration, not a claim that all probe outcomes are physically independent. Probes can share domains, templates, and endpoint-specific response modes, so discrepancies may be correlated within blocks. We therefore interpret the resulting p-value together with same-endpoint controls and robustness tests. For high-stakes operational deployments, auditors should also report repeated same-endpoint audits over fresh probe sets and, when enough per-domain data is available, a domain-block bootstrap or a conservative effective-sample-size correction.
Some relays may substitute only a fraction of requests. KBF handles this setting with a two-round audit that concentrates extra queries on the probes most likely to expose routing. The extension assumes a fixed routing probability π during the audit: each request is served by the claimed reference endpoint with probability 1 − π and by a substitute endpoint with probability π . The detection test does not require knowing the substitute identity. Two-round statistic. The first round runs the standard suspect endpoint audit on all N = |Pref | enrolled probes and records the mismatch count W1 . The second round requeries only the W1 probes that mismatched in round one and records the additional mismatch count W2 . KBF uses
3.5. Phase 3: Suspect Endpoint Audit
T = W 1 + W2
Phase 3 tests whether a suspect endpoint reproduces the enrolled reference fingerprint. KBF sends each enrolled prompt to the suspect endpoint Osus under the audit configuration caudit . It parses the returned values with the same domain rules and compares each value against the stored reference consensus. Per-probe discrepancy. For each enrolled tuple (qi , di , ai , matchdi ) ∈ Pref , let âi be the value returned by the suspect endpoint. KBF defines one discrepancy indicator: ( 0, if âi is valid for di and matchdi (ai , âi ) = 1, Zi = 1, otherwise.
as the two-round statistic. Under the reference-consistency null, round-one mismatches are expected to occur at rate at most p0 , and repeated mismatches among those probes are also bounded by the same calibrated reference noise. The null distribution of T is computed from W1 ∼ Binomial(N, p0 ), W2 | W1 = w ∼ Binomial(w, p0 ).
KBF rejects when the observed T falls above the onesided α cutoff of this distribution. The second round helps because substituted probes that mismatch once are more likely to mismatch again, while ordinary reference-side noise is already bounded by p0 . Routing-fraction estimate. When the likely substitute S is known, the same two-round statistic can estimate the routed fraction π . For each enrolled probe, define its type by whether a fresh reference query and a substitute query mismatch the stored reference value. Let nab count probes where the reference mismatch bit is a ∈ {0, 1} and the substitute mismatch bit is b ∈ {0, 1}. Under independent fixed-probability routing,
Missing, unparseable, out-of-range, and valid-butnonmatching answers all count as discrepancies. The observed P audit size is N = |Pref |, the discrepancy count is k = i∈Pref Zi , and the discrepancy rate is rdisc = k/N . Decision rule. The null hypothesis H0 says that the suspect endpoint is consistent with the reference endpoint on the enrolled probe set, with expected discrepancy rate at most p0 . The alternative hypothesis H1 says that the suspect endpoint is inconsistent with the reference endpoint, with expected discrepancy rate above p0 . KBF calculates the one-sided tail probability of observing k or more discrepancies under the null bound: N X N i P (X ≥ k) = p0 (1 − p0 )N −i . i
E[T ] = n01 π(1 + π) + n10 (1 − π)(2 − π) + 2n11 . Setting this expectation equal to the observed Tobs gives (n01 + n10 ) π 2 + (n01 − 3n10 ) π + (2n10 + 2n11 − Tobs ) = 0.
The root in [0, 1] gives the point estimate π̂ . This estimator is a diagnostic after a deviation has been flagged; the primary audit decision remains the two-round hypothesis test above. Unknown substitute. If the substitute identity is unknown but belongs to a candidate pool C , KBF reports a routing interval instead of a point estimate. Let pR be the reference self-discrepancy rate and let pmin and pmax be the minimum S S and maximum substitute discrepancy rates over C on the enrolled probes. The first-round mismatch rate W1 /N gives
i=k
If the p-value is below α = 0.05, KBF rejects H0 and reports D IFFERENT. Otherwise, it reports S AME. The primary relayaudit decision uses the CPγ -calibrated binomial rule. The implementation also records McNemar’s paired test [24] against the stored self-test outcomes as an auxiliary diagnostic for quantization and provider-comparison analyses. Interpretation of the audit. The decision is endpointlevel evidence under the audited interface. A D IFFERENT result means that the suspect endpoint’s observed behavior is statistically inconsistent with the claimed reference endpoint on the enrolled probe set. A S AME result means that the audit did not find enough discrepancies to exceed the calibrated reference-noise bound.
π̂min =
W1 /N − pR , pmax − pR S
π̂max =
W1 /N − pR , pmin − pR S
with both endpoints clipped to [0, 1]. This interval is conservative when the candidate pool covers the true substitute. If the relay routes by prompt content, user identity, timing, or
8
audit recognition, the two-round test still provides endpointlevel inconsistency evidence, but the recovered π̂ should be interpreted only under the fixed-routing assumption.
the recovered value is domain-valid and matches the stored consensus.
4. Empirical Evaluation
3.7. Implementation Details
This section evaluates KBF along five dimensions: 1) Accuracy: does KBF detect economically relevant substitutions without false rejections on same-reference controls? 2) Cost: what is the one-time cost of enrolling reference fingerprints, and what does each online audit cost after enrollment? 3) Robustness: does the calibrated test remain conservative under deployment variation, adversarial prompt settings, quantization, temporal drift, and threshold changes? 4) Performance under adaptive routing: how much partial substitution can KBF detect, and can it estimate the routed fraction once an endpoint is flagged? 5) Real-world effectiveness: what does KBF flag on shadow API endpoints that advertise flagship models?
Algorithms 1 and 2 define the statistical protocol. The implementation must also make several request-facing choices: how probes are rendered into prompts, when cheap screening is used, how batched answers remain aligned with probes, and how numerical responses are parsed. These choices affect cost and reliability, but they do not change the enrolled-probe definition or the CPγ -calibrated decision rule. Prompt rendering. Each domain renders candidate records with a short cloze-style template. For example, a boilingpoint record becomes “The boiling point of x at 1 atm is ◦ C.” The endpoint repeats and completes the sentence rather than returning a bare lookup answer. This prompt shape encourages ordinary sentence completion from parametric memory while leaving a stable numerical slot for parsing. It reduces output variance and reference self-noise, and it keeps the query surface fixed across reference-consistency checks, self-calibration, and suspect-endpoint auditing. Other prompt formats can be used when audit recognition is a concern, as long as they preserve the same probe semantics and recoverable numerical answer. Contrastive screening. After the first reference query under caudit , KBF can query a small contrast endpoint on the same prompt. We use Qwen3.5-9B as the default contrast endpoint. If the contrast endpoint matches the provisional reference value under matchd , KBF drops the candidate before the full reference-consistency checks. This screen saves reference calls and biases the retained set toward candidates with higher expected separation from cheap substitutes. It is a probe-selection heuristic rather than part of the statistical null: the online audit still compares the suspect endpoint only against the reference endpoint’s enrolled behavior. We disable the screen for reference models where it removes too much of the usable probe set. Batching and slot recovery. Oracle calls are issued in domain-homogeneous batches of ten probes. A batch therefore shares one valid range and one match rule. The prompt requests numbered short answers, which lets the parser map each numeric value back to its intended probe when an endpoint skips a line, inserts commentary, or changes answer order. Numerical parsing and failure recovery. KBF uses one numerical parser for Ai , ãi , and âi . The parser removes hidden reasoning tags, markdown separators, tables, and empty lines; maps numbered answers to probe slots; normalizes minus signs and commas; extracts the final numeric token from each answer; and rejects values outside the domain’s valid range. This recovery matters in practice: some endpoints emit long thinking traces before the final answer, add explanatory text, reorder slots, or refuse one item in a batch while answering the others. KBF keeps recoverable slots when their alignment is clear. Formatting failures or ambiguous slot alignment trigger a retry. After parsing, a probe succeeds only when
4.1. Experimental Setup We evaluate KBF in controlled settings where the claimed endpoint, provider route, probe set, and decision rule are fixed. This design makes endpoint mismatches attributable to model behavior rather than avoidable routing noise, and gives a clean baseline for the later field audit. Models. Table 2 summarizes the endpoint pool. We evaluate KBF on 16 production LLM API endpoints spanning eight model families and three price tiers: six T1 flagship endpoints, five T2 mid-range endpoints, and five T3 budget endpoints. We use OpenRouter input prices recorded in March 2026. The prices range from $0.05 to $5.00 per million input tokens, giving a 100× cost spread between the cheapest and most expensive endpoints. This spread creates the economic room for silent substitution: a relay can advertise an expensive model while serving a cheaper endpoint. Reference endpoints. For each claimed model, we enroll the reference fingerprint from the OpenRouter route pinned to the provider in Table 2. We use explicit provider selection and disable fallback routing, preventing OpenRouter from changing the upstream provider during the controlled pairwise evaluation. For each reference endpoint, Table 2 reports the retained probe count and the fresh self-disagreement rate used to calibrate the model-specific null bound. Audit protocol. For each endpoint, KBF enrolls a reference probe set and a 99% CP null bound using the construction in Section 3. We audit each target endpoint with the corresponding reference fingerprint and apply Algorithm 2 at α = 0.05. The pairwise matrix measures controlled substitution behavior. Same-endpoint controls, configurationstress trials, and temporal self-tests in Sections 4.2, 4.4, and 4.5 measure whether the calibrated decision rule remains conservative under deployment variation. Metrics. In our experiments, a false positive (FP, lower is better) is a substitution alarm raised against an endpoint
9
single bare query at temperature=0; a false positive occurs when the Top-1 retrieved fingerprint is not the queried model. MET [10] compares two independent batches of N = 250 same-model completions using its b = 1000 permutation test at α = 0.05. ZeroPrint [11] compares fingerprints with Pearson similarity and applies a single global Youdenoptimal threshold, with reference fingerprints collected in an independent run. A false positive occurs when the similarity between a query and the fingerprint of a different model exceeds this threshold. The baseline false positives concentrate on closely related models from the same family or vendor. KBF avoids these failures by calibrating each audit against the claimed reference endpoint and retaining only probes that are stable for that reference.
that serves its claimed reference, and a true positive (TP, higher is better) is a correctly detected economically relevant substitution.
4.2. Detection Accuracy We first test the cleanest relay-auditing setting: each provider-pinned reference endpoint is compared against every endpoint in the 16-model pool. KBF flags all 155 economically relevant substitutions at p < 0.05 under the 99% CP binomial test, while producing no false-positive rejection on the 16 same-reference controls. Substitution Detection. An economically relevant pair (Mr , Mt ) is one where the target endpoint Mt is cheaper than the claimed reference Mr , or where both endpoints are in the same price tier. Lower-tier substitutions capture direct cost savings. Same-tier substitutions capture smaller but practical incentives: price differences inside a tier, easier access, compliance burden, and regional availability. The only cross-tier directions excluded from this threat model are upgrades where the target is more expensive than the claimed reference, such as T2→T1 and T3→T2/T1. These upgrade directions are not used to estimate either TPR or FPR. Figure 3 summarizes the full 16×16 detection matrix. Colored cells are economically relevant replacement directions; all 155 are detected. Diagonal cells are same-reference controls, and all remain below the rejection threshold. Blank cells are upgrade directions outside the threat model. Table 3 shows representative detections spanning three substitution types: within-family swaps in both directions, a cross-family substitution, and a same-tier replacement. The hardest cases are the most informative, and even they reject at p < 0.001. KBF detects all 12/12 within-family economically relevant substitutions and all 143/143 crossfamily substitutions, showing that the signal is not limited to visibly different model families. False-Positive Control. The same 16×16 matrix also gives the controlled false-positive test. Only the diagonal cells estimate false-positive risk, because they compare each reference endpoint against itself. Across these 16 same-reference controls, KBF produces 0/16 false-positive rejections. The 99% CP null bound and the reference consistency filtering are therefore conservative on honest same-reference endpoints in this setting. This diagonal test is a sanity check rather than a complete false-positive calibration. With 0 failures in 16 trials, the exact one-sided 95% Clopper–Pearson upper bound on the false-positive probability is 17.1%. Sections 4.4 and 4.5 add configuration, quantization, and temporal sameendpoint stress tests, but a high-stakes operational workflow should repeat same-endpoint audits across accounts, days, regions, provider routes, and fresh probe sets and report confidence intervals for that larger control sample. We compare against three black-box auditing methods on the same 16-model pool used in their original evaluations. Table 4 reports the resulting FPRs, and Appendix B gives the full reproduction configuration of each baseline. LLMmap [12] runs in open-set fingerprinting mode with a
4.3. Audit Cost A relay-auditing method is useful only if third parties can rerun it without provider cooperation. We separate KBF’s cost into one-time reference enrollment and repeated online audits. Reference enrollment. Table 5 reports the reference-side cost for the six T1 reference models in the manual cost run. The run generated 3,696 retained probes for $7.07 using 1.94M tokens, with most of the expense coming from Claude Sonnet 4.6, Claude Opus 4.6, and GPT-5.4. This cost is amortized across later audits of endpoints that claim the same reference model version. Online audit. After a reference fingerprint is enrolled, the online cost is only the cost of sending the stored probes to a suspect endpoint. Figure 4 reports this cost for each model X : we take the probe set enrolled from X and query endpoint X once with that probe set. This isolates the online audit cost and excludes reference enrollment. The 16 per-model audits together cost approximately $0.39 at OpenRouter list prices from March 2026. The most expensive case is Claude Opus 4.6 at $0.243, because its audit uses 681 probes against the highest-priced endpoint; every other model costs under $0.05 per audit, and all five T3 budget endpoints cost under $0.002 each.
4.4. Configuration Robustness Relay APIs are often deployed as application endpoints rather than bare chat models: a provider may place the model behind a role prompt, use a nonzero temperature, attach retrieval context, or wrap responses for a product workflow. A relay audit should stay conservative when these wrappers preserve the claimed backend, and it should still detect economically motivated substitution behind the same wrappers. Key takeaway. Under shared deployment-wrapper changes, KBF remains stable whereas every baseline does not, raising many false alarms and sometimes missing substitutions. We test this setting at two levels. First, we run a shared comparison against three baselines on five representative
10
TABLE 2: The 16 LLM endpoints evaluated in this work. Provider is the OpenRouter inference provider used for all queries (via provider pinning). Price∗ reports input- and output-token costs in USD per million tokens. #Probes is the number of configuration-invariant numerical probes generated for each reference model. Self-Err is the reference model’s self-disagreement rate, measured during a fresh deployment run. ∗ Prices were recorded from OpenRouter in March 2026. Tier
Model
Model Family
Model Provider
T1 T1 T1 T1 T1 T1 T2 T2 T2 T2 T2 T3 T3 T3 T3 T3
Claude Opus 4.6 Claude Sonnet 4.6 GPT-5.4 Gemini 3 Flash GLM-5 Qwen3.5-397B-A17B DeepSeek-V3.2 GPT-4.1-mini GLM-4.7 Kimi-K2-0905 Qwen3.5-27B GPT-4.1-nano LLaMA-4-Scout Qwen3.5-9B GLM-4.7-Flash Gemini 2.5 Flash Lite
Anthropic Anthropic OpenAI Google Z.AI Alibaba DeepSeek OpenAI Z.AI Moonshot Alibaba OpenAI Meta Alibaba Z.AI Google
Amazon Bedrock Google OpenAI Google Z.AI Alibaba Google OpenAI Z.AI Moonshot AI Alibaba OpenAI Groq Together DeepInfra Google
TABLE 3: Representative economically motivated substitutions flagged by KBF, including the within-family Claude swap in both directions. Note gives the substitution type. All cases reject at p < 0.001. Reference
Target
Model
TABLE 4: FPR on the 16-model pool. KBF is the only method with no false alarm. Baseline failures concentrate on closely related models from the same family or vendor. FPR
KBF (ours) LLMmap (bare query, temp=0) MET (permutation, b=1000) ZeroPrint (Pearson, Youden threshold)
0 / 16 6 / 16 1 / 16 2 / 16
0.0% 37.5% 6.3% 12.5%
Self-Error Rate
681 224 317 315 415 244 364 134 356 300 115 109 146 105 309 210
4.3% 1.3% 1.6% 2.2% 4.1% 1.7% 3.3% 6.0% 4.6% 4.7% 4.3% 7.3% 11.7% 3.8% 16.2% 13.8%
Probes
Total cost (USD)
Total tokens
664 724 531 716 613 448
$2.17 $2.06 $1.59 $0.44 $0.50 $0.31
346,316 280,718 297,718 348,041 345,303 317,916
Total 3,696 Measured in May 2026.
$7.07
1,936,012
Claude Sonnet 4.6 Claude Opus 4.6 GPT-5.4 Gemini 3 Flash GLM-5 Qwen3.5-397B-A17B
Claude Opus 4.6 Claude Sonnet 4.6 ✓ < 0.001 within-family Claude Sonnet 4.6 Claude Opus 4.6 ✓ < 0.001 within-family Qwen3.5-397B-A17B DeepSeek-V3.2 ✓ < 0.001 cross-family GPT-5.4 Gemini 3 Flash ✓ < 0.001 same-tier hardest
FP / 16
#Probes
TABLE 5: One-time T1 reference-enrollment cost by reference model.
p-value Note
Method
Price∗ ($/M Tokens) Input Output 5.00 25.00 3.00 15.00 2.50 10.00 0.50 2.50 0.72 2.20 0.39 1.20 0.26 0.42 0.40 1.60 0.38 2.00 0.40 2.50 0.20 0.30 0.10 0.40 0.08 0.30 0.05 0.10 0.06 0.20 0.10 0.40
only numerical-recall probes that are stable for the claimed reference endpoint, so role prompts, RAG wrappers, and moderate temperature changes do not erase the reference signal. The baselines fail in different ways. MET detects every substitution, but it rejects 29 of 30 same-reference configuration trials because the sampled output distribution shifts under role and RAG wrappers. LLMmap has a lower FPR than MET but misses 14 of 60 substitutions; its response-feature fingerprint moves under the same wrapper changes. ZeroPrint detects 45 of 60 replacements while producing 23 false positives on same-reference trials. The comparison shows why relay auditing needs a signal that remains stable under benign endpoint wrappers while still separating substituted endpoints. Extended KBF stress tests. We sample five agent configurations from PromptConfFactory [12], including role prompts, chain-of-thought variants, and RAG injections, and apply each to all 16 models for 80 same-reference stress tests. All T1 and T2 models remain stable, producing zero false positives. The only false positives come from a T3 model: GLM-4.7-Flash, whose high baseline self-error makes
models: Claude Sonnet 4.6 (T1), GPT-5.4 (T1), GLM-4.7 (T2), GPT-4.1-mini (T2), and Qwen3.5-9B (T3). The six configurations in Table 6 vary only the deployment wrapper: role prompt, temperature, and RAG status. All methods receive output-format instructions appropriate to their own protocol, so the variable under test is deployment variation rather than parser noise. In this experiment, a false positive is a substitution alarm raised against an endpoint under a changed configuration; this comparison contains 30 same-reference trials and 60 replacement trials. Table 6 shows the main result. KBF raises no false alarms in all 30 same-reference trials and detects all 60 substitutions. This behavior follows from the construction: KBF retains
11
T1
T2
T3
Opus 4.6
4.3
22
26
22
27
37
28
51
33
26
60
62
79
65
68
85
Sonnet 4.6
11
1.3
19
12
16
28
17
46
22
20
53
54
73
61
61
96
GPT-5.4
14
18
1.6
12
20
28
20
41
27
20
56
59
72
63
65
78
Gem-3-Flash
18
27
28
2.2
33
38
23
54
36
27
66
65
78
70
68
91
GLM-5
17
21
26
17
4.2
27
21
47
19
21
59
63
73
60
67
92
Qwen3.5-397B
18
23
18
15
14
1.7
14
48
23
19
46
59
74
60
66
88
3.3
48
25
18
51
62
76
59
59
88
29
6.0
28
34
50
61
56
59
54
93
21
44
4.6
21
57
55
74
60
60
90
19
48
29
4.7
54
58
69
62
64
88
18
41
20
20
4.3
52
64
56
57
86
Gem-2.5-Lite
13.8 67
50
56
79
GPT-4.1-nano
28
7.3
28
26
33
GLM-4.7-Flash
46
55 16.2 62
59
59
68
71 11.7 92
28
37
21
100%
DeepSeek-V3.2 GPT-4.1-mini GLM-4.7 Kimi-K2
Detection summary TP 155/155 FP 0/16
Qwen3.5-27B
T3
Llama-4-Scout
25
3.8
50%
25%
0%
O
pu So s 4 nn .6 et 4. 6 G P G T em 5 -3 .4 -F la sh Q G w L en M -5 3. D ee 5-3 9 pS ee 7B k G PT -V3 .2 -4 .1 -m in G LM i -4 .7 Ki Q m w i -K en 2 3. 5G em 27 B -2 . G PT 5-L ite -4 .1 G LM -n -4 ano .7 Ll am -Fl a- ash 4Sc Q w ou en t 3. 59B
Qwen3.5-9B
Self-error (diagonal) Not applicable
75%
Cross-Error Rate
T2
Reference Model
T1
Target Model Figure 3: Detection heatmap over all 16×16 model pairs. Cell color indicates cross-model mismatch rate on the reference probe set. Colored cells are economically relevant substitutions, and all 155 such pairs are detected at p < 0.05. Diagonal cells are same-reference endpoint controls. Blank cells correspond to upgrade directions outside our threat model. it genuinely unstable, accounts for 2 of the 80 trials, for a global FPR of 2.50%. This gives configuration-specific evidence that the 99% CP null bound remains conservative under ordinary endpoint wrappers.
rejections are two adversarial prompt settings with concrete causes: one that explicitly requests approximate answers, and one small probe set with high baseline self-error. Adversarial prompts and identity spoofing. A dishonest relay may target the audit surface directly. We test two variants: adversarial prompt settings that make numerical extraction harder, and identity-spoofing prompts that ask a substitute model to present itself as the reference. For prompt-level stress, we subject three models (DeepSeek-V3.2, Kimi-K2-0905, GPT-4.1-nano) to five settings: temperature 1.0, an “approximate answers” system prompt, a non-English system prompt with English probes, 2000-character padding, and contradictory few-shot examples. 13 of 15 trials pass; both failures have clear root causes. Kimi-K2-0905 reaches a 20.0% mismatch rate under the “approximate answers” prompt, which directly instructs the model to abandon numerical precision. GPT-4.1-nano reaches
4.5. Additional Robustness Analysis We next stress KBF outside the deployment-wrapper setting. The tests cover adversarial prompt settings and identity spoofing, quantized inference, temporal drift, and threshold choice. Each test asks whether same-endpoint variation can push the calibrated audit into a false rejection. Table 7 summarizes these additional stress tests. Key takeaway. KBF stays conservative across every stress test: identity spoofing, routine quantization, threshold sweeps, and agent-CLI probing all stay below the rejection criteria, and temporal self-tests hold for 47 days. The only false
12
TABLE 6: Configuration-robustness comparison on five representative models under six shared deployment wrappers. Each row reports false positives on same-reference trials (FP, lower is better) and detected substitutions on replacement trials (TP, higher is better). ID Role
Temp
RAG Status
C0 Clean Baseline C1 Finance-Compliance Assistant C2 Enterprise RAG Policy Assistant C3 Customer-Support Workflow Assistant C4 Medical RAG Triage Assistant C5 Academic-Writing Editor Total
0 0.2 0 0.3 0.1 0.7
Without RAG Without RAG With RAG Without RAG With RAG Without RAG
Per-model online audit cost (total = $0.39 for all 16 models) Claude Opus 4.6 $0.049 Claude Sonnet 4.6 $0.036 GPT-5.4 $0.014 GLM-5 $0.0099 Gemini 3 Flash $0.0043 Qwen3.5-397B $0.011 Kimi-K2-0905 $0.0100 GLM-4.7 $0.0045 DeepSeek-V3.2 $0.0032 GPT-4.1-mini $0.0008 Qwen3.5-27B $0.0014 Gemini 2.5 Flash Lite $0.0008 LLaMA-4-Scout $0.0007 GPT-4.1-nano $0.0006 GLM-4.7-Flash Qwen3.5-9B $0.0003
10 3
10 1 10 2 Cost per audit (USD, log scale)
KBF (ours) MET LLMmap ZeroPrint FP TP FP TP FP TP FP TP 0/5 10/10 4/5 10/10 3/5 6/10 4/5 7/10 0/5 10/10 5/5 10/10 3/5 10/10 5/5 7/10 0/5 10/10 5/5 10/10 3/5 8/10 4/5 7/10 0/5 10/10 5/5 10/10 3/5 7/10 3/5 8/10 0/5 10/10 5/5 10/10 4/5 7/10 3/5 8/10 0/5 10/10 5/5 10/10 3/5 8/10 4/5 8/10 0/30 60/60 29/30 60/60 19/30 46/60 23/30 45/60
which compares per-probe agreement against the reference and separates three of the five GLM-4.7 providers (Venice fp4 p = 6 × 10−5 , DeepInfra fp4 p = 0.008, AtlasCloud fp8 p = 0.035). On DeepSeek-V3.2 it isolates the Novita provider (p = 0.005) despite its nominal fp8 label, revealing backend inconsistency that the declared quantization level alone does not expose. Provider pinning in the main detection experiment (Section 4.1) removes this source of backend variation from the 155-pair matrix. Temporal Robustness. Probe sets are generated from a reference snapshot, while providers may silently update their backends. For all 16 models, we re-run each reference self-test across five snapshots spanning 64 days, from T0 (2026-03-15) through T4 (2026-05-18), applying the same 99% CP binomial test as the main detection to each snapshot against its T0 baseline. Probes stay valid through 7 weeks, where all 16 models remain stable. At the week-9 measurement, Qwen3.5-397B-A17B crosses α = 0.05 (self-error 1.7% → 7.8%, p = 0.023), a shift we reproduced on the official API, while every other model remains stable. The drift is one-directional toward higher self-error, consistent with a silent backend update rather than sampling noise. It points to a probe-refresh cycle of roughly nine weeks, and versioned model identifiers (e.g., qwen3.5-397b-a17b:20260315) help preserve probe validity across update cycles. Threshold Robustness. The main evaluation uses the 99% CP null bound determined by each reference model’s self-test. As a diagnostic check, we compare this rule against additive margins from 1% to 10% across all 155 economically relevant pairs. At margins of 1%–4%, all 155 pairs are detected (100% TPR). At 5%–7%, detection remains at 99.4% TPR (154/155 TP); the single missed pair (GPT-5.4 vs. Gemini 3 Flash) sits at the borderline and is correctly detected under the 99% CP binomial test with the full probe set. At 10%, detection falls to 94.1% TPR as the threshold absorbs the signal from moderately distinguishable pairs. Across all 27 false-positive control trials, the FPR is 0 for every diagnostic margin value from 1% to 10%. The sweep confirms that KBF’s behavior is stable over a wide threshold range. Agent-interface probing. We finally test whether KBF stays stable under the system prompt of a real coding agent, so that a user can audit the API behind an agent simply by
$0.243
T1 T2 T3
100
Figure 4: Online audit cost. Each bar is the cost of one query to an endpoint with its own probe set. The 16 audits total $0.39, none exceeding $0.25.
18.5% at temperature 1.0; this case combines a small probe set (109 probes) with elevated baseline self-error, leaving limited statistical separation. DeepSeek-V3.2 passes all five settings, with a maximum error shift of 3.3%. For identity spoofing, we prepend to each of the other 15 models a system prompt asserting “You are Claude Opus 4.6” at temperature 0.0, with the impersonated reference itself excluded. Under the same 99% CP binomial test, all 15 endpoints stay consistent with their own baseline (0/15 FP). An asserted identity does not shift the boundary numerical recall that KBF measures, so the audit signal remains robust to whatever identity a model is prompted to claim. Quantization Robustness. Providers may serve quantized variants (fp8, fp4) under the same model name. This test measures whether such backend choices change KBF’s same-endpoint decision. We route GLM-4.7 queries through five OpenRouter providers spanning fp8 and fp4, anchored to the full-precision Z.AI reference. The error-rate gap introduced by quantization is small, so separating these backends requires a finer-grained comparison at the level of individual probes rather than aggregate error alone. For this we apply a more sensitive test, McNemar’s paired test,
13
TABLE 7: Robustness summary beyond benign deploymentwrapper variation. Scale is the number of independent trials. Key result reports the main result. Dimension Scale
TABLE 8: The eight reference–substitute pairs used in the adaptive-routing study. For each flagship reference we take the two substitutes with the lowest mismatch rate pS , the setting most favorable to the adversary. n01 counts the probes where the substitute mismatches while the reference stays correct, and drives detectability more than the total probe count N .
Key result
Adversarial 15 prompts, 13/15 prompts † and 15/15 spoofs pass 15 spoofs Quantization 9 providers separates 3/5 GLM-4.7, 1/4 DeepSeek-V3.2 Temporal 5 snapshots stable to ∼7 wk, first drift at ∼9 wk Threshold 7 margins TPR ≥ 99.4% at 1–7% margin Agent CLI 3 agents 4/4 match the API baseline † Both failures have identified root causes: an explicit approximate-answer
instruction (Kimi) and an insufficient probe count with high self-error (GPT-4.1-nano).
sending it the probes. We evaluate two references (Claude Sonnet 4.6 and GPT-5.4) across three widely used agent CLIs (Claude Code, OpenClaw, and Codex). On OpenClaw we connect both models for testing, while for the other two official CLIs we query the official APIs directly. In each, the agent reads the probe document and answers inline within its own session, under the agent’s full system prompt (the OpenClaw harness alone adds roughly 30 KB) and without temperature control; we then parse the replies and apply the standard test. All four configurations pass: under the 99% CP binomial test, every agent-side result stays consistent with the model’s own API baseline and produces no false positive (0/4 FP). A user can therefore audit the model behind an agent by sending it the probes directly, without API access.
Reference
Substitute
N
n01
pS
Opus 4.6 Opus 4.6 Sonnet 4.6 Sonnet 4.6 Gemini 3 Flash Gemini 3 Flash GPT-5.4 GPT-5.4
Kimi-K2 GLM-5 GLM-5 DeepSeek DeepSeek Kimi-K2 Kimi-K2 GLM-5
667 666 224 224 315 313 316 317
152 157 33 35 67 77 60 61
0.259 0.266 0.161 0.170 0.229 0.265 0.196 0.199
observed KBF signal. For each of the resulting eight pairs we simulate fixed-probability routing on a 5% grid over π ∈ {0.05, 0.10, . . . , 1.00} from recorded per-probe behavior, apply the two-round test, and estimate MDR65 , MDR80 , and MDR95 by probe-level Monte Carlo. Table 8 lists the pairs, and Figure 5a reports their TPR curves. MDR95 ranges from 5% (Opus → Kimi-K2) to 50% (both Sonnet-based pairs). The driver is n01 : Opus-based pairs have large n01 and expose small routing fractions, while Sonnet → GLM-5 has small n01 and stays hard even at a comparable N . Routing-fraction recovery. Our main protocol answers whether the endpoint is inconsistent with the claimed reference, and the extended protocol estimates how much traffic was routed away. We evaluate the routing-fraction estimator from Section 3.6 under two scenarios on the same eight pairs. Scenario A assumes the substitute model is identified, for example from a prior audit, and reports the point estimate π̂ with a 95% delta-method confidence interval. Scenario B treats the substitute as unknown but drawn from the fourmodel candidate pool and reports the interval [π̂min , π̂max ]. We sweep πtrue ∈ {0.05, 0.10, . . . , 0.95} with M = 10,000 probe-level Monte Carlo trials per cell. Figure 5 shows the estimator on the rank-1 substitute of each reference. In Scenario A, the estimate tracks the diagonal closely: |bias| < 0.005 everywhere, and SE shrinks as n01 grows. For the most informative pairs (Opus → KimiK2 and Opus → GLM-5), SE stays below 0.04 across the grid. For the smallest pair (Sonnet → GLM-5, n01 = 33), SE reaches about 0.07 at π = 0.5, still enough to separate coarse routing regimes such as 25%, 50%, and 75%. Scenario B adds substitute-identity uncertainty. When the auditor only knows that S lies in C , the candidate interval widens to about 0.15–0.30 across most of the grid. This width does not disappear by repeating the same probes; it reflects uncertainty about which substitute generated the mismatches. Mean coverage exceeds 80% when the actual substitute lies inside the candidate range, and drops toward 50% at candidate-set extremes. In practice, the auditor should use a candidate pool broad enough to cover plausible substitutes
4.6. Adaptive Routing Detection A relay can evade more subtly by routing only part of its traffic to a substitute. The relay claims to serve a flagship model R, but routes each request to a cheaper substitute S with probability π . We aim to answer two questions: • How small can π be before KBF detects the deviation? • Once detected, how well can KBF estimate the routed fraction? Key takeaway. KBF detects mixed routing when the reference and substitute differ on enough boundary probes. In our eight-pair study, Opus-based substitutions reach 95% TPR at 5–10% rerouting, while the hardest Sonnet-based pairs require 50%. When the substitute is known, the tworound estimator is essentially unbiased (|bias| < 0.005); when the substitute is only known to lie in a candidate pool, the estimate widens by the candidate-identification uncertainty. Partial-Routing Detection. We use the two-round extension from Section 3.6 and report the minimum detectable routing fraction MDRX : the smallest π for which the audit reaches at least X% TPR at α = 0.05. We evaluate four flagship references, each paired with its two strongest substitutes (lowest mismatch rate pS ) from the candidate pool C = {GLM-5, Qwen3.5-397B-A17B, Kimi-K2-0905, DeepSeek-V3.2}, which gives the adversary the smallest
14
' H W H F W L R Q 3 R Z H U
7 Z R 5 R X Q G ' H W H F W L R Q 3 R Z H U Y V 5 R X W L Q J ) U D F W L R Q
5 H I H U H Q F H 6 X E V W L W X W H 2 S X V . L P L . * H P L Q L . L P L . 2 S X V * / 0 * H P L Q L ' H H S 6 H H N 6 R Q Q H W ' H H S 6 H H N * 3 7 * / 0 6 R Q Q H W * / 0 * 3 7 . L P L .
low and upgraded pricing tiers, and the low tiers are flagged much more often. This within-platform contrast is difficult to explain by prompt format, reference drift, or ordinary deployment noise, and supports KBF’s endpoint-level audit signal. Background and Field Setting. The same agent-driven push toward cheap reseller endpoints described in the introduction has produced a sizable shadow ecosystem of API platforms that advertise access to flagship models at a fraction of the official price, with no technical means for the user to verify that the served model matches the claim. Independent measurement work has already documented this pattern at industry scale, including 17 distinct shadow APIs cited in 187 academic papers and quantitative evidence of widespread model substitution [1]. What is still missing is an off-the-shelf protocol that any user can run on any platform. We close that gap here by deploying KBF, unmodified, on six currently operating shadow APIs and asking two questions: (Q1) does the controlled-evaluation behavior of no same-reference false positives and 155/155 controlled substitution detections survive in the wild, and (Q2) are endpoint inconsistencies distributed uniformly across models, or are they economically targeted? Setup. We audit six platforms, anonymized as Platform 1 through Platform 6; a salted SHA-256 mapping is provided in Appendix C. Each platform exposes an OpenAI-compatible chat-completions interface and markets itself as a low-cost gateway to flagship models. For each platform we query whichever subset of six flagship-tier targets (Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, Gemini 3 Flash, GLM-4.7, DeepSeek-V3.2) the platform actually exposes; endpoints that the platform does not list, or that are unavailable through the upstream gateway, are left blank. Every audited endpoint uses the reference probe set, temperature=0, and the same 99% CP binomial test used in Section 4.2. The most expensive flagged endpoints (Claude on Platform 1 and Platform 5) are re-measured across three independent rounds to rule out transient noise; the remaining endpoints are single-round. The full survey of 27 platform×model endpoints costs roughly $10 at OpenRouter list prices. Flagged Inconsistencies Concentrate on Premium Proprietary Models. Table 9 reports the per-endpoint mismatch rates and verdicts. Rows are ordered by per-model flag rate; the right-most column gives the fraction of platforms on which the model is flagged. The pattern is immediate. Claude Opus 4.6 is flagged on 3 of 4 platforms that serve it, and Claude Sonnet 4.6 on 3 of 6. Among the four remaining models, three (GPT-5.4, DeepSeek-V3.2, Gemini 3 Flash) are consistent on every platform that exposes them, and GLM-4.7 is flagged on a single platform only (Platform 2). Aggregating across rows, Claude accounts for 6 of the 7 flagged endpoints in the survey: the two most expensive proprietary models in our pool concentrate the vast majority of endpoint inconsistencies. Figure 6 renders the same permodel consistency split as a horizontal stacked bar chart, making the Claude concentration visible at a glance. This pattern is consistent with the basic economic incentive: replacing a $3–$5/M token endpoint with a cheaper backend
5 R X W L Q J ) U D F W L R Q
(a) True-positive rate vs. routing fraction π (eight pairs).
2 S X V . L P L . Q
6 R Q Q H W * / 0 Q
* H P L Q L ' H H S 6 H H N Q
* 3 7 . L P L . Q
( V W L P D W H G
( V W L P D W H G
7 U X H 5 R X W L Q J ) U D F W L R Q
7 U X H 5 R X W L Q J ) U D F W L R Q
(b) Routing-fraction recovery π̂ vs. πtrue (best pair per reference).
Figure 5: Adaptive routing results under the two-round protocol. (a) True-positive rate as the routed fraction π increases across eight reference–substitute pairs; Opus-based pairs reach 95% TPR at π = 5–10%, while Sonnet-based pairs require π = 50%. (b) Routing-fraction recovery on the rank-1 substitute of each reference. Legend: Scenario A: 95% sampling band (known S , ±1.96 · SE); Scenario B: identification range (unknown S in 4-candidate pool); y = x (perfect estimate). when S is not already identified.
4.7. Shadow API Audit We finally run KBF as a black-box field audit on six shadow API platforms that advertise low-cost access to flagship models. Key takeaway. KBF flags 7/27 platform-model endpoints, with 6 of the 7 flags concentrated on Claude endpoints. The strongest validation comes from the tier split shown in Figure 7: the same advertised model identifier is served under
15
1
3
Claude Sonnet 4.6
3
1
GLM-4.7
Number of platforms (n = 7)
3
Claude Opus 4.6
3 6
GPT-5.4 4
DeepSeek-V3.2
Flagged Not flagged
3
Gemini 3 Flash 0
1
2
3
4
5
6
Number of shadow API platforms
7
Low tier High tier
6
6
5
5
5
5
4 3 2
2
2
1 0
Opus SAME
Sonnet SAME
1
1
Opus Flagged
Sonnet Flagged
Figure 6: Per-model consistency split across the six-platform shadow API field audit. Each bar covers all platforms that expose the model; the red segment counts platforms on which KBF rejects the null at p < 0.05 (flagged as inconsistent), and the green segment counts platforms on which the null is retained (consistent). Six of the seven flagged endpoints are concentrated on the two Claude endpoints.
Figure 7: Tier-dependent inconsistency across 7 shadow API platforms (n = 7). Bar colors: Low tier, High tier. Grouped by verdict: Opus SAME, Sonnet SAME, Opus Flagged, Sonnet Flagged. Low-tier endpoints are flagged more often, while high-tier endpoints are more likely to be consistent with the reference.
yields a much larger margin than replacing a $0.10/M endpoint. GPT-5.4, despite priced just below Sonnet at $2.50/M, is consistent on all six platforms that expose it—suggesting that supply-side availability of the official endpoint also matters alongside headline price. The cross-platform view of the same table also confirms KBF’s behavior in the wild. Platform 4 flags 0 of 6 models, with the Opus self-comparison yielding b/c=6/6, indistinguishable from the controlled-evaluation diagonal of Section 4.2. GLM-4.7 is flagged only on Platform 2 (17.4%, three rounds stable) while remaining within tolerance on the other three platforms that expose it (6.8–8.7%); KBF therefore separates a platform-specific supply problem from a model-wide trust issue. The aggregate flag rate across the entire survey is 7/27 (26%), in the same order of magnitude as—but more conservative than—the 45.8% fingerprintfailure rate reported by [1], which is consistent with our stricter binomial rule and finer-grained probes. Case Study. Seven audited platform-tier endpoints expose the same published model names under two pricing tiers. Figure 7 compares the number of platform-tier endpoints that KBF finds consistent or flagged for Claude Opus 4.6 and Claude Sonnet 4.6. On Claude Opus, the low tier is flagged on 5 of 7 endpoints while the upgraded tier is flagged on only 1 of 7. On Claude Sonnet, the low tier is flagged on 2 of 7 endpoints while the upgraded tier is flagged on 1 of 7. The high tiers therefore behave much closer to the official reference, while the low tiers show a clear increase in substitution-like mismatches. This pattern is especially striking because it appears under the same advertised model identifiers on the same platforms: users paying the upgrade fee receive behavior that is statistically closer to the reference, whereas low-tier users are more likely to receive a different backend. Summary. Across the six-platform survey, KBF retains its controlled-evaluation behaviour: the only fully consistent platform (Platform 4) is correctly recognised as such, and every flagged endpoint exceeds the per-model 99% CP null
bound by a wide margin. The flagged inconsistencies are overwhelmingly concentrated on the most expensive proprietary endpoints (Claude), and two platforms exhibit tier-dependent behavior consistent with serving different backends under the same model identifier. Together these results turn the controlled false-positive evidence of Section 4.2 into a usable field-level claim: a single $10 audit run is sufficient to separate endpoints that are statistically consistent with the reference from premium-model endpoints requiring further investigation.
4.8. Operational Use and Limitations
KBF should be used as a statistical triage tool, not as standalone proof of provider intent or backend identity. A rejected audit means that the suspect endpoint is statistically inconsistent with the reference endpoint on the enrolled probe set under the audited interface. Provider logs, independent route confirmation, or repeated audits under fresh private probes are needed before making a definitive claim about provider intent, routing policy, or contractual breach. The current evaluation also leaves three empirical questions open. First, the false-positive evidence is conservative across the controls we ran, but it is not yet the severalhundred-trial calibration that a high-stakes operational deployment should prefer. Second, the primary p-values use a binomial model even though probes can be correlated by domain and prompt template; a domain-block bootstrap or effective-sample-size correction would make the reported significance less dependent on this simplifying assumption. Third, the main results evaluate the full KBF pipeline with optional contrastive screening enabled when appropriate. An unscreened ablation would separate the contribution of knowledge-boundary probe generation from the extra power gained by filtering against likely substitutes.
16
TABLE 9: Field audit of six anonymized shadow API platforms with KBF. Platform columns use salted SHA-256 identifiers (see Appendix C for the full mapping). Each entry shows the suspect-side mismatch rate; ✓ indicates that the binomial test does not reject the null (consistent), and × indicates rejection at p < 0.05 (flagged as inconsistent). Numbers in parentheses next to each model name are the reference self-error. “—” marks platforms that do not expose the model or whose gateway was unavailable. The Platform 5 entry for Claude Opus refers to the default tier. Model (self-err) Claude Opus 4.6 (4.3%) Claude Sonnet 4.6 (1.3%) GLM-4.7 (4.6%) GPT-5.4 (1.6%) DeepSeek-V3.2 (3.3%) Gemini 3 Flash (2.2%) Flagged / tested †
P1
P2
P3
P4
P5 †
P6
Flagged / tested
× 19.7% × 13.4% ✓ 6.8% ✓ 3.2% ✓ 8.3% ✓ 2.5%
× 8.2% ✓ 2.8% × 17.4% ✓ 4.4% ✓ 3.3% ✓ 5.7%
— ✓ 2.7% — ✓ 0.9% — —
✓ 4.3% ✓ 2.2% ✓ 8.4% ✓ 2.2% ✓ 3.0% ✓ 2.5%
× 47.9% × 43.8% ✓ 8.7% ✓ 1.6% ✓ 3.6% —
— × 13.8% — ✓ 1.9% — —
3/4 3/6 1/4 0/6 0/4 0/3
2/6
2/6
0/2
0/6
2/5
1/2
7 / 27 (26%)
Default tier on Platform 5.
5. Additional Related Work
[32], [33], [34]. These techniques are valuable for content provenance and ownership verification when the provider participates. Relay auditing asks whether an uncooperative intermediary served the claimed upstream endpoint. A watermark becomes useful in this setting only when the official provider exposes a trusted verification mechanism to relay users. Verifiable inference. Verifiable inference provides cryptographic or hardware-backed evidence about provider-side computation, through zero-knowledge proofs, trusted execution environments, or selective verified audits [6], [7], [8], [16], [17]. These systems can provide stronger guarantees when the provider participates. KBF targets the complementary case where the relay user has only ordinary black-box API access.
Additional black-box fingerprinting. LLMPrint builds fingerprints from prompt-injection prompts that enforce modelspecific token preferences [13]. TRAP uses adversarial suffix honeypots to elicit target responses from selected models [19]. These probes are effective for model identification and compliance checks. In relay auditing, the same structure exposes visible security artifacts, can trigger safety layers, and gives a relay a clear pattern to special-case. ZeroPrint also treats TRAP as a targeted, semi-black-box baseline because it assumes stronger source-model access than untargeted methods [11]. Other black-box methods use learned or representationspace evidence. FDLLM trains a detector over labeled generations from a fixed model set [20]; Sentence-Embedding Fingerprinting (SEF) averages response embeddings over task prompts [11]; and Yang and Wu authenticate ownership from logit or probability vector spaces with owner-side victim-model evidence [25]. These methods are useful for provenance or candidate-set classification. Relay auditing instead needs a low-false-positive consistency test for one advertised endpoint under wrappers, updates, and mixed routing. We therefore use LLMmap, MET, and ZeroPrint as primary empirical baselines: they match our access level and cover behavioral retrieval, distributional equality testing, and perturbation-based fingerprints. White-box and grey-box fingerprinting. Other provenance methods assume stronger access than an ordinary relay user has. Some insert model-specific signals before deployment, through backdoors, instructional fingerprints, domain-specific watermarks, or memorized ownership evidence [3], [5], [26], [27]. Others read internal computation or richer output channels, including attention differences, gradients, logits, or log probabilities [9], [18], [28], [29], [30]. These methods fit model theft, lineage, and owner-side provenance settings. A relay auditor cannot observe weights, logits, routing logs, hidden prompts, or provider metadata; it can only query the official reference endpoint and the suspect endpoint. Watermarking. Watermarking embeds detectable signals into sampling, semantics, or model behavior [4], [31],
6. Conclusion This paper studies model-identity auditing for relay and reseller APIs under black-box access. We introduced KBF, which builds endpoint-specific fingerprints from stable numerical recall near a reference model’s knowledge boundary and uses a CP binomial test to compare suspect endpoints against the enrolled reference. Across 16 production endpoints spanning eight model families, KBF detected all 155 economically relevant substitutions with no false-positive rejection on same-reference controls. It remained robust under deployment-wrapper changes and detected mixed routing at economically meaningful substitution rates. In realworld shadow API auditing, KBF found that 7 of 27 tested endpoints were statistically inconsistent with their references, with the inconsistencies concentrated on premium Claude AI accesses. Future work. KBF audits whether the endpoint is serving a claimed model. Future work should extend this view in two directions. First, stronger statistical calibration is needed for high-stakes operational use: repeated same-endpoint audits across days, accounts, provider routes, regions, and fresh private probe sets, together with cluster-robust tests or conservative effective-sample-size corrections. Second, auditors should extend the target of measurement beyond
17
model identity. Routing transparency is one direction: a relay may fall back, mix providers, or apply user-specific routing policies while still advertising a single model. Message integrity is another important direction. Recent work shows that malicious intermediaries can inject tool payloads and observe secrets in transit [35]; auditors need black-box tests for request rewriting, response mutation, and credential exposure. Last but not least, billing transparency, including inflated token counts, hidden cache hits, batch-price arbitrage, and account or free-credit resale, should also be included in a full-fledged model serving audit solution.
[14] Y. Mahajan, N. Bansal, and S. K. Karmaker, “The daunting dilemma with sentence encoders: Success on standard benchmarks, failure in capturing basic semantic properties,” arXiv preprint arXiv:2309.03747, 2023. [15] C. Mao, S. Geng, J. Yang, X. Wang, and C. Vondrick, “Understanding zero-shot adversarial robustness for large-scale models,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=P4bXCawRi5J [16] M. Sabt, M. Achemlal, and A. Bouabdallah, “Trusted execution environment: What it is, and what it is not,” in 2015 IEEE Trustcom/BigDataSE/Ispa, vol. 1. IEEE, 2015, pp. 57–64. [17] F. Tramer and D. Boneh, “Slalom: Fast, verifiable and private execution of neural networks in trusted hardware,” arXiv preprint arXiv:1806.03287, 2018.
Acknowledgment
[18] Y.-Y. Tsai, C. Guo, J. Yang, and L. van der Maaten, “Rofl: Robust fingerprinting of language models,” arXiv preprint arXiv:2505.12682, 2025.
References [1]
Y. Zhang, Y. Jiang, Z. Chen, M. Backes, X. Shen, and Y. Zhang, “Real money, fake models: Deceptive model claims in shadow apis,” arXiv preprint arXiv:2603.01919, 2026.
[2]
K. Li, S. Zhuang, Y. Zhang, M. Xu, R. Wang, K. Xu, X. Fu, and X. Cheng, “I’m spartacus, no, i’m spartacus: Measuring and understanding llm identity confusion,” arXiv preprint arXiv:2411.10683, 2024.
[3]
[19] M. Gubri, D. Ulmer, H. Lee, S. Yun, and S. J. Oh, “Trap: Targeted random adversarial prompt honeypot for black-box identification,” in Findings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 11 496–11 517. [20] Z. Fu, J. Chen, L. Zhang, T. Yang, J. Niu, H. Sun, R. Li, P. Liu, J. Wang, F. He et al., “Fdllm: A dedicated detector for black-box llms fingerprinting,” in 2025 IEEE 24th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom). IEEE, 2025, pp. 1374–1379.
Y. Adi, C. Baum, M. Cisse, B. Pinkas, and J. Keshet, “Turning your weakness into a strength: Watermarking deep neural networks by backdooring,” in 27th USENIX security symposium (USENIX Security 18), 2018, pp. 1615–1631.
[4]
M. Russinovich and A. Salem, “Hey, that’s my model! introducing chain & hash, an llm fingerprinting technique,” arXiv preprint arXiv:2407.10887, 2024.
[5]
J. Xu, F. Wang, M. Ma, P. W. Koh, C. Xiao, and M. Chen, “Instructional fingerprinting of large language models,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024, pp. 3277–3306.
[6]
H. Sun, J. Li, and H. Zhang, “zkllm: Zero knowledge proofs for large language models,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, pp. 4405–4419.
[7]
M. M. Maheri, H. Haddadi, and A. Davidson, “Telesparse: Practical privacy-preserving verification of deep neural networks,” arXiv preprint arXiv:2504.19274, 2025.
[8]
Y. Guo, W. Qu, L. Wu, S. Zhai, L. Z. Wang, M. Xu, Y. Liu, B. Yuan, D. Song, and J. Zhang, “Immaculate: A practical llm auditing framework via verifiable computation,” arXiv preprint arXiv:2602.22700, 2026.
[9]
A. Nasery, E. Contente, A. Kaz, P. Viswanath, and S. Oh, “Are robust llm fingerprints adversarially robust?” arXiv preprint arXiv:2509.26598, 2025. [Online]. Available: https://arxiv.org/abs/ 2509.26598
[21] NASA Jet Propulsion Laboratory, “JPL Small-Body Database Lookup,” https://ssd.jpl.nasa.gov/tools/sbdb lookup.html, accessed May 9, 2026. [22] The OEIS Foundation Inc., “The On-Line Encyclopedia of Integer Sequences,” https://oeis.org/, accessed May 11, 2026. [23] M. M. Wagner-Menghin, “Binomial test,” Encyclopedia of statistics in behavioral science, 2005. [24] Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947. [25] Z. Yang and H. Wu, “A fingerprint for large language models,” arXiv preprint arXiv:2407.01235, 2024. [26] T. Gloaguen, R. Staab, N. Jovanović, and M. Vechev, “Robust llm fingerprinting via domain-specific watermarks,” in ICML 2025 Workshop on Reliable and Responsible Foundation Models, 2025. [27] Z. Xu, M. Han, and W. Xing, “Evertracer: Hunting stolen large language models via stealthy and robust probabilistic fingerprint,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 7019–7042. [28] H. Zhang, Z. Xu, J. Li, S. Sheng, D. Kong, and M. Han, “Attndiff: Attention-based differential fingerprinting for large language models,” arXiv preprint arXiv:2604.05502, 2026. [29] Z. Wu, Y. Zhao, and H. Wang, “Gradient-based model fingerprinting for llm similarity detection and family classification,” arXiv preprint arXiv:2506.01631, 2025.
[10] I. Gao, P. Liang, and C. Guestrin, “Model equality testing: Which model is this api serving?” arXiv preprint arXiv:2410.20247, 2024.
[30] M. Finlayson, X. Ren, and S. Swayamdipta, “Every language model has a forgery-resistant signature,” arXiv preprint arXiv:2510.14086, 2025.
[11] S. Shao, Y. Li, H. Yao, Y. Chen, Y. Yang, and Z. Qin, “Reading between the lines: Towards reliable black-box llm fingerprinting via zeroth-order gradient estimation,” in Proceedings of the ACM Web Conference 2026, 2026, pp. 2637–2648.
[31] J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein, “A watermark for large language models,” in International conference on machine learning. PMLR, 2023, pp. 17 061–17 084.
[12] D. Pasquini, E. M. Kornaropoulos, and G. Ateniese, “{LLMmap}: Fingerprinting for large language models,” in 34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 299–318.
[32] S. Dathathri, A. See, S. Ghaisas, P.-S. Huang, R. McAdam, J. Welbl, V. Bachani, A. Kaskasoli, R. Stanforth, T. Matejovicova et al., “Scalable watermarking for identifying large language model outputs,” Nature, vol. 634, no. 8035, pp. 818–823, 2024.
[13] Y. Hu, Z. Jiang, M. Li, O. Ahmed, Z. Huang, C. Hong, and N. Gong, “Fingerprinting llms via prompt injection,” arXiv preprint arXiv:2509.25448, 2025.
18
astronomy, biology, mathematics, programming history, cryptographic parameters, medical half-lives, pop culture, internet culture, Chinese history, Chinese geography, Chinese internet events, and Chinese literature. Each domain has an audit prompt template, a valid numerical range, and a comparison rule. Integer-valued domains use exact agreement after rounding; chemistry domains use absolute tolerances; continuous scientific domains use relative tolerances. The main text reports only the high-level domain families; the implementation stores the per-domain prompt template, value range, and tolerance together so that generation, verification, and auditing use the same matching rule. Audit prompt rendering. Several domains use a short fill-in-the-blank sentence as the audit prompt template, such ◦ as “The boiling point of X at 1 atm is C.” This reduces response formatting variance, keeps the requested numerical slot unambiguous, and allows the same parser to be used during recovery, reference-consistency checks, self-tests, and online audits. Domains whose facts are more naturally expressed as direct questions use the same short-answer interface. Candidate parsing and de-duplication. Candidate generation asks the reference endpoint for name | value records. The parser removes list markers, formatting artifacts, common unit symbols, Unicode minus signs, commas, and approximate-value markers before extracting the first numeric value. Candidates with duplicate normalized names or values outside the domain range are discarded before any auditformat verification query is issued. Audit-format parser. All online comparisons use the same position-aware numerical parser. Batched probes are numbered as (N). If the response preserves numbering, the parser maps each numeric answer back to its intended slot; if numbering is absent, it falls back to sequential line-by-line extraction. Empty lines, markdown separators, tables, and hidden reasoning tags are ignored. A parsed value outside the domain’s valid range is treated as invalid. Invalid or missing answers do not receive semantic repair; they count as non-matches once the corresponding reference value is valid. Filtering counts and metadata. For each reference model, the implementation records the generated candidates, the probes that pass deterministic audit-format recovery, the probes removed by optional contrastive screening, the threerun consensus values, the difficulty tier of each retained probe, and the final self-test result. The saved self-test includes the per-probe match vector, self-discrepancy count, total enrolled probes, self-error rate, request usage, timestamp, and audit configuration. These fields are used by the CPγ calibration step in Section 3 and by the empirical robustness checks in Sections 4.4 and 4.5.
[33] A. Hou, J. Zhang, T. He, Y. Wang, Y.-S. Chuang, H. Wang, L. Shen, B. Van Durme, D. Khashabi, and Y. Tsvetkov, “Semstamp: A semantic watermark with paraphrastic robustness for text generation,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024, pp. 4067–4082. [34] J. Huo, S. Liu, B. Wang, J. Zhang, Y. Yan, A. Liu, X. Hu, and M. Zhou, “Pmark: Towards robust and distortion-free semantic-level watermarking with channel constraints,” arXiv preprint arXiv:2509.21057, 2025. [35] H. Liu, C. Shou, H. Wen, Y. Chen, R. J. Fang, and Y. Feng, “Your agent is mine: Measuring malicious intermediary attacks on the llm supply chain,” arXiv preprint arXiv:2604.08407, 2026.
Appendix A. Experimental Setup This appendix records the request-side configuration used in the KBF pipeline. We focus on the parameters that affect probe content and audit behavior: prompt templates, temperatures, batch sizing, provider pinning, and the number of batched requests implied by a single audit.
A.1. Prompts and Request Configuration Across the pipeline we use only two short, task-oriented prompt types, a system prompt and a user prompt, and avoid complex behavioral instructions. The fixed system prompt (R ECALL S YS) is “Follow the user’s instructions exactly. Output only what is requested.”, and the recall user prompt (R ECALL U SER) is “TASK: Answer these factual recall questions using only values stored in your weights. Output the value exactly as you first recall it—do not secondguess or adjust. [. . . ] Do not use any context from system instructions or prior conversation.” The canonical audit instruction is simply “Fill in the blank. Output only the number.”, which the batched R ECALL U SER realizes for numbered multi-probe requests. Probe generation runs three request-facing stages: candidate generation, stability verification, and optional contrastive screening. A standard audit then takes the screened probes of one reference model and sends them to a target or suspect endpoint in fixed-size batches at temperature 0.0, reusing the R ECALL S YS + R ECALL U SER pair so the audit measures only whether the target reproduces the same stored numerical values. The main paper reports the resulting mismatch count and 99% CP binomial test. Table 10 reports the model and temperature settings for the three probe-generation stages and for a single standard audit.
A.2. Implementation Details for Probe Construction
Appendix B. Baseline Implementation Details
This subsection documents the concrete implementation choices behind the screened probe sets used in the evaluation. These details support reproducibility because they determine which prompts survive into the final audit set. Domains and templates. KBF searches over 15 numerical domains: boiling points, melting points, physical constants,
We adapt each baseline to the same endpoint pool used by KBF and configure it for the corresponding black-box relay-audit decision.
19
TABLE 10: Request-side model and temperature settings across the three probe-generation stages and a single standard audit. Phase
Stage
Model / comparator
Temperature
Probe generation
Candidate generation Stability verification Contrastive screening
Reference model Reference model Qwen3.5-9B
0.0 t0 =0.0, ta0.7 =0.7, tb0.7 =0.7 0.0
Single audit
Standard audit
Target endpoint
0.0
Appendix C. Shadow API Platform Mapping
LLMmap. LLMmap is an enrollment-and-retrieval fingerprint. We extend the pretrained open-set library with 16 new model templates. Enrollment collects 75 prompt configurations with 8 queries each, yielding 600 API responses per model. At test time, the unknown endpoint answers 8 queries in the bare-query setting at temp=0. LLMmap then converts the responses into a feature vector and ranks all enrolled templates by Euclidean distance.2 We report Top-1 consistency in the main comparison because relay auditing requires a concrete claimed-reference decision. Topk retrieval is a model-discovery metric.
The box below lists the mapping from the shortened identifiers used in Section 4.7 to their salted SHA-256 digests. Shadow API platform mapping Platform 1 Platform 2 • Platform 3 • Platform 4 • Platform 5 • Platform 6 • •
MET. MET is a two-sample distributional test. We use 25 multilingual Wikipedia continuation prompts sampled from a 500-record candidate pool. For each model and batch, we collect 10 completions per prompt, giving 250 completions per batch and two independent batches per model. Completions are encoded as Unicode codepoints and padded or truncated to length 1000. The test statistic is MMD with a Hamming kernel; p-values are estimated with 1000 permutations for the false-positive and cross-model tests. ZeroPrint. ZeroPrint uses structured perturbations to estimate local response sensitivity. We query 2 HumanEval code-completion prompts. For each prompt we generate four variants by replacing three eligible words with one of their top-10 GloVe neighbors, and we issue each prompt and its variants 20 times, for 200 requests per model. The implementation computes a Jacobian-style fingerprint from the response changes and compares fingerprints with Pearson similarity. We do not reuse the predefined similarity threshold from the original ZeroPrint evaluation, because it is fixed on base-and-derivative model pairs and does not suit the relay-auditing setting. We instead fingerprint each endpoint in two independent collections and build a reference library from the first collection. We then pool all query–reference similarity scores and set a single global decision threshold at the Youden-optimal operating point on this pool, matching the original ZeroPrint protocol of selecting one threshold over the full set of scores. A query is assigned to a claimed reference when its similarity to the library fingerprint of that reference meets the global threshold, providing the concrete claimed-reference decision that relay auditing requires.
2. We follow the public implementation and use Euclidean distance for the ranking computation.
20
↔ ↔ ↔ ↔ ↔ ↔
SH-87b19c56 SH-3c82a49c SH-4f754cc3 SH-75f73c6e SH-0989276f SH-031d44cf