S TEMMA : I NDUCED D ECISION R EGIONS R EVEAL LLM P ROVENANCE Keyu Zhang Vadim Safronov Andrew Martin Department of Computer Science, University of Oxford {keyu.zhang,vadim.safronov,andrew.martin}@cs.ox.ac.uk
arXiv:2607.25880v1 [cs.CR] 28 Jul 2026
A BSTRACT LLM provenance testing asks whether a suspect LLM belongs to the same lineage as a source. Existing black-box methods largely infer this relationship from response-level characteristics, but these characteristics may shift under adaptation or deployment even when the underlying meaning remains unchanged, weakening the reliability of provenance evidence. To address this limitation, we introduce induced decision regions by mapping open-ended outputs into a finite decision space, thereby abstracting away surface-form variation and reframing provenance testing as measuring the inheritance of decision regions. Empirical analysis shows that the source’s induced regions are preserved more strongly in related models than in unrelated models. Building on this signal, we propose Stemma, a practical black-box LLM fingerprinting method that operationalises stability, robustness, and specificity as complementary probe-selection principles for reliably estimating induced decision region inheritance. Across 770 source–suspect pairs drawn from 56 public checkpoints and spanning diverse model-weight transformations, Stemma achieves 0.967 AUC and 87.8% TPR at 1% FPR, substantially outperforming four representative baselines. It further achieves 0.995 AUC and 93.5% TPR at 1% FPR on 1,260 pairs covering 91 deployment instances, demonstrating robustness to diverse inference-time deployment settings.
1
I NTRODUCTION
Large language models (LLMs) have become widely deployed infrastructure for text generation, code assistance, and reasoning. Across their development and deployment lifecycles, models may undergo fine-tuning, merging, compression, or distillation, while prompt templates and decoding settings further shape their observable behaviour (Dettmers et al., 2023; Yu et al., 2024; Shi et al., 2024). This complexity obscures model provenance, raising intellectual-property and licensing concerns when protected models are reused without authorisation (Stalnaker et al., 2025), as well as accountability concerns when unsafe behaviours, biases, or vulnerabilities propagate to downstream models (Hammoud et al., 2024; Zhang et al., 2024). These concerns motivate model provenance testing, which assesses whether a suspect model belongs to the same lineage as a source. Black-box model fingerprinting provides a practical approach to provenance testing. It characterises a source model through observable behavioural signatures that are preserved across provenancerelated models while remaining distinguishable from independently developed models (Jiang et al., 2026). Because verification requires only query access to the suspect model, it remains applicable when model weights, internal activations, training data, and development history are unavailable. The central challenge is to identify signatures that are both persistent under adaptation and specific to the source. In classification models, the fixed label space naturally partitions inputs into decision regions. Existing methods thus exploit this structure by selecting boundary-adjacent, adversarial, or error-region inputs and measuring whether source and suspect models assign them to the same regions (Cao et al., 2021; Godinot et al., 2025; Guan et al., 2022). Derived models often inherit these fine-grained structures, whereas independently trained models are less likely to reproduce them. This paradigm does not transfer directly to LLMs. Open-ended generation produces an unstructured output space in which the same meaning may take many surface forms, making decision regions difficult to define and compare from black-box outputs. Existing black-box LLM fingerprinting meth1
ods instead rely largely on output similarity, elicited behaviours, or perturbation responses (Nikolic et al., 2025; Gubri et al., 2024; Shao et al., 2026). Because these signals depend on particular response realisations, they may shift under prompting, deployment, or model adaptation, weakening the reliability of provenance testing. We address this limitation by introducing induced decision regions for LLMs. By mapping outputs into a finite decision space, we abstract away surface-form variation, make these regions observable, and reframe provenance testing as measuring the inheritance of decision regions. Our analysis shows that related models preserve the source’s induced regions more strongly than unrelated models. Building on this formulation, we propose Stemma1 as a black-box LLM fingerprinting method for provenance testing. Stemma detects induced decision region inheritance through probes selected for stability across equivalent input representations, robustness to boundary shifts, and specificity against unrelated models. Across various model families, adaptation types, and deployment settings, Stemma achieves stronger and more consistent provenance separation than existing methods. Contributions.
Our contributions are summarised as follows.
• We introduce induced decision regions for LLMs, reframing black-box provenance testing from comparing response-level characteristics to measuring the inheritance of decision regions, and show that related models preserve these regions more strongly than unrelated models. • We propose and open-source Stemma, a black-box LLM fingerprinting method that tests provenance through induced decision region inheritance, with fingerprint construction guided by stability, robustness, and specificity. • We conduct comprehensive evaluations across 56 public checkpoints and 91 deployment instances, spanning diverse model families, adaptation types, and deployment settings. Stemma substantially outperforms four representative black-box baselines across multiple metrics, demonstrating consistently stronger provenance separation.
2
BACKGROUND AND P ROBLEM S ETTING
2.1
R ELATED W ORK
Methods for model ownership and provenance testing fall into watermarking and fingerprinting (Ye et al., 2026). Watermarking embeds identifiable signals into model parameters or behaviours, requiring prior control of the protected model and restricting verification to models marked in advance (Uchida et al., 2017; Adi et al., 2018; Li et al., 2023; Gloaguen et al., 2025). Fingerprinting is noninvasive, instead exploiting distinctive characteristics that arise naturally during model development. Depending on access to the suspect model, fingerprinting methods can be classified as white-box or black-box: white-box methods use parameters, activations, or internal representations and therefore require internal access (Jia et al., 2021; Zhang et al., 2025; Yu & Wang, 2024), whereas black-box methods rely only on behaviours elicited through controlled queries, offering broader applicability but making reliable provenance testing more challenging. Existing black-box LLM fingerprinting methods can be broadly grouped by how they obtain and characterise observable behaviour. The first group directly characterises responses to selected queries. Model Provenance Testing measures next-token agreement between the source and suspect relative to independently trained control models, while DuFFin compares response similarity over sampled questions (Nikolic et al., 2025; Yan et al., 2026). LLMmap learns representations from selected queries and responses for model identity recognition, whereas ErrorTrace derives distinctive error signatures for model family attribution (Pasquini et al., 2025; Zang et al., 2025). Recent API-auditing methods further exploit output-rank distributions or response stability near estimated knowledge boundaries (Zhu et al., 2026; Fang et al., 2026). The second group constructs adversarially optimised prompts to elicit target behaviours. TRAP uses such prompts for model identification, while ProFLingo and RoFL test whether responses persist in downstream models (Gubri et al., 2024; Jin et al., 2024; Tsai et al., 2025). LLMPrint induces and 1 The name Stemma is inspired by stemmatic analysis, which infers textual lineage from inherited variants. Appendix K develops this analogy.
2
tests inheritance of preferences between target-token pairs (Hu et al., 2026). A third line, represented by ZeroPrint, characterises response variation under semantic perturbations through zeroth-order estimates of local Jacobians (Shao et al., 2026). Across these approaches, fingerprints are derived from response-level characteristics, including response patterns, elicited behaviours, and perturbation effects. These characteristics may shift with prompting, deployment, or weight adaptation without commensurate semantic changes, limiting the stability and comparability of provenance evidence. Prior work on deep classifiers shows that decision regions provide a structured and directly comparable basis for black-box provenance testing. IPGuard constructs probes near the source’s decision boundaries and tests whether the suspect model preserves the corresponding region assignments (Cao et al., 2021). Model Lineage Closeness Analysis measures preservation of decision geometry through differences in decision boundary distances and prediction agreement (Tang et al., 2025). ADV-TRA extends boundary-sensitive probes to adversarial trajectories that traverse decision boundaries, improving tolerance to boundary shifts (Xu et al., 2024). Open-ended LLM generation, however, does not naturally expose a finite decision space over which analogous regions can be defined and compared. Stemma therefore maps open-ended outputs into such a space, abstracting away surface-form variation and testing provenance through the inheritance of the decision regions. 2.2
P ROBLEM S ETTING
We study pairwise black-box model provenance testing for LLMs. Given a source model S and a suspect model instance T , the objective is to determine whether T is provenance-related to S. A model instance comprises an underlying model and the deployment configuration through which its behaviour is exposed. The pair is related if the underlying models of S and T are identical or linked through model derivation, possibly under different deployment configurations. Otherwise, the pair is unrelated, even if the models share similarities in architecture, training, or performance. Access assumptions. We assume access to the source model’s next-token logits for fingerprint construction, whereas the suspect model is available only through a black-box interface that accepts textual inputs and returns generated responses without exposing its parameters, logits, activations, training data, or development history. This asymmetric setting captures a practical provenanceauditing scenario in which the evaluator controls the source model and seeks to identify related model variants or deployed instances through exposed interfaces. Adversarial scope. We consider a non-adaptive setting in which the provider may know the general fingerprinting method but not the test-specific probes. The suspect model may undergo ordinary model development and deployment transformations, but neither the model nor its interface is adapted to evade the method by detecting, filtering, or altering responses to fingerprint queries.
3
I NDUCED D ECISION R EGIONS FOR LLM P ROVENANCE
3.1
F ORMALISATION
Formally, let Q denote the query space, O the observable model-output space, and D = {1, . . . , C} a finite decision space. For an LLM instance M , let oM (q) ∈ O denote its output on query q ∈ Q. Given a score extraction rule g̃ : Q × O → RC and a decision extraction rule g : Q × O → D, we define the model-specific soft and discrete decision maps as h̃M (q) = g̃ q, oM (q) ∈ RC , hM (q) = g q, oM (q) ∈ D. (1) The soft map h̃M assigns each query a real-valued score vector representing the model’s relative preferences over the available decisions, whereas the discrete map hM assigns it the selected decision. Depending on the available access, the value hM (q) is obtained either by taking arg maxd∈D h̃M,d (q), with ties resolved by a fixed deterministic rule, or by extracting it directly from generated text. In both cases, the resulting decision lies in the same finite space D. The discrete decision map hM induces a partition of the query space by grouping queries assigned to the same decision. For each decision d ∈ D, we define the induced decision region of M as RM d = {q ∈ Q | hM (q) = d}. 3
(2)
Conceptually, we refer to the arrangement of these regions and their separating boundaries as the induced decision geometry of M . For provenance testing, we use induced decision region inheritance to denote the extent to which induced decision geometry is preserved between a source model S and a suspect model T . Since this geometry cannot be compared exhaustively over Q, we assess its preservation using a finite probe set P ⊂ Q. When soft decision vectors are available for both models, inheritance can be assessed at a finer resolution through their similarity over P: X d P (S, T ) = 1 Sim sim h̃S (q), h̃T (q) , |P|
(3)
q∈P
where sim(·, ·) denotes a similarity measure between soft decision vectors. When comparison is restricted to discrete decisions, inheritance is assessed through alignment with the source region assignments over P: 1 X AbP (S, T ) = 1{hT (q) = hS (q)}. (4) |P| q∈P
d P and AbP indicate stronger estimated inheritance and hence stronger evidence Higher values of Sim of a provenance relationship. 3.2
E MPIRICAL E VIDENCE
We next examine whether induced decision region inheritance is observable in LLM behaviour. In general, any evaluation interface that maps open-ended outputs to a finite decision space can induce such regions, including constrained response formats, semantic output categories, and verifierdefined behavioural tests. In this paper, we instantiate this idea using multiple-choice questions, where the answer options define the decision space D, as formalised in Appendix A. To construct the visualisation in Figure 1, we use MMLU questions (Hendrycks et al., 2020) and cyclically permute each question’s answer options, treating each permutation as a separate query. For each query, we compute the next-token choice-label logits, normalise them over the four choices, and map the probabilities back to the canonical options. Averaging these probabilities across permutations yields a four-dimensional soft decision vector for each model–question pair while mitigating order and label biases (Zheng et al., 2024a; Pezeshkpour & Hruschka, 2024). We then order the vector dimensions by the source’s averaged preference ranking, such that Ri corresponds to its i-th ranked option, and apply the same ordering to the suspect vectors. Finally, each vector is projected into two dimensions. Details of the projection and annotation metrics are provided in Appendix B. R1
R4
Sim=0.75 A=32/40
Random: Related
R2
R1
R3
R4
R2 R1
Sim=0.56 A=21/40
Random: Unrelated
R3 R4
Sim=0.80 A=39/40
Stemma: Related
R2
R1
R3
R4
R2
Sim=0.33 A=10/40
Stemma: Unrelated
R3
Figure 1: Induced decision region inheritance on MMLU using 40 randomly sampled questions (left) and 40 Stemma-selected questions (right). The source is Qwen-2.5-7B, while the related and unrelated suspects are Qwen-2.5-7B-Instruct and Qwen3-1.7B, respectively. Each point represents a model’s projected decision for one question: grey points denote source decisions, while green and orange points denote suspect decisions that preserve or change the corresponding source-region assignment, respectively. Lines connect the source and suspect decisions corresponding to the same question. Annotations report average option-preference similarity (Sim) and the fraction of matching region assignments (A). 4
For the randomly sampled probes, the related suspect exhibits higher soft decision similarity than the unrelated suspect, 0.75 versus 0.56, and preserves more source region assignments, 32/40 versus 21/40. Thus, random probes already reveal inheritance in both soft decision preferences and discrete region assignments. Because induced decision geometry cannot be exhaustively observed, any finite probe set captures only a partial view of region inheritance, with its discriminative strength depending on which parts of the geometry are sampled. Stemma-selected probes yield substantially clearer separation, with soft decision similarity of 0.80 versus 0.33 and source-region alignment of 39/40 versus 10/40. Stemma therefore does not create the underlying provenance signal, but makes induced decision region inheritance more observable and discriminative.
4
S TEMMA D ESIGN
4.1
OVERVIEW
Stemma is a black-box LLM fingerprinting method that tests provenance through induced decision region inheritance. It instantiates induced decision regions using multiple-choice questions, whose candidate options define a finite decision space. The method comprises three stages, as illustrated in Figure 2. Decision interface calibration selects and fixes a prompt template for each model to enable reliable decision extraction. Fingerprint construction selects informative probes by evaluating the source model’s induced decisions for stability, robustness, and specificity. Fingerprint verification queries the suspect model with these probes and estimates induced decision region inheritance by measuring how strongly the suspect preserves the source decisions recorded in the fingerprint. Fingerprint Construction
Question: {question} {options} Answer:
Fingerprint Verification
Cyclic Permutations
Calibrated Prompt Template
Filter & Rank
A
A
A
A
B
B
B
B
Stability
C
C
C C
Source
C
B
D
A
......
Q2
B
Suspect
C
B
C
A
......
Q3
D
Q1
C
C
C
C
D
D
D
D
Robustness Which planet is closest to the Sun?
A
Venus
B
Mercury (Gold)
C margin
B
Background Model
Source Model
Specificity
C ≠ B C
Earth (Decision)
D
Mars
Verification
Fingerprint
C
&
Q4
A
......
......
Suspect Model
C ≠ B
gold
bg
Alignment Score A(S,T) ∈ [0, 1]
Candidate MCQs
Figure 2: Overview of the Stemma workflow. 4.2
D ECISION I NTERFACE C ALIBRATION
Different multiple-choice prompt formats may elicit substantially different output behaviours, including unparseable responses and strong label bias, making induced decisions difficult to extract reliably. Stemma therefore calibrates each model’s decision interface on held-out questions by excluding severely label-biased templates from a predefined set and selecting the template with the highest valid-choice extraction rate among the remainder. Because these templates preserve the same underlying questions and candidate options, calibration varies their interface rendering, while extracted decisions remain comparable in the same canonical decision space. Calibration is performed once before fingerprint construction and verification, after which the selected template is fixed for all subsequent queries to that model. The candidate prompt templates and calibration configuration are provided in Appendix C. 4.3
F INGERPRINT C ONSTRUCTION
Stemma constructs a finite source fingerprint by selecting question probes whose induced decisions are stable, robust, and specific. 5
Stability. Stability requires a probe to remain in the same induced decision region across semantically equivalent input representations, reducing sensitivity to option order and label preferences. Accordingly, Stemma evaluates each candidate under cyclic permutations of its answer options, such that every semantic option appears once under each displayed choice label. After mapping the resulting decisions to the canonical options, stability is measured by their consistency rate across permutations, with a higher rate indicating greater stability. Robustness. Robustness requires a probe to lie well within its assigned induced decision region, so that its region assignment remains stable under small boundary shifts. A probe may be consistent across option permutations yet remain close to a competing boundary, making its region assignment vulnerable to downstream adaptation or deployment changes. Stemma measures robustness by computing, for each option permutation, the log-probability margin between the source-assigned option and the highest-scoring alternative, and then averaging these margins across permutations. Larger average margins indicate greater robustness. Specificity. Specificity requires a probe’s source decision to be uncommon among unrelated models, so that its preservation provides discriminative evidence of induced decision region inheritance. Because gold-answer decisions are often shared across related and unrelated models, Stemma first filters out questions whose source decisions match their gold answers. However, non-gold decisions may still be common because of question ambiguity, annotation errors, or attractive distractors. Stemma therefore measures how often an unrelated background model reproduces the source decision, with lower reproduction rates indicating greater specificity. Probe Selection and Fingerprint Formation. For each candidate q, Stemma obtains sourcemodel next-token choice-label logits under cyclic option permutations, maps the resulting decisions to the canonical options, and defines hS (q) as the most frequently selected canonical decision. The three selection principles are operationalised using the permutation consistency rate s(q) for stability, the average log-probability margin r(q) for robustness, and the background decision alignment rate b(q) together with the non-gold requirement hS (q) ̸= d∗ (q) for specificity. Here, d∗ (q) denotes the gold answer. Stemma first applies the eligibility filters to the candidate pool, yielding Qelig = {q ∈ Qcand | s(q) > τs , r(q) ≥ τr , b(q) < τb , hS (q) ̸= d∗ (q)} , (5) S where τs and τr are the minimum stability and robustness thresholds, respectively, and τb is the maximum background-alignment threshold. It then ranks the eligible candidates and forms the fingerprint from the top K probes: n h io FS = (q, hS (q)) q ∈ TopK ′ elig wr re(q ′ ) − wbeb(q ′ ) , (6) q ∈QS
where re and eb denote the transformed robustness and background alignment scores, respectively, and wr and wb control their contributions. Exact score transformations, weights, and thresholds are provided in Appendix C. 4.4
F INGERPRINT V ERIFICATION
Given a source fingerprint FS and a suspect model T , Stemma queries each selected probe through T ’s calibrated prompt template using the same cyclic option permutations as in fingerprint construction. A rule-based extractor maps each generated response to either a displayed option or an invalid outcome ⊥. Valid extracted options are mapped back to the canonical decision space and compared with the stored source decision, while invalid outcomes are treated as non-aligned. These permutation-level comparisons are aggregated using the fingerprint alignment score: h i X X 1 A(S, T ) = 1 dˆT (q, π) = hS (q) . (7) KC (q,hS (q))∈FS π∈Π(q)
Here, Π(q) denotes the C cyclic permutations of probe q, and dˆT (q, π) ∈ D ∪ {⊥} denotes the suspect outcome after mapping each valid extracted option back to the canonical decision space. Thus, A(S, T ) ∈ [0, 1] is the fraction of the KC probe–permutation observations that preserve the source region assignment, with invalid outcomes contributing zero. Higher values indicate stronger induced decision region inheritance and hence stronger evidence of a provenance relationship. 6
5
E XPERIMENTS
5.1
E XPERIMENTAL S ETUP
Benchmark and evaluation protocol. We curate 7 provenance groups with 14 source models from publicly available Hugging Face checkpoints (Hugging Face, 2026), considering model provenance, documentation quality, download popularity, and diversity across model families, scales, and downstream transformations. Each group contains one pretrained source, one instruction-tuned source, and 6 additional variants covering task- or domain-specific fine-tuning, parameter-efficient adaptation, model merging, quantisation, and distillation. The benchmark spans Qwen, Llama, Mistral, Falcon, and OLMo, with model sizes ranging from 1.7B to 14B parameters. Three Qwen-based groups provide hard negatives with similar architectures and training recipes but distinct provenance. Each source is evaluated against every non-self checkpoint. A pair is labelled positive when the source and suspect belong to the same curated provenance group and negative otherwise. This yields 770 ordered source–suspect pairs, comprising 98 positive and 672 negative pairs. Each checkpoint is queried using the interface recommended by its model card or tokenizer configuration, with completion-style checkpoints receiving raw prompts and chat-oriented checkpoints using their tokenizer chat templates. By default, responses are generated using stochastic decoding with temperature 0.7, top-p 0.9, and top-k 50. The complete benchmark is provided in Appendix D. Experimental setup and baselines. Unless otherwise stated, Stemma constructs a fingerprint of 40 probes for each source model from a candidate pool of 3,000 questions randomly sampled from MMLU. Phi-3.5-mini-instruct is used as the default background model, which has no known provenance relationship with any benchmark model. We compare Stemma with representative black-box LLM fingerprinting baselines, including LLMmap, LLMPrint, Model Provenance Testing (MPT), and ZeroPrint (Pasquini et al., 2025; Hu et al., 2026; Nikolic et al., 2025; Shao et al., 2026). All methods are evaluated under the benchmark and evaluation protocol described above. Experiments are conducted on NVIDIA L40S GPUs with 48 GB of memory. Detailed Stemma and baseline configurations are provided in Appendices C and F, respectively. Evaluation metrics. We evaluate all methods using their continuous provenance scores and report AUC to assess overall ranking performance. Since false-positive provenance claims can have substantial legal and reputational consequences, we additionally report standardised pAUC over the FPR range [0, 0.05] and TPR at 1% FPR to assess performance under stringent low-FPR constraints. We also report the discriminability index d′ to quantify the separation between positive and negative score distributions. For baseline comparisons, the best and second-best results are shown in bold and underlined, respectively. Exact implementation details for these metrics are provided in Appendix E. 5.2
E FFECTIVENESS IN P ROVENANCE T ESTING
Table 1: Overall comparison between Stemma and baselines on the main provenance benchmark. Pretrained sources
Method LLMmap LLMPrint MPT ZeroPrint Stemma
Instruct sources
All sources
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
0.480 0.428 0.461 0.535 0.964
-0.298 -0.183 -0.276 0.003 3.045
0.851 0.615 0.722 0.775 0.970
1.471 0.265 1.045 1.199 2.945
0.665 0.516 0.556 0.617 0.967
0.628 0.054 0.500 0.603 2.951
0.487 0.498 0.498 0.496 0.959
0.000 0.020 0.020 0.020 0.918
0.712 0.580 0.692 0.755 0.938
0.388 0.143 0.327 0.469 0.857
0.568 0.541 0.613 0.610 0.944
0.102 0.071 0.194 0.194 0.878
Table 1 shows that Stemma consistently outperforms all baselines for pretrained and instructiontuned sources, as well as overall, with gains in overall discrimination and low-FPR performance. The gap is pronounced for pretrained sources, where source fingerprints use raw completion prompts while many deployed suspects are queried through chat templates. Under this interface mismatch, baselines approach random performance, suggesting that their provenance signals are sensitive to prompt-induced changes in response realisation. Baselines improve for instruction-tuned sources, where both fingerprint construction and verification use chat templates, but do not close the gap to Stemma despite this prompt-format alignment. This suggests that induced decision region in7
heritance offers both reduced sensitivity to surface-form variation and stronger separation between related and unrelated model pairs. To isolate interface mismatch, we further evaluate an all-raw setting in which both fingerprint construction and verification use raw prompts. Stemma remains competitive in overall ranking, achieving 0.965 AUC compared with 0.988 for MPT, while attaining the strongest overall low-FPR performance with 0.952 pAUC and 0.898 TPR at 1% FPR, versus 0.938 and 0.816, respectively, for the strongest baseline. Thus, even under conditions favourable to the baselines, Stemma is more reliable under strict false-positive constraints. Detailed results are provided in Appendix G. 5.3
ROBUSTNESS UNDER D EPLOYMENT VARIATIONS Table 2: Robustness under inference-time deployment variants. Pretrained sources
Method LLMmap LLMPrint MPT ZeroPrint Stemma
Instruct sources
All sources
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
0.481 0.499 0.477 0.623 0.995
-0.190 0.050 -0.252 0.210 3.414
0.901 0.615 0.780 0.868 0.996
1.760 0.413 1.225 1.697 4.158
0.689 0.551 0.570 0.670 0.995
0.793 0.196 0.563 0.789 3.603
0.487 0.489 0.487 0.496 0.965
0.000 0.000 0.000 0.000 0.929
0.754 0.522 0.727 0.813 0.980
0.452 0.000 0.452 0.607 0.964
0.599 0.516 0.621 0.642 0.974
0.113 0.000 0.226 0.208 0.935
To evaluate robustness to inference-time deployment changes, we construct 11 alternative variants for each of the 7 instruction-tuned checkpoints, spanning system and role-play prompts, chain-ofthought prompting, retrieval augmentation, and decoding settings. Together with the 14 source checkpoints, this yields 91 model instances and 1,260 source–suspect pairs, comprising 168 positive and 1,092 negative pairs. Full details are provided in Appendix H. Table 2 shows that Stemma maintains near-perfect AUC and substantially stronger low-FPR performance across these variations. Existing methods perform particularly poorly for pretrained sources and improve for instruction-tuned sources, but still fall consistently below Stemma. These results demonstrate Stemma’s robustness under diverse black-box deployment configurations, consistent with reduced sensitivity to deployment-induced surface-form variation. A BLATION S TUDY
Performance
1.0 0.9 0.8 0.751 0.7 0.6 0.614 0.5 Random probes
0.948 0.844
0.871
0.967
0.95
0.944 pAUC
5.4
0.756 Filtered pool
0.90
Candidate pool size 1000 4000 2000 5000 3000 6000
0.88
AUC pAUC
Non-gold pool
0.93
10
Stemma
20
80
90 100
0.973 0.952
MMLU
pAUC
0.960 0.946
MMLU-Pro
AUC
0.959 0.939
0.967 0.944
CosmosQA
1.0 0.9 0.8 0.7 0.6 0.5
0.967 0.944
0.948 0.929
CSQA
Performance
0.925 0.815
pAUC
0.948 0.868
Performance
AUC
40 50 60 70 Number of probes (k)
Pool size and probe budget
Probe selection 1.0 0.9 0.8 0.7 0.6 0.5
30
Phi-3.5
Yi-1.5
Gemma-2
Combined
Background model
Probe dataset
Figure 3: Ablation study of probe selection, pool size and probe budget, probe dataset, and background model. The ⋄ symbol marks the default configuration. The probe selection panel of Figure 3 isolates successive selection stages under the same query budget. The Random probes setting retains the same cyclic-permutation evaluation and aggregation but samples uniformly from the full candidate pool, corresponding to the random setting in Section 3.2. Its performance confirms that induced decision region inheritance is observable without targeted selection, although random probes provide limited discrimination. Restricting candidates to the non-gold pool improves specificity, while subsequent filtering removes probes that fail the 8
stability or robustness requirements. Finally, ranking the retained candidates prioritises the most informative probes and yields the strongest performance. Together, these incremental gains support the proposed selection principles, while the larger gains in pAUC indicate that targeted selection is particularly valuable for distinguishing difficult-to-separate model pairs. The pool size and probe budget panel examines how performance changes with the number of candidate questions and selected probes. A pool of 1,000 questions yields noticeably weaker performance. Results generally improve at 3,000–4,000 questions compared with 2,000, while increasing to 5,000–6,000 provides no consistent further benefit, indicating that larger pools are not necessarily better. For the probe budget, performance largely plateaus once k reaches 40, and most larger-budget settings cluster around a pAUC of 0.94, indicating limited sensitivity to the exact budget. The corresponding AUC results follow the same overall pattern, but with less pronounced differences. Thus, Stemma performs consistently across a broad range of pool sizes and probe budgets. Differences across probe datasets are more pronounced in pAUC than in AUC, with MMLU-Pro and MMLU achieving stronger low-FPR performance than the two commonsense-oriented datasets. The background-model panel shows that performance is relatively insensitive to background choice. Although combining all three backgrounds yields modest improvements, a single unrelated background model remains competitive. Overall, the results suggest that knowledge-oriented questions with clearer answer boundaries may provide more specific provenance evidence, while a single unrelated background model appears practically adequate. Detailed results are provided in Appendix I. 5.5
C OMPUTATIONAL C OST
Runtime (s)
Figure 4 compares the wall-clock costs under the 29380 same GPU setting, with measurement details pro104 vided in Appendix J. Stemma requires approxi631 103 269 mately 4.5 minutes for one-time fingerprint con55 102 50 51 struction, remaining within the range of existing 38 17 20 methods and over an order of magnitude faster 101 3 than LLMPrint. Verification takes 38 seconds, only 100 tens of seconds slower than the faster baselines and LLMmap LLMPrint MPT ZeroPrint Stemma modest relative to the substantial detection gains. Collectively, Stemma maintains reasonable con- Figure 4: Median runtime cost of fingerprint construction (blue) and verification (orange). struction cost and practical verification time.
6
D ISCUSSION AND C ONCLUSION
Limitations and Future Work. Our current instantiation relies on multiple-choice questions and therefore assumes that the evaluated models can reliably process such questions and produce stable option-level decisions. Although practical provenance testing is likely to focus on capable models, this requirement may limit applicability to weaker models. However, multiple-choice questions are only one mechanism for inducing decision regions, and future work could explore more general interfaces for obtaining structured and observable decisions. We also do not consider adaptive attackers with knowledge of the probe distribution. Because our candidate probes are drawn primarily from a single public question set, an attacker could deliberately alter the suspect model’s behaviour on the corresponding question distribution, thereby weakening detection. This risk could be mitigated by constructing candidate pools from a diverse mixture of datasets or privately constructed question sets. Future work should further characterise the robustness limits of provenance testing under adaptive evasion. Conclusion. We introduced induced decision regions for LLMs by mapping open-ended outputs into a finite decision space, abstracting away surface-form variation and reframing provenance testing as measuring the inheritance of decision regions. Building on this formulation, we proposed Stemma, a black-box LLM fingerprinting method that selects probes according to stability, robustness, and specificity, allowing region inheritance to be assessed with a limited query budget. Across extensive evaluations, Stemma consistently demonstrated strong provenance separation, showing that induced decision region inheritance provides a reliable signal for LLM provenance testing. 9
E THICS S TATEMENT Stemma scores provide statistical evidence of a potential provenance relationship rather than definitive proof of model ownership or unauthorised reuse. The method could be misused to support false provenance claims, and false positives may cause legal or reputational harm. Stemma should therefore not be used as the sole basis for attribution or enforcement decisions, but interpreted alongside model documentation, licensing and development records, and other technical evidence. Our experiments use only publicly available model checkpoints and benchmark datasets and involve no human subjects or private user data.
R EPRODUCIBILITY S TATEMENT The complete Stemma procedure is described in Section 4, with prompt calibration, fingerprint construction, fingerprint verification, and hyperparameter settings specified in Appendix C. The models used in the main experiments, provenance-group construction, and pair-labelling protocol are documented in Appendix D. Appendix E provides the exact evaluation-metric definitions and implementation details, while Appendix F describes the baseline implementations and configurations. The all-raw and deployment-robustness benchmark settings are specified in Appendices G and H, respectively. Full ablation results and runtime measurements are reported in Appendices I and J. The open-source implementation and configurations for Stemma are available at https://github.com/kerryzhangcode/Stemma. ACKNOWLEDGEMENTS We gratefully acknowledge support from a Department of Computer Science Scholarship at the University of Oxford, funded by a generous sponsor. We also acknowledge the use of the University of Oxford Advanced Research Computing (ARC) facility in carrying out this work (10.5281/zenodo.22558).
R EFERENCES Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In Proceedings of the 27th USENIX Conference on Security Symposium, SEC’18, pp. 1615–1631, USA, 2018. USENIX Association. ISBN 978-1-931971-46-1. Tara L. Andrews. Analysis of variation significance in artificial traditions using Stemmaweb. Digital Scholarship in the Humanities, 31(3):523–539, September 2016. ISSN 2055-7671. doi: 10.1093/ llc/fqu072. Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. IPGuard: Protecting Intellectual Property of Deep Neural Networks via Fingerprinting the Classification Boundary. In Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security, ASIA CCS ’21, pp. 14–25, New York, NY, USA, June 2021. Association for Computing Machinery. ISBN 978-14503-8287-8. doi: 10.1145/3433210.3437526. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. In Advances in Neural Information Processing Systems, volume 36, pp. 10088–10115. Curran Associates, Inc., 2023. Yijia Fang, Yiqing Feng, Bingyu Li, and Mingxun Zhou. KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing, May 2026. Thibaud Gloaguen, Robin Staab, Nikola Jovanović, and Martin Vechev. LLM Fingerprinting via Semantically Conditioned Watermarks. In The Fourteenth International Conference on Learning Representations, October 2025. Augustin Godinot, Erwan Le Merrer, Camilla Penzo, François Taı̈ani, and Gilles Trédan. Queries, representation & detection: The next 100 model fingerprinting schemes. In Proceedings of the 10
Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence, volume 39 of AAAI’25/IAAI’25/EAAI’25, pp. 16817–16825. AAAI Press, February 2025. ISBN 978-1-57735-897-8. doi: 10.1609/aaai.v39i16.33848. Jiyang Guan, Jian Liang, and Ran He. Are You Stealing My Model? Sample Correlation for Fingerprinting Deep Neural Networks. In Advances in Neural Information Processing Systems, volume 35, pp. 36571–36584. Curran Associates, Inc., 2022. Martin Gubri, Dennis Ulmer, Hwaran Lee, Sangdoo Yun, and Seong Joon Oh. TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 11496–11517, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.683. Hasan Abed Al Kader Hammoud, Umberto Michieli, Fabio Pizzati, Philip Torr, Adel Bibi, Bernard Ghanem, and Mete Ozay. Model Merging and Safety Alignment: One Bad Model Spoils the Bunch. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 13033–13046, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. findings-emnlp.762. Tuomas Heikkilä and Teemu Roos. Thematic Section on Studia Stemmatologica. Digital Scholarship in the Humanities, 31(3):520–522, September 2016. ISSN 2055-7671. doi: 10.1093/llc/ fqw038. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations, October 2020. Yuepeng Hu, Zhengyuan Jiang, Mengyuan Li, Osama Ahmed, Zhicong Huang, Cheng Hong, and Neil Zhenqiang Gong. Fingerprinting LLMs via Prompt Injection. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (eds.), Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11795–11810, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.541. Hugging Face. Hugging face hub. https://huggingface.co/models, 2026. Hengrui Jia, Christopher A. Choquette-Choo, Varun Chandrasekaran, and Nicolas Papernot. Entangled Watermarks as a Defense against Model Extraction. In 30th USENIX Security Symposium (USENIX Security 21), pp. 1937–1954, 2021. ISBN 978-1-939133-24-3. Yongqi Jiang, Yansong Gao, Chunyi Zhou, Hongsheng Hu, Si Chen, Anmin Fu, and Willy Susilo. Intellectual property protection for deep learning model and dataset intelligence. Engineering Applications of Artificial Intelligence, 163:113024, January 2026. ISSN 0952-1976. doi: 10. 1016/j.engappai.2025.113024. Heng Jin, Chaoyu Zhang, Shanghao Shi, Wenjing Lou, and Y. Thomas Hou. ProFLingo: A Fingerprinting-based Intellectual Property Protection Scheme for Large Language Models. In 2024 IEEE Conference on Communications and Network Security (CNS), pp. 1–9, September 2024. doi: 10.1109/CNS62487.2024.10735575. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. Advances in Neural Information Processing Systems, 35:22199–22213, December 2022. Linyang Li, Botian Jiang, Pengyu Wang, Ke Ren, Hang Yan, and Xipeng Qiu. Watermarking LLMs with Weight Quantization. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3368–3378, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.220. 11
Ivica Nikolic, Teodora Baluta, and Prateek Saxena. Model Provenance Testing for Large Language Models. In Advances in Neural Information Processing Systems, volume 38, pp. 34126–34153. Curran Associates, Inc., 2025. Dario Pasquini, Evgenios M. Kornaropoulos, and Giuseppe Ateniese. LLMmap: Fingerprinting for Large Language Models. In 34th USENIX Security Symposium (USENIX Security 25), pp. 299–318, 2025. ISBN 978-1-939133-52-6. Pouya Pezeshkpour and Estevam Hruschka. Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2006–2017, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. findings-naacl.130. Shuo Shao, Yiming Li, Hongwei Yao, Yifei Chen, Yuchen Yang, and Zhan Qin. Reading between the lines: Towards reliable black-box LLM fingerprinting via zeroth-order gradient estimation. In Proceedings of the ACM Web Conference 2026, Www ’26, pp. 2637–2648, New York, NY, USA, 2026. Association for Computing Machinery. ISBN 979-8-4007-2307-0. doi: 10.1145/3774904. 3792196. Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujiu Yang, and Wai Lam. A Thorough Examination of Decoding Methods in the Era of LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8601–8629, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.489. Trevor Stalnaker, Nathan Wintersgill, Oscar Chaparro, Laura A. Heymann, Massimiliano Di Penta, Daniel M German, and Denys Poshyvanyk. An Empirical Analysis of Machine Learning Model and Dataset Documentation, Supply Chain, and Licensing Challenges on Hugging Face. ACM Transactions on Software Engineering and Methodology, November 2025. ISSN 1049-331X. doi: 10.1145/3776739. Chen Tang, Lan Zhang, Qi Zhao, Xirong Zhuang, and Xiang-Yang Li. Model Lineage Closeness Analysis. Proceedings of the AAAI Conference on Artificial Intelligence, 39(19):20796–20804, April 2025. doi: 10.1609/aaai.v39i19.34292. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023. Yun-Yun Tsai, Chuan Guo, Junfeng Yang, and Laurens van der Maaten. RoFL: Robust Fingerprinting of Language Models, May 2025. Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin’ichi Satoh. Embedding Watermarks into Deep Neural Networks. In Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, ICMR ’17, pp. 269–277, New York, NY, USA, June 2017. Association for Computing Machinery. ISBN 978-1-4503-4701-3. doi: 10.1145/3078971.3078974. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long 12
Papers), pp. 2609–2634, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.147. Noah Wang, Z.y. Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang, Jie Fu, and Junran Peng. RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 14743–14777, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.878. Tianlong Xu, Chen Wang, Gaoyang Liu, Yang Yang, Kai Peng, and Wei Liu. United We Stand, Divided We Fall: Fingerprinting Deep Neural Networks via Adversarial Trajectories. In Advances in Neural Information Processing Systems, volume 37, pp. 69299–69328. Curran Associates, Inc., 2024. doi: 10.52202/079017-2213. Yuliang Yan, Haochun Tang, Shuo Yan, and Enyan Dai. DuFFin: A Dual-Level Fingerprinting Framework for LLMs IP Protection. In Vera Demberg, Kentaro Inui, and Lluı́s Marquez (eds.), Findings of the Association for Computational Linguistics: EACL 2026, pp. 5168–5184, Rabat, Morocco, March 2026. Association for Computational Linguistics. ISBN 979-8-89176-386-9. doi: 10.18653/v1/2026.findings-eacl.273. Peigen Ye, Huali Ren, Zhengdao Li, Anli Yan, Hongyang Yan, Shaowei Wang, and Jin Li. Securing Large Language Models: A Survey of Watermarking and Fingerprinting Techniques. ACM Computing Surveys, 58(7):1–35, May 2026. ISSN 0360-0300, 1557-7341. doi: 10.1145/3773028. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. In Proceedings of the 41st International Conference on Machine Learning, pp. 57755–57775. PMLR, July 2024. Runpeng Yu and Xinchao Wang. Neural Lineage. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4797–4807, June 2024. doi: 10.1109/CVPR52733. 2024.00459. Chuanchao Zang, Xiangtao Meng, Wenyu Chen, Tianshuo Cong, Zha Yaxing, Dong Qi, Zheng Li, and Shanqing Guo. ErrorTrace: A Black-Box Traceability Mechanism Based on Model Family Error Space. In Advances in Neural Information Processing Systems, volume 38, pp. 68550– 68578. Curran Associates, Inc., 2025. Jie Zhang, Dongrui Liu, Chen Qian, Linfeng Zhang, Yong Liu, Yu Qiao, and Jing Shao. REEF: Representation Encoding Fingerprints for Large Language Models. International Conference on Learning Representations, 2025:48092–48117, May 2025. Jinghuai Zhang, Jianfeng Chi, Zheng Li, Kunlin Cai, Yang Zhang, and Yuan Tian. BadMerging: Backdoor attacks against model merging. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, Ccs ’24, pp. 4450–4464, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 979-8-4007-0636-3. doi: 10.1145/3658644.3690284. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large Language Models Are Not Robust Multiple Choice Selectors. International Conference on Learning Representations, 2024:19426–19454, May 2024a. Mingqian Zheng, Jiaxin Pei, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. When “A Helpful Assistant” Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 15126– 15154, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.888. Xiaoyuan Zhu, Yaowen Ye, Tianyi Alex Qiu, Hanlin Zhu, Sijun Tan, Ajraf Mannan, Jonathan Michala, Raluca Ada Popa, and Willie Neiswanger. Auditing black-box LLM APIs with a rankbased uniformity test. In The Fourteenth International Conference on Learning Representations, 2026. 13
A
M ULTIPLE -C HOICE I NSTANTIATION OF I NDUCED D ECISION R EGIONS
Let Q be a set of multiple-choice questions, where each question q ∈ Q contains C candidate options defining the finite decision space D = {1, . . . , C}. When decision scores are available, let h̃M,d (q) denote the score assigned by model M to option d, such as a next-token logit or normalised choice probability. The soft decision vector is h̃M (q) = h̃M,1 (q), . . . , h̃M,C (q) ∈ RC , (8) with the corresponding discrete decision given by hM (q) = min arg max h̃M,d (q),
(9)
d∈D
where ties are resolved in favour of the lowest-indexed decision. Under text-only access, we extend the observable outcome space with an invalid outcome: D⊥ = D ∪ {⊥},
(10)
where ⊥ denotes a response from which no valid option can be extracted. The extracted discrete outcome is then hM (q) = g q, oM (q) ∈ D⊥ . (11) Thus, valid outcomes under both observation modes lie in the same canonical decision space D. For each decision d ∈ D, the corresponding induced decision region is RM d = {q ∈ Q | hM (q) = d}.
(12)
The collection {RM d }d∈D partitions the valid-decision subset {q ∈ Q | g(q, oM (q)) ∈ D} of the multiple-choice question space, defining the induced decision geometry of M . Responses mapped to ⊥ lie outside this partition and are excluded from fingerprint construction. For a probe set P ⊆ Q, induced decision region inheritance can be assessed at different resolutions. When soft decision vectors are available for both models, as in the visualisation presented in Section 3.2, preservation is assessed by averaging the similarity between corresponding soft decision vectors over P, X d P (S, T ) = 1 sim h̃S (q), h̃T (q) . (13) Sim |P| q∈P
When such fine-grained information is unavailable, as in the black-box verification setting described in Section 4.4, preservation is instead assessed through alignment of discrete decisions over P, 1 X AbP (S, T ) = 1 {hT (q) = hS (q) ̸= ⊥} . (14) |P| q∈P
Thus, invalid outcomes are treated as non-aligned observations. When answer-option permutations are used, extracted scores and decisions are mapped back to the underlying answer options before aggregation or comparison, as specified in the corresponding sections.
B
I NDUCED D ECISION R EGION V ISUALISATION
This appendix describes how the visualisation in Figure 1 is constructed and how its panel annotations are computed. The visualisation is used solely for diagnostic analysis and does not affect Stemma probe selection or provenance scoring. Source-relative projection. For each question, we rank the answer options by the source model’s permutation-averaged option probabilities. This defines a source-relative labelling of the finite decision space D. In the four-choice MMLU setting, the corresponding induced decision regions are visualised as R1 , . . . , R4 , where R1 corresponds to the source model’s most preferred option, R2 to its second most preferred option, and so on. 14
The same source-defined ordering is applied to every model M . For each question q, the resulting soft decision vector is h̃M (q) = h̃M,1 (q), h̃M,2 (q), h̃M,3 (q), h̃M,4 (q) , where the entries are the permutation-averaged probabilities assigned by M to the source-ranked options R1 , . . . , R4 . To project this four-dimensional vector into two dimensions, we arrange the four regions as the quadrants of a square, with R1 , . . . , R4 occupying the upper-left, upper-right, lower-right, and lower-left quadrants, respectively. The projected coordinates are defined as h i xM (q) = tanh τ max{h̃M,2 (q), h̃M,3 (q)} − max{h̃M,1 (q), h̃M,4 (q)} , h i yM (q) = tanh τ max{h̃M,1 (q), h̃M,2 (q)} − max{h̃M,3 (q), h̃M,4 (q)} . The x-coordinate compares the strongest preference among the right-side regions with that among the left-side regions, while the y-coordinate compares the strongest preference among the upperside regions with that among the lower-side regions. The tanh transformation bounds the displayed coordinates to [−1, 1], and τ controls the projection scale. We use τ = 2 in the figure. Displayed points and region preservation.
For each model–question pair, let
zM (q) = (xM (q), yM (q)) denote its displayed two-dimensional coordinate. The figure plots zS (q) for the source model S as a grey point and zT (q) for the suspect model T as a coloured point. The connecting line represents the projected displacement zT (q) − zS (q) and is used only as a visual aid, with shorter lines indicating smaller shifts in the projected space. Under this source-relative labelling, the discrete decision map is hM (q) = min arg max h̃M,i (q), i∈{1,2,3,4}
where the outer minimum resolves ties in favour of the first decision under the fixed ordering. The corresponding induced decision region is RM i = {q ∈ Q | hM (q) = i}, which is represented by quadrant Ri in the visualisation. Suspect points are coloured green when hT (q) = hS (q), indicating source region alignment, and orange otherwise, indicating a source region change. Annotation metrics. Let P denote the displayed question set. Each panel reports soft decision similarity. For this visualisation, we instantiate the similarity function for normalised soft decision vectors as 1 sim(a, b) = 1 − ∥a − b∥1 . 2 The resulting similarity score is 1 X 1 d Sim = SimP (S, T ) = 1− h̃T (q) − h̃S (q) . |P| 2 1 q∈P
Each panel also reports source region alignment, 1 X A = AbP (S, T ) = 1{hT (q) = hS (q)}. |P| q∈P
Higher values of Sim and A indicate stronger preservation of the source model’s soft decision preferences and discrete region assignments, respectively. 15
C
S TEMMA I MPLEMENTATION AND C ONFIGURATION D ETAILS
This appendix specifies the implementation and main experimental configuration of Stemma’s three stages: prompt calibration, fingerprint construction, and fingerprint verification. Unless otherwise stated, all random sampling and stochastic generation use a fixed random seed of 42 for reproducibility. Dataset. Probe candidates are sampled from the MMLU test split across all subjects (Hendrycks et al., 2020). Unless otherwise stated, we randomly sample 3,000 candidate questions for each source model. Each candidate is evaluated under cyclic permutations of its answer options to assess the stability of the source decision across different option orders and label assignments. Prompt calibration. Before fingerprint construction and verification, Stemma independently calibrates a multiple-choice prompt template for each evaluated model using 100 examples from the MMLU validation split, which is disjoint from the candidate probe pool. Calibration selects a prompt format that reliably elicits parseable option labels, facilitating both white-box choice label scoring during fingerprint construction and black-box answer extraction during fingerprint verification. We evaluate 6 simple prompt templates. To avoid severe label bias, we discard any template for which the most frequently predicted label accounts for more than 0.7 of the calibration predictions. Among the remaining templates, we select the one with the highest valid-choice extraction rate, breaking ties according to the predefined template order, and fix it for all subsequent queries to that model. The set of prompt templates used in calibration is shown below. In these templates, {question} denotes the question text, {options} denotes the formatted answer options, and {choices} denotes the available option labels. Simple answer cue {question} {options} Answer:
MMLU-style prompt Question: {question} {options} Answer:
Strict label-only prompt Read the following multiple-choice question carefully. Question: {question} {options} Choose the best answer. Output exactly one option label from {choices}. Do not output any explanation. Answer:
Single-character response Question: {question} {options} Your response must be exactly one character from {choices}. Response:
High-constraint prompt Read the following multiple-choice question carefully. Choose the best answer from the listed options. Question: {question} {options} Answer with only one option label from {choices}. Answer:
16
Answer cue with trailing newline {question} {options} Answer:\n
Fingerprint construction. For each source model, Stemma computes next-token choice-label logprobabilities for every candidate question under all cyclic option permutations. Choice labels are scored using tokenizer-aware matching. When multiple token IDs correspond to the same visible option label, their log-probabilities are combined using log-sum-exp. Under each permutation, the highest-probability option is selected and mapped back to its canonical option identity. The canonical source decision is then determined by voting across the resulting permutation-level decisions. Candidate questions are filtered and ranked according to stability, robustness, and specificity. During hard filtering, a candidate is retained only if the same canonical wrong option receives more than half of the source model’s permutation votes (> 0.5), has a non-negative average wrong-label top margin (≥ 0), and is selected by fewer than half of the background model’s permutation-level predictions (< 0.5). Unless otherwise stated, we use microsoft/Phi-3.5-mini-instruct as the background model for specificity evaluation. During soft ranking, the surviving candidates are ranked using the following standardised score: Score(q) = 1.0 · clip z(r(q)), −2, 2 − 0.5 · z(b(q)), (15) where r(q) is the average wrong-label top logit margin across permutations, b(q) is the fraction of background-model predictions that select the same canonical wrong option, and z(·) denotes standardisation over the surviving candidate pool for the corresponding source model. The standardised robustness term is clipped to [−2, 2] to prevent extreme margins from dominating the ranking. The resulting source-specific fingerprint comprises the 40 highest-ranked probes and their canonical wrong labels, as determined by permutation voting. Fingerprint verification. For each source–suspect pair, the suspect model is queried on the corresponding source probes under the same cyclic option permutations used during fingerprint construction. Generation is limited to 16 new tokens. A rule-based regular-expression extractor maps each response to either a predicted option label or an invalid outcome ⊥, with valid labels mapped back to their canonical option identities. The fingerprint alignment score is computed as defined in Equation 7, with invalid outcomes treated as non-aligned. With 40 probes and four cyclic permutations, the main configuration yields 160 probe–permutation observations for each model pair.
D
M ODEL B ENCHMARK
Table 3 presents the complete model benchmark used in our experiments. It comprises 56 publicly available Hugging Face checkpoints organised into 7 provenance groups across 5 model families, covering both fine-grained relationships among closely related models and diverse modeldevelopment transformations. Model selection considers provenance clarity, documentation quality, download popularity, and diversity across model families, scales, and transformation types. Benchmark composition. Each provenance group follows a consistent structure, containing 8 checkpoints: 1 pretrained checkpoint, 1 instruction-tuned checkpoint, and 6 additional variants. For these additional variants, we aim to cover a diverse set of common model-development operations. Specifically, where suitable public checkpoints are available, each group includes 3 task- or domainspecialised variants covering both full fine-tuning and parameter-efficient adaptation, together with 1 model-merge variant, 1 quantised variant, and 1 distillation variant. The only exception is the OLMo-2-1124-7B group, for which we did not find a suitable public distillation variant, so we include an additional fine-tuned or adapter-based variant instead. This design gives each provenance group comparable coverage while capturing the diversity and complexity of real-world model development, thereby providing a more rigorous testbed for evaluating fingerprinting methods under realistic transformation scenarios. For evaluation, within-group pairs are labelled positive, whereas cross-group pairs are labelled negative. 17
Table 3: Complete model benchmark. Provenance group
Repository ID
Model type
Query interface
Qwen-2.5-7B
Qwen/Qwen2.5-7B Qwen/Qwen2.5-7B-Instruct Qwen/Qwen2.5-Coder-7B-Instruct zjudai/flowertune-medical-lora-qwen2.5-7b-instruct SeeFlock/task-12-Qwen-Qwen2.5-7B-Instruct Locutusque/StockQwen-2.5-7B Qwen/Qwen2.5-7B-Instruct-GPTQ-Int8 Lansechen/Qwen2.5-7B-Open-R1-Distill
Pretrained Instruct Fine-tune Adapter Adapter Merge Quantisation Distillation
raw chat chat chat chat chat chat chat
Qwen-2.5-14B
Qwen/Qwen2.5-14B Qwen/Qwen2.5-14B-Instruct oxyapi/oxy-1-small Qwen/Qwen2.5-14B-Instruct-1M ToastyPigeon/qwen-story-test-qlora v000000/Qwen2.5-14B-Gutenberg-Instruct-Slerpeno Qwen/Qwen2.5-14B-Instruct-GPTQ-Int8 alibaba-pai/DistilQwen2.5-DS3-0324-14B
Pretrained Instruct Fine-tune Fine-tune Adapter Merge Quantisation Distillation
raw chat chat chat chat chat chat chat
Qwen3-1.7B
Qwen/Qwen3-1.7B-Base Qwen/Qwen3-1.7B mlabonne/Qwen3-1.7B-abliterated HuggingFaceTB/qwen3-1.7b-gsm8k-sft txmedai/ClinicalEase-Qwen3-1.7B kurakurai/Luth-1.7B-Instruct Qwen/Qwen3-1.7B-GPTQ-Int8 prithivMLmods/Regulus-Qwen3-R1-Llama-Distill-1.7B
Pretrained Instruct Fine-tune Fine-tune Adapter Merge Quantisation Distillation
raw chat chat chat chat chat chat chat
Llama-3.1-8B
meta-llama/Llama-3.1-8B meta-llama/Llama-3.1-8B-Instruct RedHatAI/Llama-3.1-8B-tldr chchen/Llama-3.1-8B-Instruct-PsyCourse-fold7 zjudai/flowertune-medical-lora-llama-3.1-8b-instruct Xiaojian9992024/Llama3.1-8B-ExtraMix hugging-quants/Meta-Llama-3.1-8B-Instruct-GPTQ-INT4 arcee-ai/Llama-3.1-SuperNova-Lite
Pretrained Instruct Fine-tune Adapter Adapter Merge Quantisation Distillation
raw chat raw chat chat chat chat chat
Mistral-7B-v0.3
mistralai/Mistral-7B-v0.3 mistralai/Mistral-7B-Instruct-v0.3 KurmaAI/AQUA-7B chaymaemerhrioui/mistral-Brain Model ACC Trainer zjudai/flowertune-medical-lora-mistral-7b-instruct-v0.3 grimjim/Mistral-7B-Instruct-demi-merge-v0.3-7B RedHatAI/Mistral-7B-Instruct-v0.3-GPTQ-4bit eganwo/mistral7b-distilled-from-deepseek-r1-qwen32b
Pretrained Instruct Fine-tune Adapter Adapter Merge Quantisation Distillation
raw chat chat chat chat chat chat chat
Falcon3-7B
tiiuae/Falcon3-7B-Base tiiuae/Falcon3-7B-Instruct ehristoforu/falcon3-ultraset jahyungu/Falcon3-7B-Instruct-v1-Easy jahyungu/Falcon3-7B-Instruct-v1-Hard suayptalha/Falcon3-Jessi-v0.4-7B-Slerp tiiuae/Falcon3-7B-Instruct-GPTQ-Int8 RedaAlami/Falcon3-7B-Instruct-Distill-DS-v1
Pretrained Instruct Fine-tune Adapter Adapter Merge Quantisation Distillation
raw chat chat chat chat chat chat chat
OLMo-2-7B
allenai/OLMo-2-1124-7B allenai/OLMo-2-1124-7B-Instruct allenai/OLMo-2-1124-7B-SFT allenai/OLMo-2-1124-7B-Instruct-preview jahyungu/OLMo-2-1124-7B-Instruct-Humanities jahyungu/OLMo-2-1124-7B-Instruct-Social-Sciences Alelcv27/Olmo2-7B-Breadcrumbs-v1 kaitchup/OLMo-2-1124-7B-Instruct-AutoRound-GPTQ-4bit
Pretrained Instruct Fine-tune Fine-tune Adapter Adapter Merge Quantisation
raw chat chat chat chat chat chat chat
18
Model types. The model type column reports the development operation associated with each checkpoint. Pretrained denotes the base model trained before instruction tuning. Instruct denotes an instruction-tuned checkpoint derived from the pretrained model. Fine-tune denotes a checkpoint further trained on task-specific, domain-specific, or instruction-following data. Adapter denotes parameter-efficient adaptation, where additional or low-rank trainable parameters are used to specialise the model. Merge denotes a checkpoint produced by combining weights from multiple models. Quantisation denotes a numerically compressed checkpoint with reduced weight precision. Distillation denotes a checkpoint derived from a base model and further trained or fine-tuned using teacher-generated data, such as model-generated responses or reasoning traces from a stronger model. Query interfaces. We also distinguish two query interfaces, following the recommended usage in each checkpoint’s model card or tokenizer configuration. The raw interface denotes completionstyle prompting, where the prompt is passed directly to the causal language model without a chat wrapper. It is used for pretrained checkpoints and for downstream checkpoints that are released or documented as completion-style models. The chat interface denotes chat-template prompting, where the calibrated prompt is first treated as a user message and then rendered using the tokenizer’s chat template before generation. It is used for instruction-tuned and chat-oriented checkpoints. Chat templates may add model-specific system, user, and assistant formatting, thereby changing the effective input received by the model. For example, when no explicit system message is provided, the Qwen2.5-Instruct tokenizer renders a single user message as follows: Rendered input under the Qwen2.5-Instruct chat template <|im_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|> <|im_start|>user {user_prompt}<|im_end|> <|im_start|>assistant
Here, {user prompt} denotes the input prompt used for evaluation. We do not explicitly provide a manually constructed system message. Instead, when a checkpoint’s tokenizer chat template defines a default system message, it is inserted automatically during template rendering. Using the interface specified by each checkpoint’s model card or tokenizer configuration helps ensure that the model is evaluated under its intended usage pattern, better matching common usage and realistic deployment practice.
E
E VALUATION M ETRICS
Each evaluated source–suspect pair is assigned a continuous provenance score, where higher values indicate stronger evidence of a provenance relationship. Related pairs are treated as positive and unrelated pairs as negative. AUC and partial AUC.
The area under the receiver operating characteristic curve is defined as Z 1 AUC = TPR(u) du, 0
where u denotes the false-positive rate. To emphasise performance in the low-FPR regime, we additionally report standardised partial AUC over FPR ∈ [0, α], with α = 0.05. Let Z α Aα = TPR(u) du 0
denote the unstandardised partial area. We apply McClish standardisation: Aα − α2 /2 1 pAUCα = 1+ . 2 α − α2 /2 Under this standardisation, random ranking corresponds to 0.5 and perfect ranking to 1. 19
Low-FPR true-positive rate. TPR at 1% FPR is the highest empirical true-positive rate attained without exceeding a false-positive rate of 0.01. We do not interpolate between operating points. Discriminability. We quantify the standardised separation between the positive and negative score distributions using µ+ − µ− d′ = q , 2 + σ 2 )/2 (σ+ − 2 2 where µ+ and µ− are the mean positive and negative scores, and σ+ and σ− are their sample variances. Positive values indicate that related pairs receive higher scores on average, while larger values indicate clearer separation in the intended direction. Negative values indicate reversed mean ordering.
F
BASELINE I MPLEMENTATIONS
We implement four black-box LLM fingerprinting baselines to compare against Stemma. Unless otherwise stated below, we use the settings and hyperparameters specified in the original papers or official repositories for each baseline, in order to reproduce its intended performance. Since a finegrained comparison requires a continuous provenance score rather than only a binary decision for each source and suspect model pair, we use the underlying fingerprint similarity or matching score produced by each baseline as its pairwise provenance score. This is natural for these methods, as their final decisions are based on fingerprint similarity even when their decision rules differ. LLMmap. LLMmap fingerprints a model by encoding its behaviour on a fixed set of diagnostic prompts into an open-set feature space. We use the 8 input queries from the original paper and the pretrained open-set feature extractor released with the official implementation. To construct each source gallery representation, we evaluate 100 prompt configurations over these queries and average the resulting feature vectors. During verification, the suspect model is evaluated once on the same 8 queries and its feature vector is compared with the source gallery representation. For both gallery construction and verification, generated responses contain at most 100 new tokens and are truncated to 650 characters before feature extraction. Following the original comparison procedure, we compute the Euclidean distance d between the source and suspect fingerprints and convert it into a similarity score using the inverse-one-plus mapping, s = 1/(1 + d). This score is used as the pairwise provenance score for each evaluated source and suspect model pair. LLMPrint. LLMPrint constructs black-box fingerprints by learning adversarial suffix probes that induce model-specific preferences over target word pairs. Its original configuration optimises suffixes for 300 word pairs using 1,000 GCG steps per pair. Based on pilot runtime measurements, running the full configuration independently across all three evaluation settings is estimated to require over 240 GPU-days in our environment. We therefore use a reduced configuration consistently across all source models. For each source model, we sample 100 word pairs from the fixed category vocabulary provided by the official repository, such that both words are represented by a single token under the source tokenizer. We then optimise one adversarial suffix per pair for 200 GCG steps under the fixed LLMPrint prompt template. The learned suffixes are used to query both source and suspect models. Following the original scoring procedure, we perform 100 repeated one-token generations for each word pair and record how often each target token is generated. The more frequently generated token determines the model’s binary preference for that pair, and the agreement rate between the source and suspect preferences is used as the pairwise provenance score. Model Provenance Testing. Model Provenance Testing (MPT) detects provenance by comparing next-token continuation behaviour between a source model and a suspect model. Following the official implementation, we sample 5,000 sentence-level prompts from the MPT prompt pool provided in the official repository. For each source model, we construct a continuation cache by querying the model on all prompts with single-token generation, using max new tokens=1. Each suspect model is then evaluated on the same prompt set, and its next-token continuations are matched against 20
the cached source continuations. We use the resulting hit rate, namely the fraction of prompts on which the suspect continuation matches the source continuation, as the pairwise provenance score. ZeroPrint. ZeroPrint constructs black-box LLM fingerprints by measuring the response sensitivity of a model to query perturbations. We follow the original setup by using openai humaneval as the query source, converting each task prompt into a completion-style query, and truncating the query to 20 words. Following the configuration reported in the original ZeroPrint paper, we sample 2 base queries and generate 4 word-substitution variants for each query, where each perturbation replaces 3 words with independently selected top-10 nearest neighbours under GloVe embeddings. Responses are generated with at most 128 new tokens and are not truncated after generation. To reduce generation noise, each model is queried 20 times per prompt, and the resulting responses are embedded with sentence-transformers/all-mpnet-base-v2, matching the official implementation. ZeroPrint then estimates a response-sensitivity fingerprint for each base query from the input and output embedding changes using the Jacobian-based estimator with ridge regularisation α = 0.001. The resulting fingerprints are aggregated by mean pooling, and the Pearson correlation coefficient between source and suspect fingerprints is used as the pairwise provenance score. Implementation consistency. For all baselines, we use the same benchmark split, model query interfaces, source-model set, and suspect-model set. Each baseline produces a pairwise provenance score for every evaluated source–suspect pair. We compute and report the final results from these pairwise scores using the same evaluation metrics as Stemma.
G
A LL -R AW Q UERY S ETTING
Table 4 provides the full results for the all-raw query-interface setting. In this control setting, all checkpoints are queried with raw completion-style prompts during both fingerprint construction and verification, without applying tokenizer chat templates. By removing model-specific chat formatting, this setting isolates the effect of prompt-format mismatch in the main experiments and evaluates baselines under more favourable input-matched conditions, although it is less representative of realistic deployment. Table 4: Overall comparison under an all-raw interface setting. Pretrained sources
Method LLMmap LLMPrint MPT ZeroPrint Stemma
H
Instruct sources
All sources
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
0.577 0.541 0.993 0.738 0.968
0.223 0.213 3.258 0.847 3.419
0.724 0.738 0.991 0.868 0.963
0.801 0.908 3.333 1.615 2.988
0.652 0.625 0.988 0.807 0.965
0.517 0.507 2.713 1.214 3.049
0.501 0.496 0.946 0.528 0.966
0.000 0.000 0.878 0.020 0.918
0.555 0.602 0.972 0.710 0.944
0.102 0.163 0.939 0.367 0.878
0.519 0.550 0.938 0.604 0.952
0.041 0.082 0.816 0.133 0.898
ROBUSTNESS B ENCHMARK
In addition to the main checkpoint-level benchmark, we construct a robustness benchmark to evaluate provenance signals under common deployment-time variations. It follows the same 7 provenance groups and uses their pretrained and instruction-tuned checkpoints as source models. Deployment variants are created from the corresponding instruction-tuned checkpoints by changing inferencetime settings without modifying model weights. Pretrained checkpoints use the raw query interface, whereas instruction-tuned and chat-oriented checkpoints use the chat interface specified by their tokenizer configuration. We consider five categories of deployment variation. For all prompt-based variants, the additional instruction is supplied through the system-message field of the model’s chat template, while the original multiple-choice prompt remains unchanged in the user message. 21
System prompts. We evaluate two system-prompt variants that reflect commonly used assistantstyle deployment settings. The first is a minimal generic helpful-assistant prompt, while the second uses the longer safety-oriented system prompt reported for Llama 2-Chat (Touvron et al., 2023). Minimal assistant system prompt You are a helpful assistant.
Safety-oriented assistant system prompt You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
Role-play prompts. We evaluate two role-play prompt variants, motivated by prior studies of persona and role prompting (Zheng et al., 2024b; Wang et al., 2024). The first represents a simple occupational role, where the model is assigned a profession-based persona. The second represents a more detailed persona-style role, where the model is given a specific character description and instructed to imitate the assigned personality. Occupational role prompt You are a teacher.
Persona-style role prompt You are Stephen Hawking, your description is: You are a brilliant physicist who defies the limitations of your physical disability to become a world-renowned scientist, exploring the mysteries of the universe while facing personal challenges in your relationships and health. Now please answer some questions to accurately show your personality traits! Your speaking style should fully imitate the personality role assigned to you! Please do not expose that you are an artificial intelligence model or a language model, you must always remember that you are only assigned one personality role. Don't be verbose or too formal or polite when speaking.
Chain-of-thought prompts. We evaluate two reasoning-prompt variants that are commonly used to encourage intermediate reasoning during inference. The first is the classic zero-shot chain-ofthought prompt (Kojima et al., 2022), which uses a short instruction to elicit step-by-step reasoning. The second follows the plan-and-solve prompting style (Wang et al., 2023), where the model is first asked to understand the problem and devise a plan before carrying out the solution. Zero-shot chain-of-thought prompt Let's think step by step.
Plan-and-solve prompt Let's first understand the problem and devise a plan to solve the problem. Then, let's carry out the plan and solve the problem step by step.
Retrieval-augmented generation. We evaluate two retrieval-augmented generation variants using SQuAD-v2 as the retrieval corpus and all-mpnet-base-v2 as the retriever. Each SQuAD-v2 context passage is treated as one retrieval document, and the original question text is used as the retrieval query. The top-1 or top-3 retrieved passages are prepended to the original multiple-choice prompt in descending similarity order. Retrieved passages are inserted in a fixed context block before the original prompt. Decoding settings. We evaluate four decoding variants that cover deterministic and stochastic generation regimes. The first uses greedy decoding, while the remaining three use stochastic decoding with moderate, high, and low sampling settings. Moderate sampling is the default decoding configuration used in the main benchmark and in the default robustness setting. The exact decoding parameters are summarised in Table 5. 22
Table 5: Decoding settings used in the robustness benchmark. Setting
Decoding mode
Temperature
top-p
top-k
Greedy decoding Moderate sampling (default) High sampling Low sampling
Deterministic Stochastic Stochastic Stochastic
– 0.7 1.2 0.3
– 0.9 1.0 0.9
– 50 0 10
Together, the robustness benchmark uses 7 pretrained checkpoints as source models and includes the corresponding 7 instruction-tuned checkpoints as both instruct-source models and default suspect models under the moderate-sampling setting. Each instruction-tuned checkpoint is further redeployed under 11 alternative inference-time settings, yielding 77 additional deployment variants. In total, the benchmark contains 91 model or deployment instances: 7 pretrained checkpoints, 7 default instruction-tuned deployments, and 77 additional deployment variants. Under the same pairconstruction rule, this gives 1,260 evaluated pairs, including 168 positive pairs and 1,092 negative pairs.
I
F ULL A BLATION R ESULTS
Tables 6–10 report the detailed results underlying the ablation analyses in Section 5.4, including source-type breakdowns where applicable.
Probe selection. All variants use the same candidate pool and final budget of 40 probes. The Random probes variant uniformly samples probes from all candidate questions. Non-gold pool samples after restricting candidates to questions whose source decision differs from the gold answer. Filtered pool additionally applies the stability, robustness, and background-specificity filters, but samples uniformly without soft ranking. Stemma further ranks the filtered candidates and selects the top-scoring probes. All variants use the same cyclic-permutation verification and scoring procedure. Table 6: Detailed ablation results for probe selection stages. Pretrained sources
Selection stage Random probes Non-gold pool Filtered pool Stemma
Instruct sources
All sources
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
0.761 0.812 0.936 0.964
0.953 1.264 2.242 3.045
0.741 0.878 0.957 0.970
0.821 1.746 2.551 2.945
0.751 0.844 0.948 0.967
0.875 1.494 2.340 2.951
0.675 0.710 0.818 0.959
0.347 0.367 0.571 0.918
0.616 0.795 0.904 0.938
0.224 0.551 0.796 0.857
0.614 0.756 0.871 0.944
0.143 0.439 0.674 0.878
Pool size and probe budget. We jointly vary the candidate pool size from 1,000 to 6,000 questions and the final probe budget from 10 to 100, while keeping all other settings fixed. Tables 7 and 8 report the corresponding pAUC and AUC results, respectively. The default configuration is shown in bold. As the sweep was conducted independently from the main experiment, the defaultsetting result differs slightly from the main-run result due to stochastic decoding and hardware-level nondeterminism. Table 7: Detailed pAUC results across candidate pool sizes and probe budgets. Number of probes k
Pool size 1,000 2,000 3,000 4,000 5,000 6,000
10
20
30
40
50
60
70
80
90
100
0.857 0.881 0.878 0.903 0.908 0.887
0.888 0.934 0.945 0.935 0.929 0.900
0.904 0.930 0.949 0.940 0.941 0.920
0.915 0.934 0.946 0.951 0.939 0.946
0.927 0.932 0.956 0.946 0.946 0.938
0.920 0.928 0.950 0.951 0.946 0.942
0.929 0.932 0.950 0.950 0.942 0.945
0.921 0.937 0.953 0.944 0.946 0.944
0.916 0.937 0.950 0.945 0.946 0.950
0.915 0.936 0.948 0.943 0.940 0.949
23
Table 8: Detailed AUC results across candidate pool sizes and probe budgets. Number of probes k
Pool size 1,000 2,000 3,000 4,000 5,000 6,000
10
20
30
40
50
60
70
80
90
100
0.920 0.946 0.955 0.955 0.958 0.945
0.949 0.958 0.972 0.970 0.958 0.956
0.954 0.962 0.969 0.968 0.963 0.959
0.956 0.960 0.968 0.970 0.965 0.963
0.955 0.960 0.970 0.970 0.965 0.963
0.957 0.961 0.971 0.970 0.966 0.964
0.959 0.962 0.971 0.969 0.967 0.964
0.959 0.962 0.970 0.967 0.969 0.964
0.958 0.965 0.969 0.969 0.971 0.966
0.958 0.965 0.970 0.968 0.970 0.965
Probe dataset. For the dataset ablation, candidate questions are drawn from the test splits of MMLU and MMLU-Pro and the validation splits of CommonsenseQA and CosmosQA, using up to 3,000 questions per dataset. Decision-interface calibration uses 100 questions from the validation splits of MMLU and MMLU-Pro and the training splits of CommonsenseQA and CosmosQA, ensuring that calibration and fingerprint construction use disjoint splits. The selection procedure and probe budget remain fixed across datasets. Table 9: Detailed ablation results for probe dataset choice. Pretrained sources
Dataset CommonsenseQA CosmosQA MMLU-Pro MMLU (default)
Instruct sources
All sources
AUC ↑ pAUC ↑ TPR ↑
d ↑
AUC ↑ pAUC ↑ TPR ↑
d ↑
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
0.915 0.889 0.947 0.964
1.856 1.748 2.736 3.045
0.977 0.951 0.948 0.970
2.936 2.601 2.673 2.945
0.948 0.925 0.948 0.967
2.232 2.031 2.635 2.951
0.770 0.783 0.938 0.959
0.490 0.551 0.878 0.918
′
0.961 0.874 0.916 0.938
0.898 0.694 0.837 0.857
′
0.868 0.815 0.929 0.944
0.714 0.582 0.857 0.878
Background model. For the background-model ablation, we compare Phi-3.5-mini-instruct, Yi-1.5-9B-Chat, and Gemma-2-9B-IT, while keeping all other settings fixed. In the combined setting, we pool the permutation-level decisions from all three models and compute background alignment as the fraction matching the source decision. Table 10: Detailed ablation results for background model choice. Pretrained sources
Background model Phi-3.5 (default) Yi-1.5 Gemma-2 Combined
J
Instruct sources
All sources
AUC ↑ pAUC ↑ TPR ↑
d ↑
AUC ↑ pAUC ↑ TPR ↑
d ↑
AUC ↑ pAUC ↑ TPR ↑
d′ ↑
0.959 0.940 0.959 0.957
3.045 2.531 2.654 2.983
0.970 0.964 0.968 0.979
2.945 2.853 2.901 3.077
0.967 0.959 0.960 0.973
2.951 2.628 2.700 2.950
0.964 0.958 0.955 0.966
0.918 0.857 0.918 0.918
′
0.938 0.955 0.942 0.958
0.857 0.898 0.878 0.918
′
0.944 0.939 0.946 0.952
0.878 0.857 0.878 0.898
RUNTIME M EASUREMENT
Measurement protocol. We measure the wall-clock runtime of all methods on the same NVIDIA L40S GPU with 48 GB of memory. Fingerprint construction is measured per source model and reported as the median over the 14 source checkpoints. Fingerprint verification measures the suspectside computation required to evaluate one source–suspect pair and is reported as the median over the corresponding verification executions, assuming that the source fingerprint has already been constructed. The measurements cover the principal computational procedures, such as model inference and, where applicable, prompt optimisation involving backward passes, but exclude model loading. Reported runtimes are rounded to the nearest second. LLMmap. The reported construction time includes evaluating each source under 100 prompt configurations on the 8 diagnostic queries, encoding the resulting responses, and averaging the feature vectors to construct the gallery representation. The reported verification time includes evaluating each suspect once on the same 8 queries and encoding its responses using the pretrained feature extractor. LLMPrint. The reported construction time includes optimising one adversarial suffix for each of 100 word pairs, each consisting of two single-token words, over 200 GCG steps, and estimating the source model’s preference direction for each pair using 100 repeated one-token generations. The reported verification time includes querying each suspect model with the fixed optimised suffixes, 24
estimating its preference direction for each pair using 100 repeated one-token generations, and comparing these directions with the stored source preferences. MPT. The reported construction time includes querying each source on 5,000 prompts with singletoken generation. The reported verification time includes querying each suspect once on the same prompt set and matching the resulting continuations against the cached source continuations to compute the hit rate. ZeroPrint. The reported construction time includes querying each source on 2 HumanEval base prompts and 4 word-substitution variants per prompt, with 20 generations for each query, embedding the responses, and constructing the Jacobian-based fingerprint. The reported verification time includes applying the same procedure to each suspect. Stemma. The reported construction time includes calibrating the source model and the configured background model on 100 held-out validation questions, followed by evaluating them on 3,000 candidate questions under cyclic option permutations to obtain the source-decision and backgroundalignment statistics used for probe selection. The reported verification time includes calibrating each suspect on the same held-out set and querying it on the 40 source-specific probes under four cyclic permutations.
K
T HE S TEMMATIC A NALOGY B EHIND S TEMMA
Stemmatic analysis. Manuscript studies draw on several complementary approaches, including palaeography, codicology, and stemmatics. Palaeography examines handwriting and scribal practices, while codicology examines the material construction and production of manuscripts. Stemmatics instead focuses on reconstructing the transmission history of works preserved in multiple manuscript copies, particularly when the original text and many intermediate copies no longer survive and only a set of extant manuscripts, commonly termed witnesses, remains (Heikkilä & Roos, 2016). To infer the genealogy of the transmitted text, stemmatic analysis aligns witnesses at corresponding textual locations and compares their variant readings, including substitutions, omissions, additions, and transpositions. It therefore relies on structured patterns of textual variation rather than treating overall visual or stylistic resemblance as the primary genealogical signal. The genealogical value of a shared variant depends on how readily it could have arisen independently. Variants that are widespread across the textual tradition or easily reproduced provide limited evidence, whereas distinctive variants are more informative because their preservation by multiple witnesses may indicate descent from a common exemplar. Stemmatic inference therefore considers not merely the presence of shared variation, but its genealogical informativeness (Andrews, 2016). The inferred relationships are conventionally represented by a stemma codicum, or stemma, a tree-like representation of textual descent among surviving witnesses and reconstructed intermediate exemplars. Analogy to model provenance. The name Stemma reflects this methodological analogy. Candidate questions correspond to aligned textual locations, model instances to manuscript witnesses, and induced decisions to variant readings. Existing response-based fingerprints derive provenance signals from observable characteristics of generated responses, much as palaeography and codicology draw on handwriting and material features, whereas Stemma maps model behaviour into a common finite decision space and compares structured decision variants across aligned probes. Decisions readily reproduced by unrelated models provide limited provenance evidence, just as common or independently arising readings provide limited evidence of textual affiliation. By contrast, stable, robust, and specific decisions preserved by a suspect resemble distinctive inherited variants. Stemma therefore tests model provenance through structured patterns of induced decision region inheritance rather than through response-level characteristics tied to particular surface realisations.
25