Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models Akshay Paruchuri Sanmi Koyejo Stanford University
Abstract
(a) Q: What was the title of the Freida Pinto film where her role was Usha Vance?
arXiv:2606.26079v1 [cs.CL] 24 Jun 2026
Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether orderirrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines. We introduce FACETP ROBE, a five-facet audit (option, evidencechunk, document-rank, image-set, and mixedmodality ordering) of 18 frontier and openweight MLLMs. A Bayesian item-response model separates ordering noise from per-facet bias; a same-ordering control estimates the decoder-stochastic floor for observed flips.
Different Input Orders
Different MLM Responses
Hillbilly Elegy
Desert Dancer
Love Sonia
(b)
Kimi-VLInternVL 3.5 38B
MedGemma 27B
Findings. None of the 18 MLLMs we audit are order-invariant: screened per-facet panelmean flip rates span 24–50%. A Gemini sameordering control at temperature 0 estimates a substantial ordering excess over a same-input decoder-noise floor in verified cells. Capability predicts but does not eliminate flips; the best model still flips on 13.4% of trials.
Qwen 3.5 27B ChatGPT 5.5
Claude Opus 4.7 Gemini 3.1 Pro
Implications. In our Gemini mitigation tests, training-free prompt changes are modalityconditional and do not transfer from text to visual reasoning. These results suggest that prompt-level mitigation alone is unlikely to provide general order robustness, motivating future work on training-time and architectural approaches. We propose cross-ordering flip rate as a standard reporting axis for MLLMs.
Figure 1: MLLMs Remain Order-Sensitive. (a) The same evidence, presented in three different, yet semantically equivalent, orderings, yields three different answers; only one is correct. (b) Capability vs. mean K=6 flip rate over the 5 facets, shown for 7 best-of-family models. Capability predicts but does not eliminate flips (ρ ≈ −0.95 on the full 18-model panel); the best model still flips on 13.4% of trials.
Code and Audit Artifacts: https://github. com/yahskapar/facet-probe
1
Ehsan Adeli
Introduction
et al., 2025; Yao et al., 2025), agentic tool schedulers (Shinn et al., 2023; Faghih et al., 2025; Mei et al., 2025), or clinical and consumer-facing interfaces (Haberle et al., 2024; Tierney et al., 2024; Hurt et al., 2025; Paruchuri et al., 2025), not by the end user. If a model’s answer changes under those permutations, canonical benchmark accuracy
Standard benchmarks for multimodal large language models (MLLMs) score each item on a single canonical ordering and implicitly treat semantically equivalent permutations as interchangeable (Wang et al., 2024b; Zuo et al., 2025; Jiang et al., 2024). In deployed systems, input order is set by retrieval pipelines (Lewis et al., 2020; Abootorabi 1
overstates reliability for multimodal RAG, agentic tool use, and clinical or other high-stakes settings where users reasonably expect the same evidence to yield the same answer.
ternVL3.5, Kimi-VL, and MedGemma families) over 12 datasets and >400,000 trials, paired with ODI (Ordering-Decomposed Item-Response Theory), a 2PL Bayesian item-response decomposition that separates per-item ordering noise σπ from perfacet systematic bias |δ|. Capability predicts but does not eliminate flips (Spearman ρ ≈ −0.95); the best model flips on 13.4% of trials. In our Gemini mitigation tests, training-free prompt changes are modality-conditional and do not transfer from text to visual reasoning. Our contributions are as follows: • Section 4.1: FACET-P ROBE, a five-facet ordering audit of 18 multimodal LLMs across 12 datasets, decomposed with ODI (2PL Bayesian IRT). A same-ordering control estimates a substantial ordering excess over a same-input decoder-noise floor in verified Gemini cells at temperature 0; screened image-set claims are anchored on MEDFRAMEQA rather than position-referential MANTIS - EVAL items.
MLLMs are not order-invariant. The same query under three semantically equivalent input orderings yields three different answers (Figure 1). The behavior is not confined to small or older models and is not a measurement artifact of any single benchmark. Emerging AI evaluation guidance asks that benchmarks “qualify” the order-irrelevance assumption explicitly under input perturbations (Salaudeen et al., 2025; Bean et al., 2026; Keller et al., 2026), but no evaluation in widespread use does so for multimodal ordering. Prior work touches this but does not close it. Tan et al. (2024) measures multimodal positional bias on a single ordering facet across 5–8 pre-2025 multimodal models. Promptception (Ismithdeen et al., 2025) characterizes prompt-template sensitivity on 10 multimodal models but does not study orthogonal ordering facets or the 2026 frontier generation. M3IRT (Uebayashi et al., 2026) applies item-response theory to multimodal evaluation along the cross-modal shortcut axis, not the ordering-noise axis. Kostiuk and Enevoldsen (2026) establish ranking instability on text-only and embedding-only evaluation but do not study multimodal ordering or estimate the ordering excess over same-input decoder-stochastic flips. Foundational work on option-order and dialog-turn sensitivity in text-only LMs (Pezeshkpour and Hruschka, 2024; Laban et al., 2023; Chen et al., 2024b) sets up the problem we extend to the multimodal frontier. No prior work runs a same-ordering control that estimates the ordering excess over a same-input decoder-noise floor; without that decomposition, published flip-rate numbers are ambiguous. Therefore, this paper asks the following:
• Section 4.2: Capability predicts but does not eliminate flips (panel-wide ρ ≈ −0.95; best model still flips on 13.4%); stated confidence under-predicts ordering risk by 9–62 pp; mechanism analysis recovers content rationalization as a common failure mode, with the original image-set mechanism row treated as exploratory because it was sampled before the MANTIS - EVAL position-reference screen. • Section 4.3: Training-free, prompt-level mitigations are modality-conditional and do not compose; we report a cost-Pareto frontier. We plan to release the full evaluation code and related artifacts for further use by the research community.
2
Related Work
FACET-P ROBE differs from prior work along five axes: input facets (option, evidence-chunk, document rank, image-set, mixed-modality), model generation (frontier and open-weight 2026 MLLMs), measurement target (cross-order instability), decomposition via a same-ordering control, and mitigation evaluation. The decomposition axis is central: no prior work estimates the ordering excess over same-input decoder-stochastic flips on the 2026 multimodal frontier. Ordering sensitivity in language models. LMs are sensitive to many semantically irrelevant in-
Do multimodal large language models give the same answer when the same evidence is presented in a different order? If not, how prevalent is the failure, and which interventions reduce it? This paper introduces FACET-P ROBE. FACETP ROBE is a five-facet audit of 18 multimodal large language models (6 frontier closed-source: Gemini-Pro 3.1, Gemini-Flash 3, Claude Opus 4.7, Claude Sonnet 4.6, ChatGPT 5.5, ChatGPT 5.4-mini; 12 open-weight: Qwen3.5, In2
put perturbations: few-shot demonstration order (Lu et al., 2022), prompt format (Sclar et al., 2024; Mizrahi et al., 2024), multiple-choice option order (Pezeshkpour and Hruschka, 2024; Zheng et al., 2024), position in long context (Liu et al., 2024a), two-turn challenges (Laban et al., 2023), reasoning premise order (Chen et al., 2024b), embedding-prompt instability (Kostiuk and Enevoldsen, 2026), and a theoretical impossibility for inverse-permutation learning in decoderonly transformers (Alur et al., 2025). Multimodal work documents single-facet positional bias (Tan et al., 2024), multi-image VLM position bias (Tian et al., 2025), prompt-template sensitivity on 10 models (Ismithdeen et al., 2025), and U-shaped position bias in multimodal RAG (Yao et al., 2025). Multimodal evaluation and benchmark validity. Multimodal benchmarks (Yue et al., 2024; Wang et al., 2024b; Lu et al., 2024; Chen et al., 2024a; Zuo et al., 2025; Jiang et al., 2024) score each item under one canonical ordering. MMBench (Liu et al., 2024b) mitigates MCQ positional bias via CircularEval; we generalize to five orthogonal facets and decompose the residual via ODI. Validity work (Bean et al., 2026; Salaudeen et al., 2025; Reuel et al., 2024; Jacovi et al., 2023), including NIST AI 800-2 (Keller et al., 2026), treats benchmarks as instruments whose perturbation envelope must be qualified. IRT frameworks (Uebayashi et al., 2026; Polo et al., 2024; Romanou et al., 2026) target cross-modal shortcuts or item difficulty; ODI targets ordering variance via per-item σπ and per-facet |δ|. Inference-time mitigation. Self-consistency (Wang et al., 2023) is the canonical training-free mitigation and our baseline. Permutation selfconsistency (Tang et al., 2024) extends this to listwise perturbations and runs inside our mixedmodality LLM judge (Section 3). Positiondebiasing methods include PINE (Wang et al., 2025b), attention calibration (Hsieh et al., 2024), multimodal causal-bidirectional interpolation (Tian et al., 2025), and swap-and-vote LLM-as-judge (Zheng et al., 2023). Section 4.3 evaluates these against cross-ordering flip on the 2026 multimodal frontier.
3
Pro 3.1 (gemini-3.1-pro-preview) and GeminiFlash 3 (gemini-3-flash-preview) (Pichai et al., 2025); Claude Opus 4.7 (claude-opus-4-7) and Claude Sonnet 4.6 (claude-sonnet-4-6) (Anthropic, 2026); ChatGPT 5.5 (gpt-5-5) and ChatGPT 5.4-mini (gpt-5-4-mini) (Singh et al., 2025). Open-weight (12): Qwen3.5-VL (0.8B, 2B, 4B, 9B, 27B) (Qwen Team, 2026), InternVL3.5 (4B, 8B, 14B, 38B) (Wang et al., 2025a), Kimi-VLA3B-Instruct (Team et al., 2025), and MedGemma (4B-IT, 27B-IT) (Sellergren et al., 2025). Closedsource inference uses each provider’s API at temperature 0 where accepted (omitted on Anthropic and OpenAI reasoning APIs, which reject the parameter); top-p = 1 on Gemini; max output tokens 8192 (Anthropic, OpenAI) or 24576 (Gemini); thinking budget 8192 tokens (Anthropic, Gemini) or reasoning-effort medium (OpenAI). A common prompt family is used across models, with facetspecific answer-format instructions (full prompt and per-model adapter details in Section A); briefly, prompts instruct the model to read every evidence item before answering, and closed-choice prompts require exactly one option label preceded by “Answer: ”. Open-weight models are greedy-decoded in HuggingFace transformers (≥4.57) on a single A6000 GPU, with max_new_tokens 512 (default) or 2048 when emitting reasoning traces. Models load in bfloat16 except the InternVL3.5 family (4B/8B/14B/38B) and the ≥27B variants (Qwen3.5-VL 27B, MedGemma 27B-IT), which use 4-bit bitsandbytes NF4 (double quantization, bfloat16 compute) with FlashAttention-2. Caveats. The thinking/reasoning parameter has different per-provider semantics; we use the closest cross-provider analog above and vary Gemini’s budget in the ablation (Section 4.3). The May 4–25, 2026 access window pairs with each provider ID as the reproducibility anchor. Datasets and sampling. FACET-P ROBE covers 12 datasets across four task families. MCQ reasoning (OPTION - ORDER): MMLU-Pro (Wang et al., 2024b), CommonsenseQA (Talmor et al., 2019), and MathVision (Wang et al., 2024a). Multihop QA and RAG (EVIDENCE - CHUNK - ORDER and DOCUMENT- RANK - ORDER): HotpotQA (Yang et al., 2018), MuSiQue (Trivedi et al., 2022), MultiHop-RAG (Tang and Yang, 2024); MedXpertQA’s (Zuo et al., 2025) retrieved-evidence variant also stresses EVIDENCE - CHUNK - ORDER. Multi-image VLM (IMAGE - SET- ORDER): Mantis-
Methods
Models. We audit 18 multimodal large language models served between May 4 and May 25, 2026. Frontier closed-source (6): Gemini3
Eval (Jiang et al., 2024), MedFrameQA (Yu et al., 2025b). Free-form mixed-modality RAG (MIXED MODALITY- ORDER ): MRAMG-Recipe (Yu et al., 2025a), MMDocRAG (Dong et al., 2025), MultiModalQA (Talmor et al., 2021).
a ranked list. IMAGE - SET- ORDER permutes the order of multi-image inputs to a VLM (Mantis-Eval 3–6 images, MedFrameQA 2–5 images). We apply a position-reference screen for image-set analyses: items whose question or answer options explicitly refer to image position are excluded from clean image-set summaries (52/70 Mantis-Eval items excluded; 5/200 MedFrameQA items excluded). MIXED - MODALITY- ORDER permutes the entire heterogeneous text-and-image component sequence per item in free-form RAG; outcomes are scored by an LLM judge (below) as Bernoulli goldmatch labels, paralleling the MCQ facets while adding a separate measurement layer. Permutation grammar. For each (item, facet), we sample K = 6 orderings from the facet’s permutation grammar: uniform sampling without replacement from the n! permutations of the n components when n! ≥ 6; when fewer than six unique permutations exist, we use all unique permutations and repeat as needed to keep six calls. K = 6 is set by a K-ablation on MedXpertQA: K = 3 recovers most, but not all, of the K = 6 detectable flips at half the inference cost, so K = 6 is our operating point across the panel. Permutation manifests are fixed before inference and replayed across all 18 models; the planned release includes these indices so faithful regeneration does not depend on process hash seeding. Flip-rate definition. For model m on item i under facet f , let yi,m,f,k ∈ Yf be the normalized answer under ordering k (source-option index for OPTION ORDER and IMAGE - SET- ORDER, normalized shortanswer string for EVIDENCE - CHUNK - ORDER and DOCUMENT- RANK - ORDER , LLM-judge verdict for MIXED - MODALITY- ORDER; Section E). We define an item-level any-flip indicator flipi,m,f = 1 {yi,m,f,k : k = 1, . . . , K} > 1 ,
Within each dataset we sample a fixed item subset by deterministic uniform shuffling with seed 42, then apply dataset-specific filtering to admit only items with the required structural support for the target facet. Per-dataset N (after filtering) ranges 70–200, with mixed-modality datasets totaling N =597 across 3 datasets; full filter list and perdataset N in Section A. We attempt the same item set for all 18 models within each (facet, dataset) cell, with retrieved evidence (RAG and multi-hop) precomputed once and replayed verbatim across models; derived analyses report parse or drop rates when a model has unusable outputs. We do not filter to gold-evidence-containing items: HotpotQA and MuSiQue use their distractor-augmented splits, so the gold answer may or may not be derivable from the visible evidence, mirroring open-retrieval deployment. Demoted dialog-turn and few-shot facets and their datasets are reported in Section F. Ordering facets. FACET-P ROBE audits five ordering facets, each defined by the unit of permutation and the scoring rule. OPTION - ORDER permutes the mapping between option content and displayed label (the option contents themselves are unchanged; only which content gets which letter is reordered). Scoring maps the model’s predicted letter back to the source option index via the inverse permutation, so a flip denotes a change in selected option content, not a change in letter; flip on letters is uninformative because the gold letter moves with the permutation. Per item, the number of options permuted equals the dataset’s native choice count (CSQA fixed at 5; MMLU-Pro 4–10, median 5; MathVision 4–5). EVIDENCE - CHUNK - ORDER permutes the order of evidence passages within a single flatlist context (HotpotQA, MuSiQue, MedXpertQA retrieved variant), where each unit is a paragraph or short passage; 3–6 chunks per item. DOCUMENTRANK - ORDER permutes the order of ranked retrieved documents in a RAG prompt (MultiHopRAG, HotpotQA-replay), where each unit is a complete document at a specific retrieval rank; 2–6 documents per item. The distinction from EVIDENCE CHUNK - ORDER is the unit of permutation: chunks are sub-document passages presented as a flat list, documents are complete retrieval units presented as
and report the panel flip rate Nf
1 X FlipRatem,f = flipi,m,f . Nf i=1
We use item-level any-flip rather than pairwise disagreement because it answers the deploymentnatural question (“can the answer change under permutation?”); it is K-monotone (larger K can only discover more disagreements). Our K-ablation shows that K = 3 recovers most, but not all, of the K = 6 detectable-flip signal, with K = 6 as the operating point for the panel. 4
outcome Ym,i,f,o ∈ {0, 1} is correctness under that ordering (deterministic scoring for non-MIXED MODALITY- ORDER facets; LLM-judge gold-match for MIXED - MODALITY- ORDER). The correctness logit is logit pm,i,f,o = αi θm − βi − δf, d(i), o − γi, o , 2 γi, o ∼ N 0, σπ,i , 2 log σπ,i ∼ N µf (i) , τf (i) ,
Same-ordering control. To estimate ordering excess over decoder-stochastic flips, we run the canonical ordering K = 6 times (same input, independent API calls or decoder seeds) and compute a within-canonical flip rate FlipRatesame as an upper bound on decoder noise. We then report ∆order = FlipRateperm − FlipRatesame . We interpret ∆order as an estimate of excess instability attributable to ordering, not as a literal decomposition of individual flips; the subtraction can be negative under decoder-noise-dominated cells and does not propagate per-cell uncertainty. The temperature-0 decomposition is scoped to a 12-cell panel on Gemini-Pro 3.1 and Gemini-Flash 3 (6 OPTION - ORDER + 4 IMAGE - SET- ORDER + 2 EVIDENCE - CHUNK - ORDER cells); we report 8 verified cells here (4 IMAGE - SET- ORDER plus MMLUPro OPTION - ORDER and MedXpertQA EVIDENCE CHUNK - ORDER on both models). Generalization beyond these cells, especially at deployment temperatures, is a stated limitation (Section 6). LLM-judge scoring for free-form output. The non-MIXED - MODALITY- ORDER facets use deterministic scoring: OPTION - ORDER and IMAGE - SETORDER map predicted labels back to source content under the inverse permutation, while EVIDENCE CHUNK - ORDER and DOCUMENT- RANK - ORDER use exact-match against gold short answers. MIXED - MODALITY- ORDER yields paragraph-form output on 2 of 3 benchmarks (MRAMG-Recipe, MMDocRAG); MultiModalQA gold is short factoid. For all three, we define a Bernoulli correctness outcome via an LLM judge (Gemini Pro as primary judge, ChatGPT 5.4-mini as cross-vendor check) that scores the K=6 item outputs with a structured equivalence prompt. Text-flip (raw paraphrase variation) is reported alongside as an upper bound that the judge correction collapses by 10– 90 pp depending on model and benchmark. Full prompt, inter-judge Cohen’s κ, and a judge-free MMQA gold-anchor cross-check are in Section E and Section H. The MIXED - MODALITY- ORDER outcome enters ODI (below) as a Bernoulli goldmatch label, while the judge-based measurement layer requires the caveat in Table 2. ODI. ODI (Ordering-Decomposed ItemResponse Theory) is a 2PL hierarchical Bayesian item-response model (Birnbaum, 1968; Embretson and Reise, 2025). For model m, item i, facet f , and permutation index o ∈ {1, . . . , K}, the binary
where αi is item discrimination, βi is item difficulty, θm is model ability, d(i) is the dataset containing item i, and δf,d,o is a per-(facet, dataset, permutation-index) systematic latent offset hierarchically pooled within facet via σδ,f . The item2 , also level random effect γi,o has variance σπ,i hierarchically pooled within facet through a logNormal hyper-prior. We report per-facet posterior medians of the item-pooled σπ,i and the facetpooled |δf,d,o | as facet-level summaries in Table 2. Indexing note and limitations. Here o is the permutation index over the K = 6 orderings sampled per item, not a slot-position; δf,d,o therefore captures average outcome shift across permutation indices within a (facet, dataset) cell. Content-vsposition decomposition (whether a specific option content or image is more salient than a specific slot) is absorbed into βi and αi rather than modeled separately, and the model has no model-level random effects beyond θm . These are deliberate simplifications: ODI is a summary instrument for orderingassociated variance, not a content-saliency decomposition. Inference uses NUTS (Hoffman et al., 2014) in NumPyro / JAX (Bradbury et al., 2018; Phan et al., 2019) (4 chains × 3000 tune + 1500 draws, targetaccept 0.99, fp64). We fit two outcome variants: modal-outcome (Y = 1 iff the answer under ordering o matches the model’s modal answer for that item, emphasizing ordering-associated instability rather than item difficulty) for the facet-level σπ and |δ| summaries; correct-outcome (Y = 1 iff the answer matches gold) for θcorrect in Figure 1. Priors, NUTS settings, R̂, ESS, and divergence diagnostics are in Section B.
4
Findings
4.1
Cross-Facet Ordering Sensitivity
Q1: How prevalent is cross-ordering flipping across the 18-model panel? Crossordering flipping is widespread across models and 5
Figure 2: Per-facet ordering sensitivity. Capability (θcorrect , IRT-derived) vs per-facet K=6 flip rate across the 5 facets, for n=7 best-of-family models (vendor-logo markers): Gemini-Pro 3.1, Claude Opus 4.7, ChatGPT 5.5, Qwen3.5-VL 27B, InternVL3.5 14B, Kimi-VL-A3B-Instruct, MedGemma 27B-IT. Dashed references mark frontier (n=3) and open-weight (n=4) best-of-family means. MIXED - MODALITY- ORDER uses LLM-judge sem-flip (Section H). Mixed-modality is highest, doc-rank is lowest, and image-set is reported after the MANTIS - EVAL position-reference screen. Table 1: Per-facet flip rate by model. Mean K=6 flip across 5 facets, plus a 5-facet mean column. Each facet column is a cell-equal mean over its balanced main-panel datasets; the 5-facet mean averages the facet columns equally. Per-column best bold, second-best underlined. MIXED - MODALITY- ORDER is LLM-judge sem-flip averaged across 3 mm datasets (MMDocRAG, MMQA, MRAMG). IMAGE - SET- ORDER uses the clean position-referencescreened subset. Aggregation details in Section C; capability in Figure 1. Model
OPTION - ORDER
EVIDENCE - CHUNK - ORDER
DOCUMENT- RANK - ORDER
IMAGE - SET- ORDER
MIXED - MODALITY- ORDER
Mean (5-facet)
Claude Opus 4.7 Claude Sonnet 4.6 ChatGPT 5.5 ChatGPT 5.4-mini Gemini-Pro 3.1 Gemini-Flash 3
0.13 0.20 0.09 0.23 0.11 0.17
0.24 0.28 0.23 0.40 0.13 0.20
0.11 0.33 0.26 0.27 0.08 0.13
0.13 0.21 0.15 0.31 0.10 0.14
0.26 0.30 0.24 0.36 0.25 0.26
0.17 0.26 0.19 0.31 0.13 0.18
Qwen3.5-VL 0.8B Qwen3.5-VL 2B Qwen3.5-VL 4B Qwen3.5-VL 9B Qwen3.5-VL 27B
0.79 0.67 0.46 0.40 0.27
0.72 0.57 0.47 0.41 0.34
0.41 0.41 0.23 0.28 0.29
0.35 0.58 0.33 0.20 0.26
0.84 0.71 0.53 0.52 0.38
0.62 0.59 0.40 0.36 0.31
InternVL3.5 4B InternVL3.5 8B InternVL3.5 14B InternVL3.5 38B
0.45 0.44 0.40 0.29
0.50 0.46 0.42 0.42
0.28 0.19 0.19 0.37
0.23 0.33 0.26 0.25
0.65 0.57 0.61 0.60
0.42 0.40 0.38 0.39
Kimi-VL-A3B-Instruct MedGemma 4B-IT MedGemma 27B-IT
0.45 0.57 0.43
0.46 0.58 0.46
0.28 0.34 0.25
0.18 0.19 0.19
0.59 0.77 0.51
0.39 0.49 0.37
Panel mean (n=18)
0.36
0.41
0.26
0.24
0.50
0.35
facets (Table 1, Figure 2): per-facet panel-mean K=6 flip spans 0.24–0.50 (MIXED - MODALITYORDER 0.50, EVIDENCE - CHUNK - ORDER 0.41, OPTION - ORDER 0.36, DOCUMENT- RANK - ORDER 0.26, screened IMAGE - SET- ORDER 0.24); MIXED MODALITY- ORDER sem-flip is 0.50 median across 54 (model×benchmark) cells, with raw MIXED MODALITY- ORDER benchmark cells spanning 0.09–0.89.
cells show a substantial ordering excess over a same-input decoder-noise floor, but the share is substrate- and temperature-conditional. Across 8 verified cells of the 12-cell panel (Section D), the MMLU-Pro OPTION - ORDER cells show an ordering-excess share of about 51–75%. On the clean MedFrameQA IMAGE - SET- ORDER subset, the same-ordering floor remains below the crossordering rate (0.03/0.10 vs 0.10/0.14 on Pro/Flash), while MedXpertQA EVIDENCE - CHUNK - ORDER shows a similar but smaller excess (0.11/0.15 vs
Q2: How much exceeds the same-input decodernoise floor? At temperature 0, verified Gemini 6
4.2
0.20/0.21). Mantis-Eval is excluded from the clean image-set decomposition because most items contain position-reference language. At T ≥ 0.7, a broader appendix sweep shows the consistencyshare estimator is no longer a stable decomposition target, whereas decoder-cleaned accuracy swing remains informative (Section D).
Q5: Does capability scale with ordering robustness? Capability predicts flips but does not eliminate them. Panel-wide, capability and ordering robustness are tightly coupled: Spearman ρ(θcorrect , mean K=6 flip) ≈ −0.95 across the 18model panel (screened 5-facet flip means with screened θcorrect ) (Figure 1(b)); best-of-family cluster means are 17% (n=3 frontier) vs 36% (n=4 open-weight). The 18 models are not independent samples (families share training-data lineages, architectures, and vendor-specific serving behaviors), and θcorrect is recovered from the correct-outcome ODI fit on the same trial outcomes that produce flip rate; we therefore read this correlation as a descriptive cross-panel summary rather than a generalizable law. Scaling helps but does not eliminate: the best model (Pro) still flips on 13.4%. Within open-weight families, parameter count reduces screened 4-facet flip monotonically up to mid-scale (Qwen3.5-VL 0.8B→27B: 0.57 → 0.29; InternVL3.5 4B→14B: 0.37 → 0.32, slight 38B uptick to 0.33; Figure 4) but never closes the gap to the frontier mean. Within the frontier cluster, capability rank does not fully determine robustness rank (within-frontier Spearman ρ ≈ −0.89, n=6, qualitative): Opus over-performs its capability rank while Sonnet under-performs (Table 1; permodel θ and flip rates in Section B). The spread is mechanism-conditional (Section 4.2).
Q3: Where are the null effects? We observe two reliable nulls (tool-description ordering under disambiguating queries, and few-shot math on GSM8K), both detailed in Section F. Q4: Which facets carry per-item ordering noise, and which carry systematic bias? Ordering noise varies by substrate, with mixed-modality carrying the largest per-item ordering variance in the screened modal-outcome fit. The ODI decomposition (Section 3) separates per-item ordering noise (σπ ) from per-facet systematic bias (|δ|) on the screened modal-outcome fit (Table 2): IMAGE SET- ORDER is elevated in that modal-outcome fit (σπ 1.73×, |δ| 8.25× the OPTION - ORDER baseline), but the clean image-set interpretation is anchored on MEDFRAMEQA after the positionreference screen. MIXED - MODALITY- ORDER has the largest σπ overall (3.03×); its large |δ| is a latent IRT offset estimated from LLM-judge goldmatch labels, so the judge mechanism may add measurement variance and inflate the magnitude (Table 2, †). Posterior medians, 89% HDIs, crossfacet KS separability, and per-parameter diagnostics are in Section B. A companion correct-outcome fit supplies θcorrect for the capability analysis in Figure 1 and this section.
Q6: Does stated confidence calibrate to ordering risk? Self-reported confidence under-predicts cross-ordering flip rate across all 7 models tested; mismatch shrinks with capability but persists. On a MedXpertQA multi-turn elicitation (4 turns, with a 0–1 confidence after each of three evidence chunks; protocol in Section D), we define mismatch as flip rate−(1−turn-3 confidence). Across 7 models tested, mismatch ranges 8.8–62.1 pp (Opus +8.8 best to ChatGPT 5.4-mini +62.1 worst; Spearman ≈ −0.57 vs θcorrect ). In this protocol, the orderingrisk component is not reflected in the self-reported calibration signal, even with the same-ordering noise proxy.
Table 2: Per-facet ordering noise and systematic bias. Posterior median σπ and |δ| from the screened modaloutcome ODI fit; ratios vs OPTION - ORDER in parentheses. Bold marks the largest non-MIXED - MODALITYORDER facet; MIXED - MODALITY- ORDER is shown separately because its LLM-judge outcome mechanism adds measurement variance. The IMAGE - SET- ORDER row should be read with the position-reference screen caveat in Section 6. †: MIXED - MODALITY- ORDER outcomes are LLM-judge gold-match Bernoulli labels; because |δ| is a latent IRT offset estimated from those labels, the judge mechanism may add measurement variance and inflate the value. Facet OPTION - ORDER DOCUMENT- RANK - ORDER EVIDENCE - CHUNK - ORDER IMAGE - SET- ORDER MIXED - MODALITY- ORDER
σπ (ratio)
|δ| (ratio)
0.088 (1.00×) 0.097 (1.10×) 0.105 (1.20×) 0.152 (1.73×) 0.265 (3.03×)
0.003 (1.00×) 0.010 (3.81×) 0.012 (4.50×) 0.022 (8.25×) 4.796†
Capability, Calibration, and Mechanism
Why ordering changes the answer. We sample up to N =50 cross-ordering-flipped items per source-dataset cell and submit each to a GeminiPro judge classifying the failure into one of six modes (Section I). Content rationalization is the dominant primary-judge label across the nondialog categorical cells in this sample (MATH 7
Table 3: Cost-Pareto for mitigation policies. Mean accuracy across 26 screened (model × facet × dataset) cells; ∆ vs P1 . P2 (K=2 abstain, 79.7% coverage) is the only non-baseline deployable Pareto improvement; P5 /P6 are oracle bounds. P2 ’s mean is selective accuracy (items where the two calls agree); treating abstention as incorrect, all-items accuracy is ≈ 0.59. We assume abstained items trigger escalation; the selective number is the operational accuracy under that assumption.
VISION , MEDXPERTQA , and MMLU - PRO ; 58%,
n=126). The original IMAGE - SET- ORDER mechanism row also labels content rationalization as modal (80%, n=49), but it was sampled before the MANTIS - EVAL position-reference screen and is therefore treated as exploratory rather than a clean image-set estimate. Cross-family agreement on the 3-class collapse is fair on the Landis-Koch scale (Fleiss κ = 0.30, n=225 items, 3 judges; Section I). The best-supported mechanism claim is that categorical-order flips commonly rationalize over answer or evidence content; the analogous clean visual-substrate claim requires a screened resample. 4.3
Mitigation Menu
Training-free mitigation can detect or reduce some ordering sensitivity, but does not reliably eliminate it and trades off cost, coverage, or modality transfer. We test three categories: detection/abstention (K=2 disagreement screen), aggregation (K-majority voting / self-consistency), and prompt transformation (CTA and related prompts).
Policy
Cost
Mean acc.
∆ vs P1
P1 single-pass P2 K=2 abstain P3 K=3 majority P4 K=6 majority
1 2 3 6
0.629 0.648 0.648 0.640
— +0.019 +0.019 +0.011
P5 worst-ord. oracle (l.b.) P6 best-ord. oracle (u.b.)
6 6
0.580 0.674
−0.049 +0.046
(a)
Detection and aggregation: K-policies. A K=2 disagreement screen (two repeated queries, abstain or escalate on disagreement) is the cheapest deployable policy: it is precision-1 by definition against its own detection target, with recall 0.47–0.80 against K=6 flip on non-degenerate non-image text cells. Clean Mantis image-set cells are too small/degenerate for a stable recall headline. Across 26 screened cells, K=2 abstain delivers +0.019 selective accuracy over K=1 at cost 2; K=3 majority matches that gain at cost 3; K=6 underperforms (+0.011 at cost 6; Table 3). K=2 is therefore the only non-baseline Pareto improvement among deployable policies. A practical benefit of recording multiple orderings is that aggregation and abstention policies can be evaluated post hoc on any completed panel; extending the K-policy analysis beyond Gemini mainly requires scoring existing multi-order outputs, whereas prompt transformations such as CTA require new inference.
(b)
Figure 3: Mitigation outcomes (per-cell n = 50 or 100, K=6, T =0). (a) CTA with Gemini 3.1 Pro at an 8,192-token think budget: −40% on the high-baseline text cell, little change on lower-baseline text cells, and no measured effect on the visual cell. (b) Think-budget for Pro (solid) and Flash (dashed), hard MEDXPERTQA (EVIDENCE - CHUNK - ORDER, blue) vs easy MMLU - PRO (OPTION - ORDER, green). Per-cell tables in Section G.
Prompt transformation: Canonicalize-thenAnswer (CTA) is capability- and modalityasymmetric. CTA prefixes the question with a model-generated canonicalization of the evidence. On text MEDXPERTQA (Pro), CTA reduces flip 0.30 → 0.18 (−40%); on the uncertainty-clean subset, Pro improves further to 0.135 → 0.029 (−79%). On Flash, MedXpertQA CTA is null
to slightly regressing (0.26 → 0.30); on lowerbaseline text MCQ, CTA is marginal (Pro MMLU PRO 0.10 → 0.09; COMMONSENSEQA 0.06 → 0.07). On visual reasoning, CTA does not re8
duce flips (MATHVISION: Pro 0.29 → 0.29, Flash 0.35 → 0.35; Figure 3a) despite baselines comparable to MedXpertQA, consistent with, but not proving, a visual-substrate account (Section 4.2). Stacking CTA with multipass reconciliation is antisynergistic on MEDXPERTQA (Pro 0.18 → 0.29, Flash 0.30 → 0.40): pick one prompt-level intervention, not several. This is the main negative result for prompt-level mitigation on multimodal LLMs. Think-budget and other prompt-level knobs. Think-budget is dataset-conditional (Figure 3b): on hard MEDXPERTQA (EVIDENCE - CHUNK - ORDER) Pro flip drops monotonically with budget (1k → 24k: 0.41 → 0.28); on easy MMLU - PRO (OPTION ORDER ) both Pro and Flash stay essentially flat (0.08–0.12) while accuracy rises into 0.93. Orderaware minimal-disclaimer (P15), CanonicalizingCoT (T4), and PINE-lite are null or marginal in mean (per-model deltas in Section G).
5
trol is a methodological check: it estimates the repeated-call stochasticity floor, so reported flip rates can be interpreted relative to same-input instability. Training-time and architectural interventions are a path forward. Think-budget reduces flip on hard items but plateaus on easy at linear token cost; prompt-level mitigation is modality-conditional and does not compose. This trade-off points toward training-time, objective, or architectural interventions (Egressy and Stühmer, 2026) over inferencetime compute. One concrete hypothesis is that pre-training or post-training mixtures overrepresent fixed ordering conventions (canonical option letters, retrieval ranks, or image slots); future work should test whether permutation-augmented or contrastive ordering objectives reduce the effect (Alur et al., 2025; Egressy and Stühmer, 2026). Documenting input ordering as a deployment configuration parameter is an interim safeguard.
Discussion
None of 18 multimodal LLMs are order-invariant: screened per-facet panel-mean K=6 flip ranges 24– 50%. At temperature 0, the same-ordering Gemini control estimates a substantial ordering excess over decoder noise in verified cells, but this decomposition is substrate- and temperature-conditional. Capability predicts flips but does not eliminate them (panel-wide ρ ≈ −0.95; best model still flips 13.4%). Mechanism analysis shows content rationalization as a common categorical-cell failure mode; the original image-set mechanism row is exploratory until repeated on the screened image-set subset. CTA reduces flip from 0.30 to 0.18 on the high-baseline text cell (a 12 pp absolute drop, 40% relative reduction) and has no measured effect on the comparable-baseline visual cell. Order robustness as a reporting axis. Canonical benchmark accuracy leaves the order-robustness dimension uncaptured: no frontier or open-weight model card we audited reports a cross-ordering flip rate, despite an active research literature on the failure mode (Pezeshkpour and Hruschka, 2024; Chen et al., 2024b; Laban et al., 2023). Following Mitchell et al. (2019) on model-card disclosure, Raji et al. (2022) on undocumented brittleness, and NIST AI 800-2’s “qualified claims” principle (Keller et al., 2026), we propose cross-ordering flip rate as a standard reporting axis for multimodal LLMs, alongside accuracy. The same-ordering con9
6
Limitations
cross-facet |δ| comparisons should be read with this caveat. In the Gemini MMQA temperature-sweep gold-anchor check, sem-flip exceeds direct gold-match flip by 4.8–11.1 pp, so the judge-corrected statistic is not identical to a judge-free correctness flip.
• Same-ordering control is scoped, not panelwide. The estimate of ordering excess over decoder-stochastic flips runs on a 12-cell Gemini subset at temperature 0 (8 verified cells reported; Section 3); extending the control to all (model, facet) cells would tighten the orderingexcess claim. The appendix temperature sweep adds deployment-temperature evidence, but the open-weight extension is at T ∈ {0.7, 1.0} and is therefore a protocol extension rather than a matched temperature-0 comparison.
• Mechanism labels are diagnostic, not causal. The six mechanism classes come from an LLMjudge classification with cross-family Fleiss κ = 0.30 (fair on the Landis-Koch scale); the categorical-cell rationalization result is directionally judge-stable but the labels do not constitute a causal account of internal model behavior. The original image-set mechanism sample was collected before the position-reference screen and is exploratory until a screened resample is run.
• K=6 is a subsample of n! and the metric is K-sensitive. For items with n! > 6, reported flip rates are a lower bound on true sensitivity at higher K; the item-level any-flip indicator is K-monotone (Section 3), so cross-paper comparisons should match K.
• Dataset coverage is bounded. Our 12 datasets cover MCQ, multi-hop QA / RAG, multi-image VQA, and free-form mixed-modality RAG; open-ended generation, agentic tool-use beyond the disambiguating-tool null, multi-turn dialog beyond MULTICHALLENGE, non-English content, and in-the-wild user–LLM distributions (Zhao et al., 2024; Chou et al., 2025) are out of scope.
• Temperature-0 API calls are not strictly deterministic. Providers may include nondeterministic backend components even at T =0; we use the same-ordering control to quantify this floor rather than assume it is zero. • Clean image-set inference is anchored on MedFrameQA. The image-set positionreference screen excludes 52/70 Mantis-Eval items and 5/200 MedFrameQA items, leaving a clean image-set subset of 18 Mantis-Eval and 195 MedFrameQA items. We therefore avoid treating Mantis-heavy image-set rows as clean estimates of visual order robustness.
• Mitigation experiments are scoped to Gemini Pro and Flash. The modality-asymmetric CTA result, K=2 disagreement-flag thresholds, and think-budget regime characterization all run on two frontier models; Claude and ChatGPT replication would push the prompt-level findings to be vendor-agnostic.
• Closed-source drift and stack asymmetry. Provider APIs can update silently within their version IDs; we pin the May 4–25, 2026 access window (Section 3) as the reproducibility anchor for closed-source experiments. Separately, closed-source uses provider APIs while openweight uses our local transformers stack, so cross-cluster comparisons mix model behavior with stack-level differences (within-family open-weight scaling in Figure 4 partially controls within open-weight).
• Training-time interventions are not directly tested. Our finding that prompt-level mitigation is modality-conditional and does not compose is evidence against a sufficient prompt-level fix; it is not evidence for the sufficiency of trainingtime, objective, or architectural interventions, which we identify as directions for future work.
• LLM-judge sem-flip on MIXED - MODALITYORDER carries measurement variance. Mixed-modality outcomes are LLMjudge gold-match Bernoulli labels like the deterministic-scored facets, but the judge mechanism introduces variance beyond deterministic scoring (Table 2, †); direct 10
References
Susan E Embretson and Steven P Reise. 2025. Item response theory: Foundations for psychologists and social scientists. Routledge.
Mohammad Mahdi Abootorabi, Amirhosein Zobeiri, Mahdi Dehghani, Mohammadali Mohammadkhani, Bardia Mohammadi, Omid Ghahroodi, Mahdieh Soleymani Baghshah, and Ehsaneddin Asgari. 2025. Ask in any modality: A comprehensive survey on multimodal retrieval-augmented generation. Findings of the Association for Computational Linguistics: ACL 2025, pages 16776–16809.
Kazem Faghih, Wenxiao Wang, Yize Cheng, Siddhant Bharti, Gaurang Sriramanan, Sriram Balasubramanian, Parsa Hosseini, and Soheil Feizi. 2025. Tool preferences in agentic llms are unreliable. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20965– 20980.
Rohan Alur, Chris Hays, Manish Raghavan, and Devavrat Shah. 2025. The impossibility of inverse permutation learning in transformer models. arXiv preprint arXiv:2509.24125.
Tyler Haberle, Courtney Cleveland, Greg L Snow, Chris Barber, Nikki Stookey, Cari Thornock, Laurie Younger, Buzzy Mullahkhel, and Diego Ize-Ludlow. 2024. The impact of nuance dax ambient listening ai documentation: a cohort study. Journal of the American Medical Informatics Association, 31(4):975–979.
Anthropic. 2026. Introducing Claude 4.7. Accessed: June 25, 2026. Andrew M Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan Eghlidi, Chris Schmitz, Karolina Korgul, Hunar Batra, and 1 others. 2026. Measuring what matters: Construct validity in large language model benchmarks. Advances in Neural Information Processing Systems, 38.
Matthew D Hoffman, Andrew Gelman, and 1 others. 2014. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15(1):1593–1623. Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T. Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. 2024. Found in the middle: Calibrating positional attention bias improves long context utilization. Preprint, arXiv:2406.16008.
Allan Birnbaum. 1968. Some latent trait models and their use in inferring an examinee’s ability. Statistical theories of mental test scores. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and 1 others. 2018. Jax: composable transformations of python+ numpy programs.
Ryan T Hurt, Christopher R Stephenson, Elizabeth A Gilman, Christopher A Aakre, Ivana T Croghan, Manpreet S Mundi, Karthik Ghosh, and Jithinraj Edakkanambeth Varayil. 2025. The use of an artificial intelligence platform openevidence to augment clinical decision-making for primary care physicians. Journal of Primary Care & Community Health, 16:21501319251332215.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and 1 others. 2024a. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056–27087.
Mohamed Insaf Ismithdeen, Muhammad Uzair Khattak, and Salman Khan. 2025. Promptception: How sensitive are large multimodal models to prompts? arXiv preprint arXiv:2509.03986.
Xinyun Chen, Ryan A Chi, Xuezhi Wang, and Denny Zhou. 2024b. Premise order matters in reasoning with large language models. arXiv preprint arXiv:2402.08939.
Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. Preprint, arXiv:2305.10160.
Christopher Chou, Lisa Dunlap, Koki Mashita, Krishna Mandal, Trevor Darrell, Ion Stoica, Joseph E Gonzalez, and Wei-Lin Chiang. 2025. Visionarena: 230k real world user-vlm conversations with preference labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3877–3887.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. 2024. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483. Drew Keller, Ryan Steed, Tony Wang, Stevie Bergman, and Peter Cihon. 2026. Practices for automated benchmark evaluations of language models. NIST AI 800-2 Initial Public Draft.
Kuicai Dong, Yujing Chang, Shijie Huang, Yasheng Wang, Ruiming Tang, and Yong Liu. 2025. Benchmarking retrieval-augmented multimodal generation for document question answering. Preprint, arXiv:2505.16470.
Yevhen Kostiuk and Kenneth Enevoldsen. 2026. One prompt is not enough: Instruction sensitivity undermines embedding model evaluation. arXiv preprint arXiv:2605.22544.
Beni Egressy and Jan Stühmer. 2026. Set-llm: A permutation-invariant llm. Advances in Neural Information Processing Systems, 38:62798–62834.
11
Philippe Laban, Lidiya Murakhovs’ ka, Caiming Xiong, and Chien-Sheng Wu. 2023. Are you sure? challenging llms leads to performance drops in the flipflop experiment. arXiv preprint arXiv:2311.08596.
Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459– 9474.
Du Phan, Neeraj Pradhan, and Martin Jankowiak. 2019. Composable effects for flexible and accelerated probabilistic programming in numpyro. arXiv preprint arXiv:1912.11554. Sundar Pichai, Demis Hassabis, and Koray Kavukcuoglu. 2025. A new era of intelligence with gemini 3. Mountain View, CA: Google.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a. Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics, 12:157–173.
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. 2024. tinybenchmarks: evaluating llms with fewer examples. arXiv preprint arXiv:2402.14992.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024b. Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281.
Qwen Team. 2026. Qwen3.5: Towards native multimodal agents. Inioluwa Deborah Raji, I Elizabeth Kumar, Aaron Horowitz, and Andrew Selbst. 2022. The fallacy of ai functionality. In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, pages 959–972.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, KaiWei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR).
Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J Kochenderfer. 2024. Betterbench: Assessing ai benchmarks, uncovering issues, and establishing best practices. Advances in Neural Information Processing Systems, 37:21763–21813.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming fewshot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098.
Angelika Romanou, Mark Ibrahim, Candace Ross, Chantal Shaib, Kerem Oktar, Samuel J Bell, Anaelia Ovalle, Jesse Dodge, Antoine Bosselut, Koustuv Sinha, and 1 others. 2026. Brittlebench: Quantifying llm robustness via prompt sensitivity. arXiv preprint arXiv:2603.13285.
Lingrui Mei, Jiayu Yao, Yuyao Ge, Yiwei Wang, Baolong Bi, Yujun Cai, Jiazhi Liu, Mingyu Li, Zhong-Zhi Li, Duzhen Zhang, and 1 others. 2025. A survey of context engineering for large language models. arXiv preprint arXiv:2507.13334.
Olawale Salaudeen, Anka Reuel, Ahmed Ahmed, Suhana Bedi, Zachary Robertson, Sudharsan Sundar, Ben Domingue, Angelina Wang, and Sanmi Koyejo. 2025. Measurement to meaning: A validitycentered framework for ai evaluation. arXiv preprint arXiv:2505.10573.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229.
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In International Conference on Learning Representations, volume 2024, pages 25055–25083.
Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics, 12:933–949.
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, and 1 others. 2025. Medgemma technical report. arXiv preprint arXiv:2507.05201.
Akshay Paruchuri, Maryam Aziz, Rohit Vartak, Ayman Ali, Best Uchehara, Xin Liu, Ishan Chatterjee, and Monica Agrawal. 2025. " what’s up, doc?": Analyzing how users seek health information in large-scale conversational ai datasets. arXiv preprint arXiv:2506.21532.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement
12
learning. Advances in neural information processing systems, 36:8634–8652.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2024a. Measuring multimodal mathematical reasoning with math-vision dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, and 1 others. 2025. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. Preprint, arXiv:1811.00937.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025a. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021. Multimodalqa: Complex question answering over text, tables and images. Preprint, arXiv:2104.06039.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. Preprint, arXiv:2203.11171.
Zhijie Tan, Xu Chu, Weiping Li, and Tong Mo. 2024. Order matters: Exploring order sensitivity in multimodal large language models. arXiv preprint arXiv:2410.16983.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024b. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290.
Raphael Tang, Xinyu Zhang, Xueguang Ma, Jimmy Lin, and Ferhan Ture. 2024. Found in the middle: Permutation self-consistency improves listwise ranking in large language models. Preprint, arXiv:2310.07712.
Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham Kakade, Hao Peng, and Heng Ji. 2025b. Eliminating position bias of language models: A mechanistic approach. In International Conference on Learning Representations, volume 2025, pages 91212–91239.
Yixuan Tang and Yi Yang. 2024. Multihop-rag: Benchmarking retrieval-augmented generation for multihop queries. Preprint, arXiv:2401.15391. Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, and 73 others. 2025. Kimi-VL technical report. Preprint, arXiv:2504.07491.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. 2025. Identifying and mitigating position bias of multi-image vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10599–10609.
Jiayu Yao, Shenghua Liu, Yiwei Wang, Lingrui Mei, Baolong Bi, Yuyao Ge, Zhecheng Li, and Xueqi Cheng. 2025. Who is in the spotlight: The hidden bias undermining multimodal retrieval-augmented generation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15194–15204.
Aaron A Tierney, Gregg Gayre, Brian Hoberman, Britt Mattern, Manuel Ballesca, Patricia Kipnis, Vincent Liu, and Kristine Lee. 2024. Ambient artificial intelligence scribes to alleviate the burden of clinical documentation. NEJM Catalyst Innovations in Care Delivery, 5(3):CAT–23.
Qinhan Yu, Zhiyou Xiao, Binghui Li, Zhengren Wang, Chong Chen, and Wentao Zhang. 2025a. Mramgbench: A beyondtext benchmark for multimodal retrieval-augmented multimodal generation. arXiv preprint arXiv:2502.04176.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multihop questions via single-hop question composition. Preprint, arXiv:2108.00573.
Suhao Yu, Haojin Wang, Juncheng Wu, Cihang Xie, and Yuyin Zhou. 2025b. Medframeqa: A multi-image medical vqa benchmark for clinical reasoning. arXiv preprint arXiv:2505.16964.
Shunki Uebayashi, Kento Masui, Kyohei Atarashi, Han Bao, Hisashi Kashima, Naoto Inoue, Mayu Otani, and Koh Takeuchi. 2026. Evaluating cross-modal reasoning ability and problem characteristics with multimodal item response theory. arXiv preprint arXiv:2603.02663.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.
13
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9556– 9567. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations, volume 2024, pages 19426–19454. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623. Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. 2025. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362.
14
A
Extended Dataset Details
line, with no other content on that line. The full prompt and per-facet variants will be included in the planned full code release; deviations across models are limited to the provider-required systemmessage structure (e.g., Anthropic system field vs Gemini systemInstruction). Per-model adapters. Closed-source inference uses the official provider SDKs at the model IDs and access window listed in Section 3. Openweight inference uses HuggingFace transformers (≥4.57) on a single A6000 with max_new_tokens 512 (default) or 2048 when emitting reasoning traces. Models load in bfloat16 except the InternVL3.5 family and the ≥27B variants (Qwen3.5VL 27B, MedGemma 27B-IT), which use 4-bit bitsandbytes NF4 with bfloat16 compute and FlashAttention-2. Image inputs are converted to each provider’s expected format (base64 PNG for Anthropic and Gemini, URL or base64 for OpenAI, PIL.Image for HuggingFace processors) without resizing or quality reduction. Dataset licensing. All 12 main-facet datasets carry open licenses that permit research use; the two NC-licensed datasets (MEDXPERTQA, MED FRAMEQA) are used for non-commercial academic evaluation, consistent with their terms. We redistribute no dataset content; the planned full code release will include permutation indices, model outputs, and aggregation scripts that reproduce the audit against the upstream sources. Items that contain identifiable patient information were filtered upstream by the dataset creators; we add no additional content beyond permutation metadata.
This appendix expands the dataset coverage map referenced from Section 3: per-dataset N , license, and primary facet, plus the prompt family and per-model adapter notes that the main paper compresses to a pointer. Per-dataset summary. Table 4 lists each dataset’s audited N (after deterministic seed-42 uniform sampling and facet-specific filtering), license, and the facet it primarily stresses. Per-dataset raw audited N ranges 70–200 across the 12 datasets; the clean image-set subset after the position-reference screen is 18 Mantis-Eval and 195 MedFrameQA items. The mixed-modality benchmarks total N =597 across 3 datasets (MRAMG-Recipe 197 after image-load filter, MMDocRAG 200, MultiModalQA 200). Table 4: Per-dataset coverage. Audited N , license, and primary facet; image-set rows also report the clean position-reference-screened N in parentheses. “Open” denotes a license permitting research use (CC-BY, MIT, Apache-2.0, or equivalent); see the per-paper citation for exact terms. Dataset MMLU - PRO COMMONSENSEQA MEDXPERTQA MATHVISION HOTPOTQA MUSIQUE MULTIHOP - RAG MANTIS - EVAL MEDFRAMEQA
MRAMG-Recipe MMDocRAG MultiModalQA GSM 8 K (demoted) HUMANEVAL (demoted) MULTICHALLENGE (demoted)
N
License
200 200 150 190 199 200 171 70 (18 clean) 200 (195 clean) 197 200 200
MIT MIT CC-BY-NC-4.0 CC-BY-4.0 CC-BY-SA-4.0 CC-BY-4.0 MIT Apache-2.0 CC-BY-NC-4.0 CC-BY-4.0 Apache-2.0 MIT
200–500 156 100–200
MIT MIT CC-BY-4.0
Primary facet OPTION - ORDER OPTION - ORDER EVIDENCE - CHUNK - ORDER OPTION - ORDER EVIDENCE - CHUNK - ORDER , DOCUMENT- RANK - ORDER EVIDENCE - CHUNK - ORDER DOCUMENT- RANK - ORDER IMAGE - SET- ORDER IMAGE - SET- ORDER MIXED - MODALITY- ORDER MIXED - MODALITY- ORDER MIXED - MODALITY- ORDER
few-shot few-shot dialog-turn
Facet × dataset coverage. The 4 deterministicscored non-MIXED - MODALITY- ORDER facets cover all 18 models. Table 1 uses a balanced two-dataset aggregation for each deterministic nonMIXED - MODALITY- ORDER facet; additional facet stress cells, such as MATHVISION and MEDX PERTQA, are reported in targeted analyses. MIXED MODALITY- ORDER covers the 3 free-form RAG benchmarks listed above. Demoted facets (dialogturn, few-shot) cover a subset of (model, dataset) cells; see Section F.
B
Full ODI Methodology
This appendix expands the ODI specification from Section 3: model statement, priors, inference settings, and posterior diagnostics for the two fits whose summaries appear in Table 2 and Figure 1. Model specification. For model m, item i, facet f (i), dataset d(i), and permutation index o ∈ {1, . . . , K}, the correctness logit is logit pm,i,f,o = αi θm − βi − δf, d, o − γi, o , 2 γi, o ∼ N 0, σπ,i , 2 log σπ,i ∼ N µf (i) , τf (i) .
Prompt family. A common prompt family is used across all 18 models and all 5 main facets, with facet-specific instructions for whether the expected output is a single MCQ letter, a short factoid, or a paragraph-form response. The MCQ-style prompt instructs the model to read every evidence item before answering and to output exactly one option label preceded by “Answer: ” on its final
The outcome Ym,i,f,o ∈ {0, 1} is correctness under ordering o (deterministic scoring for the 4 non-MIXED - MODALITY- ORDER facets; LLMjudge gold-match for MIXED - MODALITY- ORDER). 15
Priors. The screened fit uses non-centered parameterizations for the item difficulty, systematic-offset, and ordering-noise hierarchies: zβ,i , zδ,f,d,o , zσ,i ∼ N (0, 1), βi = µβ,f,d + σβ,f,d zβ,i with µβ,f,d ∼ N (0, 1) and σβ,f,d ∼ HalfNormal(0.5); αi ∼ LogNormal(0, 0.3); δf,d,o = σδ,f zδ,f,d,o with σδ,f ∼ HalfNormal(0.3); log σπ,i = µf (i) + τf (i) zσ,i with µf ∼ N (−1.2, 0.5) and τf ∼ raw ∼ HalfNormal(0.1). Model abilities use θm N (0, 1) and are centered to mean zero for identifiability. The hierarchical log-Normal pool on σπ,i within facet keeps item-level ordering-noise estimates shrunken toward the facet-level scale.
point summary. Table 5: Posterior uncertainty for ODI facet scales. Medians and 89% highest-density intervals are computed from 6,000 posterior draws of the screened modaloutcome fit. Facet
σπ median [89% HDI]
σδ [89% HDI]
OPTION - ORDER DOCUMENT- RANK - ORDER EVIDENCE - CHUNK - ORDER IMAGE - SET- ORDER MIXED - MODALITY- ORDER
0.086 [0.050, 0.119] 0.093 [0.054, 0.137] 0.103 [0.058, 0.146] 0.147 [0.078, 0.225] 0.246 [0.095, 0.416]
0.012 [0.000, 0.033] 0.027 [0.000, 0.063] 0.028 [0.000, 0.060] 0.064 [0.000, 0.143] 2.364 [1.973, 2.764]
Cross-facet KS distinguishability. Pairwise Kolmogorov–Smirnov tests on posterior-mean σπ,i item estimates from the screened modal-outcome fit show all 10 facet pairs are separated (p < 0.001; many underflow to 0 numerically). This strong separation reflects the current model’s facet-level shrinkage of σπ,i , so we interpret it as evidence of facet-scale separability rather than a content-level item classifier. Per-model θcorrect . Per-model posterior means and 95% credible intervals from the correctoutcome fit will be included as a CSV in the planned full code release, with items sorted by descending θ. The screened fit ranks Gemini Pro ≻ Gemini Flash ≻ Opus ≻ ChatGPT 5.5 ≻ ChatGPT 5.4-mini ≻ Sonnet ≻ best open-weight; all six closed-source frontier models remain separated from the open-weight cluster on θcorrect . What ODI is and is not. 2PL Bayesian IRT is well-established (Birnbaum, 1968; Embretson and Reise, 2025; Polo et al., 2024); our contribution is the particular per-item σπ / per-facet-and-dataset |δ| decomposition applied to the ordering audit, not the IRT machinery itself. Item-pooled σπ,i is currently shrunken within facet rather than per-itemdiscriminating; content-vs-position decomposition is absorbed into αi , βi rather than modeled separately. These are deliberate simplifications: ODI is a summary instrument for ordering-associated variance, not a content-saliency decomposition.
Inference. We run NUTS (Hoffman et al., 2014) in NumPyro / JAX (Bradbury et al., 2018; Phan et al., 2019) (fp64; chain_method=parallel; target_accept=0.99; 4 chains × 3000 warmup + 1500 posterior draws). Wall-clock is 12–20 hours on a CPU node for the 18-model 5-facet panel. Two fits. We fit two outcome variants on the same trial set. Modal-outcome: Y = 1 iff the answer under ordering o matches the model’s modal answer for item i. This emphasizes ordering-associated instability rather than item difficulty and is the source of the per-facet σπ and |δ| summaries in Table 2. Correct-outcome: Y = 1 iff the answer matches gold. This recovers θm,correct used in Figure 1(b). Posterior diagnostics. The screened modaloutcome fit converges cleanly: R̂max = 1.000, ESSbulk,min = 898, 0 divergent transitions, nitems = 3,612. The screened correct-outcome fit has 0 divergences and well-mixed paper-critical θcorrect parameters (R̂max = 1.000, ESSbulk,min = 1,468), but its global diagnostic is weaker because several dataset-difficulty hypermeans mix slowly (R̂max = 1.15, ESSbulk,min = 19). The per-facet variance and systematic-bias parameters remain well mixed (σδ R̂max = 1.000, ESSbulk,min = 830). We therefore use the correct-outcome fit for descriptive θcorrect ranking and capability correlations, while the per-facet decomposition in Table 2 comes from the cleaner modal-outcome fit. Full per-parameter posterior CSVs will be included in the planned full code release.
C
Per-Facet × Per-Model Tables
This appendix clarifies how Table 1 compresses per-(model, facet, dataset) cells to facet means and supplies the within-family scaling figure. Reading guide. The 5 main facets cover all 18 models. Table 1 uses a balanced main-panel aggregation: two datasets per deterministic nonMIXED - MODALITY- ORDER facet and three freeform RAG benchmarks for MIXED - MODALITY-
Facet-scale posterior intervals. Table 5 reports compact uncertainty summaries for the screened modal-outcome decomposition. The σπ column is the draw-wise per-facet median of σπ,i . The σδ column is the direct posterior hyper-scale for systematic permutation offsets; Table 2 separately reports mean absolute posterior-mean δf,d,o as its 16
Mean K = 6 flip rate (4 deployment-natural facets)
0.7
tion). This is the main decomposition referenced from Section 4.1 Q2. We run the canonical ordering K=6 times on Gemini-Pro 3.1 and GeminiFlash 3 at temperature 0 across a 12-cell panel: 6 OPTION - ORDER cells (MMLU - PRO, COMMON SENSEQA , MATHVISION × Pro/Flash), 4 IMAGE SET- ORDER cells ( MANTIS - EVAL, MEDFRAMEQA × Pro/Flash), and 2 EVIDENCE - CHUNK - ORDER cells (MEDXPERTQA retrieved variant × Pro/Flash). The main-paper interpretation uses the positionreference-clean image-set rows: the orderingexcess share is about 51–75% on MMLU - PRO OPTION - ORDER ; on the clean MEDFRAMEQA IMAGE - SET- ORDER split, cross-ordering exceeds the same-ordering floor at T =0 (floor 0.03/0.10 vs cross 0.10/0.14 on Pro/Flash) and the floor approaches cross-ordering only at T ≥0.7. MANTIS EVAL is excluded from the clean decomposition because most items are position-referential (Section 6). MEDXPERTQA EVIDENCE - CHUNK ORDER likewise has a cross-ordering excess at T =0 (floor 0.11/0.15 vs cross 0.20/0.21 on Pro/Flash) and decoder-dominated only at T ≥0.7. MULTICHALLENGE dialog-turn is excluded because same-ordering accuracy (0.007) makes flip rate uninformative.
Qwen3.5-VL InternVL3.5
0.6 0.5 0.4 Open-weight mean: 32%
0.3 0.2 0.1
Frontier mean: 14%
0.8B
2B
4B
8B
16B
Parameter count (log scale)
32B
Figure 4: Within-family scaling of ordering robustness. Parameter count vs screened mean K=6 flip rate (4-facet) for Qwen3.5-VL and InternVL3.5 families. Scaling reduces flip monotonically within Qwen and within InternVL through 14B. ORDER (MRAMG-Recipe, MMDocRAG, Multi-
ModalQA). Per-cell flip rates and the OrderingStability Index (OSI: normalized entropy of crossordering answers, 1 stable, 0 uniform) will be included as CSVs in the planned full code release so readers and downstream users can re-aggregate at any granularity. The 2 demoted facets (dialog-turn, few-shot) cover the subset of (model, dataset) cells where data was collected; see Section F for the interpretive treatment. Within-family scaling of ordering robustness. The per-(model, facet) cells in Table 1 aggregate to per-family scaling curves on the 4 deploymentnatural facets; Figure 4 plots these for the Qwen3.5-VL and InternVL3.5 families. Scaling reduces 4-facet flip monotonically within Qwen3.5VL (0.8B→27B: 0.57 → 0.29) and within InternVL3.5 through 14B (0.37 → 0.32, with a slight 38B uptick to 0.33), but neither family closes the gap to the frontier mean. These screened means use the position-reference-clean image-set subset and the screened θcorrect fit.
D
Temperature sweep and open-weight extension. We extend the same-ordering analysis on the clean categorical facets (OPTION - ORDER, EVIDENCE CHUNK - ORDER, DOCUMENT- RANK - ORDER) with Gemini at T ∈ {0, 0.7, 1.0} and the open-weight families at T ∈ {0.7, 1.0}. This is an appendix extension rather than a matched replacement for the temperature-0 Gemini control: the open-weight grids have no local T =0 match. The consistency-share estimator is masked by decoder noise at deployment temperatures: Gemini ∆order shrinks from T =0 to average T >0 on OPTION ORDER (Flash 0.049 to 0.022; Pro 0.052 to 0.017), EVIDENCE - CHUNK - ORDER (0.048 to 0.013; 0.051 to 0.013), and DOCUMENT- RANK - ORDER (0.022 to 0.008; 0.024 to 0.008). The decoder-cleaned accuracy-swing estimator remains more stable over the same sweep (Flash 0.173 to 0.141 and Pro 0.134 to 0.102 on OPTION - ORDER; analogous T>0 Gemini means are 0.100 on EVIDENCE CHUNK - ORDER and 0.052 on DOCUMENT- RANK ORDER ). Under that estimator, the open-weight families show a persistent robustness gap at T >0: mean local acc-swing is 0.458 on OPTION - ORDER, 0.291 on EVIDENCE - CHUNK - ORDER, and 0.164 on
Multi-Method Robustness Analysis
This appendix supports the same-ordering decomposition referenced from Section 4.1 Q2 and the calibration protocol referenced from Section 4.2. The methods below show that the temperature-0 Gemini panel is not merely a decoder-noise artifact, and then extend the decomposition at deployment temperatures with a decoder-cleaned accuracy-swing estimator. Same-ordering noise floor (12-cell decomposi17
DOCUMENT- RANK - ORDER, about 3–4× the corresponding Gemini T>0 means. K=3 multi-trial averaging. As a separate corroborating check on the 6 frontier closed-source models, we average each (item, ordering) cell across K=3 repeated calls. The majority-voted cross-flip is within ±0.04 of the K=1 cross-flip across all 6 closed-source models, consistent with the K=1 facet-panel data not being noise-dominated. Permodel K=3 within-cell disagreement in this diagnostic is: Opus 0.063, Sonnet 0.270, ChatGPT 5.5 0.287, ChatGPT 5.4-mini 0.310, Pro 0.229, Flash 0.188. The stricter Gemini same-ordering noisefloor runs give Pro 0.127 and Flash 0.187. Within-dialog three-turn quasi-replication. Multi-turn elicitation forces 3 commits at the same input; the fraction of dialogs whose 3 turns produce ≥ 2 distinct answers is 44.7% (Opus), 49.2% (Flash), 50.3% (ChatGPT 5.4-mini), 51.7% (Pro), 52.0% (ChatGPT 5.5), and 68.3% (Sonnet). Multi-turn re-evaluation adds variance, so within-dialog 3-turn is an upper bound on same-ordering noise; we use it as a noise-floor proxy for the calibration analysis (Section 4.2). Bootstrap 95% CIs. A 1000-iteration itemlevel bootstrap on per-model cross-ordering flip outcomes yields CIs of approximately ±8 pp at N =150 (Pro) and ±12–15 pp at N =50 (the 4 closed-source counterparts on the add-on cells). The per-cell bootstrap CIs will be included as CSVs in the planned full code release. Stability-subset Kendall τ . We compare permodel rankings on the full panel versus the stability subset (items where 6/6 orderings agree). Kendall τ = +1.000 (p = 0.006) across the 6 closedsource models: rankings are exactly preserved. The ordering signal is not driven by a borderline-item subpopulation. Random-answer permutation test. We pool answer letters globally, resample per item, and recompute flip rate (1000 iterations). The observed flip rates for the 6 closed-source models are far below the pooled-shuffle null (0.22–0.54 observed versus null means near 0.99). This is a sanity check that models produce item-specific (non-random) answers; it is not a significance test for the ordering effect itself. Calibration protocol (Section 4.2). For Q6, we elicit confidence on MEDXPERTQA via a 4-turn protocol: the model receives one evidence chunk at a time over 3 turns and emits a 0–1 confidence
after each; the 4th turn collects the final answer commit. Mismatch is computed as flip rate − (1 − turn-3 confidence); positive mismatch means the model is over-confident relative to its own ordering instability.
E
LLM-Judge Methodology
The paper uses three LLM-judge instruments: dialog-rescore for the demoted dialog-turn facet (Section F), mechanism classification for the substrate-conditional split (Section 4.2), and mixedmodality sem-flip for the 5th main facet (Section H). Each is documented below, followed by the shared protocol used across all three. Judge 1 — dialog-rescore. We pair a crossfamily judge (Qwen3-VL-8B-Instruct, local) with a cross-generation judge (Gemini 2.5 Flash Lite) and rescore content-equivalence on dialog-turn outputs that pass raw exact-match but differ semantically. Inter-judge Cohen’s κ = 0.38 (fair on the LandisKoch scale); per-stratum ARI = 0.53. The fair-notsubstantial agreement is reported as a measurement caveat in Section F. Judge 2 — mechanism classification. Three judges (Gemini Pro primary, ChatGPT 5.5 crossfamily, Claude Opus 4.7 cross-family) each classify the same N =225 cross-ordering-flipped items into one of six mechanism modes (Section I lists the taxonomy and the verbatim prompt). Fleiss’ κ on the 3-class collapse (reasoning instability, anchoring, other) is 0.30 (fair). The categorical-cell finding is preserved at all reported granularities: the modal failure is content rationalization over answer or evidence content. The original image-set mechanism row was sampled before the MANTIS - EVAL position-reference screen and is retained only as an exploratory diagnostic until a screened resample is run. Judge 3 — mixed-modality sem-flip. The primary judge is Gemini Pro with one structured equivalence prompt over the K=6 item outputs; ChatGPT 5.4-mini serves as the cross-vendor check. Cross-vendor pooled 2-class (equivalent / not-equivalent) Cohen’s κ by output-model family: ChatGPT-output cells κ = 0.81 (substantial), Gemini-output cells κ = 0.73 (substantial), Claude-output cells κ = 0.59 (moderate; the lower number reflects a prevalence paradox where Claude cells have a low not-equivalent base rate, which inflates the marginal probability of chance agreement). For Gemini-family mixed-modality 18
Few-shot order (demoted). On the Gemini Pro/Flash few-shot runs, GSM 8 K (math) is nearzero (0–0.5% flip at n = 200), while HUMANEVAL (code) is much higher (32–54% on the n = 156 filtered run; the earlier n = 50 diagnostic spanned 20–58%). Interpretation: math has unique correct answers, so demonstration ordering rarely toggles correctness for these frontier runs; code admits multiple equivalent outputs whose preference can be demonstration-order-conditional. Demoted because the dichotomy is a Gemini diagnostic rather than a panel-wide claim, and few-shot prompting is increasingly outside the deployment-default regime. Tool-description ordering: a query-conditional null (Section 4.1 Q3). Two regimes on Pro and Flash. Under K=5 disambiguating queries, 0 of 200 model–dataset items (1,200 trials) show a content flip and gold-match accuracy is 96–100%; this is the reliable null referenced in the main paper. Under a harder glaive_k10 stress set with 6 orderings (not restricted to the disambiguating subset), Pro flip is 0.517 and Flash flip is 0.550 at gold-match accuracy 0.77–0.78. The tool-description null is therefore query-conditional, not unconditional; the main-paper Q3 claim takes the disambiguatingqueries regime as the null.
cells, the appendix reports cross-vendor ChatGPT/Claude agreement checks rather than relying on the same model family as the sole validation signal. Judge-free gold-anchor cross-check (MMQA). MMQA has short-factoid gold on n = 181 items; we directly compute gold-match flip and compare against sem-flip. In the Gemini MMQA temperature sweep (Pro/Flash at T ∈ {0, 0.7, 1.0}), semflip exceeds gold-anchor flip by 4.8–11.1 pp in the predicted direction. MRAMG and MMDocRAG have ≤ 13 short-gold items each; the MMQA n=181 subset is the calibration anchor. Shared guardrails. All judge instruments use fixed structured prompts and constrained label sets. The dialog-rescore judge additionally uses permutation self-consistency with Ksc = 3 over shuffled response labels; mixed-modality sem-flip and mechanism classification instead use one structured prompt per item and rely on cross-vendor checks for validation. Parser checks and pre-flight sanity probes are applied where the output format requires clustering or structured JSON. Limitations. The dialog-rescore κ = 0.38 is fair, not substantial; human-judge calibration on a 100item subset is future work. The mixed-modality and mechanism checks use cross-vendor secondary judges rather than a single-judge protocol, but human-rating validation is also future work for all three uses.
F
G
Extended Mitigation Results
This appendix expands Section 4.3: full cost-Pareto data backing Table 3, per-model deltas for the prompt-level mitigations the main paper reports in aggregate, and the matched think-budget sweep on easy items. Full cost-Pareto data. The 156 [cost, accuracy] policy points across 6 policies and 26 (model, facet, dataset) cells will be retained in the planned full release artifact; the aggregated values tabulated in Table 3 are means across the 26 cells after applying the image-set screen, with K=2 abstain coverage of 79.7% (the fraction of items the screen retains rather than flagging for escalation). Per-policy ∆-accuracy and ∆-cost. Versus the K=1 baseline in the screened aggregate, the deployable policies are: K=2 abstain-on-disagree (+0.019 selective accuracy, cost 2, 79.7% coverage), K=3 majority (+0.019, cost 3), and K=6 majority (+0.011, cost 6). The oracle bounds are −0.049 for worst-order selection and +0.046 for best-order selection. Across 26 screened cells, the disagreement-flag screen is the only non-baseline
Additional Facet Results
This appendix houses (i) the two demoted facets (dialog-turn, few-shot) that did not earn a mainpaper row but carry useful signal, and (ii) the tooldescription null referenced from Section 4.1 Q3. Dialog-turn order (demoted). MULTICHAL LENGE same-ordering accuracy is 0.007, a degenerate floor that makes flip rate uninterpretable, so the same-ordering decomposition cannot run. The synthetic dialog benchmarks (musique_synth, hotpotqa_synth) show modest but nonzero exactmatch flip on Pro and Flash (0.085–0.185, equivalently 0.815–0.915 stability) with macro-accuracy 0.69–0.95. Separately, LLM-judge rescoring on MT-E VAL/MULTICHALLENGE dialog samples gives content-flip rates 0.50–0.85 depending on judge and model, so semantic instability can be larger than exact-match scoring suggests. We retain the data as a demoted result because the scoring regimes are heterogeneous; the main-paper claim rests on the 5 retained facets. 19
H
low-cost Pareto improvement among deployable policies. Order-aware disclaimer (P15) and Canonicalizing-CoT (T4). Per-model ∆flip rate versus no-mitigation baseline: on the 5 models tested (Pro, Flash, MedGemma 4B-IT, Qwen3.5-VL 8B, InternVL3.5 8B), P15 is +2.0 pp on average, with Flash and MedGemma regressing +4 to +6 pp. T4 is null-to-negative on Pro and Flash: flip rate changes by +2.7 and +1.3 pp, respectively, while accuracy drops by 1.7–1.8 pp. Neither beats the no-mitigation baseline on mean, consistent with the main-paper statement that prompt-level knobs other than CTA are null or marginal.
Mixed-Modality Facet: Extended Details
This appendix expands the 5th main facet (MIXED MODALITY- ORDER ), which the main paper summarizes in Section 3 and Section 4.1 but which needs more depth for reproducibility and for the multi-judge and gold-anchor validation the main paper relies on. Setup. MIXED - MODALITY- ORDER permutes the entire heterogeneous text-and-image component sequence per item, not just one kind of content. Three free-form RAG benchmarks: MRAMG-Recipe (paragraph-form recipe synthesis, 3–6 components per item), MMDocRAG (financial-document paragraph QA, 13–17 interleaved text/chart/table components), and MultiModalQA (short-factoid answers with 4–15 captions + images + tables). Percell N ∈ {197, 200, 200}, K = 6 orderings, 7,164 trials per model class. All 18 models are evaluated.
Temp-SC vs ordering-SC head-to-head. At matched K ∈ {1, 2, 3, 6}, the two SC modes are orthogonal: on IMAGE - SET- ORDER Temp-SC is higher by ∼ 40 pp in the original pre-screen, Mantis-heavy diagnostic panel (a canonical-order advantage regime), not the clean image-set headline; on OPTION - ORDER Temp-SC leads by 5–11 pp; on EVIDENCE - CHUNK - ORDER ordering-SC matches or beats Temp-SC at K=6 by up to 9 pp. Disagreement-signature analysis suggests the two methods capture distinct uncertainty sources, so combining them (Temp-SC across orderings) is a candidate for future work but is more expensive than the cost-Pareto policies we report.
LLM-judge sem-flip protocol. For each item, we submit the K=6 outputs to the Gemini-Pro judge in one structured equivalence prompt. The judge returns an equivalent, partial-flip, full-flip, or unparsable verdict; sem-flip is partial-flip plus fullflip. Text-flip (the raw exact-string-mismatch rate) is reported alongside as an upper bound that the judge correction collapses by 10–90 pp depending on model and benchmark. Cross-vendor multi-judge validation. Pooled 2-class (equivalent / not-equivalent) Cohen’s κ for the primary judge against a cross-vendor secondary: ChatGPT-output cells κ = 0.81 (substantial), Gemini-output cells κ = 0.73 (substantial), Claude-output cells κ = 0.59 (moderate, with the prevalence-paradox caveat noted in Section E).
CTA + multipass anti-synergy detail. On MEDX PERTQA , CTA alone reduces flip to 0.180 (Pro) and 0.300 (Flash); stacking multipass reconciliation on top of CTA regresses both back to 0.290 and 0.400 respectively (+11 pp on Pro, +10 pp on Flash). One hypothesis is that multipass after CTA introduces fresh permutation noise that the canonicalized context cannot absorb. The negative result motivates the main-paper guidance to pick one prompt-level intervention, not several.
Judge-free gold-anchor cross-check (MMQA). On the n = 181 MMQA items with unambiguous short-factoid gold, we compute gold-match flip directly and compare against sem-flip. In the Gemini MMQA temperature sweep (Pro/Flash at T ∈ {0, 0.7, 1.0}), sem-flip exceeds gold-anchor flip by 4.8–11.1 pp in the predicted direction. MRAMG and MMDocRAG have ≤ 13 short-gold items each; the MMQA subset is the calibration anchor.
Think-budget plateau detail. The main paper reports monotonic improvement with thinkbudget on hard MEDXPERTQA for Pro (1k → 24k: 0.41 → 0.28). On easy MMLU PRO , the later matched Pro/Flash sweep remains essentially flat across budgets: Pro 1k/2k/8k/24k flip 0.110/0.100/0.110/0.110, and Flash 0.120/0.100/0.110/0.080; accuracy reaches 0.927–0.938 on Pro and 0.910–0.932 on Flash. The interpretation in the main paper holds: think-budget buys robustness only where there is robustness headroom.
Headline findings. Per-cell sem-flip across the 18 frontier and open-weight cells × 3 benchmarks spans 0.09–0.89 (median ∼ 0.28 on frontier, ∼ 0.62 on open-weight). By benchmark difficulty (frontier mean): MRAMG < MMDocRAG < 20
MMQA. Within-vendor capability ordering (Opus ≻ Sonnet, Pro ≻ Flash, ChatGPT 5.5 ≻ ChatGPT 5.4-mini on stability) replicates the pattern in the other 4 main facets. Text-flip is 0.81–0.99 across all cells (paraphrase upper bound). Per-cell numbers will be included as CSVs in the planned full code release. IRT integration on MIXED - MODALITY- ORDER. Per-trial gold-match labels (Gemini Pro primary, ChatGPT 5.4-mini cross-check) feed the screened ODI modal fit as the 5th main-facet row, jointly estimated with the other four main facets on the 18-model panel. See Table 2 for the σπ and |δ| row. Limitations specific to MIXED - MODALITYORDER . (i) Free-form scoring depends on LLM-judge semantic equivalence; we mitigate via cross-vendor multi-judge plus the judgefree gold-anchor cross-check on MMQA. (ii) The MIXED - MODALITY- ORDER outcomes are Bernoulli gold-match labels like the deterministicscored facets, but the |δ| posterior is a latent IRT offset estimated from judge-produced labels; for that reason we omit the per-cell ratio against OPTION - ORDER in Table 2 and flag the value with †. (iii) Human-rating validation of the judge verdicts is future work.
I
ings: component-order indices, normalized answer labels, and short answer-text excerpts. The original question text was not uniformly stored for this diagnostic pass, so the judge is asked to classify the dominant failure pattern visible in the crossordering answer summaries into one of six modes (positional anchoring, content rationalization, hallucination, refusal shift, semantic drift, other) and return a JSON object with the label plus a onesentence rationale. The full prompt text will be included in the planned full code release. Per-judge classification distribution. Across the three judges (Gemini Pro, ChatGPT 5.5, Claude Opus 4.7), content rationalization is the modal aggregate label on the two OPTION - ORDER source cells. The main-text 58% headline is the primary judge’s share over the three non-dialog categorical cells (MATHVISION, MEDXPERTQA, and MMLU PRO ; n = 126). On the pre-screen IMAGE - SETORDER Mantis-Eval sample, content rationalization is modal for all three judges (primary 80%, Claude 98%, ChatGPT 55% for n = 49), but this row is not a clean estimate after the positionreference screen. Hallucination is most frequent on MATHVISION (14% in the primary judge) and is 0% on the MEDXPERTQA/MMLU - PRO text cells. 3-judge Fleiss κ. The raw 6-class taxonomy gives Fleiss κ = 0.28 (fair agreement; judges disagree on fine-grained boundaries). The main-text 3-class collapse (“rationalization” vs “anchoring” vs “other”) gives κ = 0.30, matching a separate reasoninginstability collapse that groups hallucination with rationalization. The categorical-cell finding that flipped answers often rationalize over answer or evidence content is preserved under this collapse. The visual analogue awaits a screened image-set resample.
Mechanism Classification: Extended Details
This appendix supports the substrate-conditional mechanism finding in Section 4.2: sampling protocol, judge prompt summary, the 3-judge Fleiss κ derivation, and the per-judge mode distribution. Sampling protocol. We sample up to N = 50 cross-ordering-flipped items per source-dataset cell from the panel data, retaining 225 flips across five source-dataset cells: two OPTION - ORDER cells (MMLU - PRO and MATHVISION), one EVIDENCE CHUNK - ORDER cell ( MEDXPERTQA ), one IMAGE SET- ORDER cell ( MANTIS - EVAL ), and one demoted dialog-turn cell (MULTICHALLENGE). Percell item selection is uniform random over the items with at least one detected K = 6 flip. The imageset representative dataset in this original mechanism sample is Mantis-Eval; because the positionreference screen later flags most Mantis-Eval items, that row is an original-panel diagnostic rather than a clean image-set estimate. Judge prompt. The production prompt gives the judge the facet/dataset label, the available gold answer label, and compact summaries of the six order-
Aggregate mode distribution. On the 3-class collapse, aggregated across the original-panel 225 items and three judges: reasoning instability (content rationalization plus hallucination) is 59% (95% bootstrap CI 52–64%), positional anchoring 18% (CI 14–23%), and other modes 24% (CI 19–30%). Bootstrap is 1000 iterations at the item level. Limitations. (i) The taxonomy is judge-derived; we do not claim it is the final word on flip mechanisms. (ii) N = 50 per cell is small relative to the full panel; per-cell and aggregate numbers should be read as suggestive rather than precise. (iii) The diagnostic prompt used compact answer summaries rather than full item text, so the labels should not 21
be read as fine-grained causal annotations. (iv) The image-set row is pre-screen and Mantis-heavy; clean visual-mechanism claims require a screened resample. (v) Adding a fourth judge family (openweight) is future work.
J
ORDER , evidence chunks with rank metadata for EVIDENCE - CHUNK - ORDER / DOCUMENT- RANK ORDER , image list for IMAGE - SET- ORDER , and heterogeneous component sequence for MIXED MODALITY- ORDER. Datasets whose HuggingFace cards already declare these fields (via features schema) require no adapter; datasets without standard schemas require a ∼ 30-line loader that maps to the canonical per-item record. The planned full code release will include loaders for the 12 datasets in Section A and a template for new ones. Adding a new model. Closed-source models slot in with a provider-API adapter (set the endpoint, model ID, and provider-specific system-message structure). Open-weight models slot in with a HuggingFace transformers adapter (set the processor class, generation kwargs, and any provider-specific image preprocessing). Both adapters are ∼ 40 lines apiece in the current implementation. Scope of the release. The planned full code release will reproduce the audit on the 12 datasets, 5 facets, and 18 models reported in this paper, plus a template adapter and a registration script for adding new datasets, facets, or models. It will include permutation indices and aggregation scripts; upstream dataset content will be loaded from the original HuggingFace sources rather than redistributed.
Extending FACET-P ROBE via HuggingFace Dataset Metadata
This appendix sketches how FACET-P ROBE can be extended to new datasets and new ordering facets with minimal engineering. The audit’s per-facet permutation grammar and scoring rule are decoupled from the underlying dataset interface, so a new (dataset, facet) cell drops in once two contracts are satisfied: a permutation-unit specification and a gold-comparison rule. HuggingFace-metadata-driven facet routing. The 12 datasets in our audit are all loaded via the HuggingFace datasets library. We map each (dataset, configuration) pair to a list of admissible facets through a small declarative spec: the spec lists the dataset’s structural support for each facet (e.g., MMLU-Pro supports OPTION - ORDER because it carries ≥ 4 MCQ choices; MultiHop-RAG supports DOCUMENT- RANK - ORDER because it carries ranked retrieved documents). Dataset-card YAML fields (task_categories, modalities, size_categories) populate sensible facet defaults; the spec overrides them where the dataset shape requires a non-default treatment (e.g., MedXpertQA’s retrieved-evidence variant is routed to EVIDENCE - CHUNK - ORDER rather than OPTION ORDER). Adding a new facet. A new ordering facet (e.g., citation-order on retrieval QA with explicit citation markers) requires three components: (i) a permutation grammar (Section 3) that specifies the unit being permuted and how to sample K orderings; (ii) a scoring rule that maps the model’s raw output back to a content label invariant under the permutation (e.g., for OPTION - ORDER, the inverse-permutation mapping from displayed letter to source option index); (iii) a same-ordering control that runs the canonical ordering K times to quantify the decoder-stochastic floor for the new facet. The first two components slot directly into the facet registry; the third reuses the existing sameordering harness. Adding a new dataset. A new dataset slots into the audit once its loader exposes the per-item fields the target facet needs: choice list for OPTION 22